Facial Emotion Recognition

The problem

Facial emotion recognition sits behind real products — accessibility tools, UX research, driver monitoring, moderation. The task here: classify a face into happy, sad, neutral or surprise from a 48×48 grayscale image. At that resolution an eyebrow is a few pixels; the difference between sad and neutral can be a shadow.

Two things make this harder than a standard image-classification problem. The input is stripped of almost everything a modern vision model expects — no color, no texture detail, no context around the face, and a total of 2,304 pixels to work with. And the labels themselves are soft: humans disagree about these four categories too, particularly between sad and neutral, which puts a real ceiling on achievable accuracy. A model that hit 99% here would be evidence of a leak, not of skill.

So the question was which kind of model earns its complexity on small, low-resolution, single-channel data.

The approach

Four architectures went head-to-head under the same training and evaluation protocol:

• VGG16, ResNet and EfficientNet — ImageNet-pretrained backbones adapted via transfer learning, the default answer for most vision tasks and the thing a reviewer would ask about if it were missing.
• A custom CNN — a compact convolutional network designed from scratch for 48×48 grayscale input, sized deliberately to the data rather than to the problem's reputation.

Making that comparison mean anything required holding everything else still: the same data splits, the same augmentation, the same evaluation on a held-out test set touched only at the end. The pretrained backbones expect 224×224 three-channel input, so adapting 48×48 grayscale to them is itself a decision that shapes the result — worth being explicit about, because it's where a lot of transfer-learning comparisons quietly go wrong.

Image augmentation carried more weight here than usual. With a small dataset and a model training from scratch, augmentation is what stands between a compact network and immediate overfitting. Everything runs in TensorFlow/Keras, from the data pipeline through augmentation and training to final evaluation, reproducible end to end from the notebook.

The results

Four architectures, one held-out test set. The custom CNN reached 82% test accuracy — 16 points clear of the best pretrained backbone:

• Custom CNN — 82%
• VGG16 — 66% · best of the transfer models
• EfficientNet — 59%
• ResNet — 54%

Every ImageNet-pretrained backbone finished well behind a network designed for the data, and the explanation is domain mismatch rather than model quality. ImageNet features are built from millions of full-color, high-resolution photographs of objects in scenes; the early layers encode color gradients and textures that do not exist in a 48×48 grayscale face. The pretrained models arrive carrying representational machinery for a problem they aren't being asked to solve.

The ordering among them says the same thing. VGG16, the shallowest, did best; ResNet, the deepest, did worst — a twelve-point spread running in the opposite direction to model capacity. On data this small and this far from ImageNet, depth was a liability.

The compact model also trained faster and is far cheaper to serve. When human label agreement caps the achievable accuracy, extra capacity buys nothing, so the smaller, cheaper model is the one to ship.

What I learned

Pretrained weights only help when the pretraining data resembles yours. Reaching for the largest pretrained backbone is the reflex, and on specialized, low-resolution data it is frequently the wrong one. ImageNet holds no 48×48 grayscale faces, so here the pretrained weights did not help. Measuring beat assuming, and it beat it by enough to change the conclusion.

Capacity has to be earned against the data you actually have. A model with more parameters than the dataset can constrain will find patterns that aren't there. Sizing the architecture to 2,304 grayscale pixels is what made the model generalize. The ordering among the pretrained models makes the same point from the other direction: VGG16 beat ResNet by twelve points, and the difference between them is mostly depth.

The pipeline is most of the work. Clean data handling and augmentation moved accuracy more than architecture tweaks did — which is exactly what four years of production data engineering would predict, and a useful corrective to how much attention architecture gets relative to the plumbing underneath it.