Deep learning: a three-stage vision pipeline

What it is

A pipeline, not a model: sort what can be described, repair what can be repaired, then describe.

What was hard

Three models to train, and one constraint tying them together: denoising a clean image damages it, so you have to know before you act.

What it proves

The captioner reaches the practical ceiling of its architecture family, and the whole pipeline can be watched running.

A five-person project, within the deep learning block of the engineering programme. The client is fictional: an image archive to describe automatically.

Describing an archive of damaged images

A heterogeneous image archive to describe automatically. Photographs, but also diagrams, scanned text and drawings, in uneven condition. Describing a damaged image yields a damaged description; describing a diagram as though it were a photograph yields an absurd one.

Repairing only what needs it

The denoiser had to run only on images that needed it: denoising a clean image degrades it. That required a noise detector upstream rather than blanket processing.

Three numerical targets were set in advance, before any experiment. Two were not met, and the Limits section says so rather than burying it in a footnote.

Three models in cascade

Three-stage vision chain

One gate, then two treatments

1. Image

   of any kind

2. Stage 1, sort

   photograph, or not?

   Rejected: never goes further

   hence recall before precision

3. Stage 2, denoise

   and only when needed

4. Stage 3, describe

   a caption, word by word

Three stages in cascade, each one’s output shaping the next one’s input.

Stage 1: sort. A convolutional network trained from scratch separates photographs from non-photographs. Anything that is not a photograph is dropped, with its probability, and never goes further down the chain.

Stage 2: denoise, but only where needed. A detector measures how much neighbouring pixels disagree, a quantity noise drives up, against a threshold calibrated at start-up. Above the threshold the image is judged noisy and goes through a convolutional autoencoder; below it, the image continues untouched.

Stage 3: describe. A frozen InceptionV3 encoder produces a feature grid, projected and then read by a recurrent decoder with Bahdanau attention. Generation uses beam search with a length penalty.

constraint met

The noise detector catches every noisy image in the test set. The price is measured and accepted: one clean image in twenty is denoised needlessly.

The best model is not the one I kept

Measuring the measurement noise before comparing anything. Before ranking five architectures, the same model was retrained with three different seeds to establish the gap below which two results are indistinguishable: 0.04dBvalidation setMeasurement noise between seeds: 0.04 dB (validation set, n = 3).. Without that threshold, ranking close models is reading noise.

The best model was not the one I kept. Five architectures were compared under an equal protocol: same data, same noise, same budget. The U-Net wins. The convolutional autoencoder that was kept sits 1.39dBtest setPSNR gap between the model kept and the best one: 1.39 dB (test set, n = 148). below it, with 112,579parametersproductionSize of the denoising model that was kept: 112,579 parameters (production). and a markedly lower inference cost. The gap is well above the measurement noise. It is real, and deliberately accepted.

This is not a concession. In a three-stage pipeline where denoising is only an intermediate step, a mid-chain quality gain that does not improve the final description does not justify its cost. The only criterion that counts is what reaches stage 3.

An idea tested and dropped. A sharpening post-process applied after denoising looked like it should improve the output. Once measured, it degrades PSNR. It was removed. That is the only thing to do with a hypothesis that fails the measurement, but it is only useful if you took the trouble to measure before deciding.

documented trade-off

Five architectures compared under an equal protocol; the retained model is deliberately less capable than the best one, at a markedly lower cost.

documented trade-off

Sharpening post-process tested after denoising, then dropped because it degrades the measurement instead of improving it.

What the pipeline produces

Every number below is measured on test sets held out from training, and each carries its method.

Stage 1. The headline number says almost nothing. With the negative class three times larger, a classifier that always predicted “non-photograph” would already reach 0.7547accuracytest setAccuracy of a classifier that learns nothing: 0.7547 accuracy (test set, n = 6111). accuracy; the model reaches 0.7766accuracytest setOut of a hundred images, seventy-eight land on the right side. Compare that with the seventy-five a model always answering "non-photo" would get: the real gain is three points., two points above. The AUC, 0.8892AUCtest setRuns from 0.5 to 1: 0.5 is pure chance, 1 is perfect. At 0.89 the sorter almost always puts a photograph ahead of a non-photograph when shown both., establishes that it did learn something. But the task only becomes legible once the errors are broken down by type of non-photograph.

It is not one task but four, of wildly unequal difficulty. Scanned text and drawings are separated without a single error: 0% errortest setError rate on scanned text: 0 % error (test set, n = 1500).. Diagrams are moderately difficult. Paintings, however, are classified as photographs in 0.796error ratetest setError rate on paintings, before targeted correction: 0.796 error rate (test set, n = 1500). of cases, with an average confidence of 0.63probabilitytest setMean "photo" probability assigned to paintings: 0.63 probability (test set, n = 1500).: the model is not hesitating, it is confidently wrong. The global binary metric is therefore an average over a trivial problem and an unsolved one, and it hides both. What this classifier can do is reject text; what it cannot do is tell a photograph from a painting.

No published reference combines the three characteristics of this task: heterogeneous negative class, training without transfer, three-to-one imbalance. Work on photograph versus painting reports far better numbers, 95.7% accuracy (fine-tuned ResNet50)(Scientific Reports, 2026, not comparable)95.7% accuracy (fine-tuned ResNet50) · Scientific Reports, 2026 · Binary classification of photograph versus painting, two homogeneous classes, on a Kaggle dataset whose size could not be verified. · pretrained · comparable: no, A homogeneous negative class, paintings only, where ours mixes paintings, diagrams, drawings and text; and a backbone pretrained on ImageNet where our network starts from scratch. for instance, but it sets two homogeneous classes against each other and starts from a pretrained backbone (a methodological reference, not a single figure(Kornblith, Shlens and Le, 2019, not comparable)a methodological reference, not a single figure · Kornblith, Shlens and Le, 2019 · A general study of transfer from ImageNet to twelve image classification datasets. · not applicable · comparable: no, Does not address our task. Cited to establish that transfer outperforms training from scratch, which justifies not comparing the two regimes.). The trivial floor therefore remains the only solid reference.

Stage 2. Here too the number alone is not enough, and two floors bracket it. The untreated noisy image sits at 12.0125dBtest setThe starting point: at 12 decibels the noisy image is unrecognisable. That is the bar any treatment has to clear.. A median filter, with no learning whatsoever, reaches 18.22dBtest setWhat a three-line classic filter reaches, with nothing learned at all. A trained model that did not beat it would be pointless.: that is the bar a learned model has to clear to justify its existence. The model that was kept reaches 24.428dBtest setA decibel scale: below 15 the image is unrecognisable, around 20 it becomes readable again, past 30 the eye sees no noise at all. The denoiser takes the image from 12 to 24. and 0.6769SSIMtest setRuns from 0 to 1: at 0 the structure is lost, at 1 the image matches the original exactly. Noise drops it to 0.14, the denoiser lifts it back to 0.68. structural similarity.

A third comparison point says more than the other two: a dense autoencoder, tested under the same protocol, tops out at 13.42dBtest setPSNR of a dense autoencoder: 13.42 dB (test set, n = 148)., below the median filter, with 686 times more parameters than the convolutional model eventually kept. On this task convolution is not a refinement, it is a precondition. The test set, however, holds only one hundred and forty-eight images, with no confidence interval estimated.

Stage 3, and this is the page’s result. Captioning reaches 0.2291BLEU-4test setRuns from 0 to 1, but nobody comes near 1: two people describing the same photo do not use the same words. Above 0.22 is good, and this family of models tops out around 0.24. and 0.6003CIDErtest setRuns from 0 to about 1.2 on this dataset. Above 0.5, the generated captions use the words people judged important when describing the image. on four thousand COCO val2017 images, a set the model never saw. Beam search gives a clear gain over greedy decoding, which tops out at 0.1867BLEU-4test setBLEU-4 of the captioning, greedy decoding: 0.1867 BLEU-4 (test set, n = 4000)..

That number needs its reference point, without which it means nothing: the chosen architecture family, a frozen convolutional encoder followed by a recurrent decoder with attention, tops out in practice at 0.24BLEU-4test setThe best published score for this family of architecture. The project's model sits a hair below it, meaning further progress would come from changing architecture, not from tuning. on this set. The model is therefore within a few thousandths of what its architecture allows, and what still separates it from that ceiling comes from the architecture choice, not from tuning.

The demo shows the whole pipeline at work on twelve freely licensed images, unfiltered.

Where it falls apart

The model collapses outside its training distribution, and the demo shows it. On five images whose subjects are absent from the COCO categories, 5out of 5 imagestest setWrong captions outside the training distribution: 5 out of 5 images (test set, n = 5). are wrong. The failure mode is specific: the model falls back on the nearest COCO scene for any unfamiliar image. On 3out of 5 imagestest setImages with no person described as containing one: 3 out of 5 images (test set, n = 5)., it describes people where none are present: the desert becomes a beach, an aurora a group of people, a painting a gathering by the water.

Five images are not an evaluation set. This illustrates the failure mode; it does not measure an error rate.

Stage 1 caps the whole pipeline, and the F1 was hiding it. Recall on the photograph class is 0.8746recalltest setGenuine photographs the sorter lets through: 0.8746 recall (test set, n = 1499). against a precision of only 0.5269precisiontest setPrecision on the photo class: 0.5269 precision (test set, n = 1499).: the gate catches most photographs but lets through almost as many non-photographs. And stage 1 is a gate, not a corrective filter: the 12.5%test setPhotographs wrongly rejected by the gate: 12.5 % (test set, n = 1499). of photographs wrongly rejected are neither denoised nor described, nothing downstream flags them, and stage 1’s recall therefore puts a hard cap on what stage 3 can process, however good stage 3 may be.

The targeted correction traded recall for precision, in the wrong direction. Hard negative mining moved the error rate on paintings from 0.796error ratetest setError rate on paintings, before targeted correction: 0.796 error rate (test set, n = 1500). to 0.647error ratetest setError rate on paintings, after targeted correction: 0.647 error rate (test set, n = 1500)., but the error rate on photographs from 0.046error ratetest setPhotographs wrongly rejected, before targeted correction: 0.046 error rate (test set, n = 1499). to 0.125error ratetest setPhotographs wrongly rejected, after targeted correction: 0.125 error rate (test set, n = 1499)., and the AUC edged down, from 0.8908AUCtest setAUC before targeted correction: 0.8908 AUC (test set, n = 6111). to its final value. On a gate those two movements do not cancel out: a painting wrongly accepted gets a useless caption, a photograph wrongly rejected disappears. The gain was measured on the binary metric, not on the pipeline’s objective.

Inference cost rules out real-time use. One caption takes 5,126msproductionGenerating one caption on CPU: 5,126 ms (production, n = 12). on CPU, plus 643msproductionFeature extraction on CPU: 643 ms (production, n = 12). of feature extraction. The demo is therefore served statically: the result is identical, computed once.

Beam search settings were chosen on validation, not on test. The protocol is sound, but the selection remains a human choice, not an automatic setting a third party could verify.

The original dataset is not redistributable, which prevents anyone from replaying the training exactly. The demo images had to be replaced with freely licensed ones for that reason.

If I started over

I would evaluate outside the distribution from the start, not at the end. The result that taught me most about this model is none of the test metrics. It is the demo. All the project’s measurements were made on data drawn from the same distribution as training, and they only say how the model behaves on what it already knows.

An out-of-distribution control set, even a few dozen hand-picked images, would have taken half a day and would have been the most informative number in the project. It did not exist, and that is a methodological omission, not a lack of time.

I would measure the whole pipeline, not each stage in isolation. The chain is run end to end on the real dataset, and you can watch stage 1 discard non-photographs, but that is a demonstration, not a measurement. No number says what sorting and denoising contribute to the final description quality, which is the only question that justifies a cascade architecture. Two decisions suffered for it: the denoising trade-off, settled on cost because it could not be settled on effect, and above all the targeted correction of stage 1, optimised on a binary metric blind to the fact that recall caps everything downstream.

I would start from a pretrained sorting encoder. Training stage 1 from scratch was a defensible pedagogical choice; it is also the most likely reason the target was missed. Transfer from an existing model would have cost less and returned more.