Anub Sinha
← Anub Sinha · the library

Essay · Research notes

Esteva et al. 2017: if it's "just fine-tuning Inception v3," what was actually hard?

Research notes, 15 May 2026.

The provocation

“Esteva et al. took an off-the-shelf Inception v3, pretrained on ImageNet, replaced the final classification layer with their disease classes, and fine-tuned on dermatology images. That recipe was already known. What was the real innovation?”

The premise is mostly correct. The model architecture is unmodified Inception v3 (Szegedy et al. 2015). The weights were initialized from a public ImageNet-1K checkpoint. The training is standard transfer learning. The losses are vanilla cross-entropy. In terms of the ML method, the paper introduces essentially nothing new.

It is also worth being honest that Esteva et al. was not even the first medical-AI paper to use exactly this recipe at scale. Gulshan et al. (Google) published a JAMA paper in November 2016 — about two months earlier — using Inception v3 with ImageNet pretraining for diabetic retinopathy detection from retinal fundus photographs, with ~128,000 training images. Similar architecture, similar recipe, similar scale. The Esteva paper didn’t invent the medical-AI transfer-learning recipe; it landed in Nature with a dramatic framing while a parallel result was already in print.

So the question stands: if the ML method is borrowed and the recipe was already in the air, what is the actual contribution? The honest answer is that the contribution lives almost entirely outside the model. The hard parts were the dataset, the taxonomy, and the evaluation. The Inception v3 inside is just the engine.

This is the opposite shape from the AlphaFold case. AlphaFold’s contribution was largely a stack of ML inventions on a relatively well-curated benchmark (PDB). Esteva’s contribution was largely data engineering and clinical-evaluation rigor on an unchanged ML method. Both are real contributions; they’re contributions of different kinds, and conflating them is part of why this work gets misread.

Below is what was actually difficult, in roughly the order it bites you when you try to do this naively.

1. The dataset didn’t exist — they had to build it

Pre-Esteva, the largest public dermatology-AI dataset was on the order of a few thousand images, mostly dermoscopic (taken through a hand-held magnifying device — controlled lighting, cropped, polarized, oily skin removed). Dermoscopic images are a clean, near-laboratory imaging modality. They are not what most patients or general practitioners present with.

What clinicians actually see is clinical photographs — taken with whatever camera (smartphone, point-and-shoot, dermatology clinic SLR), under whatever lighting, at whatever distance, with whatever skin tone and body part visible. The deployment target is messy. The training data needs to match.

The Esteva team assembled 129,450 clinical images from 18 different sources, including:

Each source had different conventions:

Harmonizing these into a single trainable dataset was a substantial data-engineering project. This is the kind of work that doesn’t have a method-section paragraph but takes person-months of dermatologists and engineers working together. The paper acknowledges this only briefly; the supplementary information is more honest about the scale of effort.

Why this is “hard”: there is no formula for it. You can’t grid-search your way through “which 18 datasets do I combine, and how do I de-duplicate, and how do I handle disagreements between source diagnoses, and how do I split train/val/test so I don’t leak the same lesion seen at two angles into both splits.” It’s craft work. And it scales the result — the model is only as good as what’s behind it.

2. The labels needed a taxonomy, not a flat class list

Skin disease isn’t a flat 1000-way classification problem like ImageNet. It’s a hierarchical taxonomy — diseases are organized into a tree by clinical and pathological similarity. For example:

skin lesion
├── benign
│   ├── benign melanocytic
│   │   ├── nevus (mole)
│   │   ├── lentigo
│   │   └── …
│   └── benign keratinocyte
│       ├── seborrheic keratosis
│       └── …
└── malignant
    ├── malignant melanocytic
    │   └── melanoma
    └── malignant keratinocyte
        ├── basal cell carcinoma
        └── squamous cell carcinoma

In total, the Esteva team’s taxonomy had 2,032 disease classes at the leaves — but those classes are wildly imbalanced. Common conditions (acne, nevi) have thousands of images. Rare conditions have a handful. Training a flat 2,032-way classifier on this distribution is hopeless: the rare classes never see enough gradient.

But training a 3-way classifier (benign / malignant melanocytic / malignant keratinocyte) wastes information — you’re collapsing thousands of meaningful distinctions into three buckets, and the network gets very weak supervision per image.

The team’s solution — and this is one of the few honest ML contributions of the paper — is to:

  1. Train at the leaves (2,032 fine-grained classes). The cross-entropy loss is computed over the leaf distribution. Even rare classes contribute a gradient signal, and the network learns fine-grained features.
  2. Evaluate by summing probabilities up the tree. At test time, the probability of “is this malignant?” is the sum of leaf probabilities under the “malignant” subtree.
  3. Pick the evaluation node based on clinical relevance. For the carcinoma test, evaluate at the keratinocyte-carcinoma vs. seborrheic-keratosis node. For the melanoma test, evaluate at the malignant-melanocytic vs. benign-melanocytic node.

This is a partitioning algorithm that bridges fine-grained training and coarse-grained, clinically-meaningful evaluation. It is the closest thing the paper has to a methodological contribution. It is also unusual — most subsequent medical-AI papers simplify to flat classification because their domains don’t have the same taxonomy structure.

Building this taxonomy required dermatology expertise — multiple board-certified dermatologists curated the structure and resolved label disputes. It’s not a thing you can crowd-source on Mechanical Turk.

Why this is “hard”: it’s neither pure ML nor pure clinical work. It’s the boundary work — the kind that requires both a deep learning engineer and a dermatologist in the room simultaneously, iterating on how to map a clinical taxonomy into a loss function.

3. The evaluation set is what makes it credible — and is the hardest part to build

Many medical-AI papers (then and now) get torn apart in peer review because of weak evaluation. The two most common failures:

The Esteva team did three specific things to make their headline result credible:

(a) Biopsy-proven test sets. For both evaluation tasks (carcinomas and melanomas), every test image had a tissue-biopsy-confirmed diagnosis. Biopsy is the actual medical gold standard — a pathologist looking at a stained tissue slice under a microscope. This means the ground truth is not “what dermatologists thought it was looking at the photo” but “what was actually there in the tissue.” That’s a much harder bar to clear.

Practically: assembling biopsy-proven test images is expensive. You need access to medical records that link clinical photographs to subsequent biopsy results, and you need IRB approval for the data. This is a non-trivial logistical effort, not a click-and-download.

(b) Head-to-head with multiple dermatologists. The team had 21 board-certified dermatologists (for the carcinoma task) and 22 (for the melanoma task) score the same test images. The model’s ROC curve was compared against where each individual dermatologist sat on the sensitivity/specificity plane. The headline claim — that the model is “on par” with dermatologists — comes from showing the model’s ROC curve passing through or above the cloud of dermatologist operating points.

This is the right way to evaluate a medical AI system. It’s also significantly more work than just “compute test accuracy.”

(c) Clinically meaningful binary tasks. The two evaluation tasks weren’t picked because they were easy. They were picked because they matter:

Picking these tasks is a clinical judgment call, not a technical one. It is also the choice that determined whether the paper would matter outside the AI community.

Why this is “hard”: every piece of this — getting biopsy-proven data, recruiting 21+ dermatologists, designing the comparison protocol, selecting clinically meaningful tasks — requires a clinical co-author who genuinely knows the domain. You cannot do this from a pure ML lab.

4. Image augmentation and preprocessing — small but real

Clinical photographs come in at arbitrary orientations and scales. A nevus on the back might be photographed upright; the same nevus on a forearm might be photographed at any rotation. The model needs to be invariant to this.

The team used standard augmentation — random rotation, horizontal flip, scaling, cropping — applied per training image. This is straight from the ImageNet playbook. What’s slightly less standard:

These are not novel techniques. But the team had to tune them specifically for dermatology — too aggressive and the color cues for malignancy (e.g., melanoma’s characteristic dark, irregular pigmentation) get destroyed; too gentle and the model overfits to a specific camera/source.

Why this is “hard”: not technically hard, but the kind of taste judgment that comes from iterating with a dermatologist over many failed runs.

5. What was inherited vs. invented

Honest accounting:

Inherited (from prior work):

Invented (or first-time-applied at this scale):

Calling Esteva 2017 a “transfer learning paper” undersells the data work. Calling it a “deep learning paper” undersells the clinical-evaluation work. The honest description is: it’s a data and evaluation paper with a deep learning model inside it.

6. Why it mattered, despite the above

Given that the ML method was borrowed and Gulshan had a parallel result, why is Esteva 2017 the canonical reference for medical-AI transfer learning?

A few reasons:

(a) Nature placement and framing. JAMA is a top medical journal but reaches a clinical audience. Nature reaches everyone. The “dermatologist-level” framing — explicit head-to-head comparison with named expert clinicians — gave the paper a story arc that the diabetic retinopathy work didn’t push as hard.

(b) The scope of the disease taxonomy. Gulshan classified diabetic retinopathy into 5 severity grades. Esteva covered 2,032 conditions. The breadth signaled “this scales to a full clinical specialty,” not just one disease.

(c) The visual story. Dermatology images are recognizable to a general audience in a way that retinal fundus photographs are not. A reader can look at a mole and viscerally understand the question being asked. This matters for paper impact even though it shouldn’t.

(d) Timing. ImageNet had been “solved” in 2015 (ResNet hit super-human top-5 error). The field was looking for the next demonstration that deep learning could clear hard real-world bars. Esteva landed in that window.

(e) It catalyzed the medical-AI wave. Within 18 months of Esteva 2017, comparable papers appeared in radiology (CheXNet, pneumonia detection), pathology (lymph-node metastasis detection), ophthalmology, cardiology. Esteva 2017 is what most of those papers cited as the methodological template.

7. Honest limitations (what the paper does not show)

Worth being clear about. These are the standard critiques of Esteva 2017, mostly raised in follow-up work over the next several years:

These don’t invalidate the paper. They contextualize the headline.

8. Why “just supervised learning” undersells it

Refreshing the AlphaFold-essay framing — what made this a hard supervised learning problem? — the answers are very different from AlphaFold’s case:

  1. The dataset didn’t exist. Most of the project’s effort went into building it.
  2. The label space wasn’t flat. A taxonomy of 2,032 classes had to be turned into a trainable, evaluable structure.
  3. The evaluation gold standard wasn’t free. Biopsy-proven test data + multi-dermatologist comparator panel is a logistics-and-IRB project, not a train_test_split call.
  4. The task choice was a clinical judgment, not a benchmark choice. Picking carcinoma-vs-keratosis and melanoma-vs-nevus required clinical insight into what mattered.
  5. The boundary work was the actual work. Almost everything hard about this paper happens at the intersection of dermatology and ML — neither field alone would have produced it.

Calling that “just supervised learning with deep neural networks” is like calling a clinical trial “just biology and statistics.” Technically true. Skips over every part that was actually hard.

9. TL;DR

If a junior ML engineer believes Esteva 2017 is “fine-tune Inception v3 on some dermatology data,” ask them:

None of those questions have technical answers. They have organizational, clinical, and ethical answers. Solving them — with a deep learning model inside — is what the paper actually did.

The hard part wasn’t transfer learning. Transfer learning was a tool, already on the shelf, well-understood by 2016. The hard part was assembling a clinically-credible dataset, a taxonomy that bridges fine-grained ML training and coarse-grained clinical evaluation, a biopsy-proven test set, and a head-to-head dermatologist comparison — and then publishing all of it in a venue where it would catalyze a field.

The contribution is on the outside of the model, not the inside. Recognizing that is the difference between reading the paper as “they fine-tuned a CNN” and reading it as “they showed how to do this category of work, rigorously, for the first time at a level that mattered.”

Primary sources