Knowledge Distillation

Knowledge distillation is a training technique where a smaller "student" model is trained to reproduce the outputs (or output distributions) of a larger, more capable "teacher" model — effectively compressing most of the teacher's useful behavior into a much smaller, faster, cheaper package. Instead of training the student purely on raw labeled data (the traditional approach), the student learns from the teacher's rich probability distributions over possible answers, which carry more signal than a single "correct answer" label — the teacher's confidence spread across plausible options ("this is 70% likely to be a cat, 25% dog, 5% other") teaches the student subtler distinctions than a flat "the answer is cat" label would. This matters directly for SaaS builders because distilled models are frequently the right choice for high-volume, latency-sensitive, cost-sensitive production features where the full capability of a frontier model is overkill. Model families explicitly built this way include GPT-4o-mini (distilled from GPT-4o), Claude Haiku (a smaller, faster sibling in the Claude family optimized for speed and cost), and DistilBERT (an early, influential distilled version of BERT). A concrete worked example: a SaaS company runs a content-moderation feature classifying millions of user-submitted comments daily as "safe," "spam," or "flag for review." Running every comment through a frontier model like Claude Opus would be accurate but prohibitively expensive and slow at that volume. Instead, they use the frontier model to label a large training set (say, 50,000 comments with ground-truth classifications, possibly teacher-labeled), then distill that behavior into a much smaller, purpose-built classifier that runs in milliseconds at a fraction of the cost per call — achieving similar accuracy on this narrow task because the task itself doesn't require the teacher's full general-purpose reasoning ability. The broader lesson for builders: distillation (and choosing an already-distilled "mini"/"flash"/"haiku" model tier) is how you match model capability to task complexity instead of defaulting to the biggest, most expensive model for every call, which is one of the most common and costly mistakes in early-stage AI product architecture. Distillation quality depends heavily on how representative the training examples are of real production traffic — a student model distilled on a narrow slice of examples will perform well on similar inputs but can fail unpredictably on edge cases the teacher never demonstrated, which is why production distillation pipelines typically sample training data from real (or realistically synthetic) traffic distributions rather than a hand-picked, overly clean example set. Builders considering "should I use the frontier model or a distilled/mini model for this feature" should benchmark both directly against representative production examples and compare accuracy-per-dollar, rather than assuming the larger model is always meaningfully better for a specific, narrow task.