Model Serving

Model serving is the infrastructure layer that puts a trained model behind an API so applications can request predictions in real time. It handles loading the weights into memory, batching incoming requests, autoscaling GPUs, and exposing versioned endpoints. If training is a one-time event, serving is the part that runs forever — and it is where latency, cost, and reliability are actually won or lost. When you call a closed model like Claude or GPT, the vendor handles serving for you. The topic becomes hands-on the moment you self-host an open-weight model, where you reach for engines like vLLM, Text Generation Inference, or NVIDIA Triton, or managed platforms such as Modal, Replicate, and Baseten. Techniques like request batching and quantization dramatically cut the per-request GPU bill. Practical note: measure tail latency and cost per thousand requests, not just averages — one slow percentile can wreck a real-time feature, and idle GPUs quietly drain a startup's runway.