[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"glossary-inference::en":3,"gloss-cluster-inference::en":23,"gloss-next-inference::en":9},{"slug":4,"category":5,"name":6,"definition":7,"meta_desc":8,"faq":9,"schema_markup":9,"related":10},"inference","core-ai","Inference","Inference is the runtime process of feeding input into an already-trained model and generating an output — it's the \"using\" phase, as distinct from training, which is the earlier, far more expensive process of teaching the model its weights in the first place. Every API call to Claude, GPT, or any other LLM is an inference request: the model doesn't learn or update its weights from that call, it simply computes a forward pass through its (fixed) network to produce a response. This distinction matters enormously for SaaS builders because training and inference have completely different cost, latency, and infrastructure profiles: training a frontier model costs tens to hundreds of millions of dollars and happens rarely (a new model version every several months to a year), while inference happens on every single user request and its cost\u002Flatency directly determines your product's unit economics and responsiveness. Inference cost and speed are driven by model size (parameter count), the length of input and output (tokens), the hardware it runs on (GPUs like NVIDIA H100s, or specialized chips like Google's TPUs), and optimization techniques like quantization (reducing numeric precision to speed up computation) and caching (reusing computation for repeated prompt prefixes). A concrete example: a customer-support SaaS handling 10,000 chat messages a day makes 10,000 separate inference calls to an LLM API — each one independent, each one billed by tokens in and out, each one taking somewhere between a few hundred milliseconds and several seconds depending on model size and output length. This is why builders obsess over inference optimization at scale: switching from a large frontier model to a smaller, distilled model for a simple classification task can cut inference cost by 10-20x with minimal accuracy loss, and techniques like prompt caching (reusing the computation for a repeated system prompt across calls) or batching requests can meaningfully reduce both cost and latency in high-volume production systems. Inference cost and latency are also affected by whether a request is served synchronously (the caller waits for the full response) or asynchronously via streaming (tokens arrive incrementally as they're generated) — streaming doesn't reduce total inference cost, but it dramatically improves perceived latency for user-facing features, which is why chat interfaces almost universally stream responses rather than waiting for the complete answer before displaying anything. Builders evaluating a new AI feature's unit economics should model inference cost per request from day one (tokens in + tokens out, multiplied by the provider's per-token pricing), since AI inference cost — unlike most SaaS infrastructure costs — scales directly and linearly with usage rather than benefiting from the fixed-cost economics of traditional compute.","Inference is running a trained AI model on new input to produce output — as opposed to training, when the model learns.",null,[11,14,17,20],{"slug":12,"name":13},"gpu","GPU (Graphics Processing Unit)",{"slug":15,"name":16},"latency","Latency",{"slug":18,"name":19},"quantization","Quantization",{"slug":21,"name":22},"throughput","Throughput",[24,28,32,36,39,42,45,48,51,54,57,60],{"slug":25,"category":5,"name":26,"updated_at":27},"agentic","Agentic AI","2026-08-24T02:46:36+00:00",{"slug":29,"category":5,"name":30,"updated_at":31},"alignment-tax","Alignment Tax","2026-08-24T02:46:37+00:00",{"slug":33,"category":5,"name":34,"updated_at":35},"artificial-intelligence","Artificial Intelligence (AI)","2026-08-24T02:46:38+00:00",{"slug":37,"category":5,"name":38,"updated_at":27},"attention","Attention",{"slug":40,"category":5,"name":41,"updated_at":35},"beam-search","Beam Search",{"slug":43,"category":5,"name":44,"updated_at":31},"benchmark-contamination","Benchmark Contamination",{"slug":46,"category":5,"name":47,"updated_at":31},"catastrophic-forgetting","Catastrophic Forgetting",{"slug":49,"category":5,"name":50,"updated_at":35},"computer-vision","Computer Vision",{"slug":52,"category":5,"name":53,"updated_at":31},"constitutional-ai","Constitutional AI",{"slug":55,"category":5,"name":56,"updated_at":27},"context-window","Context Window",{"slug":58,"category":5,"name":59,"updated_at":35},"deep-learning","Deep Learning",{"slug":61,"category":5,"name":62,"updated_at":27},"diffusion-model","Diffusion Model"]