Self-Consistency

Self-consistency is a prompting/inference technique that improves the reliability of chain-of-thought reasoning by sampling multiple independent reasoning paths for the same question (by running the same prompt several times with a non-zero temperature, so each run can reason differently) and then selecting the final answer by majority vote across the runs, rather than trusting a single generation. Introduced alongside chain-of-thought research, self-consistency exploits the fact that an LLM's incorrect reasoning paths tend to diverge (different mistakes look different) while multiple correct reasoning paths tend to converge on the same answer even via different routes — so a majority-vote answer is statistically more likely to be right than any single sample, similar in spirit to ensemble methods in classical machine learning. For SaaS builders, self-consistency is a practical reliability lever for tasks where a wrong automated answer is costly (financial calculations, medical-adjacent triage, legal document classification) and where the added cost of 3-5x API calls is justified by the accuracy gain. It's implemented entirely in application code, not as a special API parameter: call the same prompt N times (typically 3, 5, or 7 — odd numbers avoid ties) at a moderate temperature (e.g., 0.7), parse each response's final answer, and return whichever answer appeared most often; for free-text answers, a cheaper "judge" call can cluster semantically similar answers before voting. The obvious trade-offs are cost (N× the API spend) and latency (mitigated by running the N calls in parallel rather than sequentially). Concrete worked example: an AI tax-category classifier for a bookkeeping SaaS runs each ambiguous transaction description through the classification prompt 5 times in parallel at temperature 0.7. For the transaction "Adobe Creative Cloud subscription," 4 of 5 runs return "Software & Subscriptions" and 1 returns "Office Supplies" — the system takes the majority answer, "Software & Subscriptions," and additionally logs the 1-in-5 disagreement rate as a confidence signal, flagging transactions with low agreement (e.g., 3-2 splits) for human review instead of auto-categorizing them. This turns a single noisy classification into both a more accurate answer and a built-in uncertainty estimate.