Compare exact models, then test the work
An API model affects what data you can send, how you validate the answer, and what a successful task costs. “Claude 4” is a family name, not a sufficiently precise implementation or billing specification.
This documentation-based comparison was reviewed on 27 September 2026. It covers the named model families, not necessarily the latest models available. The earlier unsupported client-testing stories, benchmark conclusions, and two-million-token Gemini claim have been removed. No private comparison runs are asserted here.
Documented capabilities
| Exact model or variant | Documented distinction | Implementation consequence |
|---|---|---|
gpt-5 | 400,000-token context; up to 128,000 output tokens; text and image input, text output | Budget prompt, reasoning, and output within the applicable limits |
| Claude Sonnet 4 and Claude Opus 4 | Separate models introduced with ordinary and extended-thinking modes | Record the variant and thinking configuration rather than assigning one price to the family |
gemini-2.5-pro | 1,048,576 input tokens and 65,536 output tokens; multimodal input, text output | A large input allowance does not establish retrieval accuracy |
gemini-2.5-flash | 1,048,576 input tokens and 65,536 output tokens | Evaluate Flash separately from Pro |
Sources: GPT-5 reference, Claude 4 launch, Gemini 2.5 Pro, and Gemini 2.5 Flash.
A repository that fits in a context window can still contain stale documents, conflicting definitions, and irrelevant generated files. Selecting the evidence required for a question is often more useful than sending every file. Check model availability for the deployment account before relying on an older model ID.
Coding: measure the change
A persuasive explanation and a correct patch are different outcomes. A patch may also compile while weakening authorization or removing behavior.
Prepare a small evaluation set: a bug with a failing regression test, a feature with acceptance criteria, and a refactor that must preserve behavior. This is a proposed evaluation method, not a report of experiments performed for this article.
For each run, retain:
- Repository revision, exact model ID, prompt, tool permissions, and time budget.
- Test results before and after the change, including tests not written by the model.
- Human review notes covering correctness, security boundaries, readability, and unnecessary edits.
- Elapsed time and billed usage across retries.
Repeat tasks if discussing reliability, and report attempts as well as successes. A preference for one model's writing style is an opinion. A recommendation for an engineering workflow needs evidence from that workflow.
Long context: retrieval and synthesis are separate tests
Create questions with known answers and references to supporting passages. Include questions whose answers are absent. Otherwise, a model that invents missing information can appear to outperform one that correctly declines to guess.
Separate locating one fact from combining several facts. Include similarly named components and contradictory document revisions, then require the answer to identify its evidence. For a codebase, ask for a request path through entry point, authorization, business logic, persistence, and error handling. Check every cited symbol.
This procedure can expose an incorrect answer even when the response sounds coherent. It does not produce a universal model ranking; the result applies to the collection and tasks tested.
Pricing: cost per accepted result
The GPT-5 reference lists standard text-token rates of $1.25 per million input tokens and $10 per million output tokens, plus a separate cached-input rate. The Claude 4 launch listed Sonnet 4 at $3/$15 and Opus 4 at $15/$75 per million input/output tokens. The Anthropic figures are historical launch prices; check current account, deployment, and pricing terms before budgeting. GPT-5 pricing, Claude 4 launch pricing.
For illustration, 20,000 uncached input tokens and 2,000 billed output tokens at those GPT-5 rates cost $0.045, excluding tools and other charges. That is arithmetic, not measured usage. Reasoning usage, retries, caching, and external tools can change the bill.
Use:
cost per accepted task = total billed usage and tool costs / accepted tasks
Record human review time separately. A cheap completion that repeatedly needs repair can be expensive to use; a more expensive model may also offer no measurable advantage for a constrained task.
API modalities are not consumer application features
The GPT-5 model reference lists native audio and video as unsupported. A voice feature in a consumer application does not change that contract. Gemini 2.5 Pro accepts audio, images, video, text, and PDF, but returns text; it is not listed as an audio-generation or Live API model. GPT-5 modalities, Gemini 2.5 Pro capabilities.
Draw each pipeline step explicitly: recording to transcript, transcript to structured result, result to application action. Each has its own latency, cost, failure handling, and validation requirements. Choose a model that satisfies that contract and passes your evaluation, rather than treating a family name as a guarantee.
Sardar Mukhshaf
Full-Stack & DevOps Engineer
I build scalable web applications, secure backend systems, and production-ready cloud infrastructure. My engineering focus spans interface design, API architecture, deployment pipelines, and observability. Follow for more deep dives into system architecture, performance engineering, and emerging technologies.
More from the practice
Keep the useful ideas close.
Follow new articles with your preferred RSS reader.
Open the RSS feed ↗
