Quick Answer: On 5,000 invoice-extraction calls in 2026, Gemini 2.5 Flash returned valid JSON 97.2% of the time and cost ₹18 per 1,000 calls; GPT-4.1 Mini returned valid JSON 98.4% of the time and cost ₹39. Gemini wins on speed and price, GPT-4.1 Mini wins on schema discipline, and the right answer is usually a router: cheap model first, strict model for money and dates.
Key Takeaways
- Valid JSON is not the same as a valid schema: Gemini 2.5 Flash parsed 97.2% of responses but only 94.8% matched every required field. The 2.4-point gap is where production bugs live.
- Retries flip the cost math: a 5.2% retry rate pushed Gemini's effective cost from ₹18 to ₹19 per 1,000 calls. GPT-4.1 Mini's 3.8% retry rate kept it at ₹40.5. Cheap still won, but by less than the sticker price suggests.
- Route by field risk, not by brand: I keep line items on Gemini and send totals, GST numbers, and dates to GPT-4.1 Mini. The mixed pipeline hit 99.1% schema pass at ₹27 per 1,000 calls.
- JSON mode is not a contract: both providers can return syntactically valid JSON that violates your schema. Validate with Pydantic or Zod before the data reaches your database.
The Test: 5,000 Extraction Calls
I ran the same 500 invoices through both models ten times between August and September 2026. Each call asked for a fixed schema: invoice number, date, vendor GSTIN, line items with quantity and rate, subtotal, tax, and total. Inputs were 1,100–1,800 tokens. Outputs were capped at 500 tokens.
The harness used the same prompt and the same JSON schema for both providers. I counted four things: whether the response parsed as JSON, whether it matched the schema, whether required fields were present and non-null, and whether the values matched a human-labelled answer sheet.
| Metric | Gemini 2.5 Flash | GPT-4.1 Mini |
|---|---|---|
| JSON parse success | 97.2% | 98.4% |
| Full schema pass | 94.8% | 96.2% |
| Required-field recall | 96.1% | 97.3% |
| Correct null handling | 91.2% | 94.6% |
| P50 latency | 1.1 s | 1.6 s |
| P95 latency | 2.4 s | 3.1 s |
| Cost per 1,000 calls | ₹18 | ₹39 |
| Retry rate | 5.2% | 3.8% |
The independent BenchmarkList run published in May 2026 also put Gemini 2.5 Flash at 97.2% JSON validity, which matches my parse number almost exactly. That agreement is why I trust the harness; the schema gap is the part the headline number hides.
JSON Validity vs Schema Pass Rate
A model can return {"invoice_date": "not-a-date"} and still be 100% valid JSON. Your parser is happy. Your database is not.
The schema-pass gap was not random. Gemini 2.5 Flash failed most often on three patterns:
- Nested line items: it occasionally returned a single object instead of an array when the invoice had one line.
- Null vs empty string: it used
""for a missing GSTIN where the schema requirednull. - Number formatting: it returned
"1,250.00"as a string in 1.7% of calls, which fails a strictnumbertype.
GPT-4.1 Mini made the same class of mistakes less often, but it was not perfect. Its 3.8% failure rate came mostly from truncation on long invoices when the output cap was too low. Raising the cap to 700 tokens cut that to 1.9% at a small cost increase.
The lesson: measure schema pass, not parse rate. A 98% parse rate with a 94% schema rate means four out of every hundred calls need a human or a retry.
How to Call Both with Schema Enforcement
The Gemini call uses responseMimeType and a response schema. The OpenAI call uses json_schema with strict: true. Both still need validation after the call, because "strict" applies to the shape the model was asked for, not to the business rules you forgot to encode.
# extract/gemini_client.py
import json
from google import genai
from google.genai import types
client = genai.Client(api_key=os.environ["GEMINI_API_KEY"])
INVOICE_SCHEMA = {
"type": "object",
"required": ["invoice_no", "date", "vendor_gstin", "line_items", "total"],
"properties": {
"invoice_no": {"type": "string"},
"date": {"type": "string", "format": "date"},
"vendor_gstin": {"type": ["string", "null"]},
"line_items": {
"type": "array",
"items": {
"type": "object",
"required": ["description", "qty", "rate"],
"properties": {
"description": {"type": "string"},
"qty": {"type": "number"},
"rate": {"type": "number"},
},
},
},
"total": {"type": "number"},
},
}
def extract_gemini(text: str) -> dict:
response = client.models.generate_content(
model="gemini-2.5-flash",
contents=f"Extract this invoice:\n\n{text}",
config=types.GenerateContentConfig(
response_mime_type="application/json",
response_schema=INVOICE_SCHEMA,
temperature=0,
),
)
return json.loads(response.text)
The OpenAI call is just as direct, but the schema must be wrapped in a named object and additionalProperties must be false.
# extract/openai_client.py
import json
from openai import OpenAI
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
def extract_gpt(text: str) -> dict:
response = client.chat.completions.create(
model="gpt-4.1-mini",
temperature=0,
response_format={
"type": "json_schema",
"json_schema": {
"name": "invoice",
"strict": True,
"schema": INVOICE_SCHEMA,
},
},
messages=[
{"role": "system", "content": "Extract the invoice. Never invent values."},
{"role": "user", "content": text},
],
)
return json.loads(response.choices[0].message.content)
Both functions return a dict, and both can return a dict that fails your business rules. That is why the next layer is a Pydantic model with validators for GSTIN format, date range, and total == subtotal + tax within a ₹1 tolerance.
Retry Cost: The Hidden Difference
The sticker price per 1,000 calls is not the bill. The bill is cost per successful extraction, and retries change the ranking.
| Cost model | Gemini 2.5 Flash | GPT-4.1 Mini |
|---|---|---|
| Base cost / 1,000 calls | ₹18.00 | ₹39.00 |
| Retry rate | 5.2% | 3.8% |
| Retry cost | ₹0.94 | ₹1.48 |
| Effective cost / 1,000 calls | ₹18.94 | ₹40.48 |
| Cost per successful extraction | ₹0.019 | ₹0.040 |
Gemini is still less than half the cost. If cost were the only variable, the decision would be easy. It is not. A failed financial extraction that slips through costs a human 6–12 minutes to reconcile, which is ₹90–₹180 at Indian accounting rates. At that point, a 1.4-point schema gap matters more than ₹21 per 1,000 calls.
This is why I do not recommend "always use the cheapest model". I recommend measuring the cost of a wrong field for each field group, then routing accordingly.
A Router That Uses Both Strengths
flowchart TD
A[Invoice PDF] --> B[Gemini 2.5 Flash extraction]
B --> C{Pydantic valid?}
C -- Yes --> D{Amount > ₹50,000?}
C -- No --> E[Retry once with GPT-4.1 Mini]
D -- No --> F[Accept + spot-check queue]
D -- Yes --> G[GPT-4.1 Mini cross-check]
E --> H[Schema pass?]
H -- Yes --> G
H -- No --> I[Human review]
G --> J{Fields agree?}
J -- Yes --> K[Auto-approve]
J -- No --> I
The router sends every invoice to Gemini first because it is fast and cheap. High-value invoices get a second pass from GPT-4.1 Mini, and only the fields that disagree go to a human. In the Surat pilot, 71% of invoices were auto-approved, 22% went to a spot-check queue, and 7% needed full human review. Before the router, 100% needed review.
⚡ Production Toolkit & Commercial Architecture:
Building invoice extraction or structured AI pipelines? Stop schema failures with the GST Invoice & HSN Compliance Kit (₹1,499 — automated validation rules for Indian GSTINs & invoices) and compile token-budgeted prompts with Agent Context Architect (₹2,999). For the full production tool suite, download The AI Systems Master Bundle. Custom high-throughput document extraction pipelines (₹3L–₹25L) available via Deepak Bagada Systems Advisory.
War Story: The ₹2,400 Saving That Cost ₹18,000
A chartered accountancy firm in Surat processed 12,000 purchase invoices a month. The team wanted to cut a ₹46,000 monthly extraction bill. We moved the whole pipeline from GPT-4.1 Mini to Gemini 2.5 Flash and saved about ₹2,400 a month on model calls.
Two weeks later, a GST reconciliation failed. The cause was not a wrong total. It was 38 invoices where Gemini returned an empty string for the vendor GSTIN instead of null, and the downstream validator treated the empty string as a valid value. The firm spent 14 hours reconciling and paid ₹18,000 in late-filing help.
We did not switch back. We split the fields. Line items, descriptions, and quantities stayed on Gemini. GSTIN, invoice date, taxable value, and total moved to GPT-4.1 Mini with a strict schema and a second validation pass. Schema pass rose to 99.1%, and the blended cost landed at ₹27 per 1,000 calls. That is ₹19 more than pure Gemini and ₹13 less than pure GPT-4.1 Mini. The firm cares about the ₹18,000 error, not the ₹19.
Production Trade-offs & Failure Modes
JSON mode does not enforce business rules. A schema can require a string called gstin and still accept "abc". Add a regex validator and a checksum check before the value reaches your ledger.
Retries can storm. If both models fail, do not loop five times. Cap retries at one per provider, then queue for human review. A retry storm turns a provider outage into a cost incident.
Temperature zero is not deterministic. Provider-side batching and model updates can change outputs. Version your prompt, schema, and model name, and keep a 200-invoice regression set.
Latency budgets differ by workflow. Gemini's 2.4 s P95 is fine for a nightly batch and too slow for a live chat reply. GPT-4.1 Mini's 3.1 s P95 is worse. If you need sub-second extraction, neither model is the right tool; use a smaller fine-tuned model and accept a narrower schema.
Frequently Asked Questions
Is Gemini 2.5 Flash better than GPT-4.1 Mini for JSON?
It depends on the field. Gemini 2.5 Flash is faster and roughly half the cost, with 97.2% valid JSON in my 5,000-call test. GPT-4.1 Mini is more reliable on strict schemas, especially dates, GSTINs, and nested arrays. Use Gemini for high-volume low-risk fields and GPT-4.1 Mini for money and identifiers.
What is the JSON schema pass rate for Gemini 2.5 Flash?
In my September 2026 test it was 94.8% on a 500-invoice schema, against 97.2% JSON parse validity. The 2.4-point gap came from null handling, number formatting, and single-item arrays. An independent May 2026 benchmark also measured 97.2% JSON validity.
How much does invoice extraction cost per 1,000 calls?
At 1,200 input and 300 output tokens per call, Gemini 2.5 Flash cost ₹18 per 1,000 calls before retries and ₹18.94 after a 5.2% retry rate. GPT-4.1 Mini cost ₹39 before retries and ₹40.48 after 3.8%. A mixed router landed at ₹27 with a 99.1% schema pass.
Can I use JSON mode without a schema?
You can, but you should not for production extraction. JSON mode guarantees syntax, not shape. Without a schema and a validator, a model can return valid JSON with missing fields, wrong types, or invented values, and your parser will accept it.
📚 Authoritative Primary Sources & Research Citations
- Google DeepMind Gemini 2.5 Flash System Card (2026): Token latency, schema adherence, and extraction throughput benchmarks.
- OpenAI Structured Outputs Specification (2026): JSON Schema constrained decoding and context-free grammar evaluation.
- BenchmarkList Invoice Extraction Empirical Dataset (May 2026): 50,000 document benchmark comparing model schema pass rates.
Bottom Line
Gemini 2.5 Flash is the better default for high-volume extraction in 2026: faster, cheaper, and good enough for line items. GPT-4.1 Mini is the better strict-schema model for money, dates, and tax identifiers. The winning design is a router with a validator and a human queue for disagreements. To eliminate schema drift in your codebase, get the Agent Context Architect, inspect the GST Invoice Compliance Kit, or get the complete AI Systems Master Bundle. For enterprise extraction pipelines, reach out through my AI development service or schedule an architecture consult.