Large Language Models (LLMs) are becoming the core building blocks for modern software, with applications that use them to assist, automate, chat, and generally get things done better and faster. For example, the Tursio search platform uses LLMs for a series of tasks in context building and query processing to make search simpler and more automated for users.

Given this deep reliance on LLMs, making those requests efficient and reliable matters operationally. That means taking a close look at how we structure our requests, which is what we dig into below. Specifically, we break down how batch sizing affects the cost and reliability of LLM requests across different output parser implementations.

Study Methodology

We ran a standalone experiment on a dataset of 1,433 columns across all tables, processing each through a column-simplification [1] LLM task – just one of the several tasks in the Tursio platform for the sake of simplicity in this blog. To understand the impact better, we evaluated three different output parser implementations for this task: default String OutputParser, Pydantic OutputParser [2], and Structured OutputParser [3].

• We tested batch sizes of 20, 30, 50, and 80 for each parser implementation.
• To test scalability, we extended the batch sizes to 100, 200, and 500 to find the point at which we start failing (i.e., can no longer parse the output correctly).
• We used gpt-4.1 as the foundation model since it's the oldest model that is still supported in Tursio.
• We record the following:
  
  
  • llm_calls: The total number of LLM calls executed.
  • input/output/total tokens: The billed tokens across all calls for that specific batch size.
  • errors: The columns where the parser failed to reconcile a decision, resulting in a "failed" status.

We focus strictly on reliability and cost in this study. Evaluating the accuracy of the simplified column name, and whether it should be simplified, requires ground-truth labels and is beyond our scope here. For LLMs, cost directly correlates to token count. In this context, "quality" is defined simply — did every input column come back with a real decision, or were some quietly dropped?

Output Format Implementations

1. String OutputParser: The baseline LLM call returns raw model text with prompt instructions to generate the output as a JSON array. Then we parse the JSON array and match columns purely based on array position. If the returned array length does not exactly match the batch size, the entire batch fails.

2. Pydantic OutputParser: This implementation replaces manual JSON parsing with LangChain's Pydantic OutputParser, embedding format instructions directly in the prompt. It still relies on text-based generation and reconciliation by array length and position, meaning the schema only helps if the array comes back at the correct size.

3. Structured OutputParser: This moves to OpenAI's structured-output path using the with_structured_output() function-calling mechanism, eliminating manual parsing. Reconciliation remains a length check without per-item IDs, so any count mismatch will still fail the entire batch.

Impact on Errors and Reliability

The baseline String OutputParser and the Pydantic OutputParser both work well up to a batch size of 30. Beyond that, they start to fail — String at 50, and Pydantic at 80.

Conversely, OpenAI's structured output parser rejects malformed responses before they even reach the reconciliation step. When using Structured Output on gpt-4.1, it never lost a single column across every batch size from 20 to 500.

Performance Data

Architectural Improvements

Batch failures can have severe consequences in production environments, and we found specific structural improvements to further mitigate the failure risks:

• Index the columns with keys to retrieve the non-failed items per batch.
• Turn on strict=True schema enforcement once the schema shape supports it.
• A List[Item] shape supports strict mode, whereas a Dict[id, Item] shape does not; with a Dict shape, strict mode collapses it to additionalProperties: false and empties the payload.

Key Takeaways

• Failure Thresholds: String OutputParser begins failing at a batch size of 50, while Pydantic OutputParser manages 50 without errors but fails at a batch size of 80. Structured OutputParser maintains perfect robustness up to the maximum tested batch size of 500.
• Cost vs. Reliability Trade-off: Total token usage and cost decrease across all parsers as batch size increases. Structured Output at a batch size of 500 proves to be the most cost-efficient, dropping total token usage to 120,018 while maintaining zero errors.

[1] https://www.amazon.science/code-and-datasets/nameguess-column-name-expansion-for-tabular-data

[2] https://reference.langchain.com/python/langchain-core/output_parsers/pydantic/PydanticOutputParser

[3] https://developers.openai.com/api/docs/guides/structured-outputs?api-mode=responses