Production LLM Cost Optimization: Model Choice, Prompt Caching & Output Engineering

The four-way trade

An LLM call on a hot path (the code that runs on every request) is a bill you pay again each time. You pay it in tokens, in latency, and against someone else's rate limit (the cap a provider puts on how many requests per second, or how many at once, you may send). Cutting that bill is easy if you let performance, accuracy or observability slide, and those three are exactly what a cheaper setup tends to spend without anyone signing off on it. The job is to bring cost down while all three hold.

Almost every LLM optimization I've made comes down to a trade between cost and those three. I think of them as four corners:

Decide up front which corners you won't touch and which you're willing to spend, and prove the trade with numbers instead of swapping models, eyeballing a few outputs and hoping.

I'll go through four techniques in order of payoff. The first three make each call cheaper. The fourth changes how calls are scheduled. Each one pulls on different corners:

graph LR
    subgraph TECH["Techniques"]
        T1["1. Model choice"]
        T2["2. Cache-aware prompt"]
        T3["3. Output-aware design"]
        T4["4. Batching"]
    end
    Cost["Cost"]
    Perf["Performance"]
    Obs["Observability"]
    Acc["Accuracy"]
    T1 -->|"lowers the most"| Cost
    T1 -->|"up or down"| Perf
    T2 -->|"lowers, for free"| Cost
    T3 -->|"lowers"| Cost
    T3 -->|"spends"| Obs
    T4 -->|"lowers"| Cost
    T4 -->|"spends latency"| Perf
    TECH -.->|"all four must hold it"| Acc
    style T1 fill:#7fc47f,color:#000
    style T2 fill:#7fc47f,color:#000
    style T3 fill:#7fc47f,color:#000
    style T4 fill:#7fc47f,color:#000
    style Cost fill:#f7c47f,color:#000
    style Perf fill:#4f8cf7,color:#fff
    style Obs fill:#4f8cf7,color:#fff
    style Acc fill:#4f8cf7,color:#fff

More arrows, bigger effect. ↔ means the corner holds. n/a means the technique has no bearing on that corner at all.

The running example is a real pipeline. Guliel scans a customer's connected inbox for expenses, and one LLM call per email decides "is this an expense, and what are the numbers?". That call runs on every email, which makes it a textbook hot-path call. Every figure below is measured on the same 465 real emails, at real published prices.

flowchart LR
    Inbox["Inbox"]
    FE["1. Fetch"]
    DE["2. Classify + extract<br/>(the LLM call)"]
    OE["3. OCR the attachment"]
    CR["4. Create expense"]
    Skip["Skip"]
    Inbox --> FE --> DE
    DE -->|"expense + file"| OE --> CR
    DE -->|"expense in body"| CR
    DE -->|"not an expense"| Skip
    style Inbox fill:#f7c47f,color:#000
    style FE fill:#4f8cf7,color:#fff
    style DE fill:#7fc47f,color:#000
    style OE fill:#cf7fcf,color:#fff
    style CR fill:#4f8cf7,color:#fff
    style Skip fill:#9a9a9a,color:#fff

Rule zero: you can't trade what you can't measure

Before trying any technique, build the instrument: one test that scores all four corners in the same run, on real inputs. Without it, every change is a guess. A real test captures four things, and most capture none of them.

Providers keep the processed start of prompts they have already seen. When a new request begins with exactly the same text, those input tokens are a cache hit and are billed at a small fraction of the normal price. The rest is a cache miss, billed in full.

Cost is a dot product: multiply each token count by its own price and add them up. One total times one rate gives the wrong answer.

type Usage = {
  prompt_cache_hit_tokens: number;   // input you already paid to compute → cheap
  prompt_cache_miss_tokens: number;  // input computed fresh → full price
  completion_tokens: number;         // output (a "reasoning" model thinks in these)
};

const cost = (u: Usage, price: { hit: number; miss: number; out: number }) =>
    u.prompt_cache_hit_tokens  / 1e6 * price.hit   // e.g. $0.0036 / 1M
  + u.prompt_cache_miss_tokens / 1e6 * price.miss  // e.g. $0.435  / 1M  ← ~120× the hit
  + u.completion_tokens        / 1e6 * price.out;  // e.g. $0.87   / 1M

Two traps faked my numbers, and each one gave me a wrong conclusion before I caught it.

The first is cache contamination. Prompt caches persist between runs, so if you re-run the same test prompts they cache at ~100%. That's a gorgeous number, and it has nothing to do with production, where every input is new. The fix is to prepend a unique per-run nonce (a random string that changes with every run), so each candidate measures its true caching.

The second is non-determinism. A reasoning model (one that writes extra "thinking" tokens before its answer and bills them as output) ignores temperature=0 and re-samples on every call. A single before/after comparison can't tell a real change from a coin flip. What saves you is an A/A run: the identical prompt, run twice, and a look at how far the two results differ. That spread is your noise floor, and a change is real only if it clears it.

Put together, one evaluation run looks like this:

flowchart TD
    In["Real inputs<br/>(the same emails every time)"]
    Nonce["Per-run nonce"]
    Run["1. Run the candidate"]
    Usage["2a. Capture the usage split<br/>hit / miss / output tokens"]
    Cost["3. Cost = dot product<br/>with the price sheet"]
    Qual["2b. Precision and recall<br/>against the trusted baseline"]
    Noise["4. Compare with the<br/>A/A noise floor"]
    Keep["Keep the change"]
    Reject["Reject the change"]
    In --> Run
    Nonce --> Run
    Run --> Usage --> Cost --> Noise
    Run --> Qual --> Noise
    Noise -->|"clears the floor"| Keep
    Noise -->|"inside the noise"| Reject
    style In fill:#f7c47f,color:#000
    style Nonce fill:#f7c47f,color:#000
    style Run fill:#4f8cf7,color:#fff
    style Usage fill:#7fc4c4,color:#000
    style Cost fill:#7fc4c4,color:#000
    style Qual fill:#7fc4c4,color:#000
    style Noise fill:#cf7fcf,color:#fff
    style Keep fill:#7fc47f,color:#000
    style Reject fill:#f77f7f,color:#fff

Technique 1: Model choice is the biggest lever, and price is only part of it

Every model sits somewhere on three axes: price, throughput and quality. The cheapest token loses if the model can't keep up, or can't tell a receipt from a newsletter. So pick by your binding constraint (the one you hit first) and treat the price-per-token column as one input among three:

flowchart TD
    Q["What is your bottleneck?"]
    T["Optimize for throughput<br/>(concurrency, req/s)"]
    P["Optimize for price per token"]
    A["Optimize for precision"]
    TW["Watch out for premium models<br/>that are fast and pricey"]
    PW["Watch out for cheap models<br/>that rate-limit"]
    AW["Watch out for cheap models<br/>that over-flag"]
    Q -->|"bursty volume,<br/>a queue draining"| T --> TW
    Q -->|"steady high volume"| P --> PW
    Q -->|"ambiguous inputs"| A --> AW
    style Q fill:#f7c47f,color:#000
    style T fill:#7fc47f,color:#000
    style P fill:#7fc47f,color:#000
    style A fill:#7fc47f,color:#000
    style TW fill:#f77f7f,color:#fff
    style PW fill:#f77f7f,color:#fff
    style AW fill:#f77f7f,color:#fff

In practice, our scan's model was swapped twice, and the cost went up before it came down, because each switch fixed one corner and gave up another.

The earlier models flagged more email, but the extras were look-alikes ("rate your hotel stay", "your trial is ending"). More flags looks like better recall; it was lower precision. Where it counted, all three extracted the same amounts, so the upgrade gave up nothing real.

The three models on the three axes:

And why we moved each time. A 429 is the HTTP status a provider returns when you've hit its rate limit.

flowchart LR
    M["mistral-large<br/>~$0.66"]
    G["glm-5.1<br/>~$1.64<br/>fixed speed"]
    D["deepseek-v4-pro<br/>~$0.40<br/>first to win<br/>cost and speed"]
    M -->|"1. too slow:<br/>50% of calls 429'd"| G
    G -->|"2. 3.7× the cost"| D
    style M fill:#9a9a9a,color:#fff
    style G fill:#9a9a9a,color:#fff
    style D fill:#7fc47f,color:#000

Measure all three axes at once. A price win that loses throughput or precision along the way is a loss you'll discover in production.

Technique 2: Cache-aware prompts stop you re-paying for your own instructions

Providers cache the longest identical leading prefix of a request (the opening stretch that matches an earlier request byte for byte) and bill it at the hit rate, which is often ~100× cheaper than a fresh miss. One rule follows from that:

Every token before the first byte that changes between two calls can be nearly free. Every token after it is full price.

Most prompts are written for the human reading them, so per-request data gets spliced into the middle of the instructions. Every splice is a cache guillotine: the identical rules after it are re-billed on every call.

Green is cached at the hit rate, red is billed in full on every call. Moving the per-request line from the middle to the end pushes the cache boundary down past the whole rule block.

graph LR
    subgraph BEFORE["Before: a mid-prompt variable cuts the cache early"]
        direction TB
        b1["Rules, part 1"] -->|"cache breaks here"| b2["Attachments: receipt-4823.pdf"] --> b3["Rules, part 2"] --> b4["The input"]
    end
    subgraph AFTER["After: whole instruction block static, variables last"]
        direction TB
        a1["Rules, part 1"] --> a2["Rules, part 2"] -->|"cache breaks here"| a3["Attachments: receipt-4823.pdf"] --> a4["The input"]
    end
    style b1 fill:#7fc47f,color:#000
    style b2 fill:#f77f7f,color:#fff
    style b3 fill:#f77f7f,color:#fff
    style b4 fill:#f77f7f,color:#fff
    style a1 fill:#7fc47f,color:#000
    style a2 fill:#7fc47f,color:#000
    style a3 fill:#f77f7f,color:#fff
    style a4 fill:#f77f7f,color:#fff

"After" caches its entire rule block on every call. "Before" re-pays for half of it forever, and the answers come out byte-identical either way.

In code, two interpolations sink most prompts, and you'd write both without a second thought:

// BEFORE: two everyday interpolations, both in the wrong place
function buildPrompt(email: Email): string {
  return `Current date: ${today()}
You are an expense classifier. Rules:
[ ...hundreds of tokens of rules, identical on every call... ]
${email.hasAttachments ? `Attachments: ${email.attachmentNames.join(", ")}` : ""}
[ ...more rules... ]

Email:
${email.body}`;
}

The Current date: line at the top changes daily, so the prefix differs from byte one and the entire prompt, rules and all, misses the cache every day. This is the most common form of the mistake: Today is {date} pinned to the top of a system prompt. The hasAttachments ternary in the middle cuts the cache right where it sits, so every rule after it is re-billed on every call even though none of them ever change.

The fix is to make the instruction block a constant and let every variable fall to a tail after it:

// AFTER: one constant instruction block; all variables in the tail
const SYSTEM = `You are an expense classifier. Rules:
[ ...all the rules, identical on every call... ]
- If attachments are listed below, treat the email as likely financial.`;

function buildPrompt(email: Email): string {
  return `${SYSTEM}

Current date: ${today()}
Attachments: ${email.attachmentNames.join(", ") || "none"}
Email:
${email.body}`;
}

SYSTEM is byte-identical on every call, so it caches in full. today(), the attachment list and the body are the only parts that vary, and they sit after the cached prefix, where they cost what they should and nothing more.

In practice, reordering alone lifted the input cache-hit rate 41% → 51% and cut cost 9%, with provably identical output. It's the cheapest win in this guide: the model does exactly what it did before, and the cache finally gets to do its job.

On the four corners this one is free. Cost goes down, and accuracy and observability don't move. Always do it first.

Technique 3: Output-aware design trades insight you don't need for cost you forgot

Caching only helps with input, and input is rarely where the money is. The money is in the output, which you can never cache. On a reasoning model the bill is dominated by reasoning_tokens, the model's "thinking", billed at the full output rate. Thinking can't be cached either, so the only way to pay less for it is to ask for less.

This is the one technique that touches observability, so it's where the trade gets explicit. I made two cuts:

1. Drop fields the consumer never reads. We asked for a human-readable reason on every call. A reasoning model has already worked out its rationale while thinking, so writing it again in the answer pays for it a second time, in expensive output tokens.

2. Short-circuit the common case. ~80% of email is not an expense, and for those, every field except is_invoice: false is dead weight. So the model returns only that.

The short circuit, per email:

flowchart TD
    E["Incoming email"]
    C["1. Model decides is_invoice"]
    F["2. Return only<br/>is_invoice: false"]
    T["2. Return the full set of fields"]
    E --> C
    C -->|"false (~80% of email)"| F
    C -->|"true"| T
    style E fill:#f7c47f,color:#000
    style C fill:#7fc47f,color:#000
    style F fill:#9a9a9a,color:#fff
    style T fill:#4f8cf7,color:#fff

In practice, asking for the minimum did more than shorten the answer. Told to be terse, the reasoning model also reasoned less: output −34%, reasoning tokens −29%, and cost another −18%.

Here is what it cost us. Dropping reason gave up observability: that string was the per-email explanation in our scan log ("Marketing newsletter", "loyalty points"). Without it, the log just says "Not an expense". The rule we used to decide:

Pay for observability where someone reads it. Don't bill the user for it on the hot path.

The harness can switch reason back on for internal runs. In production it was cost without a reader, so it went. Diagnostic detail belongs in the path you run a thousand times, not the one you run a billion.

Technique 4: Batching trades latency for half the bill

The first three techniques shrink the cost of each call. Batching changes the price of a call by changing how you send it. Most providers run an asynchronous batch endpoint at roughly half price, for work that doesn't need an answer this second.

What you pay is latency. A batch is finished only when every request in it is finished, so each result arrives later. For an interactive chat that's fatal. For anything asynchronous, like a nightly job, a queue drain or an inbox scan, it costs almost nothing. If no human is watching a spinner, batching is close to a pure cost win.

Batching belongs in the scheduler. If your pipeline already rate-limits its LLM calls through a queue (at production volume it should), that queue is the natural place to batch. The scheduler collects items until it has N of them or T milliseconds have passed, sends them as one batch request and fans the results back out:

flowchart LR
    R1["Request"]
    R2["Request"]
    R3["Request"]
    Q["1. Scheduler collects<br/>until N items or T ms"]
    B["2. One batch call<br/>at ~½ price"]
    O1["3. Result"]
    O2["3. Result"]
    O3["3. Result"]
    R1 --> Q
    R2 --> Q
    R3 --> Q
    Q --> B
    B --> O1
    B --> O2
    B --> O3
    style R1 fill:#f7c47f,color:#000
    style R2 fill:#f7c47f,color:#000
    style R3 fill:#f7c47f,color:#000
    style Q fill:#7fc4c4,color:#000
    style B fill:#4f8cf7,color:#fff
    style O1 fill:#7fc47f,color:#000
    style O2 fill:#7fc47f,color:#000
    style O3 fill:#7fc47f,color:#000

The idea reaches past hosted APIs. On a self-hosted model, batching is how you use the GPU you're paying for. Serving stacks like vLLM and TGI do continuous batching out of the box: many requests are interleaved through one forward pass (one run of the model over its inputs) where a naive server would run them one at a time. The throughput you get still depends on tuning it: batch size, max tokens, how long you'll wait to fill a batch. Idle compute is the most expensive token of all.

Batching fits high-volume work that can tolerate latency. An async pipeline like the inbox scan has exactly that shape: a worker already drains a rate-limited queue, so the scheduler is sitting right there, waiting to be taught to batch. It's the only technique here that spends performance to save cost, so it earns its place only where latency is slack. Where it is, ~2× is on the table.

Putting it together: what to trade, what to refuse

With the three per-call techniques stacked, the cost per mailbox walked down like this, while throughput went up and accuracy never moved. Batching is the lever we'd reach for next, as volume grows.

flowchart TD
    S0["Original: mistral-large<br/>~$0.66 / mailbox<br/>(rate-capped)"]
    S1["~$0.52 / mailbox"]
    S2["~$0.47 / mailbox"]
    S3["~$0.38 / mailbox"]
    S0 -->|"1. model: switch to deepseek-v4-pro"| S1
    S1 -->|"2. cache: static-prefix prompt"| S2
    S2 -->|"3. output: minimized output"| S3
    style S0 fill:#9a9a9a,color:#fff
    style S1 fill:#4f8cf7,color:#fff
    style S2 fill:#4f8cf7,color:#fff
    style S3 fill:#7fc47f,color:#000

The number matters less than the position we took on each corner, decided on purpose and checked with the test:

Your feature may answer differently. A medical or legal classifier might keep every reason token and eat the cost, because there the explanation is the product. You choose what to value. The framework only makes you price each of those things, so the trade happens on purpose instead of turning up later on an invoice.

There's also a reason to cut cost that has nothing to do with margin. Inference cost is priced into what users pay, so every multiple you remove comes off their bill, and a cheaper product is one more people can afford to run. Done well, this kind of optimization is how you let more people in, and to me that matters more than what we keep.

If you read this far, there's a part of Guliel built for you: a fully typed REST API and an MCP server. You can wire your own financial operations into your code or an AI agent: issue documents, pull reports, reconcile and scan expenses. We hold that surface to the same discipline that keeps our inference honest, which is what keeps it clean enough to hand to you.

Explore the Guliel API & MCP

Sapir