Inference cost accumulates linearly with call volume

What it is

Conventional interface features have near-zero marginal cost: one more button rendered or page viewed costs the server almost nothing. Model inference breaks that intuition — every call genuinely burns power: loading weights, running the forward pass, generating tokens. Each call is small, but the total accumulates strictly linearly with call count, with no dilution by scale. A casual "let's add an AI feature" decision, multiplied by ten million daily users times dozens of calls per day, turns a near-zero-marginal-cost app into an always-on computational facility. The linear law ties inference cost directly to product decisions: every design choice of call frequency (auto-trigger or not, batched or not, skippable or not) is an energy decision.

Why it happens

Linearity comes from the structure of inference: each request independently executes the full computation, and the previous run does not discount the next (unlike human practice effects, and without classical cache-hit amortization — the same question asked twice is paid for twice). Three design variables set the slope. Model size: parameter count and per-call energy grow on roughly the same order; routing occasional hard problems to a large model and frequent simple tasks to a small one is the first determinant of slope. Generation length: autoregressive cost grows linearly with output tokens — verbose output is literally billed by the word. Triggering: auto-trigger (model called on every input) versus on-demand (called only on explicit user request) spans multiples of call volume — the biggest hidden variable, because it hides in interaction defaults rather than model configuration. Batching and specialized hardware lower per-call cost but change only the constant in the slope, never the linearity.

Studying it

Per-request energy benchmarking is the base measurement paradigm: on controlled hardware, measure single-inference energy (GPU power telemetry × duration) stratified by model size, sequence length, and batch size, building a size × length → energy lookup; on the data-center side multiply by PUE and grid carbon intensity for carbon attribution. Systems research measures real product call profiles: call-frequency distributions, input/output length distributions, and redundant-call share (repeat rate of identical requests), decomposing product-level energy down to interaction-pattern level. Methodological cautions: specialized accelerators and general-purpose GPUs differ in efficiency by an order of magnitude, so reports must pin hardware assumptions; amortizing training cost over projected calls is highly sensitive to usage frequency — short-lived features can see training amortization dominate inference, and conclusions need sensitivity analysis over expected product lifetime.

Where it stops holding

The linear law covers the compute portion of inference, bounded in two places. First, fixed costs exist: training is a one-time large outlay, and in low-frequency features the training amortization may dominate total cost — there the optimization question is "whether to build at all," not "how to call"; the linear law fits live features whose cost is call-driven. Second, the interaction layer mediates: users' sense of "intelligence" correlates non-linearly with call count — a thousand automatic-completion calls may buy less perceived fluency than one clearly requested response buys trust; so the energy-optimization headroom often lies in interaction design (removing unnecessary calls) rather than model compression (lowering each call's quality at a price).

Applying it

• Establish a call budget for every model feature: projected monthly inference cost (energy and carbon) from daily users × trigger frequency × average sequence length; exceeding budget is a design constraint surfaced up front, not discovered later.
• Default to on-demand triggering: generative features fire on explicit user action; auto-trigger (inference on every input) requires frequency-and-benefit evidence at review; high-frequency simple tasks route to small models, with large models reserved for explicitly requested hard problems.
• Constrain output length as a first-class setting: reply features carry length caps and a concise default, with full-length output behind explicit expansion.
• Verify: monthly review of call-volume distributions — redundant-call share (repeated identical requests) plus auto-trigger abandonment (inference completed but output unused); above threshold, tighten triggering conditions.

Related

• Same group: P4.15.1 Where energy is consumed decides who bears the cost · P4.15.4 The directional trade-off of caching
• Adjacent: L6 Interaction patterns of intelligent systems · P4.08.1 Energy facts of transit and computation
• Search terms: inference cost · per-request energy · LLM energy consumption