L0
Energy
Sets the price of power
The electricity and water a data center takes in, from the point they enter the facility: what
supply is contracted, what it costs, how much the connection can deliver, and the water that cooling
consumes. How much of it you control depends on
whether you own the facility, colocate, rent equipment, or buy cloud capacity with energy folded into
the rate card. Energy per token is a rate, valid within one serving configuration, not a comparison
across models or environments.
L1
Physical infrastructure
Sets cost per token
The accelerator and its generation, with the memory, interconnect, cooling, and density it ships
with, and the regions and availability zones it runs in. This is the equipment and location that
turn power into tokens. Not all tokens are equal: a low-latency token on a general-purpose GPU can
cost more than the same token on purpose-built silicon.
L2
Infrastructure management
Sets effective cost
How much of that hardware does useful work. Cluster orchestration, scheduling and queues,
autoscaling, GPU pools, quotas and multi-tenancy, and the shared memory and storage pools that
hold caches and checkpoints, all decided against a traffic shape that rarely matches the
commitment. Idle capacity raises the effective cost of the work that does run.
L3
Inference stack
Suppresses repeated work
The serving software and the runtime around it: engine, batching, KV and semantic caches, session
memory, tool calls, runtime safety checks, and the agent loop that plans, calls, and calls again.
Usually the single largest source of optimization, and the first layer where consumption is
suppressed rather than merely priced. Cache hit rate is the number that matters most, and it is
won largely through prompt and prefix stability.
L4
Model management + selection
Right-sizes what runs
Which model runs, from which provider and region, with what runtime configuration and adapted
weights, chosen from a catalog of capabilities, pricing, latency, and quality. Running a
frontier-scale model for a trivial task is the classic waste. Right-sizing is what pulls
multiplicative work back toward linear; runtime configuration and adapter strategy set what each
token costs once the class is settled.
L5
AI access + control
Bounds fan-out and retries
The governed entry point in front of every model: admission, pass-through proxying, routing,
policy and guardrails, and attribution. It decides which model serves each request, enforces
budgets and quotas, bounds agent fan-out and tool use, and stamps each request with the user, app,
agent, team, and use case it belongs to. It is the only layer that can decide a request needs no
model at all, and the only one that can stop a workload consuming without limit. Routing is an
optimization; a kill switch is a containment. A workload with no ceiling has no worst case.
L6
Demand + entry points
Sets the demand
Where consumption starts: who or what is asking, through which interface, for which use case, and
with what inputs. A person in a chat window, a scheduled job, and an agent calling through MCP
produce different request patterns against the same model. The use case sets the quality,
latency, and autonomy the work requires, and the request carries the prompts, documents, and
connected data that become context. Every layer beneath serves the demand this one creates.