Stop counting AI tokens and start measuring shipped code

Token counts obscure real output

When developers hand a coding agent a task, the agent may iterate through planning, generation, testing and debugging, consuming 10-50 times more tokens than a simple chat prompt. Per-token pricing therefore makes the most productive workflows appear expensive.

Cost per merged pull request as a performance gauge

The author proposes measuring “cost per merged pull request” – total AI-assisted spend divided by the number of changes that pass review and are merged. Tracking rework and defects alongside this metric prevents cheaper but lower-quality changes from skewing results. If the cost falls while usage rises, developers are converting more token consumption into shipped software.

Matching models to task complexity

A second recommendation is to reserve frontier or on-premises flagship models for planning and novel reasoning, while routing routine code generation, test creation and refactoring to open-weight models the organization runs itself. A gateway can direct requests based on complexity, sensitivity, cost and latency, ensuring premium inference is billed only for the exceptions that truly need it.

Owned capacity versus rented inference

The article argues that owning inference capacity shifts costs from a variable per-token bill to a largely fixed, fully-loaded expense that drops per-call as usage scales. Owned capacity also keeps proprietary code on-premises, allowing only governed prompts to reach external models. HPE’s internal hybrid platform reportedly reduced token costs by 30-50 % (conflict of interest, own or unaudited numbers) by routing over 150 applications to on-prem open-weight models.

Test the operating model

At HPE, the hybrid AI platform routes routine workloads from more than 150 internal applications to open-weight models running on premises, including on HPE Private Cloud AI. The result is lower token costs and greater control over sensitive code.

Build for adoption, not rationing

Owned capacity suits organizations with sustained AI coding demand; low or unpredictable usage still favors API-based access. Most enterprises are expected to run hybrid setups, using frontier APIs for demanding or intermittent work and owned capacity for repeatable volume, with a governed gateway deciding what leaves the building.

To learn more about HPE Private Cloud AI, please visit HPE.

This content was produced by HPE. It was not written by MIT Technology Review’s editorial staff.

Source