Tokens Are Not Value: Economic Accounting for AI Inference

Why this is an economic accounting paper

Hardware efficiency and benchmark quality matter, but neither identifies economic value. Economic value requires a counterfactual. The relevant question is not how much revenue appeared next to an AI system. It is how outcomes changed relative to what would have happened without the intervention, net of declared costs and harms.

This distinction is familiar in program evaluation [20, 12]. It becomes easy to lose in AI measurement because providers observe enormous streams of tokens and relatively few well-identified outcomes. A large intermediate data set does not compensate for a missing counterfactual.

Contribution and novelty boundary

Subword tokenization, model evaluation, energy reporting, causal inference, cost-benefit analysis, and ratio inference have mature literatures. This paper does not replace them. Its contribution is a shared accounting contract that keeps their outputs typed and forces a study to state where evidence ends.

Tokenization

Modern language models commonly operate on subword units. Byte-pair encoding for neural machine translation was developed to handle rare and open-vocabulary forms [22]. SentencePiece provides language-independent subword tokenization from raw text [13]. These methods were designed for modeling and engineering goals, not as natural units of economic output.

Tokenizer choice can affect sequence length, multilingual representation, performance, and compute. Rust et al. [21] show that tokenizer quality helps explain differences between multilingual and monolingual models. Ali et al. [1] report that tokenizer choices can alter downstream performance and training cost. This empirical literature studies more than the formal counting issue proved here. Our theorem needs only the simpler fact that different deterministic segmentations can assign different counts to the same string.

Evaluation and TEVV

HELM argues for multi-metric, scenario-based evaluation rather than one aggregate score [14]. NIST’s AI Risk Management Framework and TEVV program emphasize context, documented metrics, validity, reliability, robustness, safety, and ongoing evaluation [17, 18]. These sources support the separation between model output and validated system performance.

A task metric still does not equal realized value. Accuracy on a frozen data set is evidence about one stage. Workflow adoption and downstream outcomes require field evidence.

Inference energy

MLCommons measures full-system AC power at the wall for the accompanying benchmark configuration and warns that thermal design power or power-supply ratings are not validated measurements [16]. Henderson et al. [9] propose systematic reporting of machine-learning energy and carbon. Luccioni et al. [15] compare inference energy across model and task classes.

These studies make the denominator more credible. They do not choose the economic numerator. Carbon also differs from energy because location, time, and generation mix intervene.

AI productivity evidence

Noy and Zhang [19] ran an incentivized experiment on professional writing tasks and found faster completion and higher evaluated quality with generative AI in that setting. Brynjolfsson, Li, and Raymond [4] studied a staggered deployment to 5,172 customer support agents and estimated a 15 percent increase in issues resolved per hour, with substantial heterogeneity.

Dell’Acqua et al. [5] report field-experimental evidence from consulting tasks. Performance improved on tasks within the model’s capability frontier, while participants using AI were less likely to reach correct solutions on a selected task outside that frontier. Becker et al. [3] randomized AI access across 246 tasks completed by 16 experienced open-source developers and estimated a slowdown in that narrow setting. Humlum and Vestergaard [11] use survey and administrative data to study broader workplace and labor-market adjustment.

The mixed results are informative. They do not imply that AI is generally helpful or harmful. They show why task, worker, deployment, and horizon belong in the estimand. None of these studies jointly meters experimental-unit wall energy and counterfactual net value, so none directly estimates the ratio proposed here.

Ratios and causal inference

Fieller [7] developed confidence sets for ratios whose denominator is estimated. Gleser and Hwang [8] show why bounded confidence procedures can fail badly in related weak-denominator problems. Efron and Tibshirani [6] provide the standard bootstrap reference. Interference and spillovers require potential outcomes indexed by more than a unit’s own assignment [10, 2].

Typed stages

For an experimental unit $i$, define:

• $E_i$: operational wall energy under a named boundary;

• $C_i$: named computation events under a hardware and software stack;

• $T_{i,\kappa}$: tokens under tokenizer $\kappa$;

• $O_i$: verified task outcome under protocol $P$;

• $A_i$: adopted or realized use;

• $\Delta W_i$: counterfactual net value under functional $W$.

The path is $$E_i\to C_i\to T_{i,\kappa}\to O_i\to A_i\to\Delta W_i.$$ The token node is explicitly indexed by $\kappa$. The outcome node is indexed by task distribution, scoring rule, quality threshold, latency budget, and version. The value node is indexed by counterfactual and horizon.

Why stages do not collapse

An output token can be part of a correct answer, an unnecessary restatement, or a harmful fabrication. Verification changes the type from emitted symbol to scored outcome. Adoption changes the type again. A correct answer that nobody uses has no realized workflow effect, although it remains a correct answer.

The final stage is causal. If $Y_i(1)$ and $Y_i(0)$ are potential outcomes with and without AI access, then $$\Delta W_i=W\!\left(Y_i(1)\right)-W\!\left(Y_i(0)\right).$$ Only one potential outcome is observed. Randomization or another identification strategy is needed.

Net value

Let $G_i(z)$ be gross realized benefit, $R_i(z)$ rework cost, $F_i(z)$ failure cost, and $K_i(z)$ other nonenergy opportunity cost under assignment $z$. One declared net-value functional is $$N_i(z)=G_i(z)-R_i(z)-F_i(z)-K_i(z).$$ Energy purchase cost can be included or excluded, but the choice must be explicit. When the denominator is physical joules, excluding monetary energy cost from $N$ often avoids mixing an electricity price into both the numerator and interpretation. Environmental externalities can be reported as a separate vector component or monetized under published weights.

Boundary manifest

The denominator records model, precision, hardware, software, prompt distribution, output rule, batching, latency target, retries, idle allocation, host and memory, networking, facility overhead, and time interval. Embodied energy is separate unless a lifecycle study allocates it.

The numerator records task version, affected population, quality rule, adoption rule, rework, failure costs, counterfactual, price year, discounting, and spillover exposure. A study that measures one side at the request level and the other at the firm-quarter level needs a bridge.

Multilingual consequences

Different languages and scripts can have different token fertility under the same multilingual tokenizer. A token-priced service can therefore charge different token counts for semantically comparable content. This is an engineering and distributional issue, not proof that every language has a different economic value. Comparisons should report language mix and tokenizer version.

The legitimate fixed-tokenizer use

When tokenizer, model, precision, decoding rule, and workload are fixed, tokens per joule can diagnose serving efficiency. It can compare batching or kernel changes for the same service contract. The conclusion is not “stop measuring tokens.” It is “do not relabel the operational measure as value.”

Billing is not welfare accounting

Token billing allocates provider costs and manages demand. A billing unit need not equal a welfare unit. Postal services charge by weight even though a letter’s value is not its mass. The analogy is limited but useful: an operational meter can work well without measuring benefit.

Goodhart pressure

If teams are rewarded for tokens per joule, they can raise the numerator by changing tokenizer or verbosity. A fixed protocol reduces this gaming but does not connect tokens to usefulness. Outcome metrics should sit beside operational metrics, and economic claims should use downstream evidence.

Task schema

A task specification is $$P=(\mathcal{D},S,q^\star,L^\star,d,v),$$ where $\mathcal{D}$ is task distribution, $S$ scoring rule, $q^\star$ quality threshold, $L^\star$ latency budget, $d$ difficulty weight, and $v$ version. For unit $i$, $$O_i=d_i \mathbf{1}\{S_i\geq q_i^\star\} \mathbf{1}\{L_i\leq L_i^\star\} \mathbf{1}\{\text{no terminal failure}\}.$$

Quality and latency are joint constraints. An accurate response delivered after a contractual deadline can fail the service. A fast incorrect response also fails.

Retries

Retries consume energy and time. Let attempts $j=1,\ldots,J_i$ have energy $E_{ij}$. The task denominator includes $$E_i^{\mathrm{attempt}}=\sum_{j=1}^{J_i}E_{ij}$$ when retries fall inside the boundary. Reporting only the successful attempt conditions on success and understates energy per delivered outcome.

Retries can be automatic, user-initiated, or caused by validation failure. Their causes belong in diagnostics. Their energy belongs in the total once.

Rework

Rework includes human review, correction, additional inference, testing, and downstream repair. Monetary rework cost enters net value. Measured compute used for rework enters energy if the boundary covers the full workflow. Human labor is not converted into metabolic joules. It remains labor time or cost.

Terminal failures and missing outcomes

Timeouts, refusals, malformed responses, abandoned sessions, and unmeasured downstream results need explicit branches. Dropping them can make quality and energy metrics look better. Missingness assumptions should be stated, and bounded analyses should replace point estimates when outcomes cannot be recovered.

Reliability

One successful run does not establish a reliable service. Repeated tasks, prompt variation, temporal drift, and rare high-cost errors matter. A study can report a service frontier over quality, latency, failure probability, and wall energy rather than collapsing every dimension.

Non-overlapping components

For unit $i$, define $$E_i^{\mathrm{wall}} =E_i^{\mathrm{initial}} +E_i^{\mathrm{retry}} +E_i^{\mathrm{host}} +E_i^{\mathrm{network}} +E_i^{\mathrm{facility}} +E_i^{\mathrm{idle,alloc}} +E_i^{\mathrm{rework}}.$$ The components must be non-overlapping. If a wall meter already captures host and network energy, those terms are not added again. The equation is a schema, not a command to sum duplicate telemetry.

Idle allocation

Serving systems reserve capacity. Idle energy can be allocated by request time, peak capacity, tokens, or another rule. Rankings can change with the rule. Publish both active-only and allocated operational results when possible.

Wall measurement

Wall measurement integrates power: $$E_i=\int_{t_{i,0}}^{t_{i,1}}P(t)\,dt.$$ Clock alignment matters when task logs and power logs use different systems. Sampling must capture bursts. Calibration, uncertainty, warm-up, cooldown, and background load need documentation.

MLCommons provides a strong benchmark precedent because its measured power is valid for the accompanying system and benchmark, not every deployment [16]. A field study needs the same humility.

Control energy

The control arm can use energy. A human workflow may use laptops, search, databases, and office infrastructure. The causal denominator is $$\tau_E=\mathbb{E}[E_i(1)-E_i(0)],$$ not treatment energy alone. A system that replaces more energy-intensive work can have $\tau_E<0$. The sign changes interpretation.

Operational is not lifecycle

Operational wall energy excludes embodied equipment and construction unless allocated separately. A lifecycle study can add non-overlapping embodied components under a lifetime and utilization model. It should not call a wall meter lifecycle energy.

Potential outcomes

Let $Z_i\in\{0,1\}$ assign AI access. Define potential net value $N_i(z)$ and operational wall energy $E_i(z)$. The average effects are $$\tau_N=\mathbb{E}[N_i(1)-N_i(0)], \qquad \tau_E=\mathbb{E}[E_i(1)-E_i(0)].$$

For $\tau_E>0$, $\theta$ is expected incremental net value per additional operational joule under the study population and assignment policy. For $\tau_E<0$, the intervention saves energy; the sign and numerator should be reported separately rather than summarized with a slogan.

Identification

In a randomized experiment with no relevant interference, arm means identify $\mathbb{E}[N_i(z)]$ and $\mathbb{E}[E_i(z)]$. In observational deployment, selection into AI use can depend on task difficulty, worker skill, expected benefit, and management. Regression adjustment needs assumptions. Revenue divided by observed AI electricity does not identify $\theta$.

Treatment policy

“AI access” is incomplete. Treatment includes model and version, interface, training, price, usage limits, workflow, and enforcement. An intention-to-treat estimand measures assignment to access. A treatment-on-the-treated estimand needs uptake modeling. Selective usage can make naive user-versus-nonuser comparisons misleading.

The result is conditional, not automatic. Exclusion fails if mere access changes training time, reserved capacity, idle energy, or behavior among nonadopters. In that case assignment and adoption define different policies, and the numerator and denominator need a design that represents those routes.

Net value and transfers

Revenue is not always social value. Payments can transfer surplus. A firm perspective can use profit or contribution margin. A social perspective can include consumer surplus, labor effects, externalities, and distributional weights. The paper permits several $W$ functions but requires one to be named.

Horizon

Short-run completion time can improve while long-run learning, skill, quality, or market equilibrium changes in another direction. The estimand includes a horizon $\tau$. A one-day experiment should not be silently projected over years.

Causal version

Our target is $$\theta = \frac{\mathbb{E}[N(1)-N(0)]} {\mathbb{E}[E(1)-E(0)]}.$$ It is not $$\mathbb{E}\left[ \frac{N(1)-N(0)}{E(1)-E(0)} \right].$$ The unit-level expression may be undefined when an individual energy effect is zero, and individual potential-outcome differences are not jointly observed.

Aggregation level

Pure regrouping does not change a ratio of totals. If net value and energy are summed over the same units with the same weights and boundary, then $$\frac{\sum_i N_i}{\sum_i E_i} = \frac{\sum_g\sum_{i\in g}N_i}{\sum_g\sum_{i\in g}E_i}.$$ The ratio of expectations inherits this volume-aggregation invariance under consistent weighting. The expectation of unit ratios does not: changing from requests to workers changes which ratios receive equal weight and can also change which zero denominators are discarded.

Aggregation can still change the causal ratio when it changes weights, captures shared idle energy, or incorporates team spillovers and interference missing at request level. Equal weighting of unequal-sized clusters is not pure regrouping. A study should define its unit, boundary, and weighting rule before estimation.

Difference-in-means estimator

For treated and control sample means, $$\widehat\tau_N=\bar N_1-\bar N_0, \qquad \widehat\tau_E=\bar E_1-\bar E_0, \qquad \widehat\theta=\frac{\widehat\tau_N}{\widehat\tau_E}.$$ Under random assignment, the first two estimators have standard randomization-based interpretations. Ratio bias can remain in finite samples.

Let $$\widehat V_N=\widehat{\operatorname{Var}}(\widehat\tau_N),\qquad \widehat V_E=\widehat{\operatorname{Var}}(\widehat\tau_E),\qquad \widehat C_{NE}=\widehat{\operatorname{Cov}} (\widehat\tau_N,\widehat\tau_E).$$ These are estimated sampling variances and covariance of the two effect estimators, not raw sample variances of $N_i$ and $E_i$. The covariance matters because high-value tasks may also consume more energy.

Delta interval

When $\widehat\tau_E$ is far from zero, the delta-method variance is $$\widehat{\operatorname{Var}}(\widehat\theta) \approx \frac{\widehat V_N}{\widehat\tau_E^2} +\frac{\widehat\tau_N^2\widehat V_E}{\widehat\tau_E^4} -2\frac{\widehat\tau_N\widehat C_{NE}}{\widehat\tau_E^3}.$$ This approximation can fail near a zero denominator and can hide the correct unbounded geometry.

Fieller confidence set

For normal critical value $q$, candidate ratio $r$ belongs to the Fieller set when $$(\widehat\tau_N-r\widehat\tau_E)^2 \leq q^2(\widehat V_N-2r\widehat C_{NE}+r^2\widehat V_E).$$ Rearranging gives $$Ar^2+Br+C\leq0,$$ with $$A=\widehat\tau_E^2-q^2\widehat V_E, \quad B=-2(\widehat\tau_E\widehat\tau_N-q^2\widehat C_{NE}), \quad C=\widehat\tau_N^2-q^2\widehat V_N.$$ Depending on $A$ and the discriminant, the confidence set can be bounded, disjoint and unbounded, all real numbers, or empty under the approximation.

An unbounded result is not a software defect. It says the data do not exclude a denominator near zero strongly enough to support a bounded ratio.

Bootstrap

A stratified bootstrap resamples treated and control units separately and recomputes both effects and their ratio. It preserves arm sizes and the within-unit value-energy relation. Cluster assignment requires cluster resampling.

Percentile intervals can behave poorly near zero denominators. The software counts resamples whose incremental energy is too close to zero and refuses to report an interval when too many are invalid. Fieller output remains the primary weak-denominator diagnostic.

Multiple outcomes

Quality, latency, failures, rework, and value create multiple testing and selection risks. The analysis plan should identify primary outcomes and report the rest as a vector. Tuning the value functional after seeing results defeats its role as a declared estimand.

Why SUTVA can fail

One worker’s AI use can affect colleagues through shared queues, copied outputs, training, review load, and changed norms. One task can populate a cache or knowledge base used by later tasks. Then $$N_i(\mathbf{Z})\neq N_i(Z_i)$$ because outcomes depend on the assignment vector.

Exposure mappings

An exposure mapping $g_i(\mathbf{Z})$ can classify a unit by own treatment and neighbors’ treatment [2]. Direct, indirect, total, and overall effects are different estimands. Energy must be allocated at the same exposure level.

Cluster randomization

Randomizing teams or time blocks can reduce contamination. It lowers the effective sample size and requires cluster-aware inference. Shared facility energy may naturally live at the cluster level.

General equilibrium

A firm deployment can change prices, hiring, task volume, and demand. A local randomized effect does not equal an economy-wide equilibrium effect. Humlum and Vestergaard [11] illustrate the value of studying broader labor outcomes separately from controlled task productivity.

Rebound

Cheaper inference can increase demand. Even if value per inference joule rises, total energy may rise. The causal ratio in this paper is policy and horizon specific. It is not a forecast of aggregate energy demand.

Writing tasks

Noy and Zhang’s experiment [19] offers credible evidence that generative AI access changed completion time and evaluated quality for a specified set of professional writing tasks. In our chain, it informs the verified-outcome stage. To estimate incremental value per joule, the same units would need metered treatment and control energy, adoption and rework records, and a value functional.

Customer support

Brynjolfsson, Li, and Raymond [4] analyze deployment at scale and find heterogeneous productivity changes. Issues resolved per hour is closer to a workflow outcome than token volume. It still requires quality checks, customer effects, costs, and wall-energy measurement for our target ratio.

Consulting

The jagged-frontier study [5] is especially relevant because effects changed with task location relative to model capability. More output was not uniformly better. A task distribution that omits outside-frontier work can overstate deployment value.

Software development

Becker et al. [3] study experienced developers working in familiar repositories and estimate slower completion with the tested early 2025 tools. The sample and tool era limit generalization. The study still demonstrates that review, prompting, and context costs can outweigh generation speed in a real workflow.

What the studies do not say

The studies do not show that tokens are useless. They do not establish a universal effect of AI. They also do not measure causal value per wall joule. Their strength is design-specific evidence about intermediate and outcome stages. Combining their productivity estimates with unrelated provider energy numbers would create a synthetic ratio with mismatched units and populations.

Unit and assignment

Choose the unit at which treatment and spillovers are credible: request, task, worker, team, or time block. Record assignment before uptake. Preserve failed and abandoned units.

Treatment definition

Pin model, endpoint, date, system prompt, tools, tokenizer, decoding parameters, context policy, user interface, training, and quotas. Provider model aliases can drift and should not stand alone as version identifiers.

Control definition

The control can be usual workflow, alternative software, another model, or no AI. Record its compute and labor inputs under a compatible boundary. A zero-energy control is rarely appropriate for workplace tasks.

Energy collection

Use synchronized wall meters where feasible. Record idle, warm-up, networking, retries, and rework. If facility overhead is allocated, publish the rule and a sensitivity analysis. Report device telemetry separately.

Outcome collection

Blind evaluators to assignment when possible. Record quality, latency, reliability, adoption, rework, and harms. Predefine the task distribution and failure policy. Link downstream outcomes without dropping unmatched cases.

Value collection

Choose firm, worker, customer, or social perspective. State price year, discount rate, transfers, externalities, and horizon. Estimate numerator components before aggregation so readers can reweight them.

Analysis

Report $\widehat\tau_N$ and $\widehat\tau_E$ separately with intervals. Then report the ratio, Fieller set, bootstrap diagnostic, covariance, and weak-denominator warning. Show task-stratum and allocation sensitivity without turning exploratory subgroups into confirmatory findings.

Tokenizer rank reversal

The Python fixture compares abcdefghij with a b c d e at one joule each. Whitespace tokenization assigns counts one and five. Space-preserving character tokenization assigns counts ten and nine. The ordering reverses. No claim about semantic quality is needed.

Verbosity

At two joules, 42 has one whitespace token and The answer is 42 has four. If the task protocol extracts the same correct answer, tokens per joule changes from $0.5$ to $2.0$ while the declared outcome is unchanged.

Energy ledger

A constructed task uses $10$ joules for initial inference, $3$ for retries, $2$ for host and memory, $1$ for network, $4$ for facility overhead, $5$ allocated idle, and $6$ for downstream compute rework. The non-overlapping total is $31$ joules. If the wall meter already includes several components, the ledger must not add them again.

Randomized ratio fixture

Four treated units have net values $13,15,17,19$ and wall energies $16,17,18,19$. Four controls have values $8,10,12,14$ and energies $10,11,12,13$. Therefore $$\widehat\tau_N=5, \qquad \widehat\tau_E=6, \qquad \widehat\theta=\frac{5}{6}\approx0.8333.$$ The software retains variance $3.3333$, denominator variance $0.8333$, and covariance $1.6667$ for the effect estimators.

Under its normal approximation, the 95 percent Fieller set is approximately $[0.338,1.101]$. A 20,000-draw stratified percentile bootstrap with seed 9 returns approximately $[0.444,1.067]$. These are COMPUTATIONAL fixture results, not an empirical AI estimate.

Schemas

TaskSpec records task identifier, version, quality threshold, latency budget, and difficulty weight. TaskOutcome records score, latency, adoption, failure, retries, gross realized value, rework, and other nonenergy cost. EnergyAccount records non-overlapping operational components.

Counterexamples

Whitespace, character, and UTF-8 byte tokenizers make the representation choice inspectable. has_rank_reversal checks whether two named tokenizers reverse an ordering.

Estimators

estimate_incremental_ratio computes arm differences, sampling variances, covariance, and their ratio. fieller_confidence_set returns bounded, unbounded, all-real, or empty approximate sets. bootstrap_ratio_interval resamples within assignment arm and counts invalid weak-denominator draws.

Tests

Tests cover tokenizer counts, rank reversal, verbosity, task success, latency, adoption, retries, energy aggregation, ratio non-equivalence, difference in means, covariance, bounded and unbounded Fieller behavior, deterministic bootstrap, weak-denominator rejection, and schema validation.

Passing tests support COMPUTATIONAL claims about the code. They do not validate an external study’s randomization, meter, value functional, or spillover model.

No new field estimate

The paper proposes an estimand and implementation but supplies only constructed data. It does not estimate the return to a deployed AI service.

Value remains normative

Net value depends on perspective, horizon, prices, externalities, and distributional weights. A declared functional makes the choice visible but does not make it neutral.

Energy measurement can be expensive

Request-level wall metering is difficult in shared cloud infrastructure. Provider telemetry may omit host, network, and facility components. Allocation uncertainty can dominate statistical precision.

Rapid tool drift

Models, interfaces, user skill, and prices change quickly. A field estimate is tied to its deployment period. Replication over versions is necessary.

Fieller approximation

The implementation uses a normal approximation. Small samples, clustering, heavy tails, and complex assignment can require randomization inference or a more specialized model.

Bootstrap limits

The percentile bootstrap is simple and reproducible, but it is not reliable for every ratio problem. Near zero denominators demand unbounded-set diagnostics, not forced bounded intervals.

Interference

Exposure mappings can miss unknown spillovers. Team and market effects may remain after cluster randomization. The local estimand should not be advertised as an economy-wide effect.

Human outcomes

Productivity is not the only outcome. Skill, autonomy, satisfaction, error severity, privacy, and distribution matter. They should appear as separate outcomes before monetization.

General rank reversal construction

Let two strings have tokenizer counts $$(T_{\kappa_1}(A),T_{\kappa_1}(B))=(a_1,b_1)$$ and $$(T_{\kappa_2}(A),T_{\kappa_2}(B))=(a_2,b_2).$$ For equal energy, a reversal occurs when $$(a_1-b_1)(a_2-b_2)<0.$$ For unequal energies $E_A,E_B$, replace each count with count divided by its energy. The reference function implements this sign test.

Ratio weighting identity

For positive $E_i$, $$\frac{\sum_iN_i}{\sum_iE_i} = \sum_i \left(\frac{E_i}{\sum_jE_j}\right) \frac{N_i}{E_i}.$$ Thus the aggregate ratio is an energy-weighted mean of unit ratios. The unweighted mean answers another question.

Fieller geometry

The Fieller inequality is obtained by testing $$H_0:\tau_N-r\tau_E=0$$ with estimated variance $$\widehat V_N-2r\widehat C_{NE}+r^2\widehat V_E.$$ Inverting the tests over $r$ gives a confidence set. When the data do not exclude $\tau_E=0$, arbitrarily large ratios can remain plausible. A bounded interval would conceal that feature.