Quality-Adjusted Intelligence per Joule

Evidence status

The program uses explicit status labels:

• proved denotes a complete argument under stated assumptions;

• conditional denotes a result that depends on a measurement or identification design;

• computational denotes executable evidence;

• obstructed denotes a comparison blocked by incompatible data or assumptions;

• open denotes an unresolved question.

The tokenizer non-invariance result and the algebra of the estimator are proved. The rank-reversal fixtures, bootstrap implementation, and frontier algorithm are computational. Whether a chosen benchmark generalizes to a deployment population is conditional. A universal conversion from benchmark points to economic value is obstructed without a separate causal design.

Energy measurement

MLCommons has developed power-measurement procedures alongside MLPerf Inference. Its documentation describes full-system alternating-current measurement with approved analyzers and couples the power interval to the benchmark run [2, 1]. This is a strong operational starting point. It avoids treating thermal design power as measured energy and captures more of the system than accelerator telemetry alone.

The scope remains benchmark-specific. A wall measurement of the system under test does not automatically include upstream networking, cooling beyond the metered boundary, embodied hardware, or data-center construction. Conversely, multiplying by a power usage effectiveness value changes the boundary and adds uncertainty. An evaluator should not label a measured operational result “lifecycle energy” unless a non-overlapping lifecycle inventory has actually been constructed.

Prior work on Green AI called for reporting computational cost alongside accuracy [8]. Henderson and colleagues proposed systematic energy and carbon reporting [9]. These efforts establish that resource disclosure is part of responsible empirical work. They do not settle the output unit problem for generative or agentic systems.

Quality and scenario coverage

HELM argues that model evaluation should declare scenarios, adaptations, and metrics rather than infer broad capability from one aggregate score [3]. That structure is useful here. Energy is measured for a particular system under a particular evaluation protocol, while quality is defined over the same requests and service conditions. Neither half should be borrowed from an unmatched leaderboard.

NIST’s work on AI test, evaluation, validation, and verification emphasizes construct validity, reliability, context, and uncertainty [4]. NIST AI 800-3 distinguishes performance on a fixed benchmark from generalized performance over a broader population of similar questions [5]. That distinction applies directly to an energy-normalized ratio. A narrow confidence interval around a benchmark ratio does not prove that the ratio transports to a different user population.

Difficulty and task duration

METR defines a task-completion time horizon using the duration that human experts require for tasks that an AI system completes with a specified probability [6, 7]. Human duration is attractive because it provides an interpretable within-domain ordering and captures some forms of task length. It is not a universal difficulty scale. Tool familiarity, domain expertise, environmental setup, task ambiguity, and the selected human population all affect the duration. METR’s own methodology reports hierarchical uncertainty and limits claims to the studied task domain.

This paper therefore permits several declared difficulty conventions: expert-time bins, item-response estimates, rubric complexity, or no weighting. Comparisons must use the same convention. The unweighted and stratified results remain primary.

Task population

Let $\mathcal{T}$ be a finite or measurable set of task instances and let $\mu$ be the target distribution. A benchmark average usually substitutes the empirical distribution $$\widehat{\mu}(t_i)=\frac{1}{n}.$$ That substitution is defensible only if the benchmark itself is the target. If the target is a deployment population, the evaluator needs sampling or importance weights. For observed task $i$, let $w_i\geq 0$ denote its weight, with normalized weight $$\widetilde{w}_i = \frac{w_i}{\sum_{j=1}^{n}w_j}.$$

The distribution covers more than prompts. It includes tool availability, context length, input modality, required output format, time budget, and interaction policy. A change in any of these may define a new task population.

Difficulty

For task $t$, let $d(t)$ be a declared nonnegative difficulty descriptor. It may carry units, such as expert seconds. If difficulty enters a scalar service product, a published normalization maps it to a dimensionless weight $\delta(t)$. Raw $d(t)$ is used for stratification or model fitting and never enters a dimensionless product directly. Three reporting modes are allowed.

1. In stratified mode, $d$ assigns a bin and the evaluator reports a separate score for each bin.

2. In model-based mode, $d$ is estimated using a declared item response or regression model. The estimate and its uncertainty are reported.

3. In weighted mode, $d$ is mapped to dimensionless normative weight $\delta$ by a published normalization. The unweighted result must appear beside it.

Human expert duration may support the first two modes within a task family. A ten-minute legal research task and a ten-minute image-editing task do not become interchangeable merely because their durations match. Cross-domain weighting needs an additional service model. Rasch models provide one established approach to within-domain item scaling, but their latent scale still depends on model assumptions and identifying constraints [12].

Quality and correctness

Quality $q_i\in[0,1]$ records graded fitness for the task specification. Correctness $c_i\in[0,1]$ records factual, logical, or test-based correctness. They may coincide for a binary exact-answer task. They may differ for a translation or program patch, where a response can pass a narrow test while violating style, safety, or maintainability requirements.

The grading rule must be fixed before system outputs are inspected. Automated graders require validation against human or executable ground truth. Model-based judges should report judge identity, prompt, order effects, position randomization, and agreement. If correctness already includes every quality dimension, set $q_i=1$ rather than count the same property twice.

Reliability

Reliability $r_i\in[0,1]$ is the probability that the system reproduces the required outcome across the declared variation in seeds, retries, transient failures, and environment. It is not a decorative confidence score. The sampling unit must match the intended claim.

For a binary task with $x$ successes in $n$ independent trials, the point estimate is $\widehat{r}=x/n$. A Wilson interval is preferable to a naive normal interval near zero or one. If attempts are clustered by task family, model version, or site, a hierarchical model or cluster bootstrap is needed.

Correctness and reliability can overlap. If $c_i$ is the mean success across all retries, multiplying by a separately estimated $r_i$ would count failure twice. In that design set $r_i=1$, or redefine $c_i$ as conditional quality given a completed response. The manifest must say which design is used.

Usefulness without economic value

Usefulness $u_i\in[0,1]$ is a declared service weight inside the evaluation. It can downweight tasks that are technically valid but irrelevant to the intended workload. It cannot be inferred from token count or benchmark difficulty.

This paper stops at verified service. Revenue, cost savings, welfare, and scientific value require an adopted action and a counterfactual downstream effect. Calling $u_i$ an economic value coefficient would skip that causal work. Paper 5 in this program addresses the later chain.

Latency

Let $L^\star$ be the maximum acceptable latency. The simplest service indicator is $$\ell_i(L^\star)=\mathbf{1}\{L_i\leq L^\star\}.$$ Some applications may use a preregistered continuous penalty $\ell_i\in[0,1]$. A penalty chosen after observing model results is a ranking control, not a neutral metric.

A late answer has zero or reduced service under the declared contract, but its energy remains in the denominator. The same rule applies to invalid outputs, refusals that violate the task contract, crashes, retries, and timeouts.

Energy boundary

We distinguish four profiles.

Device operational.

Measured accelerator or device energy. Host, memory, networking, facility overhead, and embodied energy are excluded.

System operational.

Energy for the complete system under test at the wall. Warm-up, idle allocation, batching, retries, and serving overhead are declared.

Facility operational.

System energy plus measured or modeled facility overhead. Any PUE multiplier is indexed by site and interval.

Lifecycle.

A non-overlapping allocation of operational and embodied components over a declared lifetime and utilization schedule.

Results from different profiles cannot be ranked without a bridge. Device telemetry may diagnose prefill and decode behavior, but it is not a substitute for wall energy in a deployed-system claim.

Task-level contribution

For attempt $i$, define the dimensionless service contribution $$\begin{equation} s_i = \delta_i u_i q_i c_i r_i \ell_i(L^\star). \label{eq:service} \end{equation}$$ Here $\delta_i$ is one when difficulty is only stratified. If a difficulty-weighted sensitivity result is reported, $\delta_i$ is the declared dimensionless normalization of raw descriptor $d_i$. Every factor is declared and nonnegative. A product is convenient because a zero in a mandatory dimension makes the contribution zero. It is not compulsory. An evaluator may replace the product with a preregistered aggregation function $g(\delta_i,u_i,q_i,c_i,r_i,\ell_i)$. The exact function is part of the estimand.

The product is most defensible when factors represent separate interfaces. For example, a completed answer can receive a correctness score, while reliability is estimated from independent reruns and usefulness is fixed by the target task distribution. If factors are empirically redundant, the evaluator should report a joint outcome instead of multiplying correlated grades.

The expectation in the equation is over tasks drawn from $\mu$ and the system and environment data-generating law $\omega_x$. The uncertainty procedure $\pi$, such as a hierarchical bootstrap or mixed model, estimates sampling uncertainty but is not itself the source of outcomes.

The ratio uses expected service divided by expected energy. It is not the expectation of per-task ratios: $$\frac{\mathbb{E}[s]}{\mathbb{E}[E]} \neq \mathbb{E}\left[\frac{s}{E}\right] \quad\text{in general}.$$ The ratio of expectations answers a capacity question: how much expected service does the system provide for the expected energy spent on the target mix? The mean of ratios gives extreme influence to tasks with very small denominators and does not aggregate cleanly to fleet energy.

Finite-sample estimator

For weighted observations $i=1,\ldots,n$, $$\begin{equation} \widehat{\operatorname{QAIJ}}_{\mathcal{A}}(x) = \frac{\sum_i w_i s_i} {\sum_i w_i E_{b,i}}. \label{eq:estimator} \end{equation}$$ Normalization by $\sum_i w_i$ cancels. Failed and late observations enter with $s_i=0$ and their full measured energy $E_{b,i}>0$.

This property fails for an unweighted mean of task-level ratios. The pooled form also makes denominator auditing easier because reported service and energy totals remain separately inspectable.

Primary and sensitivity reports

A serious report should not print one $\operatorname{QAIJ}$ number. We recommend:

1. unweighted task service per joule;

2. service per joule by difficulty stratum;

3. a difficulty-weighted sensitivity result;

4. expected service and expected energy separately;

5. latency and reliability summaries;

6. the energy-quality frontier;

7. uncertainty intervals under alternative task mixes.

The vector makes normative choices visible. A procurement team may require a high reliability floor, while a batch research workflow may accept more variance for lower energy. A single weighted average conceals that choice.

Tokenizer non-invariance

Let a tokenizer $\kappa$ map the same output string $y$ to a sequence of tokens. Define $$\operatorname{TPJ}_\kappa(x) = \frac{\mathbb{E}[|\kappa(Y_x)|]}{\mathbb{E}[E_b(x)]}.$$

Verbosity rank reversal

The reference tests instantiate this example under the names wordy-wrong and concise-correct. The test is computational evidence that the implementation matches the proposition. The proposition itself is proved.

Output length can still matter

Output length affects latency, memory traffic, serving capacity, user reading time, and energy. It therefore belongs in the causal path. The correct interpretation is not “tokens never matter.” It is that token count is an intermediate quantity: $$E \longrightarrow \text{compute and memory traffic} \longrightarrow \text{tokens} \longrightarrow \text{graded task outcome}.$$ A shorter correct answer may reduce both energy and user burden. A longer proof may be required for verifiability. The task rubric, not token volume by itself, decides which output is better.

Distribution-specific efficiency

Consider two task classes, easy $e$ and hard $h$, each costing $10$ joules per attempt. System $A$ has service scores $$s_A(e)=1,\qquad s_A(h)=0.1,$$ while system $B$ has $$s_B(e)=0.4,\qquad s_B(h)=1.$$ Under an easy-heavy distribution $\mu_1(e)=0.9$, $\mu_1(h)=0.1$, $$\operatorname{QAIJ}_{\mu_1}(A)=0.091,\qquad \operatorname{QAIJ}_{\mu_1}(B)=0.046.$$ Under a hard-heavy distribution $\mu_2(e)=0.1$, $\mu_2(h)=0.9$, $$\operatorname{QAIJ}_{\mu_2}(A)=0.019,\qquad \operatorname{QAIJ}_{\mu_2}(B)=0.094.$$ The preferred system reverses.

This is not a defect in the metric. It reveals that “best model” is incomplete without a workload. Benchmark maintainers should publish the task weights, and deployers should rerun the estimator under their own target mix.

Difficulty shifts and Simpson effects

An aggregate can improve while every difficulty stratum worsens if task mix changes toward easier cases. Conversely, a system can improve within each stratum while its aggregate declines because the deployment admits harder work. Both are versions of a composition effect.

Every longitudinal report should therefore include:

• the marginal task distribution at each time;

• within-stratum quality and energy;

• a standardized estimate using a fixed reference distribution;

• the observed deployment estimate using the actual distribution.

The standardized estimate supports technical comparison. The deployment estimate supports operational accounting. They answer different questions.

Importance weighting

If benchmark tasks are sampled from proposal distribution $\nu$ but the target is $\mu$, importance weights are $$w(t)=\frac{\mu(t)}{\nu(t)}$$ where $\nu(t)>0$ whenever $\mu(t)>0$. The weighted ratio estimator uses the equation. Large weights increase variance and expose weak support. Trimming weights changes the estimand and must be reported.

If the benchmark contains no examples of a deployment task family, weighting cannot manufacture evidence. The appropriate status is obstructed, not an extrapolated score.

Definition

Let $\mathcal{X}$ be the feasible configurations of a deployed system. A configuration includes model, quantization, decoding, batching, hardware, parallelism, prompt policy, tool scaffold, and serving load. For $x\in\mathcal{X}$, let $S_\mu(x)$ be expected task service, $E_\mu(x)$ expected energy, and $L(x)$ a latency statistic.

A configuration $x$ dominates $y$ when $$E_\mu(x)\leq E_\mu(y),\qquad S_\mu(x)\geq S_\mu(y),$$ with at least one strict inequality, and both satisfy the same latency constraint. Dominated points should not be recommended for that task distribution unless an omitted attribute explains the choice.

Why the frontier is more informative than one ratio

A ratio draws rays from the origin. It can prefer a low-energy, low-capability configuration even when a user requires higher absolute quality. It can also ignore a saturation region where additional energy produces little service. The frontier exposes both.

Three decision rules can be applied to the same frontier:

1. maximize service subject to an energy budget;

2. minimize energy subject to a service floor;

3. maximize a declared utility over service, energy, and latency.

Only the third collapses the frontier into a scalar, and it does so by adding a decision maker’s preferences. The frontier itself remains a technical object.

Frontier uncertainty

An empirical point is an estimate, not a fixed coordinate. Near-ties require joint uncertainty in service, energy, and latency. A conservative report can classify a point as robustly dominated only when its uncertainty region is dominated. Otherwise, display the overlapping regions and avoid a categorical claim.

Operating points also drift with load. Batch size can improve energy per task while increasing queueing latency. Hardware utilization can alter idle allocation. The frontier should therefore be indexed by arrival process and service policy, not just by a model name.

Sources of uncertainty

At least five uncertainties enter $\widehat{\operatorname{QAIJ}}$:

1. task sampling from the target population;

2. stochastic model outputs and tool trajectories;

3. grading error or disagreement;

4. energy-meter calibration and sampling alignment;

5. task-distribution and shared-infrastructure weights.

Repeated runs on a fixed benchmark quantify output variation but not generalization to unseen tasks. More benchmark items quantify item variation but not meter bias. A single interval cannot hide which components were modeled.

Percentile bootstrap

The reference implementation uses the nonparametric bootstrap [10], resampling task attempts with replacement under a fixed seed. For bootstrap replicate $b$, it computes $$\widehat{\theta}^{(b)} = \frac{\sum_i w_i^{(b)}s_i^{(b)}} {\sum_i w_i^{(b)}E_i^{(b)}}.$$ The interval takes empirical quantiles. This method is transparent and useful for fixtures, but its sampling unit must be correct. If attempts are nested within tasks and task families, the resampling should follow that hierarchy. METR uses a hierarchical bootstrap for its time-horizon analysis [6].

When denominator variation is large or near zero, ratio inference needs more care. Operational energy is positive and usually well separated from zero, but the raw values and denominator distribution should still be published.

Reliability intervals

For $x$ binary successes in $n$ trials, the Wilson interval is [11] $$\frac{\widehat{p}+z^2/(2n) \pm z\sqrt{\widehat{p}(1-\widehat{p})/n+z^2/(4n^2)}} {1+z^2/n}.$$ The code computes this interval using the standard library’s normal distribution quantile. It avoids intervals below zero or above one.

For very small $n$, especially when $x=0$ or $x=n$, report a continuity-corrected Wilson or exact-binomial interval as a sensitivity check. No interval method compensates for dependent attempts or an unrepresentative perturbation set.

Reliability is conditional on the tested perturbations. Repeating the same prompt at temperature zero tests less variation than changing tool state, network response, context order, and input paraphrase. The protocol should name the perturbation set.

Generalized performance

NIST AI 800-3 separates accuracy on a fixed benchmark from generalized accuracy over a superpopulation [5]. The same distinction applies to $S_\mu$. A mixed-effects or item-response model can estimate latent system and item components, provided its assumptions are checked. The output should not be described as generalized if the target population is undefined.

For energy, generalization may require a separate load model. Energy per task at full batch and stable load may not transport to bursty production traffic. The target population therefore includes both tasks and system states.

Meter uncertainty

Let measured energy be $$\widetilde{E}_i=E_i+\epsilon_i+\beta,$$ where $\epsilon_i$ is random measurement error and $\beta$ is calibration bias. Repetition can reduce the first term but not the second. Calibration records and analyzer ranges matter.

Timestamp misalignment can attribute pre-run warm-up or post-run cooldown incorrectly. The power interval should be triggered by the benchmark harness and checked against request timestamps. If telemetry is used, sampling frequency and counter wraparound must be tested.

Multiple comparisons

Leaderboards compare many systems, scenarios, and operating points. Selecting the largest observed ratio creates winner’s curse. Report all preregistered comparisons, intervals, and task strata. For procurement, a practically meaningful non-inferiority margin is often more useful than a claim that two close point estimates differ.

Step 1: declare the decision and estimand

State whether the study supports engineering optimization, model selection, capacity planning, or scientific comparison. Declare $\mathcal{A}$, the task population, the energy boundary, and the latency contract. Do not choose weights after observing rankings.

Step 2: freeze the system

Record model identifier and checksum when available, weights, tokenizer, quantization, numerical precision, inference engine, compiler, kernels, drivers, operating system, hardware, firmware, parallelism, sampling parameters, prompt template, tool scaffold, and network dependencies.

A provider model name is insufficient when endpoints can change silently. Record request dates, endpoint version fields, and behavioral fingerprints that licensing permits.

Step 3: define the task distribution

Publish task sources, inclusion rules, deduplication, contamination checks, family labels, sampling weights, and target population. Separate fixed benchmark inference from deployment generalization.

For interactive tasks, specify human actions, tool permissions, stopping rules, and whether human time is included in the service outcome. Human labor is not converted into joules in this metric.

Step 4: define difficulty

Choose strata before evaluation. If using expert completion time, publish the expert population, timing protocol, censoring, and uncertainty. If using item response theory, publish model form, fit diagnostics, anchors, and scale identification. Always include unweighted results.

Step 5: define grading

Use executable tests where they faithfully express the task. For human or model judging, validate the rubric, blind system identity, randomize order, measure agreement, and adjudicate a sample. Archive raw outputs when privacy and licensing permit.

Step 6: measure reliability

Repeat task attempts across the variation relevant to deployment. A minimum count cannot be universal because task cost and expected failure rates differ. Publish the number of tasks, attempts per task, seeds, and transient failures.

Step 7: instrument energy

For system-operational claims, prefer synchronized wall power. Configure the analyzer range, log calibration, and record sampling rate. Device telemetry is reported as a diagnostic series, not silently substituted for wall energy.

For shared serving, define the allocation rule. A simple rule may assign dynamic energy to requests and allocate baseline idle energy by occupied time, but alternative rules should be tested. No rule is neutral when multiple tenants share hardware.

Report sensitivity to at least one credible alternative allocation rule when shared energy is material. Time-proportional and resource-proportional rules can yield different rankings under high concurrency. If task-level allocation cannot be supported from telemetry, report a pooled service-per-joule result for the shared system instead of invented precision.

Step 8: handle warm-up and idle energy

Declare whether compilation, model loading, cache warm-up, and idle periods are included. An engineering kernel benchmark may exclude setup. A deployed service study should include the setup and idle share that its operating model causes.

Report both gross and incremental energy when possible: $$E_{\mathrm{gross}} = \int_{\tau}P(t)\,dt, \qquad E_{\mathrm{incremental}} = \int_{\tau}\max\{P(t)-P_{\mathrm{idle}},0\}\,dt.$$ These answer different questions. The primary denominator must be named.

Step 9: preserve failures

Do not drop failed, refused, malformed, timed-out, or retried requests from the energy denominator. Record their failure class and energy. If the serving system retries automatically, the entire chain belongs to the attempt unless the functional unit says otherwise.

Step 10: control load and latency

Measure arrival process, concurrency, batch policy, queueing, time to first token where relevant, completion latency, and tail percentiles. Throughput comparisons under unmatched latency targets are not directly comparable.

Step 11: repeat and randomize

Randomize run order to reduce temperature, load, and drift confounding. Repeat across times or machines if the claim covers them. Monitor thermal throttling, background processes, and clock behavior. Preserve raw power traces and request logs.

Step 12: report numerator and denominator

Publish expected service, expected energy, their ratio, uncertainty, and every manifest field. A reader should be able to recompute the ratio and substitute an alternative task distribution without reconstructing hidden logs.

Scope

The module src/joule_standard/intelligence.py uses only the Python standard library. It defines:

• TaskResult, with task, system, energy, latency, tokens, quality, correctness, reliability, usefulness, difficulty, and sampling weight;

• quality_adjusted_service_per_joule, implementing the equation;

• bootstrap_qaij, a seeded percentile bootstrap;

• wilson_interval, for binary reliability;

• pareto_frontier, for non-dominated energy-quality points;

• rank_systems, for explicit $\operatorname{QAIJ}$ or token-throughput comparisons.

The module validates probabilities, nonnegative weights, positive energy, and finite inputs. It does not estimate economic value, carbon emissions, or general intelligence.

Executable claims

The test file tests/test_intelligence.py maps paper claims to fixtures.

Passing tests do not prove external validity. They show that the reference calculation obeys the published definitions and counterexamples.

Ratio implementation

The estimator computes weighted service and energy separately: $$\begin{align*} \widehat{S} &= \frac{\sum_iw_is_i}{\sum_iw_i},\\ \widehat{E} &= \frac{\sum_iw_iE_i}{\sum_iw_i},\\ \widehat{\operatorname{QAIJ}} &= \frac{\widehat{S}}{\widehat{E}}. \end{align*}$$ For a late task, the service method returns zero before multiplying quality factors. The energy object remains unchanged.

Frontier implementation

The frontier routine first filters points that violate $L^\star$. It then marks candidate $y$ dominated when another point $x$ satisfies $$E(x)\leq E(y),\qquad S(x)\geq S(y)$$ with one strict inequality. Returned points are ordered by energy. This quadratic reference algorithm favors auditability over scale. Large leaderboards may use a sorted sweep after handling ties.

Example 1: same answer, different tokenizer

Suppose two tokenizers segment the same correct answer into $10$ and $40$ tokens. The measured system energy is $10$ joules and every service factor equals one. Tokens per joule is $1$ or $4$, depending on tokenizer. $\operatorname{QAIJ}$ is $0.1$ task equivalents per joule in both cases.

The example isolates tokenization. Real tokenizers can also affect model behavior because tokenization is part of the model. In that case, task outcome or energy may change, and the evaluator should measure those changes rather than assume invariance.

Example 2: late answers

Consider one successful answer using $5$ joules in one second and one correct but late answer using $15$ joules in three seconds. With $L^\star=2$ seconds and equal task weights, $$\widehat{S}=0.5,\qquad \widehat{E}=10\ \text{joules},\qquad \widehat{\operatorname{QAIJ}}=0.05.$$ Dropping the late answer would report $0.2$, overstating deployed service by a factor of four. Counting its energy but not its service makes the contract visible.

Example 3: reliability

Suppose a tool-using agent succeeds in $8$ of $10$ independent runs. The point reliability is $0.8$, but the uncertainty is wide. A Wilson interval at $95$ percent is approximately $0.49$ to $0.94$. Reporting only $0.8$ would create false precision.

If the ten attempts reuse the same hidden state or tool cache, they are not independent. The effective sample size is smaller. The interval method cannot repair a bad experimental design.

Example 4: difficulty-weight sensitivity

Let easy and hard tasks have unweighted service scores as in 7. If a weighted scalar assigns the hard stratum five times the easy stratum, the hard specialist receives a larger composite gain. That result reflects the declared weight, not a discovery that a hard task has five times the intrinsic value.

Report the stratum vector first: $$\left( \frac{S_e}{E_e}, \frac{S_h}{E_h} \right).$$ Then show any composite under at least two plausible normalizations. A reader can see whether the decision is robust or weight-driven.

Example 5: system and device boundaries

System $A$ uses $70$ joules at the accelerator and $30$ joules elsewhere. System $B$ uses $80$ joules at the accelerator and $10$ joules elsewhere. With equal service, device-only accounting ranks $A$ first, while wall accounting ranks $B$ first: $$70<80,\qquad 100>90.$$ The result is another rank reversal, but its source is boundary choice rather than task mix. Neither number is wrong if labeled. Mixing the two in one table is wrong.

Engineering optimization

Within a fixed model, task distribution, and quality floor, $\operatorname{QAIJ}$ can compare quantization, caching, batching, kernel, and routing configurations. The frontier shows whether an optimization shifts the feasible set or merely moves along a latency tradeoff.

An optimization that raises tokens per joule but reduces correct service does not improve the quality-adjusted metric. An optimization that reduces energy while preserving task service and latency produces an unambiguous technical gain under the fixed boundary.

Model selection

For model selection, publish task-level results. The aggregate depends on $\mu$, and users may have different mixes. A routing policy can dominate a single-model policy by sending easy cases to a small model and difficult cases to a larger one, but router errors and router energy belong in the system.

The selected configuration includes the entire policy: $$x = (\text{router},\text{models},\text{serving stack},\text{hardware}).$$ Reporting only the chosen model’s energy omits the cost of selection.

Procurement

A procurement rule should use constraints before ratios. First set minimum correctness, reliability, safety, privacy, and latency. Then compare energy among qualifying systems or inspect the frontier. A high ratio from a system below the service floor is irrelevant.

Usefulness weights in procurement are stakeholder choices. Publish them and run sensitivity analysis. If stakeholder rankings conflict, keep the vector rather than invent a neutral average.

Capacity planning

Tokens remain useful for memory and throughput forecasts. $\operatorname{QAIJ}$ adds the missing service layer. A capacity model can retain both: $$\frac{\text{tokens}}{\text{second}}, \quad \frac{\text{joules}}{\text{token}}, \quad \frac{\text{verified task service}}{\text{joule}}.$$ The three quantities answer production, energy, and outcome questions.

Scientific comparison

Scientific comparisons require a versioned harness, open task data where possible, raw outputs, power traces, and code. Closed endpoints can still be evaluated, but model drift and unavailable meter access limit reproducibility. The status of a cross-provider comparison may be conditional or obstructed.

Benchmark gaming

A system can optimize against the public task set without improving the target population. Contamination, memorization, evaluator exploitation, and grader-specific style can raise measured service. Holdout tasks, transcript audits, multiple graders, and post-deployment monitoring reduce this risk but do not eliminate it. The benchmark-lottery and general-benchmark critiques show why task selection and construct scope must remain visible [13, 14].

Verbosity gaming

Tokens-per-joule discourages verbosity only if energy grows faster than token count. Tokens-per-second may reward long easy continuations. A quality-adjusted task unit avoids direct reward for volume, but the quality grader may still prefer elaborate answers. Rubrics should reward required evidence and penalize irrelevant material.

Difficulty gaming

If difficulty is based on human time, tasks can be made cumbersome without becoming cognitively valuable. Bad tooling, unclear instructions, and setup latency can inflate the duration. Difficulty protocols should separate task substance from avoidable environment friction.

Reliability gaming

Retry-until-success policies can raise observed success if retries are hidden. The functional unit must state whether one user request includes all retries. Energy, latency, and failed attempts from the retry chain must remain in the record.

Boundary gaming

Device-only energy can make off-device preprocessing, retrieval, networking, or CPU work disappear. Facility multipliers can be selected from favorable periods. Lifecycle allocation can use optimistic utilization. A boundary manifest and bridge table are the defense.

Task-mix gaming

A provider can report a favorable mixture or exclude expensive failures. Versioned task weights and a public task-level dataset allow reweighting. When privacy prevents release, publish sufficient aggregate cells and an independent audit.

Shared evaluation datasets

The field needs task-level datasets that join raw outputs, executable or human grades, reliability repetitions, request timing, wall-power traces, hardware state, and task-family metadata. Privacy-preserving aggregate releases may be necessary for production workloads.

Hierarchical estimators

Future work should model tasks nested in families, attempts nested in tasks, and machines nested in sites. A joint model can separate task, model, and hardware variation. It should retain raw ratio components and pass posterior predictive checks.

Interactive systems

Human-AI workflows complicate the functional unit. The model may reduce human time but add review or correction. Energy-normalized technical service can be measured, but productivity and value require a randomized or credible counterfactual study of the combined workflow.

Dynamic routing

Routers can allocate tasks to models based on predicted difficulty. The evaluation must include router errors, abstentions, escalation, duplicate inference, cache behavior, and total system energy. A policy frontier may be more relevant than a model frontier.

Lifecycle extension

A lifecycle extension would allocate embodied accelerator, host, cooling, and building energy across service units. Utilization and lifetime assumptions will dominate some comparisons. The operational metric should remain visible rather than being buried inside one lifecycle number.

Task service and causal value

The next paper in the AI sequence should estimate the link from verified task service to adopted actions and downstream effects. The bridge may use randomized deployments, encouragement designs, or careful observational methods. It should not convert $\operatorname{QAIJ}$ into dollars with a fixed coefficient.