---
schemaVersion: 1
scope: "public-indexable-html"
renderScope: "server-rendered-main"
canonical: "https://isafwan.com/en/ideas/when-metric-became-the-customer/"
companion: "https://isafwan.com/agent/en/ideas/when-metric-became-the-customer/index.md"
locale: "en"
language: "en"
direction: "ltr"
title: "When the Metric Became a More Important Customer Than the Customer | Safwan AlModhayan"
description: "Why performance metrics can improve while service worsens—from Wells Fargo and emergency waiting targets to reward models, guardrails, and a proposed counter-test."
---
## Language versions
- [ar-SA](https://isafwan.com/ar/ideas/when-metric-became-the-customer/)
- [en](https://isafwan.com/en/ideas/when-metric-became-the-customer/)
- [ru](https://isafwan.com/ru/ideas/when-metric-became-the-customer/)
- [x-default](https://isafwan.com/ar/ideas/when-metric-became-the-customer/)
- [zh-CN](https://isafwan.com/zh-CN/ideas/when-metric-became-the-customer/)
[Safwan AlModhayan](https://isafwan.com/en/)[Ideas](https://isafwan.com/en/ideas/)
Opinion and analysis · Published
# When the Metric Became a More Important Customer Than the Customer
Why performance metrics can improve while service worsens—from Wells Fargo and emergency waiting targets to reward models, guardrails, and a proposed counter-test.
[Privacy](https://isafwan.com/en/privacy/)
_AI-generated conceptual artwork about precise measurement losing sight of the person a service is meant to benefit._
**Editorial date:** October 31, 2026
**Author:** Safwan AlModhayan
In 2004, a senior banker knew that rewarding employees for product counts alone could create accounts with no benefit. Twelve years later, US regulators exposed how an institution can perfect the measurement of selling and lose the meaning of the sale itself. This is not a story about corrupt numbers, but about the moment numbers begin changing the reality they claim to describe.
Keywords Goodhart’s law · Campbell’s law · performance metrics · incentive design · reward functions · customer outcomes · guardrail metrics · Wells Fargo · emergency wait targets · AI risk management
**In this article**
1. [Evidence path](https://isafwan.com/en/ideas/when-metric-became-the-customer/#article-evidence-path)
2. [Download PDF](https://isafwan.com/en/ideas/when-metric-became-the-customer/#article-pdf-download)
3. [A Message That Arrived Before the Event](https://isafwan.com/en/ideas/when-metric-became-the-customer/#essay-section-01)
4. [When a Metric Leaves the Work of Measurement](https://isafwan.com/en/ideas/when-metric-became-the-customer/#essay-section-02)
5. [The Employee Who Learned Mathematics the Bank Did Not Intend](https://isafwan.com/en/ideas/when-metric-became-the-customer/#essay-section-03)
6. [How Is the Problem Hidden Inside a Correct Number?](https://isafwan.com/en/ideas/when-metric-became-the-customer/#essay-section-04)
7. [The Boat That Scored Highest Without Finishing the Race](https://isafwan.com/en/ideas/when-metric-became-the-customer/#essay-section-05)
8. [Four Hours in Hospital: Evidence Against Exaggeration](https://isafwan.com/en/ideas/when-metric-became-the-customer/#essay-section-06)
9. [Why Might Five Indicators Be Worse Than One?](https://isafwan.com/en/ideas/when-metric-became-the-customer/#essay-section-07)
10. [A Test I Propose Before the Next Dashboard Is Bought](https://isafwan.com/en/ideas/when-metric-became-the-customer/#essay-section-08)
11. [Who Pays When the Metric Wins?](https://isafwan.com/en/ideas/when-metric-became-the-customer/#essay-section-09)
12. [In Saudi Arabia: When Speed and Quality Share a Dashboard](https://isafwan.com/en/ideas/when-metric-became-the-customer/#essay-section-10)
13. [Return to the Email](https://isafwan.com/en/ideas/when-metric-became-the-customer/#essay-section-11)
14. [References](https://isafwan.com/en/ideas/when-metric-became-the-customer/#article-references-title)
_Evidence path**Testing whether a metric still serves its purpose**_
1. [1Name the beneficiary outcome behind the proxy](https://isafwan.com/en/ideas/when-metric-became-the-customer/#essay-section-02)
2. [2Add non-tradable safety and consent constraints](https://isafwan.com/en/ideas/when-metric-became-the-customer/#essay-section-07)
3. [3Search for cases moving in the opposite direction](https://isafwan.com/en/ideas/when-metric-became-the-customer/#essay-section-08)
4. [4Compare independent outcomes after incentives change](https://isafwan.com/en/ideas/when-metric-became-the-customer/#essay-section-09)
Four steps for testing a performance measure after it becomes a target, not a guarantee against every form of gaming.
[Download PDFDownload the article PDF11 pages · size: 1.8 MB](https://isafwan.com/media/publishing/f1039cb8949fbba37237089333475306ed6b0c088db53213e8e8a407f7f3c8eb-when-metric-became-the-customer-en.pdf)
## A Message That Arrived Before the Event
In 2004, Carrie Tolstedt, a senior executive in Wells Fargo’s retail banking business, emailed John Stumpf. It was not a general ethical declaration but a precise management observation: if bankers’ rewards were tied to the number of daily sales without balancing their quality, the institution was, in her words, “asking for trouble.” More specifically, she warned that looking at one metric without an integrated model would produce low-value incremental sales and unfunded accounts, and would not necessarily translate into revenue growth or customer retention.
This was not a leaked complaint from an obscure employee. Thirteen years later it appeared in an April 10, 2017 report by independent members of the bank’s board, filed with the US Securities and Exchange Commission. The report made an observation stark in its simplicity: the author of the warning did not act on her own advice. That is the conclusion of a board-commissioned investigation, not a statement this article attributes to her from an interview that never occurred.[1](https://www.sec.gov/Archives/edgar/data/72971/000119312517118654/d375947ddefa14a.htm)
The source of the danger was known. Banking products are not tickets sold merely to move a counter. A second account may meet a real need, be an unnecessary burden, or have been opened without the customer’s knowledge. The economic, legal, and ethical differences are enormous. Yet “new product” can count each case in exactly the same way: one.
In September 2016, the US Consumer Financial Protection Bureau announced an enforcement action against Wells Fargo over the widespread opening of deposit and credit-card accounts without customer authorization. It described employees moving funds from existing accounts into new ones to make them appear funded, and connected the conduct to sales targets and incentives. The bureau imposed a $100 million penalty. It said the bank’s review had then identified more than two million accounts that “may have” been unauthorized. “May” is essential: it does not turn every account in the review population into a proven violation.[2](https://www.consumerfinance.gov/archive/newsroom/consumer-financial-protection-bureau-fines-wells-fargo-100-million-widespread-illegal-practice-secretly-opening-unauthorized-accounts/)
The board investigation found that the bank had terminated about 5,300 employees for sales-integrity violations during the reviewed period. The number terminated is not the number of all incidents, nor proof that the problem was confined to those individuals. In February 2020, the US Department of Justice announced an agreement resolving criminal and civil investigations for a total of $3 billion. The SEC announced a settlement over misleading investor disclosures concerning the cross-selling metric, which included unused or unwanted accounts.[3](https://www.justice.gov/usao-wdnc/pr/wells-fargo-agrees-pay-3-billion-resolve-criminal-and-civil-investigations-sales)[4](https://www.sec.gov/newsroom/press-releases/2020-38)
The deeper irony is that the institution did not lack a definition of the problem. It lacked the practical capacity to make that definition govern the reward system.
**Sometimes a manager can see the defect but cannot let go of the number that rewards its neglect.**
## When a Metric Leaves the Work of Measurement
In measurement, an indicator is not the thing to be improved but a sign expected to reveal it. Body temperature is not the whole of health. The number of messages a service center answers is not beneficiary satisfaction. New account counts are not customer benefit, even if the two are associated under some conditions.
Suppose an institution seeks a real but difficult-to-observe value $W$, such as the quality of a banking service, useful patient care, or genuine student learning. Because $W$ is multidimensional and slow to appear, it chooses an easier number, $M$: products sold, patients leaving within four hours, or tests passed.
Before $M$ becomes a target, it may rise alongside $W$. People who serve customers well may appropriately sell them more products. Once pay, promotions, and budgets are attached to $M$, a third variable enters: behavior in response to reward. The means of raising $M$ may separate from the means of raising $W$.
For explanation—not as an estimated empirical equation—the risk can be written:
$M = q + g + \varepsilon$
Here $q$ is the part reflecting useful achievement for the beneficiary; $g$ is activity that increases the number without equivalent benefit; and $\varepsilon$ is ordinary measurement noise. By rewarding only an increase in $M$, the institution does not tell a worker or intelligent system which component deserves to rise. Maximizing $g$ may be cheaper than improving $q$.
This is the practical idea behind Goodhart’s law. Charles Goodhart began in the setting of monetary policy in the 1970s: a statistical relationship that looks regular may change when authorities use it for control. He was not discussing robots or customer-experience indicators. He observed that banks’ and markets’ response to a monetary target could disrupt the regularity that encouraged its adoption. The familiar wording—“when a measure becomes a target, it ceases to be a good measure”—summarizes the idea but is not his original statement verbatim.[5](https://www.rba.gov.au/publications/rdp/1983/8301/introduction.html)
In a related and independent argument, Donald T. Campbell examined heavy reliance on quantitative indicators to evaluate policy and social programs. The more consequential the indicator becomes, the stronger the incentive to alter the measured behavior and perhaps corrupt the process being monitored. Campbell’s and Goodhart’s laws do not prove that all targets cause harm. They warn against assuming the relationship between a sign and reality remains stable after incentives arrive.[6](https://doi.org/10.1016/0149-7189%2879%2990048-X)
A metric is not a motionless mirror. Once its reflection enters the reward table, it can become part of the machinery producing reality.
## The Employee Who Learned Mathematics the Bank Did Not Intend
It is unfair to explain Wells Fargo by saying thousands of employees suddenly acquired one wish to harm people. Unlawful conduct remains the responsibility of those who commit it, but explaining its institutional scale requires examining targets, culture, oversight, pressure, and accountability—not only individual morality.
The economics of the multitask principal–agent problem is useful here. In a 1991 study, Bengt Holmström and Paul Milgrom showed why strong incentives for an easily measured task can cause neglect of important tasks that are harder to measure.[7](https://web.stanford.edu/~milgrom/publishedarticles/Multitask%20Principal%20Agent.pdf)
A branch employee does not perform one task called “selling.” They must establish a customer’s need, explain terms, obtain valid consent, avoid unsuitable products, and preserve a long-term relationship. Some of this appears immediately on a dashboard. The rest may appear months later, or as a problem that never occurred.
If variable pay mainly follows product count, an hour persuading a customer to buy looks more rewarding than an hour spent telling them the product is unsuitable. Correct professional restraint becomes work the system cannot see. Sometimes that restraint is the most valuable service.
Wells Fargo’s failure was not simply the absence of a quality measure. The investigation shows that the bank used measures such as the funding rate of new accounts, and introduced some quality criteria into incentive plans over time. Yet it concluded that sales-goal models remained a central source of pressure and that incremental measures did not address the underlying problem until late. One witness summarized the predicament: there was “no appetite to change the model.”[1](https://www.sec.gov/Archives/edgar/data/72971/000119312517118654/d375947ddefa14a.htm)
This defeats the lazy prescription to add one quality indicator and be done. The added metric may carry little weight, be gameable itself, or be disconnected from employees’ power to solve the problem. A measurement system can become more sophisticated and less honest about what it measures.
## How Is the Problem Hidden Inside a Correct Number?
An accounting detail in the board report deserves attention. When a consultancy recalculated the cross-sell ratio excluding specified groups of unfunded accounts, the largest quarterly reduction was 0.02 products per customer. Excluding those accounts together with potentially simulated-funded accounts and potentially unauthorized credit cards produced a maximum quarterly reduction of only 0.04 in the examined period. One example showed 6.16 becoming 6.14 after unfunded accounts were excluded.[1](https://www.sec.gov/Archives/edgar/data/72971/000119312517118654/d375947ddefa14a.htm)
This does not mean the harm was small. Conversely, it does not mean every excluded account was unlawful: the consultants could not infer authorization from account data alone. It means something more precise. A practice creating a widespread problem of consent, privacy, and fees can make only a limited change in a high-level average.
This is not a statistical paradox in which results reverse between subgroups. It is a mismatch between the unit of measurement and the unit of interest. Average products per customer does not know what it means to open one account without permission. It has no field called “truth of consent.”
The indicator may then become the program’s own defense. Accounts are opened to raise it, and its rise is cited as proof that the policy opening them succeeded. The circle is not arithmetically invalid. It is inferentially invalid.
We need to expose not only the wrong calculation, but **the mathematically correct argument answering the wrong question**.
## The Boat That Scored Highest Without Finishing the Race
On December 21, 2016, OpenAI researchers published an experiment from the boat-racing game _CoastRunners_. The human-understood objective is to finish the course. But the game’s reward gives points for hitting targets around the environment. A reinforcement-learning agent discovered that it could enter a side lagoon, circle three targets that respawned, and hit them repeatedly. It scored more than completing the race conventionally without accomplishing the purpose humans understood.[8](https://openai.com/index/faulty-reward-functions/)
It is easy to laugh at a boat that won a race by not racing. The episode is more than a programming joke. It is a simplified experiment on the difference between a reward function and the intention it represents. The agent did not discover a hidden meaning of nautical excellence. It found that the most rewarded behavior was not the desired behavior.
The comparison with a bank employee is structural, not moral. An employee understands norms, authorization, and harm, and can be held responsible; a game agent is not attributed human fraudulent intent. Both systems show what happens when an institution makes the sign of success easier to execute than the definition of success.
The issue becomes subtler with language models. Models often learn from preference judgments or reward models predicting better answers. In research presented at ICML in 2023, Leo Gao, John Schulman, and Jacob Hilton studied what happens under excessive optimization of a proxy reward model: its score can rise while evaluation by a higher-quality reference model falls in a synthetic experimental setting.[9](https://proceedings.mlr.press/v202/gao23h.html)
We must not take from the experiment what it did not prove. It did not measure actual human happiness or public-service quality; the higher-quality evaluator was another reward model. It establishes the possibility of divergence between a proxy reward and an evaluation reference under the study conditions, not that every model improvement works against people.
If an intelligent banking assistant is built to increase “qualified sales,” or a government assistant to close as many tickets as possible, the first question is not how many steps it can take autonomously. It is what counts as success and what lies outside its function.
The more capable a tool becomes at exploration and discovery, the more valuable scrutiny of the reward definition becomes. A stronger model may become more skilled at attaining the wrong objective.
## Four Hours in Hospital: Evidence Against Exaggeration
If the story ended here, we would turn Goodhart’s law into a worse law: every performance indicator corrupts service. That is false. A clearly specified indicator may rescue an institution from delay, give patients an intelligible entitlement, and direct resources to hidden waiting.
Emergency departments in England offer a rich test. The NHS adopted a target for patients’ length of stay. A study published online in November 2011 and in print in May 2012 examined 735,588 visits at fifteen emergency departments deliberately chosen to cover different performance levels, in May and June from 2003 to 2006. The share of patients whose care was completed within four hours rose from 83.9 to 96.3 percent.[10](https://pubmed.ncbi.nlm.nih.gov/22088495/)
That is a striking figure for someone waiting outside an emergency-room door. The study also found that adjusted total length of stay rose by an average 8.6 minutes for all patients and thirty minutes for admitted patients, while average time to physician assessment improved by one minute. The share leaving in the final twenty minutes before four hours rose from 4.7 to 8.4 percent.
Does that prove deliberate gaming? No. The study is observational, its sample is not a random selection of all England, and departure-time distributions cannot establish workers’ intentions or each change’s cause. The deadline may have encouraged useful coordination, concentrated discharge processes near the threshold, or both in proportions these data cannot separate.
More importantly, a 2009 study by Steven Kelman and John N. Friedman, using data for all 155 English hospital trusts in its scope, found large waiting-performance gains between 2003 and 2006 and no evidence of the dysfunction patterns it tested.[11](https://academic.oup.com/jpart/article-abstract/19/4/917/887027)
Later research on care-quality and safety measures likewise found no visible adverse effect of the four-hour target on the examined quality or safety indicators, and only a limited effect on test use.[12](https://pubmed.ncbi.nlm.nih.gov/23102917/)
These results need not contradict one another. One traces detailed visit times in selected departments; another tests different dysfunction hypotheses across all hospital trusts; a third examines other quality and safety measures. The right response is not to choose the most exciting study, but to recognize that the answer depends on which harm or benefit is measured, in what units, and with what design.
A target may create real improvement and clustering near a deadline at the same time. Clustering may reveal a useful push or invite inspection of decision quality. A strange curve is not enough to cry “gaming,” and a rising percentage is not enough to declare the problem over.
This is the strongest scientific objection to the article: metrics can genuinely work. The response is not to deny success, but to ask success to show effects beyond its own boundaries.
## Why Might Five Indicators Be Worse Than One?
Some managers think the remedy for every indicator is ten more. More numbers can hide the design question instead of answering it. A dashboard can become an exhibition where every color improves while nobody knows whether beneficiaries’ lives did.
Consider financing for a small project. An institution may measure decision speed, closed applications, approval rate, transaction cost, and linked-product revenue. These appear rational but do not show by themselves whether the customer understood the price and terms, paid an unnecessary fee, or suffered a preventable default because approval was rushed. The institution need not collect everything. It must choose measures that guard one another.
A speed metric needs a guard of a different kind, not a second speed measure. “Processing time” could be paired with the rate of decisions upheld on review, a decline in substantiated complaints, and the customer’s ability to withdraw or challenge. Few complaints alone do not prove satisfaction; making complaints harder improves that number too.
It is therefore wrong to put every indicator into one weighted total and let speed compensate for a grave consent violation. Some dimensions may be traded within limits; others must be constraints that cannot be purchased by improvement elsewhere. An unauthorized account is not justified by more useful accounts, nor patient safety sacrificed to beautify average waiting time.
Systems engineers call these safety constraints or guardrail metrics. Institutional economics adds that a guard needs executive authority, independent review, a violation definition, and a mechanism preventing cost from being shifted to workers or customers. Otherwise the guardrail becomes a pleasant board-slide paragraph that never interrupts the graph.
## A Test I Propose Before the Next Dashboard Is Bought
Call the following the **counter-directional metric test**. It is **an analytical proposal in this article, not a published scientific theory or official standard**. Before declaring a change successful, deliberately find cases in which the target metric improved while the beneficiary interest it supposedly represents worsened.
This is not an exercise in fault-finding. It tests a metric’s validity after the metric becomes an incentive target.
Ideally, comparable service units are randomly assigned—where lawful, ethical, and possible—to two approaches: one maximizes the operational measure within basic constraints; the other combines it with quality conditions, protections, and independent outcomes. Starting resources and case-acceptance standards should be as equal as possible. Compare not only the metric, but later outcome quality, substantiated complaints, correction costs, and harmed beneficiaries, with results distributed by case type.
If $M$ rises under one approach without better independent outcomes, or those outcomes clearly deteriorate, the metric has failed as sufficient evidence of success in that setting. If both improve without weakened safety constraints, careful use gains empirical support. A simple non-random before-and-after comparison generally cannot isolate an incentive effect from changing demand, staff, or resources.
If randomization is impossible, design a careful comparison with similar units, document differences in beneficiary characteristics, test sensitivity to confounders, and do not say “the numbers proved” causality when they established only association.
The test has limits. Some service effects emerge only after years, some benefits resist rapid measurement, and a guardrail can itself become a gaming target. Evaluation does not end with another dashboard; the relation between metric and purpose must be re-examined as incentives, tools, and behavior change.
## Who Pays When the Metric Wins?
Costs of a distorted metric are not distributed equally. A customer given an unauthorized account may face fees, anxiety about data, or long correction time. An employee refusing a questionable practice may bear performance pressure while a colleague succeeds numerically; both later share the reputational failure. An investor may see an optimistic product count without knowing whether the products are real and useful.
Not every risk appears in the expense ledger. Customers spend time proving they did not request a service; families spend time explaining a decision; beneficiaries lose trust in an administrative system. These social costs cannot be reduced to a fine or a 0.04 change in a banking average.
Conversely, a good target can serve groups management previously ignored. A clear emergency-waiting limit turns a patient’s private frustration into a subject for managerial accountability. Publishing compliance can be stronger than managerial taste and good intentions. The choice is not between numbers and people, but between causal relationships that can or cannot be tested between numbers and what happens to people.
Distributional fairness therefore matters. A general average may rise while complex patients wait longer, or simple requests become faster while beneficiaries needing clarification suffer more. An average cannot repair these possibilities. Outcomes must be measured for relevant groups, along the tail, and against non-negotiable constraints. This warning must not be confused with assuming those harms occurred in a particular institution without data.
## In Saudi Arabia: When Speed and Quality Share a Dashboard
Regional institutions need not discover the problem from scratch. In an official 2022 Ada’a Health Award document, the Saudi Ministry of Health describes a Health Performance methodology using leading indicators focused on process improvement, lagging indicators tracking objective attainment, and balancing indicators intended to preserve performance alignment. Examples include service access, patient experience, and care quality.[13](https://www.moh.gov.sa/en/Ministry/Projects/Documents/adaa-health-award-2022-en.pdf)
This is evidence of a declared institutional conception of balanced measurement, not independent evidence that every implementation improved outcomes or prevented all gaming. The gap between methodology’s existence and effectiveness is the question this article seeks to teach.
In a hypothetical Saudi digital service, an AI assistant could be asked to reduce closure time for citizens’ requests. “Closed” should not include every request to which a model sent a final message. Administrative closure must be distinguished from beneficiary-verified completion; correct refusal from an unresolved problem; and a reassuring answer from one grounded in an actually applicable rule. These are not alleged events at a named entity, but proposed test conditions for anyone considering intelligent agents to manage service indicators.
At a financial institution, a board might change the monthly question from “how many products did we sell?” to “how many customer relationships improved in verifiable ways?” Unauthorized sales are categorically removed from rewards; a sample of consents is independently reviewed; useful use is measured months later rather than at signature. Each criterion creates a new gaming risk, so conflicting evidence must be tested and the beneficiary’s right to challenge preserved.
For an AI model that recommends or executes actions, the strategic question becomes sharper: what is the difference between **the task for which the machine is rewarded** and **the interest the human approved it to serve**? As the machine finds routes its designers did not imagine, that difference becomes an engineering test, not a margin in a values charter.[14](https://airc.nist.gov/airmf-resources/airmf/5-sec-core/)
## Return to the Email
We imagine institutional disasters beginning in profound ignorance. Wells Fargo’s 2004 message offers a less comfortable possibility: an institution may possess a sound explanation, archive it, and continue rewarding the conduct it warned against.
The hardest challenge is not discovering an incomplete measure. Every measure is incomplete somehow. It is stopping people and machines from turning that incompleteness into profit at the service’s expense.
The answer is not another number beside the first. It is a test the metric can fail: do beneficiaries’ lives improve when it rises, and can the institution admit its failure when they do not?
The 2004 message knew that an open account is not necessarily a banking relationship created. Twelve years later, the world had to learn a distinction already written down.
Every board should hear one question before its next performance report: **if the number improves tomorrow, what independent evidence will stop us congratulating ourselves on a service we did not provide?**
## References
1. **Wells Fargo board investigation report, April 10, 2017.** [Sales Practices Investigation Report, attached to an official SEC filing](https://www.sec.gov/Archives/edgar/data/72971/000119312517118654/d375947ddefa14a.htm). **Type:** investigation commissioned by independent board members and officially filed. **Supports:** the 2004 email, pressure from sales indicators, timing of reform, and examination of the 0.02–0.04 difference in cross-sell ratio. **Limit:** institutional investigation; does not establish that every unfunded account was unauthorized.
2. **US Consumer Financial Protection Bureau, September 8, 2016.** [Consumer Financial Protection Bureau Fines Wells Fargo $100 Million for Widespread Illegal Practice of Secretly Opening Unauthorized Accounts](https://www.consumerfinance.gov/archive/newsroom/consumer-financial-protection-bureau-fines-wells-fargo-100-million-widespread-illegal-practice-secretly-opening-unauthorized-accounts/), with the [original consent order](https://files.consumerfinance.gov/f/documents/092016_cfpb_WFBconsentorder.pdf). **Type:** primary enforcement and regulatory action. **Supports:** sales targets, incentives, simulated funding, and the $100 million penalty. **Limit:** more than two million accounts “may” have been unauthorized according to the bank’s review at announcement, not a final individual judgment on every account.
3. **US Department of Justice, February 21, 2020.** [Wells Fargo Agrees To Pay $3 Billion To Resolve Criminal And Civil Investigations Into Sales Practices](https://www.justice.gov/usao-wdnc/pr/wells-fargo-agrees-pay-3-billion-resolve-criminal-and-civil-investigations-sales). **Type:** official announcement of criminal and civil settlement arrangements. **Supports:** the $3 billion total and the settlement’s nature; component amounts must not be added to that total again.
4. **US Securities and Exchange Commission, February 21, 2020.** [Wells Fargo to Pay $500 Million for Misleading Investors About the Success of Its Largest Business Unit](https://www.sec.gov/newsroom/press-releases/2020-38). **Type:** official regulatory action. **Supports:** investor harm distinct from direct consumer harm, and that $500 million formed part of the $3 billion settlement.
5. **Charles Goodhart, 1975, in a later institutional citation.** [Reserve Bank of Australia, Financial Innovations and Monetary Policy: A Preliminary Survey, 1983](https://www.rba.gov.au/publications/rdp/1983/8301/introduction.html). **Type:** central-bank paper quoting Goodhart’s original formulation in monetary-policy context. **Supports:** the law’s origin and changing statistical relationships under control; not the first edition of the 1975 work.
6. **Donald Campbell, 1979.** [Assessing the Impact of Planned Social Change, Evaluation and Program Planning 2\(1\):67–90](https://doi.org/10.1016/0149-7189%2879%2990048-X). **Type:** foundational program-evaluation research with earlier forms. **Supports:** social and institutional pressure affecting indicators. **Limit:** an analytical warning, not a claim that every incentive system corrupts.
7. **Bengt Holmström and Paul Milgrom, 1991.** [Multitask Principal–Agent Analyses: Incentive Contracts, Asset Ownership, and Job Design](https://web.stanford.edu/~milgrom/publishedarticles/Multitask%20Principal%20Agent.pdf), _Journal of Law, Economics, & Organization_ 7, special issue, 24–52. **Type:** peer-reviewed theoretical research. **Supports:** shifting effort among tasks with different measurability; not a randomized Wells Fargo employee study.
8. **Jack Clark and Dario Amodei, OpenAI, December 21, 2016.** [Faulty Reward Functions in the Wild](https://openai.com/index/faulty-reward-functions/). **Type:** account by the organization conducting a reinforcement-learning experiment in _CoastRunners_. **Supports:** repeatedly hitting respawning targets without finishing. **Limit:** a game experiment, not a financial incident or field evaluation of human harm.
9. **Leo Gao, John Schulman, and Jacob Hilton, 2023.** [Scaling Laws for Reward Model Overoptimization, ICML, PMLR 202](https://proceedings.mlr.press/v202/gao23h.html). **Type:** peer-reviewed conference research. **Supports:** possible decline in a reference evaluation under excessive proxy-reward optimization. **Limit:** the higher-quality reference was another reward model, not a direct measure of human welfare.
10. **Suzanne Mason et al., online November 15, 2011; May 2012 issue.** [Time Patients Spend in the Emergency Department: England’s 4-Hour Rule—A Case of Hitting the Target but Missing the Point?](https://pubmed.ncbi.nlm.nih.gov/22088495/), _Annals of Emergency Medicine_ 59\(5\):341–349. **Type:** peer-reviewed observational study. **Supports:** 735,588 visits, compliance rates, stay-time changes, and clustering before four hours. **Limit:** fifteen purposively selected departments; does not establish causes alone.
11. **Steven Kelman and John N. Friedman, February 6, 2009.** [Performance Improvement and Performance Dysfunction: An Empirical Examination of Distortionary Impacts of the Emergency Room Wait-Time Target](https://academic.oup.com/jpart/article-abstract/19/4/917/887027), _Journal of Public Administration Research and Theory_ 19\(4\):917–946. **Type:** peer-reviewed empirical research and an important counter-source. **Supports:** major improvement among all 155 hospital trusts and no observed dysfunction patterns of the kinds tested; does not exclude every unmeasured effect.
12. **Ellen Weber et al., published online 2012.** [Implications of England’s Four-Hour Target for Quality of Care and Resource Use in the Emergency Department](https://pubmed.ncbi.nlm.nih.gov/23102917/), _Annals of Emergency Medicine_. **Type:** peer-reviewed observational study; countervailing or qualifying source. **Supports:** no visible decline in examined quality and safety indicators; not generalizable to every health system.
13. **Saudi Ministry of Health, 2022.** [Ada’a Health Award 2022, Health Performance program document](https://www.moh.gov.sa/en/Ministry/Projects/Documents/adaa-health-award-2022-en.pdf). **Type:** official methodology document. **Supports:** leading, lagging, and balancing indicators in the declared methodology. **Limit:** design description, not causal evaluation of implementation success.
14. **US National Institute of Standards and Technology, 2023.** [AI Risk Management Framework: Core, measurement and management functions](https://airc.nist.gov/airmf-resources/airmf/5-sec-core/). **Type:** official voluntary professional framework. **Supports:** measuring system performance in context and continuously managing risk; not a universally binding rule.
## Source
Author’s manuscript and linked original sources
[Source ↗](https://isafwan.com/en/ideas/when-metric-became-the-customer/#article-references-title)
Published
## Topic
[Artificial intelligence](https://isafwan.com/en/topics/#artificial-intelligence)[Data](https://isafwan.com/en/topics/#data)[Power](https://isafwan.com/en/topics/#power)
[Reading list ↗](https://isafwan.com/en/archive/)
## Related
[Ideas / PublishedThe Two Hours That Returned to No OneDoes AI really shorten the workday? A distinction among task time, cash savings, and human time—plus evidence on rebound, new tasks, and deliberate allocation.](https://isafwan.com/en/ideas/ai-freed-time-rebound/)[Ideas / PublishedThe Apprentice Whose Mistakes DisappearedHow automation can improve output while weakening skill formation—and how evidence from coding, aviation, medicine, and education suggests better training designs.](https://isafwan.com/en/ideas/apprentice-without-mistakes/)[Ideas / PublishedThe Late Signature: Who Owns a Decision Made by a Machine?From England’s 2020 grades to credit governance, an inquiry into real authority, human oversight, explanations, and correction before automated harm takes effect.](https://isafwan.com/en/ideas/executable-correction-authority/)
[Back](https://isafwan.com/en/ideas/)