The greatest obstacle to discovery is not ignorance—it is the illusion of knowledge.
Imagine a world where doctors, educators, and policymakers could quickly determine which approach works best for their patients, students, and communities. We have a powerful tool to do exactly that: A/B testing, which compares two alternatives head-to-head in controlled experiments. Yet, there's a puzzling contradiction in how we view this tool. While we readily accept untested policies implemented across entire organizations, we often bristle when someone proposes to rigorously test which of two approaches actually works better 1.
Consider the case of Pearson Education, which faced public outrage when testing educational software, despite no objections to the previous untested version 1.
Testing revealed that students who received no encouragement attempted more problems—a discovery impossible without systematic testing 1.
The A/B effect describes a consistent pattern in human judgment: people frequently approve of implementing untested policies or treatments but disapprove of randomized experiments to determine which of those same policies or treatments is superior 1.
Researchers documented this phenomenon across 16 studies involving 5,873 participants from diverse populations, spanning nine domains from healthcare to autonomous vehicle design to poverty reduction 1.
One revealing study illustrates the A/B effect perfectly. Participants were presented with a scenario about a hospital director trying to reduce deadly catheter-related infections 1.
All doctors have safety precautions printed on their hospital ID badges
All procedure rooms display a poster of safety precautions
Randomly assigning patients to badge or poster method to determine which works better
| Approach | Mean Appropriateness Rating (1-5 scale) | Percentage Rating as Inappropriate |
|---|---|---|
| Badge (A) | 3.93 | ~25% |
| Poster (B) | 4.35 | ~10% |
| A/B Test | 2.74 | ~60% |
One powerful explanation for the A/B effect is what researchers call "the illusion of knowledge"—the belief that experts already know or should know what works without needing to test 1.
This phenomenon isn't limited to the general public. In science funding, the peer review system favors incremental, conservative projects over innovative but riskier proposals 3.
People seem to demand consent for A/B tests more than for universal implementations of untested policies, despite the logical inconsistency of this position 1.
There's also a deeper aversion to the element of control in experiments. While we accept that outcomes in the real world are influenced by countless uncontrolled factors, the deliberate introduction of control in an experiment feels different.
Beyond psychological factors, structural barriers also impede the testing of alternatives. In scientific funding, the extremely low success rates of grant applications creates a hypercompetitive environment that discourages risk-taking 3.
| Organization | Success Rate | Notes |
|---|---|---|
| National Institutes of Health (NIH) | 20% | For experienced PIs, the "payline" is just 11% |
| National Science Foundation (NSF) | 26% | 2021 data |
| Gates Foundation | 1-2% | Estimate, as exact rates not released |
Whether you're testing educational methods, healthcare interventions, or business practices, having the right methodological toolkit is essential.
| Component | Function | Example Applications |
|---|---|---|
| Randomization | Eliminates selection bias by ensuring each participant has an equal chance of being assigned to any condition | Clinical trials, program evaluations, policy pilots |
| Control Groups | Provides a baseline against which to measure changes, isolating the effect of the intervention | Testing new teaching methods against standard practice |
| Blind Procedures | Reduces bias by preventing participants and/or researchers from knowing who receives which intervention | Drug trials, educational interventions, product tests |
| Pre-registration | Specifies analysis plan before data collection to prevent cherry-picking results | All experimental sciences |
| Standardized Metrics | Ensures consistent measurement across conditions for fair comparison | Standardized tests in education, medical outcome measures |
In fields facing particularly difficult testing questions, useful frameworks have emerged. In animal research, for instance, the "3Rs" framework (Reduction, Refinement, Replacement) guides scientists to use animals more ethically when alternatives exist 4.
Minimizing the number of animals used while maintaining statistical power
Improving experimental procedures to minimize pain and distress
Using non-animal methods when scientifically valid alternatives exist
Create psychological safety for admitting uncertainty and proposing tests to resolve it
Recognize that a well-designed test that reveals what doesn't work is as valuable as one that confirms a hypothesis
Shift organizational culture from decision-by-anecdote to decision-by-data
Use pilot testing to generate preliminary evidence before full implementation
Share methods and results openly to build trust and contribute to collective knowledge
The resistance to testing alternatives represents a significant barrier to progress in nearly every field. The A/B effect—where we approve of untested universal implementations but disapprove of randomized comparisons—is a robust phenomenon that persists across diverse domains and populations 1.
Overcoming this resistance requires addressing both psychological biases and structural barriers.
The stakes are high. When we reject rigorous testing in favor of untested implementations, we risk perpetuating ineffective or even harmful practices while missing opportunities to discover better approaches. The good news is that we can build a more rational approach to decision-making by normalizing uncertainty, creating better testing frameworks, and recognizing that in a complex world, systematic testing isn't a sign of ignorance—it's the hallmark of intellectual humility and wisdom.
The next time you face a choice between two alternatives, consider whether testing might be better than guessing. The answers might surprise you, and they might just lead to better outcomes for everyone involved.
As one researcher noted, "rigorously evaluating policies or treatments via pragmatic randomized trials may provoke greater objection than simply implementing those same policies or treatments untested" 1. But the temporary discomfort of testing is far preferable to the permanent ignorance of implementing untested solutions.