Primary source
Dark Data (book_cube)
Two-part summary of David Hand's book: core idea, 15 dark-data classes, and practical management implications.
Primary source
Dark Data (book_cube)
Two-part summary of David Hand's book: core idea, 15 dark-data classes, and practical management implications.
Dark Data: Why What We Don't Know Is Even More Important Than What We Do
Authors: David J. Hand
Publisher: Princeton University Press
Length: 2020
A practical lens on decision quality: robust leadership requires working with both observed evidence and structured unknowns.
OriginalManagement / Data / Statistics / Decision-making
The central message of Dark Data: decisions are usually driven by visible reports, but many failures come from data we do not see. For engineering leaders, this is an operating discipline for uncertainty, not only a data-science topic.
1. Map of 15 dark-data classes
Class 1
Known unknowns
We know some values are missing, but still read the report as if the dataset were complete.
Class 2
Unknown unknowns
The most dangerous case: we do not even see which data is absent, yet act with high confidence.
Class 3
Selection effects
Inclusion criteria narrow reality. This also covers p-hacking and reporting only convenient outcomes.
Class 4
Self-selection
Respondents and non-respondents differ systematically, but the final metric hides that gap.
Class 5
Unknown determining factor
A compelling correlation may be driven by a hidden variable or group-mixing effect.
Class 6
Counterfactual data
We cannot directly observe what would happen under an alternative decision path.
Class 7
Data changing over time
Metrics age, systems evolve, behavior shifts, while models often stay calibrated to old context.
Class 8
Misdefined data
Metric definitions change over time, making trend lines incomparable without explicit annotation.
Class 9
Data summarization
Keeping only aggregates (mean, median, variance) removes tails, distribution shape, and outliers.
Class 10
Measurement error and uncertainty
Noise in measurement and format conversion quietly distorts conclusions.
Class 11
Feedback distortion
The metric itself changes behavior: people optimize the indicator, not the actual outcome.
Class 12
Information asymmetry
Different participants hold different data, so decisions are made with unequal information quality.
Class 13
Intentionally obscured data
Facts are filtered or framed selectively to support a preferred narrative.
Class 14
Fake and synthetic data
Synthetic data can be useful, but in bad-faith scenarios it can hide fraud and noise.
Class 15
Extrapolation beyond observed range
A model can perform well in-sample but fail once conditions move outside known boundaries.
2. Practical toolkit from the book
Randomized controlled experiments
A/B tests and RCTs help estimate causal impact instead of relying on observational correlations.
Simulation
When direct observations are unavailable, simulation can stress-test decisions across plausible scenarios.
Replication and bootstrapping
Resampling gives uncertainty ranges and reduces overconfidence in one observed sample.
Bayesian approach
Prior assumptions plus sequential updates are useful when data is sparse or arrives gradually.
Privacy-preserving data collection
Sensitive signals can still be collected responsibly using privacy-aware collection patterns.
Streetlight effect reminder
Hand closes with the classic joke about searching where the light is better. It is a direct warning against making expensive decisions from convenient-but-incomplete metrics.
3. Common anti-patterns
Assuming missing data means missing risk.
Treating correlation as causation without checking hidden factors.
Making strong decisions from aggregates without distribution checks.
Deploying a model in a new context without boundary testing.
Optimizing only what is easy to measure (streetlight effect).
4. Recommended practices
For major decisions, explicitly document which data is missing and the risk impact.
Include bias, self-selection, and metric-definition drift checks in KPI reviews.
Validate at least one counterfactual scenario before high-cost commitments.
Use uncertainty intervals and scenario ranges instead of single-point forecasts.
Separate metric monitoring from the true business objective to avoid feedback traps.
5. Pre-decision checklist
- What is observed as fact, and what is assumption?
- Which data is missing, and why was it not collected?
- Which user segments might be absent from observation?
- Could a hidden variable explain the observed correlation?
- How does the decision change if assumptions fail?
- What minimal experiment removes the costliest uncertainty?