Nullius in verba
Latin for "no one's words" or "take nobody's word for it".
Optimization hierarchy of evidence
HiPPO
Highest Paid Person’s Opinion
ZEBRA
Zero Evidence But Really Arrogant
Types of metrics for experimentation
- Success metrics: Metrics that we aim to improve, tested with superiority tests.
- Guardrail metrics: Metrics that we do not want to see deteriorate more than a certain threshold, tested with non-inferiority tests.
- Deterioration metrics: Metrics that should not deteriorate, tested with inferiority tests.
- Quality metrics: Metrics that verify the integrity and validity of the experiment itself.