A/B tests: listen to the user, beat the HiPPO
Spiros Denaxas
Lokku Ltd. / Nestoria
Presentation Structure
- What is an A/B test
- Motivations
- Examples
- Technical Aspect
- Evaluation
- Conclusion
What you say?!
- {A/B | split | control | treatment | bucket} tests
- controlled online experiment
- one or more different changes are tested
- you are the test subject
- trivial concept
- control group / exposure group
Ronny Kohavi (Microsoft EXP)
Motivations
- "The fewer the facts, the stronger the opinions"
- prediction = hard
- your intuition is poor
- quick, easy, scientific and statistical evaluation
- causal relationships
- insulation
- listen to the user
- no more
HiPPO
Examples
- new homepage design
- move things around SERP page
- different algorithms
- different search ranking algorithms
vs
vs
vs
Technical Aspects
- Randomness of assignment
- 1% exposure initially up to 50%
- Automate
- Statistics::Descriptive
- mod_perl handlers
Evaluation
- OEC
- single or multiple long term objectives
- clean the data
- cleanse the data
- mine the data
- duration
Conclusions
- Fun!
- Be bold, always test novel ideas
- Be data oriented
- Test often
no HiPPOS
Thank you
