Experimentation Engine
Growth Experiments
Test everything. Measure everything. Scale what works. Data-driven experimentation framework.
3
Running
4
Completed
5
Proposed
3
Winners
75%
Win Rate
Sample Size Calculator
8,150
Per Variant
16,300
Total Required
To detect a 20% relative change from 5% to 6.0% with 95% confidence.
Experimentation Framework
Hypothesis
Clear, testable statement
Design
Sample size, duration, metrics
Build
Implement A/B variants
Run
Collect statistically significant data
Learn
Document insights, ship winners
Status:
Category:
ICE Prioritization (Proposed Experiments)
| Experiment | Impact | Confidence | Ease | ICE Score | Priority |
|---|---|---|---|---|---|
Referral Leaderboard referral | high | 70% | medium | 35 | #1 |
Instagram Story Sharing referral | high | 70% | medium | 35 | #2 |
Merchant Welcome Bonus activation | medium | 70% | low | 35 | #3 |
Streak Multiplier retention | high | 70% | medium | 35 | #4 |
Personalized Merchant Recommendations retention | medium | 70% | high | 7 | #5 |
Key Learnings from Experiments
Onboarding Gamification
Users respond strongly to visible progress. Implemented for all new users.
Merchant Landing Page A/B
"500+ merchants already growing" beats "Grow your business with data"
Coin Expiration Warning
Loss aversion works. Users with expiring coins transacted 2.1x more.
Premium Subscription Pricing
Lower price didnt improve conversion. Users see AED 39 as more premium. Revenue per user actually dropped.
WhatsApp Chatbot Onboarding
Technical issues with WhatsApp API. Paused until resolved.
Experimentation Best Practices
✓ DO
- • Define success metrics before starting
- • Run to statistical significance (95%+)
- • Document learnings, even from failures
- • Test one variable at a time
- • Consider novelty effect (wait 2+ weeks)
✗ DON'T
- • Stop experiments early when results look good
- • Run too many experiments simultaneously
- • Ignore negative results
- • Make decisions on small sample sizes
- • Change experiment mid-run
📊 Metrics
- • Primary: One clear north star metric
- • Secondary: Supporting metrics to watch
- • Guardrails: Metrics that shouldn't drop
- • Sample: Per-variant calculation required
- • Duration: Minimum 1-2 full cycles