Example 1. Bugs and Downtimes

📍 Situation

A large online marketplace frequently experiences failures and downtime, and it’s unclear what to do to reduce them.

🏋 Stakeholders

• CTO: commissions the task, reports to the CEO on product reliability
• monitoring engineers: will be affected, because their workflow will change
  • know how the incident response process works and which failures occur more often than others
• product developers: will be affected, because requirements for their code (old and new) will change
  • know how reliability is accounted for in product requirements
• product managers: will be affected, because tasks will appear in the backlog
• users: will be affected, because they will encounter fewer bugs
  • know which problems hurt the most
• media, social networks: will be affected, because they monitor the platform’s status and shout about every major outage

⚡ Pains

• Downtimes and bugs make it impossible for users to do their work, so they leave our platform for competitors.
• Wide media coverage damages the platform’s reputation, making it harder to attract and retain users, which affects profit.
• Downtimes and failures for large corporate users cause their frustration, contract cancellations, and complaints to the CEO.
• Overly strict reliability requirements paralyze product development, preventing product managers from evolving it.
• If we don’t do the task at all, there will be more and more bugs and technical issues, the company will lose money, and growth will slow.

📗 What we know

• There are N failures per month, of which M are critical (all features unavailable) — <BI report>
• We have a failure cost model. According to it, we lose X million RUB per month — <BI report, model docs>
• We learn about failures from three sources: a) triggers on technical metrics in production, b) monitoring of events in the product, c) user tickets to tech support — <docs, runbooks, repositories>
• Failure alerts are handled by on-call staff and a separate monitoring team [details] — <source>

🔖 What we don’t know

• How exactly are on-call staff notified? Is there an escalation protocol? How much time passes from problem detection to fix?
• Do detectors catch all problems? How often do we learn about a problem from users or from the media?
• How is the product tested?
• What is the reliability team’s roadmap for the quarter and year?

💡 Hypotheses

• I expect losses from failures to follow the Pareto principle. To check this, I will build a distribution of losses by problem type. If it turns out that 50% of losses are caused by 3–5 types, I will consider the hypothesis confirmed.
• I expect that the escalation process has slow steps that account for most of the downtime. To check this, I will collect statistics on events within incidents. If it turns out that 50% of the time falls on 1–2 stages, I will consider the hypothesis confirmed.

🚀 Next steps

• Go to the monitoring team to map the process and get data on alerts and outages.
• Go to 1–2 product development teams to learn how testing is set up.
• Build a distribution of losses over the last six months to find the most expensive incidents.
• Go to the CTO to show him the task statement and hypotheses.

Example 2. A/B Test

📍 Situation

Due to a new law, our existing users have become expensive to serve, so we need to attract a new segment to the product. We are adapting the product for them, and also running various marketing campaigns targeting people from this segment; we want to choose the most effective ones.

🏋 Stakeholders

• Product Manager: commissions the task, owns the product backlog
  • knows what we offer the new segment, how their processes work, and how the product for them differs from that for existing users
• marketers: design campaigns and organize the launch
  • know the profile of our current and new users
• designers: create the creative assets and prepare campaign materials
• developers: make small product changes if needed to run a campaign
  • know how the product is built, where and how data is stored, and what data is collected
• users from the new segment: will see the ads/offers/banners and may or may not come to the product
• existing users: will see product changes made for the new segment

⚡ Pains

• Because of new laws, existing users have become expensive to serve and have started to churn, so unit economics has deteriorated and the product manager cannot grow the product according to plan.
• The product’s lack of adaptation to the new segment limits the inflow of people from there, forcing the product manager to drop previously planned features from the backlog and marketers to find ways to reach new users.
• If we don’t do the task at all, the product’s unit economics will go negative with existing users.

📗 What we know

• For unit economics with the new segment to become positive, customer acquisition cost (CAC) for them must not exceed X. — [unit economics calculation]
• Marketers have tried to attract the new segment; the results are in [document].
• Creative assets and campaigns ready for launch are in [document].
• The product for the new segment is still raw: the payment and cash-out features work with bugs; a fix is in the backlog, but will take 3 sprints — [backlog, developers’ Jira]
• We have a traffic splitter in the product, but no A/B testing tool — [splitter docs]

🔖 What we don’t know

• What should customer acquisition cost for new users be? How many of these users are there in total?
• How much time is there for experiments? How many users are needed for the results to be statistically significant?

💡 Hypotheses

• I expect campaigns to differ in acquisition cost, but not in retention.
  • To check this, I will run an A/B test. If it turns out that <acceptance condition>, I will consider the hypothesis confirmed; if <rejection condition>, rejected.

🚀 Next steps