How to use a weekly experiment review to create a growth culture
This is a quick atomic essay I’ve created as part of the ship30 writing workshop.
The Weekly Experiment Review was my favorite meeting at Coursera.
Many leaders want to create a culture of experimentation. Done right, growth experimentation can be a force multiplier for a startup. Unfortunately, few companies get there. Fits and starts in experimentation eventually die down as a team scales, experiments lose rigor over time, or teams over- optimize trivial details with diminishing returns.
Why efforts to create experimentation cultures fail
Here are the most common reasons I've seen:
Experimentation is about the scientific method - hypothesis, test, observe the data , updated hypothesis. Over and over. Your whole team needs to understand this. You are only as good as your weakest link.
Your experiments are as good as your questions. Many teams follow the rituals of experimentation and check all the boxes, but ask poor questions due to fear or lack of imagination.
The best experiments are polarizing. It should be okay to propose these ideas and pursue them even if the leader thinks it might never work. This requires deliberate investment in culture, which many companies don't make.
A great Experiment Review ritual can help leaders establish the cadence, process and culture of experimentation even as they scale their organization. Here's how, step by step:
Step 1: Establish the cadence and mechanics
For consumer growth, I find a weekly experiment review at a set time works best. Create an experiment brief template as well as a results template. Review both upcoming experiment proposals for feedback on experiment design, and data from live experiments to decide on next steps. Create a wiki to catalog results and learnings from all previous experiments. Make sure there is a person running the meeting (not necessarily the leader) and agenda is sent before-hand.
Step 2: Establish the decision making norms
Data is open to interpretation, and the best experiments will spark lots of opinion and debate. Leaders need to be clear "how" decisions will be made, and who will make them -Is it the leader of the meeting, the product manager, the team as a whole, or the person who shouts loudest? When a team is new to experimentation, I prefer to have all experiments go through experiment review. Typically, the Product Manager for the experiment owns the decision and is responsible for ensuring all functional inputs are considered. If the decision is truly high stakes, then it gets escalated up the chain as appropriate.
Step 3: Shape the discussion to shape the culture
The value of the meeting lies in the quality of discussion. In the beginning, the leader will have to model the culture they want to create. Play an outsize and overactive role in asking questions and suggesting ideas, Invite input from the quiet ones and push for sharper hypotheses or rigor in data. But the most important thing a leader can do is admit when their own hypotheses don't work, or disagree and commit to a test the team believes in but they are personally skeptical about.
Experimentation is highly divergent and chaotic by nature. Teams have to move fast and pivot in response to the data. Establishing a strong cadence, process and decision making framework outlined in steps 1 and 2 will provide order and structure to your team. That will enable you and the team to focus on the high quality discussion and ideation to generate the ideas that propel your growth.