Smarter Week

How to automate it

How to automate “analyze A/B tests and experiments”

Here are 2 ways to spend less time on this, best first. Each comes with steps you can follow today and, for AI fixes, a prompt to copy.

90 min
typically, once a week
50%
of the time can be automated
Some setup
to set up

Fix 1 of 2

Software featureBest fix

Run experiments through a platform that does the stats for you

Statsig, Eppo, GrowthBook, Optimizely and Amplitude Experiment calculate results, guardrails and sample ratio checks automatically, so analysis is review, not rebuild.

Typically saves about 50% of the time8 h to set up
  1. 1Pick the platform that fits your stack (warehouse-native options read your existing metrics).
  2. 2Define standard metrics and guardrails once.
  3. 3Launch experiments through it so assignment and exposure are logged.
  4. 4Review the generated results and write the decision; dig in by hand only for odd results.

Tools: Statsig · Eppo · GrowthBook · Optimizely · Amplitude Experiment

Fix 2 of 2

AI

Have AI draft the "so what" write-up from your results

AI is good at turning tables and charts into a clear narrative for a non-technical audience. You keep the conclusions honest and the caveats in.

Typically saves about 45% of the time5 min to set up
  1. 1Paste the result tables, the original question and any context.
  2. 2Run the prompt below, naming the audience.
  3. 3Check every number and remove claims the data does not support.
  4. 4Add your recommendation in your own words.
Prompt to copy
Write a short summary of these results for [AUDIENCE]. The question was: [QUESTION]. Lead with the answer in one sentence, then three supporting points with the numbers, then caveats (sample size, data quality, what this does not show). Avoid jargon; explain any statistic in plain words. Do not claim causation unless the method supports it.

Results:
[PASTE TABLES OR DESCRIBE CHARTS]

Context:
[PASTE]

Tools: ChatGPT, Claude, Gemini or Microsoft Copilot

Who does this task

Roles in our library that list this as one of their common tasks. Each guide covers the rest of that role’s week.

Data scientist

HourLeak · the 8-minute work audit

How many hours does this cost you?

The free 8-minute check works out where your week goes and gives you your top fixes. The team scan does the same for everyone and adds it up, so you know which leaks to fix first.

Take the free check Scan my team, $49

Answers are anonymous. Leaders only see team totals.

Other common tasks for Data scientists