App Store Connect lets you run only one Product Page Optimization test at a time, for up to 90 days. With modest traffic, that might be a handful of real tests a year. Each wasted test costs you weeks.
These are the mistakes that waste them most often, and how to avoid each one.
1. Changing Everything at Once #
The variant has new captions, a new background, a new order and a new first screenshot. It wins. What did you learn? Nothing you can use in your next design, because you don't know which change did it.
Instead: change one thing per variant. If you have three variants, use them for three versions of the same idea: three different first screenshots, or three caption styles.
2. Testing a Change Too Small to Measure #
A one-word caption tweak might move conversion by 2%. At a 3% conversion rate, detecting a 2% lift takes around a million impressions per variant, which most apps don't get in 90 days. The test runs, confidence never reaches 90%, and you're no wiser.
Instead: check the math before you start with the A/B test calculator, and test bold changes if your traffic is modest. Why tests never reach confidence explains the numbers.
3. Stopping the Moment It Looks Good #
Day four: +25%, and confidence just crossed 90%. You apply it. But early numbers swing wildly, and if you check every day and stop at the first good-looking moment, you'll often "win" with a variant that is no better at all.
Instead: decide the minimum run time before you start (at least one or two full weeks) and don't stop before then, whatever the numbers say.
4. Running Over a Holiday or a Launch #
A test that runs over Black Friday, a feature on the App Store, or your own ad campaign measures a different audience than your usual one. The winner may not win in a normal week.
Instead: avoid unusual weeks, or at least write down what happened during the test so you can judge the result.
5. Too Many Variants for Your Traffic #
Three variants means your traffic is split four ways, and each variant needs the full number of impressions. On a smaller app that turns a 40-day test into a 120-day one, which Apple won't let run.
Instead: with limited traffic, run one variant against the original. Use three variants only when you have the traffic for them.
6. Testing in the Wrong Localizations #
The test only counts traffic from the localizations you include. Include one small market and you wait months. Pool very different markets and a win in one can be cancelled out by a loss in another.
Instead: start in your biggest market, and pool only markets you expect to react alike. See A/B testing localized screenshots.
7. Forgetting App Review #
Every treatment is reviewed before the test can start. Screenshots that break the App Store screenshot guidelines, like claims you can't back up or UI that isn't in the app, get the test rejected. And an alternate app icon must already be in your published app binary, so an icon test needs an app update first.
Instead: hold variants to the same standard as your product page, and plan icon tests around a release.
8. Mistakes When Uploading #
Uploading by hand means dragging a screenshot set into every localization and device size of every treatment. It's easy to drop English screenshots into the French treatment or get the order wrong, and you only notice when a result looks odd, or after App Review.
Instead: check each treatment's localizations before you submit, or let a tool do it. Screenshot Studio uploads a variant straight into the test: you pick the test and variant, and every language and device size goes into the right place.
9. Not Writing Anything Down #
Six months later: did bigger captions win, or was that the dark background? Without notes, you'll test the same idea twice.
Instead: before each test, write down the change, why you think it will win, and the minimum run time. After it, write down the result, including "no difference". Our 25 A/B test ideas use that same change / hypothesis / win format.
Make Every Test Count #
A good test is one change, big enough to measure, run long enough, in the right market. The less time it takes to build and upload variants, the more of your year goes into real tests. That's what Screenshot Studio is built for.