The number to watch is roughly 600 orders a month, which tends to line up with somewhere around $1M a year in revenue for most stores. Revenue is really just a proxy here, what actually determines whether testing works is order volume, because that's what determines whether a test can reach statistical significance in a reasonable timeframe.
Below that volume, a test simply can't gather enough data to tell a real result apart from noise without running for months. By the time a low-traffic test reaches a trustworthy sample size, you've lost the speed that makes testing valuable in the first place.
If you're under this threshold, that's not a reason to neglect your website — it's a reason to focus on getting the fundamentals right (solid UX, clear messaging, a site built on established best practice) rather than running formal tests. Your resources are usually better spent on growing traffic through marketing at this stage than on optimising a site that isn't getting enough visitors to test properly yet.
Say you've crossed 600 orders a month and you're ready to start testing properly. The next decision is who runs it, and this is where a lot of stores default to the wrong answer.
It's not a job for one ecommerce manager.
Quality A/B testing needs a copywriter, a designer, a developer, and someone who genuinely understands statistical analysis and testing software ,reading a results report properly is its own skill, separate from running your store's day-to-day operations.
Your ecommerce manager is focused on the backend: product setup, inventory, getting ready for sales. Asking them to also run a testing program on the side usually means it gets done badly, or not at all.
Between roughly $1M and $30–50M, an agency is almost always the better math. A newly crossed-threshold store is typically only running one or two tests a month, maybe one test every two to four weeks. A full in-house testing team, fully staffed with the specialists above, runs somewhere around $500,000 a year in wages. At one or two tests a month, that's an enormous fixed cost against a testing cadence that doesn't come close to using it.
An agency solves this two ways. First, the cost is shared across the agency's other clients, so you're paying for a fraction of a fully built team instead of the whole thing. Second, an agency brings pattern recognition from other stores it works with, insights and test results from a different brand's testing program often transfer directly to yours, in a way an in-house team, working only on your site, simply can't replicate.
Past roughly $30–50M, bringing testing in-house starts to make real sense. At that scale, testing cadence increases enough that the math flips, and there's a genuine case for institutional knowledge, a team that's spent a year or more working purely on your site, your customers, and your products, building depth an outside agency can't match on your specific brand alone. Even then, most brands at this stage keep some agency involvement for outside perspective, so their own testing program doesn't get stuck only ever comparing itself to itself.
Here's a distinction worth sitting with when you're deciding what to bring in-house and what to keep external: your website optimisation and your marketing are not the same kind of risk.
If your marketing team or agency drops the ball, your revenue can fall off a clif, genuinely overnight. Marketing is a pillar the business runs on. If most of your revenue comes through paid channels and that engine breaks, you don't stay flat. You go backwards, fast.
Website optimisation doesn't work that way. If you stopped testing tomorrow, your conversion rate wouldn't drop,your site just stops getting incrementally better. The agency's job is to keep moving things forward, but it isn't propping the business up day to day the way marketing or fulfilment is. That means there's meaningfully less operational risk in how you structure this function, and more flexibility in who you work with.
This is also exactly why an agency relationship for testing works well: an agency has to keep proving its value every quarter to keep the contract, in a way that's far harder to do — and far harder to walk away from, with a salaried in-house team that isn't hitting targets. If an in-house team isn't delivering ROI, unwinding six hires is slow and painful. If an agency isn't delivering, you end the contract and nothing about your day-to-day operations breaks.
If you're under 600 orders a month, hold off on formal testing and put that energy into growing traffic and getting the fundamentals right. Once you're past that line and up to somewhere around $30–50M a year, an agency almost always beats building an in-house team,the cost math, the testing cadence, and the outside pattern recognition all point the same direction. Past that scale, in-house starts to make sense, usually alongside an agency rather than instead of one.
Book a free RPV Opportunity Call. a 60-minute walkthrough where we identify 3–5 specific revenue opportunities on your store.
How many orders do I need before A/B testing is worth it? As a general guide, somewhere around 600 orders a month, which tends to line up with roughly $1M in annual revenue for most stores. Below that, tests take too long to reach a trustworthy result, and the time spent waiting usually outweighs the value of what you'd learn.
Should I hire a CRO agency or build an in-house testing team? For most stores between roughly $1M and $30–50M in annual revenue, an agency is the stronger financial case — a fully staffed in-house testing team typically costs around $500,000 a year in wages, which is hard to justify against a testing cadence of one or two tests a month. Past roughly $30–50M, in-house starts to make more sense.
Can my ecommerce manager run A/B testing for my store? Not effectively on their own. Proper A/B testing needs copywriting, design, development, and statistical analysis skills working together — a different skill set to the day-to-day store operations an ecommerce manager typically owns.
Is it riskier to outsource website testing than marketing? Generally, no — it's the opposite. Marketing is an operational pillar; if it breaks, revenue can drop sharply and quickly. Website testing is incremental — if it stopped tomorrow, your conversion rate wouldn't fall, it would just stop improving. That makes it lower risk to structure flexibly, including through an agency relationship.