Published · 9 min read
Hold a funnel over a mark on a table and drop marbles through it one at a time, noting where each one lands. Left alone, the funnel gives you a tight scatter around the mark. Now try to help by shifting the funnel after every marble to cancel out the last miss. The scatter widens, and under the most natural correction (move the funnel back by the distance of the previous miss) the variance of the landings doubles. W. Edwards Deming used this experiment in Out of the Crisis (1986), crediting its design to the statistician Lloyd S. Nelson, and called the habit tampering.
A sudden drop in conversion rate is a fall in the share of visitors who complete a page's action, large enough and fast enough (usually day on day or week on week) to be noticed before anyone knows the cause. It has three possible sources that look identical on a dashboard: a real change in how visitors behave, a change in who is arriving, and a change in how the number is measured. Underneath all three sits a fourth possibility, which is that nothing changed and this is what the page's numbers do from one week to the next.
The usual response to a bad week is quick and looks sensible. Someone asks why, a thread fills with theories, and by Friday a fix has shipped: a new headline, a shorter form, a revert of whatever went out last.
The funnel puts a question in front of all of that. Walter Shewhart, in Economic Control of Quality of Manufactured Product (1931), split variation into chance causes, which the system produces all the time, and assignable causes, which you can find and remove. A drop has to earn its investigation by landing outside the scatter the page already produces. If it lands inside, there is no cause to find, and the hunt will find one anyway.
Your page is the funnel and each week is a marble. Move the page after every bad week and next quarter's chart gets noisier, because your own changes are now stacked on top of the ordinary scatter. Whatever you shipped also takes the credit when the following week comes back to normal, which it probably would have done on its own, so the team reaches for the same kind of fix next time and the backlog fills with changes that barely move the rate.
Analytics can tell you that the number fell and at which step. It cannot tell you whether the fall is bigger than the page's normal wobble, and that has to be settled first. Recordings from the bad week will show people leaving and rage clicks firing, and so would recordings from any ordinary week; the limits of that instrument are covered in what session recordings can and cannot show.
Say a signup page gets 2,000 visitors a week and converts at 3%. An average week produces 60 signups. For a count like that, the ordinary week-to-week wobble is roughly the square root of 2,000 × 0.03 × 0.97. That is the square root of 58.2, which comes to about 7.6 signups.
Now a week arrives with 45 signups. The dashboard calls it a 25% drop. It is 15 below the average, or about two wobbles. On this arithmetic alone, a page whose visitors have not changed at all will produce a week that low roughly once a year.
The formula treats every visitor as an independent coin toss with the same odds, and real traffic is lumpier than that, so treat the figure as the least wobble to expect. Smaller pages have it worse: at 500 visitors a week and the same 3%, one ordinary wobble is about a quarter of the average.
Shewhart set his control limits at three times the spread, on purpose, because he judged that chasing chance cost more than occasionally missing a real shift. By that standard the 45-signup week is comfortably inside the limits, which on these numbers sit at about 37.
To get your own band, pull the last 8 to 12 weeks of visitors and conversions for the page, leaving out any week you already know was broken, and work out the average and the wobble with the formula above. If this week sits inside that range, you do not yet have a drop to explain.
The second thing that can move without the page changing is the meter. A consent banner update changes who gets counted. A tag stops firing after a deploy. A redirect strips campaign parameters, so a paid channel's signups land under direct traffic. In each case the visitors did exactly what they did the week before.
The check takes about ten minutes. For the same week, compare the conversions your analytics tool recorded with the ones your own systems recorded: new rows in the signups table, orders in billing, demo requests in the CRM. The two will never match exactly, and that is fine. What matters is whether the gap between them moved. If the backend held steady while analytics fell, the thing that dropped is your measurement, and no change to the page will bring it back.
A July thread on r/GoogleAnalytics shows what the gap costs: "GA4 says 80k, the backend says 100k, and you can explain consent mode and ITP all day, what the client hears is 'your dashboard is wrong' and they stop looking at it."
We tripped over a version of this on our own site this month. While reviewing our homepage, we tried to replace some placeholder numbers with real ones and pulled an average audit duration straight from our production database. The long tail of that average was made of pipeline processes that had hung, and the counts sitting beside it came from a period when the pipeline had known defects. We withdrew the numbers before they were published, and we are not quoting them here for the same reason. Nothing on the page had changed, and the numbers still looked like a finding until someone asked what was in the tail.
If the week sits outside the band and the backend agrees with analytics, you have a real signal and the root-cause hunt is the right next move. When something shipped just before it, the piece on conversion drops after a redesign covers how to find what the change removed.
Sometimes the right answer is to wait a week, and that advice is uncomfortable for anyone paid to act, us included. A Nudgent audit reads the page as it stands. It cannot tell you whether last Tuesday was noise, and it will find friction on a page that has not changed in a year, because nearly every page has some. Run one the week after an ordinary wobble and it will hand you a ranked list of real problems; you will fix the top one, the next week will come back to average, and the fix will get the credit. That is the funnel again, with our name on the adjustment. How we score, and what a finding does and does not claim, is written out in the methodology, and the audit methodology glossary entry covers why two reviewers looking at the same page will still disagree.
Waiting has its own cost. Someone on r/AskMarketing wrote in June: "You can run a great campaign and still lose the account because you couldn't explain why CAC spiked one week." The funnel does not make that pressure go away. What it changes is the honest explanation, which is sometimes "that week is inside what this page normally does, and here is the chart that shows it."
Before the next meeting about a drop, run these checks in order.
Only a week that clears all of them has earned a root-cause hunt, and then it deserves a proper one: one change at a time, measured for long enough to believe the result.
Most dashboards report the change against last week in green or red and leave the judgement of whether it matters to whoever happens to be looking. When conversion last dropped at your company, did anyone check whether it sat outside normal variation before the fixes started shipping?
Usually because something other than the page changed: the mix of visitors arriving, the way conversions are counted, or nothing beyond ordinary week-to-week variation. Check the week against the range of your last 8 to 12 weeks, compare analytics with your backend count, then look at traffic sources. If all of that is clean, the drop is real and worth investigating.
Compare the conversions your analytics tool recorded with the ones your own systems recorded for the same week, such as new signups in your database or orders in billing. The two numbers rarely match, so look at whether the gap between them changed. If the backend held steady while analytics fell, the drop is in the measurement.
Not automatically. First check whether the drop sits outside the page's normal range and whether your backend agrees with analytics. If both say the drop is real and the change is the only thing that shipped, a rollback is a reasonable test, and a recovery tells you something. If the week sits inside normal variation, rolling back is one more adjustment to a stable process, and whichever version is live when the numbers return to average will look like the fix.