Your analytics are lying to you

Alistair Davidson writes about migrating a form-heavy web application from a React SPA to a traditional server-rendered HTML-first website. The entire article is worth reading, but I want to draw attention to this bit about analytics (emphasis mine):

The results? When we launched, the number of people completing the form doubled. The analytics people didn’t even know where these users were coming from. Of course, your javascript-based analytics package doesn’t see the users you are bouncing because of javascript failures. It was a flood! We also saw my “keep a backend session, never lose user data” approach pay off. In one case, someone completed a form a month after starting it.

Web analytics are fragile. They fail in so many ways that making product decisions based wholly on your Google Analytics or Plausible data is folly of the highest degree.

Here's a subset of all the reasons your analytics package undercounts or miscounts visitors:

  • Network errors prevent your analytics script from loading.
  • Ad-blockers and tracking prevention block your script from loading (enabled by default on many browsers today).
  • A JavaScript error in an unrelated part of the page prevents the analytics script from working correctly.
  • The user loses network connectivity before the analytics script can send data to the server.
  • The user gets impatient and bounces off your website before the page can load fully and start collecting data.
  • Too much JavaScript on the page causes the browser tab to crash (a common issue on low-end devices).
  • The analytics script is blocked by a DNS rule, corporate proxy, firewall, or VPN.
  • The user has disabled JavaScript.
  • The user's browser has limited or no support for JavaScript (Opera Mini still has more than half a million downloads on Android, and it's still widely-used in Africa).
  • The user is accessing your content using a service that strips JavaScript (e.g. an RSS reader, a web archiving tool, Telegram Instant View, AMP, a read-later service, or a bookmarking service).
  • You only test your app in Chrome, so you don't realize that your website is entirely broken in Firefox and Safari.

Web analytics can only give you an approximation of what your web traffic looks like. Even when they work correctly, they paint an incomplete picture.

As I said in my post about share buttons, the number one referrer for pages on this website is "Direct/none". It's impossible for Plausible to figure out where those users are coming from. Further, my server logs report three times as much traffic as my Plausible dashboard over a seven day window. Some of this might be bot traffic and thus irrelevant, but I know for a fact that a large chunk of this traffic comes from RSS readers. Plausible will never have insight into these users.

My point is, if you rely on your analytics dashboard to make product decisions, you're excluding a large chunk of potential users who simply don't show up in your graphs.

You might be missing out on serving thousands of potential users because you can't see them in your data. These are users who want to sign up for your newsletter, buy your app, subscribe to your service. These are human beings you could help, whose lives you could improve.

I'm not saying that analytics are completely useless. They can and should have a place in your decision-making process. Just don't treat analytics data as gospel, because there will always be massive blind spots in what it tells you.

To get a real understanding of how users experience your products, test them on real devices under real conditions as much as possible. And as always, get out there and talk to your users.