Removing unauthenticated access from our APIs
Over the last year or so we’ve seen a huge increase in bot traffic to all our sites. We’ve attempted to improve performance, implement caching and various other methods but, sadly, we have now decided to remove unauthenticated access to our APIs.
To recap, the API allows people to access data from our candidates website. This includes historical data on elections and candidates. The data in it is under an open licence and API keys are free for anyone who creates a user account on the site.
We are not introducing charging to our free APIs. Authenticated API access is (more or less) unlimited. This change allows us to identify and communicate with someone who is using an API heavily enough to make it hard to keep the site running.
To put that into numbers, over the last three months about 70% of requests were from bots: around 88 million requests of the 128 million total requests.
This has been slowly increasing, with more of those requests coming in the last few weeks. In the first 12 hours after blocking unauthenticated access, we rejected about 2.2 million requests from a total of about 2.4 million.
What are they doing?
Our API contains a lot of data, so we expect to see a fair number of automated requests. However the nature of the requests we’re seeing look more like a distributed, coordinated set of scripts that are ‘mindlessly’ crawling the open web.
To take one example: Meta (formerly Facebook) sent a huge number of requests that ‘went in circles’: it requested every link on every page, even if it had seen it before. On a web page with a previous/next link, it ends up going back and forwards in a loop.
What we tried
The pattern we saw was: something would overload the site. We could log into our firewall and try to figure out what the traffic profile looked like. Initially this was simple, like blocking the Meta bot.
However, we started to see a more complex pattern of requests over time.
These include
- Requests that we think were coordinated but coming from random IP addresses. This by-passed our attempts at limiting requests based on IP.
- Bots using automated control of real browsers, making it hard to detect legitimate use from illegitimate use.
- Rotation of the ‘user agent’ or the label a browser gives itself, designed to give the impression of different browsers and bypass more classes of blocking.
There are ways of ‘fingerprinting’ requests to see if they come from the same code, but even that wasn’t a useful enough way to reliably block traffic.
The site had been taken down enough that we decided to just force legitimate users to authenticate. We’re not blocking anyone from getting the data, just making it harder for the bots, hopefully.
Using our data
As a recap: the data we’re talking about covers elections, nomination data and candidate profiles going back about 10 years. The API is useful for some things, but a lot of research and analysis can be done using the CSVs. These are still free to download without signing in, and provide the best way to get data out of the site.
If you would like to use our data, or if you have a good reason for using the API without authentication, please get in touch.
Did it work?
Since making this change we quickly saw the bots move to our CSV download. They ended up mindlessly crawling the exact same CSV over and over again, with the same profile of distributed IPs. It’s easy to download the full CSV from our site, but someone or something is requesting filtered versions of the CSV over and over again.
We’re working on making this harder for them to do, and it’s likely that we’ll see a lot more of this in the future.