The distributed attacks started immediately when we went live with Feature Upvote’s public REST API last week.
When I say “immediately”, I mean it. These probing attacks were certainly already happening before our API endpoints existed, just in case there was already a REST API.
Sigh. The Internet is full of bad actors.
(An aside: Many website owners are surprised to learn that their website is under non-stop attack. “Why are they attacking my site? There’s nothing worth getting!” they ask, puzzled, not realising that the attacks are often not targeted at any website specifically, and that all public websites are under non-stop attack.)
Luckily, defending a REST API is a common, well-documented task, made straightforward by the fact that all our site traffic is already funnelled through Cloudflare.
Our public REST API is defended in three ways:
- We drop requests that don’t match valid API endpoints
- We drop requests missing an authorization token
- Rate limiting
1. We drop requests that don’t match valid API endpoints
Our REST API is small. So it was straightforward to add each valid endpoint to a whitelist in Cloudflare. Cloudflare blocks any request to our API if it is not on that whitelist.
This alone blocks much of the generic, scripted attacks faced by the public APIs of all SaaS products all the time.
If an attacker is a bit more motivated, they’ll replace the generic list of API endpoints for probing with our specific endpoints. If we one day choose to publish an OpenAPI config file, it would be trivial to automate attacks that probe our actual endpoints.
2. We drop requests missing an authorization token (aka “Token Presence Enforcement”)
All of our API endpoints require authorization via a Bearer token. If an API request does not have an “Authorization” header whose value starts with “Bearer “, Cloudflare immediately blocks it. The request does not reach our servers.
This step is helpful against the primitive, generic attacks we are facing.
3. Rate limiting
I’ve added reasonable rate limits to all of our endpoints. For simplicity (and to avoid having to upgrade to a higher tier of Cloudflare pricing), I’ve used a simple rate limiting rule for all endpoints.
If our API receives too many requests from one IP address per minute, we block that IP address for the next minute. Again, this is handled by Cloudflare, preventing the requests that exceed the rate limit from reaching our servers.
Even better would be if we could apply the rate limit by API token as well. That would require an expensive upgrade to our Cloudflare plan, so I’ll hold off on that approach unless it becomes very clear we need it. I can also implement “rate limiting by API token” at the app level, if need be.
Bonus step: logging
All attempts to access our API at logged at various levels.
I’ve set up one specific log to show all attempts that reach our app server, having got through our Cloudflare-powered blocks. This gives me the data to be confident that our current defences are working, and to work out what future defences might be needed.
Conclusion
Implementing these three steps led to an abrupt stop of scripted bot attacks reaching our servers. This is a great result. However I’m pessimistic enough to expect some future attacks to still reach our server, so I—and the entire Feature Upvote team—remain vigilant.