When rate limiting becomes a security problem

A rate limiter is a security control. Implemented in the wrong place, it can become the weakness.

The pattern

An API in production showed a puzzling behaviour under load tests. At low concurrency (10 to about 160 users) the 95th-percentile latency was around 20 seconds. Beyond roughly 170 to 290 users it dropped to about 50 ms. Error rates did not rise and throughput stayed stable. Authenticated and anonymous requests behaved alike.

What the evidence showed

The slow log of the application server showed dozens of workers blocked at the same point: a database query inside a global middleware. That middleware ran on every request, before authorisation and before any controller, including requests that would end in a 401. The query implemented rate limiting by counting the rows in a log table for the last 60 seconds, with an OR condition on client and IP and several joins.

With few workers, each request waited on the same query and the pool was serialised. With more concurrency the contention was spread across more connections, which is why the numbers improved as load grew.

Why this is a security issue

• An unauthenticated caller could cause database work on every request.
• A handful of requests were enough to degrade the service, so low-volume denial of service was possible.
• The control meant to limit abuse was the cheapest way to cause it.

What was recommended

1. Remove database queries from the global middleware and return early for unauthorised requests.
2. Move the counting to an in-memory store such as Redis, using an atomic increment with a 60-second expiry, which costs a constant amount of work per request.
3. Keep authentication and business logic in separate middleware, and make the middleware stateless.

What to test in your own API

Run a load test at low concurrency as well as high. Check what happens before authentication. Ask what an anonymous caller makes your database do.

Based on a real engagement, with details removed. The fixes above were recommendations; we do not report measured results here.

This post is the short version. The full analysis, with diagrams, is in the white paper Rate limiting that blocks the service.