Rate limiting that blocks the service: anatomy of a low-load denial of service

Key points

• An API showed a 95th-percentile latency of about 20 seconds at low concurrency, and about 50 milliseconds at higher concurrency. Errors did not increase.
• The cause was a rate-limiting query that ran in a global middleware, on every request, before authorisation.
• With few busy workers, every request waited on the same query. Tuning the web server alone did not remove the problem.
• The same design allowed a low-volume denial of service, so it is a security finding as well as a performance one.
• The recommended fix moves the counting to an in-memory store and rejects invalid requests before touching the database.

1. The symptom

During a load test of a production API, the results looked backwards. Between 10 and about 160 concurrent users, the 95th-percentile response time was around 20 seconds. Beyond roughly 170 to 290 users it dropped suddenly to about 50 milliseconds.

Other signs pointed the same way. Authenticated and anonymous requests had almost identical curves. The time to first byte dominated. Error rates did not rise, and throughput stayed stable.

Figure 1. Latency against concurrent users, redrawn from the reported ranges. These are not raw measurements.

A system that gets faster as more people use it is unusual. It usually means requests are waiting on something shared, and that waiting is diluted when more requests arrive.

2. Finding the cause

The first step was infrastructure tuning: the number of PHP workers, concurrent connections, timeouts and buffering on the web server. It helped. The point at which the service stayed stable moved from about 60 concurrent users to much higher values. It did not remove the high latency at low load, which showed that the cause was in the application.

The slow log of the PHP process manager then showed dozens of workers blocked at the same moment, in the same place: a database call made from a global middleware. That middleware ran before authorisation and before any controller, so it ran for every request, including those that would end in a 401 response.

Figure 2. Where the rate-limit query sat in the request path.

The query implemented rate limiting. It counted the rows of a log table for the last 60 seconds, with an OR condition on the client and on the IP address, and joined two further tables to find the client's settings.

3. Why it was worse at low load

The explanation from the analysis is serialisation. With few busy workers, each request reached the middleware and waited for the same query, so the whole pool was effectively blocked on it. With more concurrent requests the contention was spread over more connections, and latency fell.

Figure 3. A simplified picture of the mechanism.

This is why a short load test at high concurrency can look healthy, while real users at moderate traffic suffer.

4. Why this is a security issue

A control meant to limit abuse had become the cheapest way to cause it.

• Pre-authentication cost. An anonymous caller made the database do work on every request.
• Low-volume denial of service. A small number of requests could occupy the worker pool and degrade the service for everyone else.
• Misleading scalability. The system looked healthy in tests at high concurrency, so the weakness went unnoticed.

In a penetration test this would be reported as an application-level denial-of-service finding, together with its evidence: the load profile, the slow log and the trace of the request path.

5. What was recommended

The recommendations were ordered by effort and risk.

Figure 4. The recommended request path.

Option	What it does	Notes
Rate limiting in Redis	Counts requests with an atomic increment and a 60-second expiry	Constant work per request. Recommended
Database tuning, temporary	Splits the OR into separate counts and adds indexes on client and IP with date	A short-term relief, not a fix
Caching of client settings	Keeps the data from the joined tables in a cache for 60 to 300 seconds	Removes repeated lookups
Early reject	Refuses obviously invalid requests without any database access	Closes the pre-authentication path

Figure 5. The recommended plan.

Beyond the immediate fix, the development guidelines were: no blocking queries in the global request path, security middleware kept separate from business logic, dedicated systems for rate limiting, and performance validation as part of the development cycle.

6. Checking your own API

1. Run a load test at low concurrency as well as high, and look at the shape of the curve.
2. List everything your API does before it knows who is calling. Each item is work an anonymous caller can trigger.
3. Turn on the slow log of your application server and look for many workers stuck at the same line.
4. Count the database queries per request, including requests that end in an error.
5. Make sure the rate limiter costs the same whether the caller is legitimate or not.

7. Limits of this paper

This paper describes one engagement. Details that could identify the client have been removed. The figures come from the load tests and the analysis of that system. The fixes above were recommendations: we do not report measured results after their application, and the expected improvement from the analysis is a forecast, not a result.

Want to know what an attacker could make your system do before it knows who they are? Ask us about a penetration test.