Rate Limiting at Scale
link to slides:
Agenda
- Why rate limit
- Envoy recap
- Extending Envoy
- Sliding window
- Distributed
Julien Bisconti
Software Engineer / SRE
slides: bisconti.cloud
contact: g.dev/julien
Why Rate Limit
- Traffic Policing vs Traffic Shaping
- Fair sharing of resources amongst users.
- Prevent abuse (scraping, breach of ToS, DDoS).
- Control cost.
Envoy
Envoy Filters
Rate Limiting Algorithm
- Token Bucket
- Leaky Bucket
- Fixed Window
- Sliding Window
What if you just want to block the outliers ?
(99.9% of users are well behaved)
Reduce the bandwidth overhead of rate limiting.
Data example
{
"counters": [{
"header": { "user": "b64encodedsecret", "service": "activity" },
"count": 3
},{
"header": { "user": "anotheruser", "service": "activity" },
"count": 8
},{
"header": { "user": "scraper"},
"count": 42
}],
"window_start": "2009-11-10T23:00:00Z",
"window_end": "2009-11-10T23:01:00Z"
}
Sliding Window
# 8 seconds window
4 2 6 8 3 0 0 9 1 1 0 2 0 4 5 0 # requests per seconds
|---------------| = 23 (t0)
|---------------| = 14 (t1)
|---------------| = 12 (t2)
Distributed System
Hashing function to determine the aggregator
# { "user": "b64encodedsecret", "service": "activity" },
# hash = "isHfNLCKDW4832bMJkosRA=="
# agg = maglev(hash) # consistent hashing
client -> proxy -> backend service
|
├──> aggregator01
├──> aggregator02
├──> ...
└──> aggregatorN
What to do
when things go wrong
Talk at KubeCon 2025
Adventures in Rate Limiting: Spotify’s Journey Writing a Scalable Envoy Rate Limiter
Oliver Soell & Peter Marsh
Resources
THANK YOU
and I'm sorry 🙏
If you had to maintain my code
I hope you learned more by maintaining it
than me by writing it
Slides made with Reveal.js and hugo-reveal
Julien Bisconti
Software Engineer / SRE
slides: bisconti.cloud
contact: g.dev/julien