Rate Limiting at Scale

link to slides:

qrcode link to slides

Agenda

  1. Why rate limit
  2. Envoy recap
  3. Extending Envoy
  4. Sliding window
  5. Distributed

Julien Bisconti

Software Engineer / SRE

Google Cloud Platform icon

slides: bisconti.cloud

contact: g.dev/julien

qrcode link to Julien Bisconti contact information

Why Rate Limit

  • Traffic Policing vs Traffic Shaping
  • Fair sharing of resources amongst users.
  • Prevent abuse (scraping, breach of ToS, DDoS).
  • Control cost.

Envoy

life of a request

Envoy Filters

  1. Extending Envoy
  2. example filter
  3. Lua filter
  4. WASM
  5. Go contrib
  6. Dynamic modules

Rate Limiting Algorithms

What if you just want to block the outliers ?

(99.9% of users are well behaved)

Reduce the bandwidth overhead of rate limiting.

Rate Limiting Algorithms

  • Token Bucket
  • Leaky Bucket
  • Fixed Window
  • Sliding Window

Sliding Window

# 8 seconds window
4 2 6 8 3 0 0 9 1 1 1 2 0 4 5 0 # requests per seconds
     |---------------|          = 24 (t0)
       |---------------|        = 15 (t1)
         |---------------|      = 13 (t2)

Data example

{
  "counters": [{
    "header": { "user": "b64encodedsecret", "service": "activity" },
    "count": 3
  },{
    "header": { "user": "anotheruser", "service": "activity" },
    "count": 8
  },{
    "header": { "user": "scraper"},
    "count": 42
  }],
  "window_start": "2009-11-10T23:00:00Z",
  "window_end":   "2009-11-10T23:01:00Z"
}

Distributed System

Hashing function to determine the aggregator

# { "user": "b64encodedsecret", "service": "activity" },
# hash = "isHfNLCKDW4832bMJkosRA=="
# agg = maglev(hash) # consistent hashing

client -> proxy -> backend service
           |
           ├──> aggregator01
           ├──> aggregator02
           ├──> ...
           └──> aggregatorN
          

Maglev: A Fast and Reliable Software Network Load Balancer

What to do

when things go wrong

Outage

Talk at KubeCon 2025

Adventures in Rate Limiting: Spotify’s Journey Writing a Scalable Envoy Rate Limiter

Oliver Soell & Peter Marsh

qrcode link to talk

Resources

THANK YOU

and I'm sorry 🙏
If you had to maintain my code
I hope you learned more by maintaining it
than me by writing it

Slides made with Reveal.js and hugo-reveal

Julien Bisconti

Software Engineer / SRE

Google Cloud Platform icon

slides: bisconti.cloud

contact: g.dev/julien

qrcode link to Julien Bisconti contact information