Inference gateway

medium 60 min Python Meta AI-enabled coding-styleAtlassian-style

1 Four times the same bad request
2 Build a feature
3 Make it fast

repo

infer/
tests/
visible/

read-only

# infer

A small gateway in front of a language model. Tenants (our customers) submit
prompts; the gateway queues them, packs them into batches, sends each batch to
the model server and keeps the results.

    infer/models.py      Request, Completion, Result
    infer/tokens.py      count_tokens(): a rough count, one token per word
    infer/errors.py      ModelError: what the model server raises, with its HTTP status
    infer/clock.py       SystemClock, and FakeClock for tests (sleep() just moves time on)
    infer/fake_model.py  FakeModel: the stand-in model server we develop and test against
    infer/scheduler.py   FairQueue: waiting requests, shared fairly between tenants
    infer/batcher.py     Batcher: sends one batch, retrying when that can help
    infer/gateway.py     Gateway: submit(), flush(), result()

How a request goes through:

1. `submit(tenant, prompt, max_tokens)` checks the request (not empty, fits in
   the model's context) and queues it. Bad requests are `"rejected"` at once.
2. `flush()` takes batches off the queue until it's empty. A batch holds at
   most `max_batch` requests whose costs (prompt tokens + `max_tokens`) add up
   to at most `max_batch_tokens`.
3. Batches are filled **fairly**: tenants take turns, one request each, oldest
   first (see `FairQueue`). One tenant with a huge backlog can't starve others.
4. The `Batcher` sends the batch. Errors the model server says are temporary
   (429, 5xx) are retried with backoff (0.5 s, 1 s, 2 s...); any other error
   fails the batch straight away. Every request in a batch that fails gets a
   `"failed"` result.

Run the tests with `python -m pytest` (or press Run in the browser).

Phase 1 of 3 0:00

Four times the same bad request

Incident #2214 · severity: 2 · from: ML platform on-call

Monday's model-server deploy started rejecting a parameter we send, so every batch came back 400 Bad Request. That part was their bug. But our gateway made it worse: the model server's logs show every batch arriving four times, and each request took 3.5 seconds to fail instead of failing at once. The README says we only retry errors that can clear up.

The retry loop is Batcher.send() in infer/batcher.py.

tests/visible/test_retries.py reproduces it. Make it pass.
On Submit, hidden tests try every kind of error: client errors fail on the first try, 429 and 5xx are still retried with the same backoff, and retrying stops at max_attempts.

Bring your own AI: paste the copied repo into Claude, ChatGPT or Gemini. In a real interview the assistant usually reads the files itself. Or download the zip, work in your own editor, and upload the files you changed.