01LLM Infra · Cost
A cost-aware gateway that routes LLM traffic by how hard the question is
−68%
inference cost
$0.48 → $0.15
per batch
93.4%
routing accuracy
0.06 ms
to decide
Most LLM spend is wasted sending easy prompts to an expensive model. This gateway puts a cheap complexity classifier in front of the models (no inference, ~0.06 ms per decision) and keeps 79% of traffic on a self-hosted Llama 3.2 via vLLM, escalating only the genuinely hard 21% to GPT-4o.
The tradeoff worth naming: a heuristic classifier is fast and free but occasionally under-routes, so the escalation path and a per-query cost dashboard exist to catch that. On an 81-query benchmark it held 93.4% accuracy while cutting cloud inference spend by roughly two-thirds, the difference between a demo you can afford and one you can't.
Stack · FastAPI · vLLM · Llama 3.2 · GPT-4o · Streamlit · Docker
Source ↗