Heavy, Medium, Light: Routing LLM Workloads by Task

LLM routing is a quality decision as well as a cost decision: the RouteLLM research learns to choose between stronger and weaker models before generating a response. We recommend organising workloads into heavy, medium and light routes, then testing each route against the work it will actually receive. Treat those names as operating choices, not fixed model sizes or a promise that three routes will outperform one.

The important questions are what an acceptable answer looks like, when it must arrive and what happens when it is wrong. We recommend answering those questions before choosing a model or buying local hardware.

Define the work before the tier

Start with a workload record that names its inputs, expected output, permitted data destinations and acceptance test. Add the deadline and the person responsible for rejected results. Keep capability requirements separate from scheduling: do not treat background work as permission to reduce the quality threshold.

For a proposed extraction job, require each returned field to point back to its source passage. For a proposed code review, specify which defects it should identify and which checks must still run outside the model. Treat both as examples of acceptance criteria, not evidence that either task belongs on a particular model.

RouteLLM's experiments evaluate model selection on academic benchmarks, including MMLU, MT Bench and GSM8K. Its authors also acknowledge that real applications may have substantially different query distributions. Use those results as a reason to evaluate routing, not as a forecast of your own savings.

Heavy means a higher capability requirement

We recommend reserving the heavy route for tasks whose acceptance tests require the strongest model you have validated. Examples to assess include ambiguous requirements, reasoning across conflicting sources and changes with expensive consequences. Set a human approval requirement separately where the proposed action warrants one.

Do not assign every interactive request to heavy merely because somebody is waiting. For a known account-status question, first test an authorised lookup with a fixed response format. For an unfamiliar architecture question, require the relevant system evidence before accepting a generated answer.

Keep a rejection route for work that no available model can complete to the required standard. Our recommendation is to return the missing evidence or request human judgement rather than manufacture a confident answer. A higher-tier retry should still pass the original acceptance test.

Medium means a validated alternative

We suggest using medium for workloads where another model meets the acceptance criteria at an acceptable total cost. Consider a self-hosted model as one candidate, alongside a hosted alternative. Do not define medium by a parameter count, ownership of hardware or the fact that a job runs overnight.

For a summarisation trial, compare candidate outputs with the source records and mark omissions as well as invented details. For extraction, measure missing items separately from malformed output. Keep the original records available to the reviewer rather than treating the generated summary as the only evidence.

Include hardware, energy, operation and review effort in the local option's cost comparison. For a hosted option, include inference charges and the cost of retries. Use an agreed accounting period and the same accepted-work definition for both options. We would not describe self-hosted inference as free merely because it has no external charge per token.

Light includes work with no model

Our recommendation for light is to begin with explicitly defined transformations and lookups. Use ordinary code for a published mapping of tags to canonical names, or for a price lookup from an authorised table. Keep ambiguous inputs out of that path unless the policy defines how to handle them.

Where the task requires interpretation, test a small model rather than calling the task deterministic. Set an allowed output vocabulary, check the result and retain an explicit unknown outcome. Do not make the weakest route the default for inputs that the router cannot classify.

In a hypothetical support intake, a fixed rule could recognise a valid reference number and retrieve its status. A model could propose a category for an unstructured message, subject to a tested acceptance rule. We recommend sending an uncertain or sensitive case for stronger assessment rather than allowing a guessed category to trigger an irreversible action.

Measure the complete route

Evaluate the proposed policy against a single-model baseline using the same representative cases. Keep a separate set of cases out of policy tuning. Report accepted results, rejected results and escalation frequency alongside cost, rather than reporting a lower bill alone.

The RouteLLM paper distinguishes routing from judging a completed response: routing selects the model beforehand. Keep a separate output check in your proposed design. Limit retries and define what happens after that limit, including the owner of work left unresolved.

For timing, measure from arrival to accepted completion, including waiting and review. Bass, Clements and Kazman's Software Architecture in Practice, fourth edition, describes throughput as dependent on workload, transaction dependencies and parallel processing in its discussion of performance models. Test the routes at expected demand, not only with an isolated request.

Start a trial with one bounded workload and retain the existing route for comparison. Widen the policy only after reviewing the failed cases and confirming that the accepted-work target still holds. Record the policy version, selected model and reason for escalation so the next evaluation can reproduce the decision.

Update, October 2026

Our later engineering article, The Cheapest Model That Can Safely Do the Job, describes a deterministic routing table with risk overrides and escalation. Read it alongside the separate local AI hardware article when deciding whether deployment capacity and task requirements justify a local route. Keep the workload acceptance test as the decision criterion, rather than copying a model name or a hardware configuration.

Sources

• Ong and colleagues, "RouteLLM: Learning to Route LLMs with Preference Data", version 4, 23 February 2025.
• Bass, Clements and Kazman, "Software Architecture in Practice", fourth edition, chapter 14, 30 June 2021.
• Dendro Logic, "The Cheapest Model That Can Safely Do the Job", 30 July 2026. Referenced only in the October update.
• Dendro Logic, "What You Can Actually Run on £4,000 of Local AI Hardware", 1 April 2026. Further reading in the October update.