Using LLM Structured Outputs For Routing

Applying a classic enterprise integration pattern to AI engineering

Choose your path

The Router is a classic software design pattern that’s useful when building AI applications. It’s a way to triage user requests and send them to the right function to take action.

In the world of traditional software architecture, it’s old news, but LLMs give it a powerful upgrade. This is because LLMs can be asked to route based on semantics, i.e. inferred intent from the user message instead of hard-coded keywords.

If you want to see it in action, here’s a gist.

Read on for an explanation.

Why it’s useful

All the major AI companies and frameworks have a take on routing but these usually focus on AI Agents or AI tool chaining. This is fine when all functions live in the AI space but sometimes you want routing outside of the AI chain.

For example, let’s say you’ve built a chatbot and it has standard tools like web search or document lookup. These are clearly in the AI space because results are typically processed by an LLM before presenting back to a user.

ChatGPT with Web Search

But let’s say the user says “save this as a CSV” or “print!”. These are easily handled through regular tooling and don’t need an AI — in fact, coding these as AI functions would be overkill.

AI Function Overkill

Flow

Implementing a router is simple and scales well. The flow is as follows:

LLM Router Flow Diagram

What’s Going On

Start with some simple classes:

class Route(str, Enum):
    structured_dog = "structured_dog"
    search = "search"
    download = "download"

class TriageDecision(BaseModel):
    route: Route

Then define a route handler and register for the various routes

RouteHandler = Callable[[str], Any]
ROUTE_TABLE: Dict[Route, RouteHandler] = {}

def register(route: Route) -> Callable[[RouteHandler], RouteHandler]:
    """Decorator that registers a handler for a route."""
    def _wrap(fn: RouteHandler) -> RouteHandler:
        ROUTE_TABLE[route] = fn
        return fn
    return _wrap

Next, we define our routes as Python functions with a decorator for @register. In this case, we have two AI functions (i.e. that call LLMs) and one native.

# AI function 1
@register(Route.structured_dog)
def _handle_structured_dog(user_query: str):
    """
    Ask the model for breed details and validate with Pydantic.
    """
    # schema = Dog.model_json_schema()
    resp = client.responses.parse(
        model="gpt-4o-mini",
        input=f"Describe the dog breed in this query: '{user_query}'.",
        text_format=Dog,
    )
    raw_json = resp.output[0].content[0].text
    return Dog.model_validate_json(raw_json)

# AI function 2
@register(Route.search)
def _handle_search(user_query: str) -> str:
    """
    Perform a real-time web search and return the model’s answer.
    """
    resp = client.responses.create(
        model="gpt-4o-mini",
        input=user_query,
        tools=[{"type": "web_search"}],  # let the model invoke it

    )
    return resp.output[1].content[0].text  # text answer follows the web_search_call

# Native function (no AI)
@register(Route.download)
def _handle_download(_: str):
    """
    Save some data
    """
    data = {'col1': [1, 2, 3], 'col2': [4, 5, 6]}
    df = pd.DataFrame(data)
    df.to_csv('output.csv', index=False)
    print("downloading your file")
    files.download('output.csv')
    return ("Check your download folder for output.csv")

Then we add our triage function that calls a small LLM and returns a route:

def triage_query(user_query: str) -> Route:
    """
    Ask the model what the user wants.
    """
    system = textwrap.dedent(
        """
        You are a router.
        Your only job is to determine which route best matches the user's query.
        Note that certain queries may imply multiple routes. In such cases,
        choose the primary route that best fits the main intent of the query.
        Respond ONLY with valid JSON.
        """
    )
    resp = client.responses.parse(
        model="gpt-4.1-nano",
        input=[{"role": "developer", "content": system},
               {"role": "user", "content": user_query}],
        text_format=TriageDecision,
    )
    print(TriageDecision.model_validate_json(resp.output[0].content[0].text).route)
    return TriageDecision.model_validate_json(resp.output[0].content[0].text).route

And wrap this all up in a simple helper function

def smart_answer(user_query: str):
    route   = triage_query(user_query)
    handler = ROUTE_TABLE.get(route, ROUTE_TABLE[Route.search])  # default
    return handler(user_query)

That’s it. Now you have a framework for effective function routing inside and outside of the AI space.

if __name__ == "__main__":
    print("Structured run →", smart_answer("Tell me about the Bernese Mountain Dog"))
    print("Search run      →", smart_answer("What are the top VC deals this week?"))
    print("Save file ", smart_answer("Save it!"))

Note: It’s a good idea to run your triage prompt through evals to increase response consistency.

Why This Pattern Scales

• Keeps routing logic clean and decoupled from implementation

• Lets you drop in a cheaper model for triage

• Doesn’t require tools or agents if you don’t need them

• Plays nice with both LLM and non-LLM logic

It’s inspired by the old-school Message Router pattern, but with a semantic twist: instead of routing based on headers or keywords, we let the model infer intent and drive execution.

Old pattern. New engine.