Model Routing for EU AI Act Readiness: Choosing the Right Model Under Compliance Constraints

Why Model Selection Is a Governance Decision

In most enterprise AI deployments, multiple models are available: large language models, small language models, specialist fine-tuned models, embedding models, and potentially external cloud models accessed through APIs. The decision about which model handles a given request is typically made by an orchestration layer based on performance criteria such as latency, cost, and accuracy.

Under the EU AI Act, this routing decision acquires governance significance. Different models have different risk profiles depending on where they run, what data they process, how their outputs are used, and whether their behavior can be audited and explained. A model hosted on an external cloud API may be unsuitable for processing personal data subject to GDPR residency requirements. A general-purpose model may lack the domain-specific accuracy required for high-risk use cases. A model without adequate logging capabilities may be unable to produce the traceability evidence that regulators expect.

Model routing, when designed with compliance in mind, becomes a governance control rather than purely an optimization mechanism. It ensures that each request is handled by a model that meets the data sensitivity, auditability, explainability, and deployment boundary requirements that apply to that specific use case.

Data Classification as the Primary Routing Dimension

The most fundamental routing constraint for regulated enterprises is data classification. Not all data should be processed by all models, and not all models operate within the same trust boundary.

A practical routing policy starts by classifying data into tiers. Public data can be processed by any approved model, including external APIs, subject to cost and performance constraints. Internal data should be routed to models running within the organization's infrastructure, whether on-premises or within a controlled private cloud environment. Confidential data, including personal data, financial records, health information, or trade secrets, must be processed exclusively by models running on-premises within the organization's security perimeter, with full logging and access control.

This classification-based routing is not merely a security preference. Under GDPR, data controllers must ensure that personal data is processed in a manner consistent with the legal basis for processing and that data transfers outside the EEA comply with Chapter V requirements. Under the EU AI Act, high-risk AI systems must maintain records that enable traceability, which is difficult to guarantee when data is sent to external services with opaque logging practices.

The routing layer must inspect the data classification label, which should be applied at the point where the request enters the AI platform, and direct the request to a model that operates within the permitted trust boundary for that classification level.

Risk-Level Routing and Use Case Constraints

Beyond data classification, the EU AI Act's risk framework introduces routing considerations based on the AI system's intended use. A model that generates marketing copy operates under different regulatory expectations than one that assists with medical diagnosis or credit assessment.

For high-risk use cases, routing policies should prefer models that offer stronger explainability, more deterministic behavior, and comprehensive audit capabilities. In many cases, this means routing high-risk workloads to smaller, domain-specific models that have been fine-tuned and evaluated for the specific task, rather than relying on general-purpose large models whose behavior is harder to characterize and explain.

Consider a financial services firm that uses AI for both customer-facing chatbot interactions and internal credit risk assessment. The chatbot, classified as limited risk, can be served by a general-purpose language model with standard logging. The credit risk assessment system, classified as high-risk, should route to a specialized model that has been validated against the firm's evaluation benchmarks, deployed with approval gate workflows, and configured with comprehensive decision logging that captures the model version, input features, retrieval context, output, and confidence score for every assessment.

Routing policies can encode these distinctions as rules that the orchestration layer evaluates for every request: if the use case tag is "credit-assessment" and the risk level is "high," route to the approved credit risk model with full audit logging enabled; if the use case is "internal-helpdesk" and the risk level is "minimal," route to the general-purpose model with standard logging.

Deployment Boundary Policies and Cloud Fallback

Many enterprises operate in hybrid environments where some models run on-premises and others are available through cloud APIs. The routing layer must enforce deployment boundary policies that prevent data from crossing trust boundaries inappropriately.

A well-designed routing architecture defines clear boundaries. The on-premises boundary includes models running on the organization's own hardware, within its own network, managed by its own teams. The private cloud boundary includes models running in a dedicated cloud tenancy with contractual data processing agreements. The public cloud boundary includes shared API services where the organization has limited visibility into data handling, logging, and model behavior.

Cloud fallback policies determine whether and under what conditions a request can be escalated from on-premises to cloud. For many regulated enterprises, the policy is that requests involving confidential or personal data never fall back to cloud models. Requests involving internal data may fall back to private cloud models if the on-premises model is unavailable, but not to public cloud APIs. Only requests involving public data or non-sensitive internal queries may use cloud APIs as a fallback.

These policies should be implemented in the routing layer as enforceable rules, not as guidelines that individual applications can override. The routing decision and the rationale should be logged for each request, creating an auditable record of boundary enforcement that can be reviewed by compliance teams and presented as evidence of data governance.

Making Routing Decisions Auditable and Explainable

For routing to function as a governance control, every routing decision must be logged and explainable. This means recording, for each request, the data classification applied, the risk level and use case tag, the set of eligible models based on policy evaluation, the model that was selected, the reason for selection, and whether any fallback was triggered.

This routing log serves multiple compliance purposes. It demonstrates that the organization is systematically directing sensitive workloads to appropriate models. It provides evidence that deployment boundary policies are being enforced. It enables retrospective analysis when a model produces an unexpected or harmful output, allowing investigators to understand why that particular model was selected for that particular request.

Compliance teams should be able to query routing logs with questions such as: which models processed personal data in the last quarter, how many high-risk requests were routed to general-purpose models, and were there any policy violations where data crossed an unauthorized boundary. These queries form part of the continuous monitoring that the EU AI Act expects for high-risk AI systems.

Sysart Consulting helps enterprises design model routing architectures that integrate compliance constraints directly into the orchestration layer. This includes defining data classification schemas, mapping risk levels to routing rules, configuring deployment boundary policies, and building the logging and monitoring infrastructure that makes routing decisions auditable. For organizations using VDF AI as their on-premises AI platform, Sysart provides guidance on configuring model routing policies that align with the organization's regulatory obligations and governance framework, ensuring that model selection decisions can be explained, defended, and continuously improved.

Featured image by Shubham Dhage on Unsplash.