What Is an AI Gateway? How It Works and How to Keep Data Off Model Servers

TL;DR:

• An AI gateway is a control point between your applications or agents and the model providers they call. It handles routing, caching, observability, access control and, in the best case, data protection.
• Most gateways cover the first four. The data layer is the one that decides what your model provider gets to keep.
• Blocking a request stops the work, and deleting values leaves the model nothing to refer to. The workable answer is to tokenize in transit and restore on the way back.
• Strac's AI Gateway detects 255 data element types across PII, PHI, PCI and secrets, swaps them for consistent tokens before the request leaves, and restores real values in the model's answer and tool calls.
• It connects with one environment variable, needs no changes to agent code, and runs either as SaaS or self-hosted in your own environment.

Teams adopted AI gateways for the same reason they adopted API gateways: once more than one service starts calling more than one model, you want a single place to manage keys, control spend, see what is happening and fail over when a provider has a bad afternoon. That part of the category is well served.

The part that is not well served is data. An agent that reads an email thread, queries an orders table and opens a scanned invoice is handling names, card numbers, addresses and health details, and all of it travels to a model provider as ordinary API traffic that file-level controls never see. This guide covers what an AI gateway does, how it differs from an API gateway and an LLM gateway, and how the data layer actually works when you cannot afford to break the agent.

✨ What is an AI gateway?

An AI gateway is a proxy that sits between the software that wants to use a model and the provider that serves it. Instead of each application holding its own provider key and calling the provider directly, applications point at the gateway, and the gateway decides what happens next: which model serves the request, whether a cached answer can be reused, what gets logged, who is allowed to call what, and what the outbound request is allowed to contain.

That last clause is the one that separates gateways from each other. A gateway that only routes is a convenience layer. A gateway that inspects and changes the content of a request is a control.

✨ What an AI gateway actually does

Five functions show up across the category. The first four are table stakes and most vendors do them well. The fifth is where the real differences are.

• Routing. Pick a model per request, fail over when a provider errors, retry with backoff, and move traffic between providers without redeploying.
• Caching. Reuse answers for repeated prompts and cut token spend. This is also where naive redaction quietly destroys value, which we come back to below.
• Observability. Logs, traces, latency and cost broken out per team, per app and per model.
• Access control. Central key management, rate limits, and rules about which applications may call which models.
• Data protection. Inspecting what is actually inside the request and deciding what the provider is allowed to receive.

If you are evaluating gateways mainly as governance for AI agents, our guide to AI agent security and the broader picture in agentic AI security cover the controls that sit around the gateway itself.

AI gateway vs API gateway vs LLM gateway

The three terms overlap enough to cause real confusion in procurement, so it is worth being precise.

• An API gateway fronts your own services. It understands HTTP: routes, methods, headers, auth, rate limits. It has no idea what a prompt is.
• An LLM gateway fronts model providers and understands model-shaped traffic: prompts, completions, streaming, token accounting. In practice most people use this term when the main job is routing and cost control across providers. We cover it in depth in what is an LLM gateway.
• An AI gateway is the broader term. It covers LLM traffic plus the agentic surface around it: tool calls, retrieved context and the data agents pull in from your systems.

There is a fourth that is frequently confused with these: an MCP gateway, which governs the Model Context Protocol path agents use to reach your tools and data sources. An AI gateway controls what goes out to the model. An MCP gateway controls what comes in from your systems. Most serious deployments end up wanting both.

✨ The data problem every AI gateway inherits

Here is the situation that sends teams looking for a gateway in the first place. To process one refund, an agent reads the customer record, the email thread and the invoice. That is a name, an email address, a card number, a postal address, sometimes a date of birth or a clinical note. Every one of those values goes into the next model request.

Three responses are common, and two of them fail:

• Block the request. Safe, and the agent stops working. The refund does not get issued.
• Delete the values. The request goes through but the model has nothing to refer to, so it cannot write the email or build the next query. You get a confused agent and a support ticket.
• Substitute the values. Replace each sensitive value with a stable token, let the model reason over tokens, then put the real values back before your agent acts. The work completes and the provider never receives the original data.

The third option is the only one that preserves the workflow. It is the approach Strac's gateway takes, and it is closely related to how data masking works for databases, applied to traffic in flight instead of data at rest.

✨ How the Strac AI Gateway works

The gateway is one hop between your agent and the model, and it runs inside your environment or ours depending on how you deploy it.

1. The agent sends its request, carrying whatever data it read from your databases, mailboxes and files.
2. The gateway detects sensitive values, swaps them for tokens, and forwards the request.
3. The model answers and asks for tool calls, using the tokens it was given.
4. The gateway restores the real values before the agent sees them, so tools run against real data in your environment.

Connecting an agent is one environment variable pointing at your gateway URL. There are no changes to agent code, prompts or tools, which matters because the agent teams and the security teams are rarely the same people. The same detection engine runs behind Strac AI DLP and MCP DLP, so a policy you write once applies across the agent, the browser and the MCP path.

✨ What the model actually receives

Tokens are not random. The same value always maps to the same token, which is what lets the model keep doing useful work: it can still tell that two records share an email address, group by customer, and refer back to a specific person across a long conversation.

Consistency has a second benefit that is easy to miss and expensive to get wrong. Because the same value always produces the same token, resent conversation history is byte-identical between turns, so prompt caching keeps working. Redaction schemes that generate a fresh mask per request silently break the cache and raise your bill.

✨ A worked example: an order agent issues a refund

Abstract descriptions of tokenization tend to raise the same objection, which is that surely the agent breaks somewhere. Here is the full path for a real task.

The agent queries the orders table and gets Jane Doe's real name, email and card. The model receives three tokens. The model replies with a tool call asking to send an email to a token. The gateway restores the real address before your agent receives that instruction, and your agent's own code sends the message from your systems. The job is done correctly and nothing sensitive left your environment.

✨ What the gateway covers

Sensitive values do not only live in prompts. They arrive in query results, command output, logs, uploaded files and scanned documents, which is where most inspection tools stop.

• Prompts and queries: prompts, query results, command output and logs.
• Structured files: JSON, CSV and documents.
• Images: OCR finds each value and covers it with a numbered box, plus a token legend so the model can still use the document.
• PDFs: each page goes as a covered image plus its tokenized text. Metadata, annotations and hidden text are dropped.

Detection spans 255 data elements across PII, PHI, PCI and secrets, including 160 national ID types, and you can add custom elements and exclusions. The same classifier powers Strac's AI DLP and MCP DLP products.

✨ Two ways to run it

Which deployment you pick usually comes down to whether your data is allowed to touch a vendor at all.

SaaS is Strac-managed on Strac's AWS. You point an agent at your tenant's gateway URL and the provider gets tokens only.

Self-hosted is a Docker container in your VPC, on-prem, or inside your customer's infrastructure. Detection models are bundled in the image and run on your CPUs with no calls home. The token vault is AES-256-GCM encrypted on your storage under a secret you hold. The only egress is gateway to provider, so your agents can sit on a network with no other route out, and audit events go as JSON to your own logging or SIEM.

See it on your own data. Send a few examples of what your agents handle and Strac returns a tuned policy plus a report of exactly what the model would have seen. Book a demo or browse the integrations your agents already touch.

When you need an AI gateway

Two situations drive most of the urgency we see.

Agents acting for your customers. Your agents run in Docker, on your infrastructure or your customer's, and handle their customers' data. Regulated data has to stay inside that boundary, and you still need the agent to do real work. This is the deal-blocker case: a security review asks what your model provider receives, and "everything the agent read" is not an answer that closes.

Production data in dev and test. Customers share production databases so your team can build and debug. Coding agents like Claude Code query that data, and every result goes to the model. For this case the gateway composes with redaction at the source: Postgres masking so the agent only ever reads masked rows, and Endpoint DLP for the files and downloads on the machine itself. Use either layer or both.

🎥 See Strac detect and redact sensitive data

The gateway runs the same detection and remediation engine Strac applies everywhere else your sensitive data lives. This walkthrough shows Strac finding and acting on sensitive data in Slack: the same classifier that decides what your model provider is allowed to receive.

Strac detecting and remediating sensitive data in Slack. The AI Gateway applies the same detection engine to the API path between your agents and the model.

Which gateway should own the slot?

Nobody runs two gateways. The entire point of one is to be the single control point between your applications and the models, so the real question is not which layer to bolt on. It is which gateway earns that position.

Routing, caching and cost dashboards are commodity. Every incumbent ships them, several give them away to sell something else, and none of them change what your model provider ends up holding. Cloudflare AI Gateway has added DLP, and it detects and blocks. Kong and Azure API Management govern traffic and enforce policy. AWS applies guardrails to prompts. Not one of them substitutes a sensitive value and puts it back afterwards, which means the moment a real agent touches real customer data you are left with the same two bad options: stop the work, or send the data.

Strac takes the slot because it answers that question without breaking the agent. Tokenize on the way out, restore on the way in, and the job completes with the provider holding nothing but tokens. Routing and cost features are shallow to add and sit on the roadmap. A classifier covering 255 data element types across text, images and PDFs, plus a reversible token vault, is not.

✨ How to choose an AI gateway

If you are shopping for an AI security gateway rather than a router, these are the questions that actually separate the options.

• Does it inspect content, or only route it? Many gateways can tell you a request went to a provider but not what was in it.
• Does it block, delete, or substitute? Blocking breaks the agent. Deleting breaks the answer. Substitution is the only one that keeps the workflow intact.
• Are tokens consistent? If not, prompt caching breaks and your spend climbs.
• Are tokens restored inside tool calls? Restoring only the final answer is not enough for agents that take actions.
• Does it cover files, images and PDFs? Text-only inspection misses the scanned invoice.
• Can it run where your data must stay? Ask specifically whether detection runs locally and whether anything calls home.
• What does the audit trail look like? You will need to show a reviewer what the provider received.

For the adjacent control on the inbound side, compare this list against our MCP gateway guide, read MCP DLP for how the same engine applies to tool calls, and see shadow AI for discovering the tools and agents already running without review.

See it on your own data. Send a few examples of what your agents handle and Strac returns a tuned policy plus a report of exactly what the model would have seen. Book a demo or browse the integrations your agents already touch.

🌶️ Spicy FAQs on AI gateways

Does Strac do routing, caching and failover?

Not yet, and the sequencing is deliberate. Routing, failover and cost dashboards are commodity: every incumbent ships them, several give them away, and none of them change what your model provider ends up holding. The difficult part is a classifier that recognizes 255 data element types across text, images and PDFs, plus a reversible token vault that lets an agent keep working after the substitution. We built that first because it is the part that decides whether AI ships in a regulated environment. Routing and cost features are on the roadmap.

Does an AI gateway slow my agents down?

It adds one hop. The Strac AI Gateway runs detection locally, in the same environment as the gateway, so the cost is detection time rather than a round trip to another service. In the self-hosted deployment the models are bundled in the image and run on your own CPUs, with no calls home. The thing that actually saves time is not having to rebuild your agent when security asks what the model provider receives.

Will tokenizing data break prompt caching?

No, and this is the question that kills most redaction approaches. A random mask on every request changes the prompt every time, so the cache never hits and your costs go up. Strac maps the same value to the same token every time, so resent conversation history stays byte-identical and caching behaves exactly as it did before.

What happens if the model needs the real value to do its job?

It does not need it. The model works with tokens and asks your agent to run a tool, for example sending an email. The gateway restores the real values inside that tool call before your agent sees it, so the agent runs the tool against real data in your environment. The work completes; the provider only ever saw tokens.

Can I run an AI gateway without sending anything to a vendor?

Yes. Strac ships a self-hosted Docker container that runs in your VPC, on-prem, or inside your customer's infrastructure. Detection models are bundled in the image, the token vault is AES-256-GCM encrypted on your storage under a secret you hold, and the only network egress is from the gateway to the model provider. Your agents can sit on a network with no other route out.

What happens when detection misses a value?

No classifier is perfect, so treat this as a layered question rather than a guarantee. Strac covers 255 data element types including 160 national ID formats, and you can add custom elements and exclusions for the formats specific to your business, which is where most misses actually come from. The audit trail records what was detected and substituted on every request, so you can review real traffic and tighten policy. For the highest-sensitivity cases, pair the gateway with masking at the source so the agent never reads the raw value in the first place.