An LLM gateway centralises keys, routing, caching and cost. Here is how it works and how to stop PII reaching your model provider without breaking prompt caching.
.webp)
TL;DR:
The LLM gateway category grew out of a practical mess. Keys were scattered across repositories, nobody could say what the monthly spend was per team, switching models meant a deploy, and when a provider degraded there was no failover. A proxy in front of the providers fixes all of that, and the incumbents handle that part competently.
What almost none of them fix is the content of the request. This guide covers what an LLM gateway does, how the terms relate to each other, and the data-protection layer that decides whether regulated workloads can use any of it.
An LLM gateway is a single endpoint that your applications call instead of calling model providers directly. It accepts model-shaped traffic, applies policy, forwards the request to whichever provider should serve it, and returns the response. Because every request passes through one place, it becomes the natural home for key management, rate limiting, caching, cost accounting and observability.
The important consequence is architectural: once all model traffic flows through one control point, that control point is also the only practical place to inspect what is being sent.
Five responsibilities, in rough order of how commonly they are implemented.
These three get used loosely, and the distinction matters when you are deciding what to buy.
A useful way to hold it: the LLM and AI gateway control the outbound path to the model, the MCP gateway controls the inbound path from your systems, and agentic deployments generally need both. Our write-ups on MCP data security and shadow MCP cover the inbound side.
Routing a request does not change what is in it. Logging it arguably makes things worse, because now the sensitive values exist in a second place. The moment an application sends a customer record, a query result or an uploaded document to a provider, that data is on someone else's servers and subject to whatever retention policy applies.
The naive fixes both fail in production:
There is a third option, and it is what makes regulated workloads viable.
Instead of removing values, substitute them. Each sensitive value is replaced with a token before the request leaves, the model reasons over tokens, and the real values are put back before your application acts on the response.
Connecting an application is a single environment variable pointing at the gateway. No changes to code, prompts or tools, which is what makes this deployable in an organization where the people who own the agents and the people who own the risk are different teams. The same detection engine runs behind Strac AI DLP and MCP DLP, so one policy covers the model path, the browser and your tool calls.
Tokens preserve structure and identity without carrying the value.
Because tokens are deterministic, the model can tell that two rows belong to the same customer, group by email, and refer to a specific person later in a thread. It simply never learns who that person is. This is the same reversible-token approach behind Strac's data masking, applied to traffic in flight instead of data at rest.
This is the question that quietly decides whether a redaction layer is usable at scale, and it deserves its own section.
Provider prompt caching works by matching a prefix of the conversation. If your filter emits a different placeholder on every request, the resent history differs each turn, the prefix stops matching, and you pay full price for every call. Teams that bolt on random masking often discover this weeks later as an unexplained cost increase.
Deterministic tokenization avoids it entirely. The same input always serialises to the same output, so the history you resend is byte-identical to the history you sent last turn and caching behaves exactly as before. Strac also re-checks the agent's history on every request before it leaves, so a value that slipped in earlier does not quietly persist.
Most filtering stops at the prompt string. Real applications send far more than that.
Images go through OCR, with each detected value covered by a numbered box and a token legend so the model can still work with the document. PDFs are sent as covered page images plus tokenized text, with metadata, annotations and hidden text dropped. Detection covers 255 data elements across PII, PHI, PCI and secrets, including 160 national ID types, plus any custom elements and exclusions you define.
A refund agent, start to finish, with the actual payloads at each step.
Step 1. The agent looks up the order, in your environment. Asked to refund order 88213 and email the customer, the agent queries the orders database and gets back real values:
SELECT name, email, card FROM orders WHERE id = 88213;
Jane Doe | jane.doe@example.com | 4111 1111 1111 1111
Step 2. The model receives tokens, at the provider. The gateway recognises the columns and values and replaces them before the request leaves your boundary. What arrives at the provider is:
[[NAME_ecc4dcdbb381c137]] | [[EMAIL_ADDRESS_f785576eceba2a00]] | [[CREDIT_CARD_c8d0a2f9aeab4158]]
Step 3. The model asks your agent to send an email, at the provider. The model cannot send anything itself. It replies with a tool call, a request to run the email tool your agent offered, with the customer still represented as a token:
send_email(to="[[EMAIL_ADDRESS_f785576eceba2a00]]", body="Hi [[NAME_ecc4dcdbb381c137]], your refund for order 88213 is on its way.")
Step 4. Your agent sends it with real values, in your environment. The gateway restores the tokens inside that tool call before your agent receives it. Your agent's own code runs the tool, so the email goes from your systems to the right person:
send_email(to="jane.doe@example.com", body="Hi Jane Doe, your refund for order 88213 is on its way.")
The refund completes and the right customer gets the email. The provider only ever saw three tokens, once in the request and once in its own reply.
Most regulated programmes come down to a boundary question: which systems hold the data, and who else receives it. A model provider that receives raw PHI or cardholder data becomes part of that answer, with the contracts, assessments and retention questions that follow.
Tokenizing before the request leaves changes the shape of the conversation. The provider receives tokens, the real values stay in systems you already assessed, and the audit trail shows what was detected and what was substituted on every request. In practice that matters most in three places: whether a model provider needs to sit inside your HIPAA business associate chain, whether prompts carrying cardholder data drag the provider into PCI DSS scope, and what your SOC 2 reviewer sees when they ask which subprocessors receive customer data. Strac's classifier is the same one behind our AI DLP and endpoint DLP coverage, so detection is consistent across the places your data lives.
Some environments cannot send data to a vendor under any framing, including the vendor providing the security control. Both deployments run the same policy engine and the same 255 data elements.
The self-hosted build is a Docker container in your VPC, on-prem, or your customer's infrastructure. Models are bundled in the image and run on your CPUs with no calls home, the token vault is AES-256-GCM encrypted on your storage under a secret you hold, and the gateway's only egress is to the model provider. Audit events go as JSON to your logging stack or SIEM.
| SaaS (default) | Self-hosted | |
|---|---|---|
| Where it runs | Strac-managed, on Strac's AWS | Docker container in your VPC, on-prem, or your customer's infrastructure |
| Connect an agent | One environment variable | One environment variable |
| Where real values go | Your environment and Strac | Nowhere outside your environment |
| Detection models | Run by Strac | Bundled in the image, run on your CPUs, no calls home |
| Token vault | Encrypted, held by Strac | AES-256-GCM on your storage, under a secret you hold |
| Network egress | Agent to gateway, gateway to provider | Gateway to provider only |
| Policy and audit | Strac console | Console or local policy file, JSON audit events to your SIEM |
See it on your own data. Send a few examples of what your agents handle and Strac returns a tuned policy plus a report of exactly what the model would have seen. Book a demo or browse the integrations your agents already touch.
The gateway runs the same detection and remediation engine Strac applies wherever your sensitive data lives.
Strac detecting and remediating sensitive data in Slack. The gateway applies the same detection engine to the API path.
Nobody runs two gateways. The entire point of one is to be the single control point between your applications and the models, so the real question is not which layer to bolt on. It is which gateway earns that position.
Routing, caching and cost dashboards are commodity. Every incumbent ships them, several give them away to sell something else, and none of them change what your model provider ends up holding. Cloudflare AI Gateway has added DLP, and it detects and blocks. Kong and Azure API Management govern traffic and enforce policy. AWS applies guardrails to prompts. Not one of them substitutes a sensitive value and puts it back afterwards, which means the moment a real agent touches real customer data you are left with the same two bad options: stop the work, or send the data.
Strac takes the slot because it answers that question without breaking the agent. Tokenize on the way out, restore on the way in, and the job completes with the provider holding nothing but tokens. Routing and cost features are shallow to add and sit on the roadmap. A classifier covering 255 data element types across text, images and PDFs, plus a reversible token vault, is not.
Routing and cost features converge quickly and every incumbent ships them. These are the questions that actually separate options.
If agents in your stack also reach into SaaS tools and databases, pair this with an MCP gateway on the inbound path, read MCP DLP for how the same engine applies to tool calls, and see MCP prompt injection for the attack side of that surface.
See it on your own data. Send a few examples of what your agents handle and Strac returns a tuned policy plus a report of exactly what the model would have seen. Book a demo or browse the integrations your agents already touch.
They are often used interchangeably. In practice LLM gateway usually describes a proxy focused on model traffic: routing across providers, caching, key management and cost tracking. AI gateway is the broader term that also covers the agentic surface, including tool calls and the data agents retrieve from your systems. If your workloads are agents rather than single prompts, the broader framing is the useful one.
Only if it inspects and rewrites request content, which many do not. Routing and logging change how a request is delivered, not what is inside it. To keep PII off a provider's servers the gateway has to detect sensitive values in the outbound request and replace them before it leaves. Strac does this with tokenization, and restores real values on the response path so the agent still works.
Yes, that is the intent. Point your applications at Strac and it becomes the single hop between them and your model providers, which is how a gateway is supposed to work. Nobody wants two proxies in the request path. Today Strac owns the decision that actually carries risk, which is what the provider receives, and forwards to whichever provider you point it at. Routing and cost features are on the roadmap.
Routing and failover matter less with one provider, but the data question does not change. A single-provider setup still sends every prompt, query result and uploaded file to someone else's servers. If that traffic contains regulated data, you need a control on the path regardless of how many providers you use.
Barely, and it can help. Tokens are short strings that usually consume a similar number of model tokens to the values they replace, so the request size is roughly unchanged. The cost risk is caching, not length: a redaction scheme that emits a different placeholder each request destroys your cache hit rate and raises the bill. Because Strac is deterministic, resent history stays identical and caching keeps working.
Strac should be the hop closest to your application, because anything upstream of it writes raw values into its own request logs before Strac ever sees them. If you are mid-migration you can run Strac in front of an existing proxy for a while, but the end state is one gateway in the path and it should be the one deciding what is allowed to leave your boundary. That is the whole reason a gateway exists.
“Don’t just alert on problems, fix them.”
Scott Roberts, Chief Information Security Officer, UiPath
.webp)
.avif){kind=icon}
Customers include
.gif)
