Ollama makes it easy to spin up open-source models with an API ready to consume. In the cloud, that same convenience makes risk skyrocket when someone publishes a “temporary” endpoint on an instance with a public IP, without authentication and without network segmentation. I’ve seen it more than once: the team only wanted an internal demo, but the endpoint ends up indexed, used by third parties, or consuming GPU out of control.
This post focuses on how to secure Ollama deployments in the cloud when you’re going to operate it as a real service: controlling who can invoke it, from where, with what limits, and how to prove it in an audit. The goal is not to “harden by theory”, but to eliminate the failures that cause shadow AI (LLM APIs spun up off the books) and turn the platform into something governed.
Real attack surface when Ollama is exposed as an API in the cloud
Ollama is usually run as an HTTP service (by default on port 11434) and many integrations assume direct connectivity. In the cloud, the typical mistake is assigning a public IP to the VM or the cluster node and opening the port “to test”. That pattern turns the model into an internet-facing service with an attack surface similar to any public API, but with a much greater economic and data impact.
The most common abuses are not “sophisticated hacking”; they are opportunistic use: scrapers that detect open ports, bots that force prompts to extract information, and automated consumption for massive inference (crypto-mining no, but “GPU-mining” tokens). If the model has access to internal data via tools (RAG, connectors, environment variables, logs), the risk stops being just cost: it becomes leakage of sensitive information via prompt injection or indirect exposure.
- Shadow AI via temporary endpoints: a team spins up Ollama “for a week”, nobody registers it, and it stays accessible for months. That ghost service usually lacks credential rotation, clear owners, and traceability.
- Exfiltration via prompts and responses: if real data is tested (tickets, emails, internal KB), responses can contain confidential information. Without controls, it’s impossible to attribute who asked for what.
- Denial of service via consumption: a single leaked key or an open endpoint can saturate CPU/GPU and degrade other workloads. In companies, this ends in availability incidents, not a “simple” cost overrun.
In practice, each bullet materializes into urgent tickets: “the demo is slow”, “the GPU bill spiked”, or “security asks to justify who accessed”. The root cause is usually the same: an inference API was treated like a lab service, but it was deployed in production (or in a production VPC) without guardrails.
Minimum secure architecture: segmentation, gateway, and private service
The architecture that best holds up to audits and day-to-day operations is simple: Ollama must not be directly accessible from the Internet. It runs in a private network (private subnet or the cluster’s internal network), without a public IP, and is published via an API Gateway or managed reverse proxy with strong authentication and rate-limiting policies. The gateway is the controlled entry point; Ollama is a private backend.
Segmentation is the difference between “we have auth” and “we have control”. Even if you put a token in a header, if the backend is reachable via alternative routes (peering, open rules, broad SGs, NodePorts), you end up with bypasses. In corporate environments, it also reduces damage when there is lateral movement: the attacker should not be able to jump from a compromised VM to your inference server without going through explicit controls.
- Private backend (Ollama): run in a private subnet, with no inbound from 0.0.0.0/0. Connectivity must come only from the gateway or from a managed bastion for operations.
- API Gateway / reverse proxy: terminates TLS, applies authentication/authorization, limits rate and payload size. This is where you centralize logs and metrics per consumer.
- Egress controls: if the Ollama host can freely access the Internet, a bad configuration (or additional tooling) can leak data. In companies, controlled egress is expected (NAT with allowlists, firewall, or egress policies in K8s).
The important thing is that each component has a clear responsibility: the network prevents accidental exposure, the gateway controls identity and abuse, and the backend stays “deaf” to the Internet. That separation lets you change the model, scale nodes, or migrate regions without redoing the perimeter.
How to do it in practice with API Gateway (AWS) and a private backend
An operable approach in AWS is: API Gateway (HTTP API) as the frontend, a VPC Link to an internal NLB, and Ollama behind it on private instances or an internal service. Alternatively, an internal ALB with OIDC authentication can work, but API Gateway usually provides better quota/ratelimit controls and observability per API consumer.
Concrete actions that get implemented in real projects (not “nice to have”): create the API, enforce authentication (JWT/OIDC or IAM), limit who can invoke it (resource policy), and ensure that the NLB/target group only accepts traffic from the VPC Link. If someone tries to bypass the gateway, they won’t reach the backend.
- Create OIDC/JWT authentication on the gateway: for example with a corporate IdP (Entra ID/Okta) so each call carries verifiable identity. This avoids shared tokens in repositories or scripts.
- Restrict the API with a Resource Policy: allow invocation only from your VPC, from corporate ranges, or via a private endpoint. This reduces exposure even if someone discovers the URL.
- Apply limits (throttling/quotas) and size: protects GPU/CPU and prevents huge prompt/streaming loads that degrade the service.
Each measure has an operational reason: OIDC allows revoking access by user/role without redistributing keys; the resource policy prevents an endpoint from “becoming public” by mistake; and throttling protects you from poorly written internal scripts (very typical) in addition to external abuse.
Resource Policy example (API Gateway) to restrict invocation to a VPC Endpoint (Private API):
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Deny",
"Principal": "*",
"Action": "execute-api:Invoke",
"Resource": "execute-api:/*",
"Condition": {
"StringNotEquals": {
"aws:SourceVpce": "vpce-0abc123def4567890"
}
}
},
{
"Effect": "Allow",
"Principal": "*",
"Action": "execute-api:Invoke",
"Resource": "execute-api:/*"
}
]
}
Validation in AWS (what I would check in a security review): confirm that the backend (NLB/instance) does not have an SG with inbound from the Internet; verify in API Gateway that the endpoint is Private (if applicable) or that the policy truly restricts; run a curl from outside the allowed network and confirm 403; and review 4XX/5XX and throttles metrics to detect abuse or poorly integrated clients.
Authentication, authorization, and traceability: avoiding the “shared token”
In many quick deployments, “access control” ends up being a static token in an environment variable or, worse, no control because “it’s internal”. In companies, that breaks as soon as there’s staff turnover, external teams, or CI/CD integrations. Also, if the token leaks in logs, tickets, or screenshots, you have no way to attribute calls to a specific identity.
For Ollama, which typically doesn’t come with an embedded enterprise auth system, the usual approach is to move that control to the gateway/proxy. There you can map users or services to scopes/roles, enforce policies per application (not per person), and record each request with consistent identifiers. Traceability is not a whim: it’s what lets you answer “who queried the model with sensitive data?” or “which client saturated the GPU?” without turning it into a witch hunt.
- Workload identity, not human identity: integrations (apps, pipelines, bots) must authenticate with their own credentials (OIDC client credentials, IAM roles, etc.). When a developer leaves, nothing breaks and no orphaned keys remain.
- Logs with correlation: include
request_id, identity, route, latency, and tokens consumed (if you measure it). This changes the conversation with security and finance: you go from “we’re being attacked” to “this consumer exceeded its quota”. - Access policies by environment: dev/test/prod truly separated. A common failure is reusing the same URL/token for everything; the day someone tests with real data in dev, you already have an incident.
An anti-pattern that shows up in corporations is “put it behind a VPN and done”: yes, the VPN reduces exposure, but it doesn’t replace per-application authentication or rate limiting. Also, when access is opened to third parties or distributed teams, the VPN becomes a governance bypass if everything inside is flat.
Recommendations for corporate environments
If you’re going to deploy Ollama in the cloud, treat it as a sensitive service from day one: private backend with no direct exposure, single entry through a gateway with TLS, auth, and limits, and segmentation that prevents alternative routes to the model. This cuts off at the root the “temporary public API” pattern that ends in shadow AI.
The part that gets paid for most in incidents is not the model, but the lack of control: shared tokens, absence of quotas, and logs that don’t allow attribution. Centralizing authentication/authorization in the gateway and requiring workload identity gives you revocation, auditability, and stable operations as usage grows.
Finally, validate what you configure: tests from outside the allowed ranges, verification of security groups/NACLs, and periodic review of throttling and 4XX metrics. In corporate environments, “being behind a VPC” is not a sufficient control if you can’t prove who accesses, from where, and under what limits.
Interested in Cloud Security?
Technical analysis, hands-on labs and real-world cloud security insights.