Running inference on a workstation or private server removes one external data path. It does not automatically protect source code from model artifacts, editor extensions, logs, retrieval indexes, exposed APIs, tools, compromised dependencies, or accidental egress.
Ollama documents that locally executed prompts and data are not sent back to its service, can disable cloud features, and binds its server to 127.0.0.1 by default. Those are useful controls, but the complete developer workflow includes the editor, plugins, shell, source repository, vector store, model downloader, package manager, telemetry, tools, backups, and operating system.
A secure design starts with a data-flow diagram and explicit classification. Mark source code, secrets, customer data, incident evidence, architecture documents, prompts, embeddings, model outputs, traces, and generated patches. For each flow, identify the process, identity, storage location, retention, network destination, and action it can trigger.
Applications should not call an Ollama, llama.cpp, or vLLM endpoint directly. Put a local gateway between clients and inference. It authenticates the caller, classifies the request, enforces repository and data policy, removes obvious secrets, selects an approved model, limits context and output, records minimal audit metadata, and blocks disallowed tools or destinations.
The same interface can route low-sensitivity work to an approved cloud model when capability is required. The decision must be policy-driven and visible: sensitivity, project, data residency, model approval, task type, context size, and user authorization. Never rely on a developer remembering which chat window is safe.
Local inference still downloads gigabytes of third-party artifacts and runs parsers, tokenizers, templates, native libraries, GPU drivers, and sometimes custom model code. Hugging Face documents the arbitrary-code risk of pickle and recommends safer tensor serialization. Prefer safetensors or constrained formats such as GGUF, reject models requiring unreviewed remote code, scan artifacts, and isolate conversion jobs.
Mirror approved models into an internal registry. Pin the repository revision, file hashes, container digest, runtime version, tokenizer, prompt template, quantization, license, and model card. Record provenance and vulnerability results. SLSA, Sigstore, and the NIST SSDF provide reusable supply-chain patterns even though model bundles require additional metadata.
A private vector database can still become a cross-project data leak. Ingestion must exclude .env files, private keys, credentials, dumps, generated binaries, dependency trees, personal data, and directories outside the approved repository. Apply the source ACL before embedding and again before returning chunks. Do not trust a metadata filter added after similarity search as the only tenant boundary.
Keep indexes separate where isolation matters, encrypt storage, version the corpus, delete removed content, and expose source path, commit, line range, and access decision with each retrieved chunk. The model should receive the minimum code needed for the task, not the entire repository because the context window allows it.
A local model that can run arbitrary shell commands, read the home directory, use SSH credentials, access cloud CLIs, or call internal APIs has more effective reach than many cloud chat systems. OWASP identifies prompt injection and excessive agency as core risks. Treat model output as untrusted input and put each tool behind a typed, policy-enforced adapter.
Use disposable worktrees or containers, non-root identities, read-only mounts by default, resource limits, command allowlists, network deny rules, path restrictions, timeouts, and explicit approval before writes, commits, package installation, database changes, or external communication. Show the exact diff and command, not a generic “allow agent” prompt.
| Surface | Typical exposure | Required control | Evidence to keep |
|---|---|---|---|
| Inference API | Server exposed on LAN, weak caller identity, broad browser origins, unlimited requests. | Loopback or private network, authenticated gateway, firewall, TLS where remote, origin allowlist, rate limits. | Caller, policy decision, model ID, timing, request class, denied attempts. |
| Model artifact | Malicious serialization, remote code, replaced tag, incompatible license. | Internal allowlist, safe format, digest pinning, scanning, signature/provenance verification, license review. | Source revision, hashes, signature, SBOM, scanner and approver. |
| Source and RAG | Secrets embedded, stale deleted code, cross-repository retrieval, excessive context. | Pre-ingestion scanning, ACL before retrieval, separate indexes, deletion propagation, context minimization. | Corpus version, chunk sources, ACL result, excluded paths, retention. |
| Editor integration | Extension telemetry, hidden cloud fallback, broad workspace access. | Approved extension build, configuration policy, egress control, restricted workspace, update governance. | Extension version, settings, destinations, permissions, update record. |
| Agent tools | Prompt injection triggers shell, Git, ticket, database, or cloud action. | Sandbox, typed adapters, least privilege, dry run, human approval, transaction limits. | Tool arguments, command, diff, approver, result, rollback identifier. |
| Logs and memory | Prompts, code, tokens, secrets, or customer data retained indefinitely. | Minimal structured logs, content redaction, encrypted storage, short retention, deletion workflow. | Retention policy, access log, deletion event, redaction version. |
“Do not reveal secrets” is not a control. Keep credentials outside indexed paths and model-visible environment variables. Use short-lived workload identities, scoped tokens, secret brokers, pre-commit and pre-ingestion scanners, and output scanning before patches or messages leave the boundary. Rotate any secret that enters a prompt, trace, embedding, or generated artifact unexpectedly.
Full-disk encryption protects a powered-off device, not a compromised editor process. Enterprise use also needs endpoint management, patching, screen-lock policy, malware protection, user separation, backup controls, and remote revocation.
Privacy can fail indirectly when a weak local model produces insecure code or developers bypass it for an unapproved service. Evaluate on the actual repositories and tasks: patch correctness, test pass rate, secret handling, authorization decisions, dependency choices, hallucinated APIs, tool behavior, latency, memory, and energy.
Quantization, context length, hardware, prompt template, and retrieval strategy change behavior. Version them and use the same evaluation pipeline as any production AI component. Route only tasks the local tier can perform within an accepted quality floor.
I would build a developer AI gateway running on managed workstations and private inference clusters. It would classify context, enforce repository policy, choose a pinned local model, retrieve ACL-approved code, create an isolated worktree, expose narrow tools, scan the proposed diff, and require approval for consequential actions.
A central control plane would distribute model manifests, policy bundles, approved extensions, eval results, revocations, and audit schemas without collecting source content. Teams would see which data stayed local, which sanitized tasks crossed the boundary, which artifacts ran, and which actions were approved.
Local inference is valuable because it can shrink the data boundary and remove a provider dependency. Security comes from controlling the whole workflow: identity, artifacts, network, retrieval, tools, storage, logs, quality, and human authority. “Runs on my machine” is the starting condition, not the security conclusion.
Article about secure local LLMs for sensitive developer workflows, including Ollama, model provenance, source-code privacy, ACL-aware RAG, secret protection, tool sandboxing, egress controls, and hybrid model routing.