An air-gapped network has no outbound internet connection. It is common in regulated industries, defence, and large organisations with strict security policies. Running an LLM locally solves most of the privacy problem, but air-gapping introduces practical constraints. Getting weights into the network requires logistics. Dependencies that assume internet connectivity need discovery and handling. Verification that nothing is phoning home requires work.
Getting model weights into an air-gapped environment
The first challenge is simple: the model weights are usually too large to carry on a USB drive. A 70-billion-parameter model at 4-bit quantisation is roughly 40 GB. Multiple copies of different models exceed reasonable transfer media quickly.
Three approaches:
Sneakernet with a connected staging machine. Download the model on a machine with internet access using Ollama or Hugging Face tools. Copy the weights to portable storage (external drive, network transfer). Transfer to the air-gapped machine and import. This is the most common approach for small-to-medium deployments. It works, requires discipline about what you transfer, and has no ongoing dependency on internet access.
Pre-loaded images. Build a container or VM image with the model weights already present. Deploy that image into the air-gapped environment. This works at scale and is reproducible, but requires DevOps infrastructure to build and manage images.
Weights stored outside the air-gap. Keep the model on a connected file server that the air-gapped network can access via a one-way transfer mechanism or jump host. This adds a network dependency and complicates the deployment, but reduces the need to physically move large files.
Most organisations use the first approach initially and move to images once the process is routine.
Model updates in an air-gapped setting
Keeping models current is harder without internet. New versions of models appear constantly. Quantised versions improve. Bug fixes and safety improvements are released.
Plan for updates before you deploy. Decide whether you will:
- Freeze a model version indefinitely and accept technical debt.
- Update on a schedule: quarterly, twice yearly. Download and test new versions on the connected staging machine, then transfer to air-gapped machines.
- Maintain a change request process for model updates, similar to infrastructure changes.
All three are defensible. The second is the most common: periodic updates through a deliberate process. Frozen indefinitely is rare in practice, because it eventually causes pain. The third is necessary only at very large scale.
Telemetry and calls home
The biggest gotcha is software that quietly assumes internet. Inference itself runs offline once the weights are local, but that is a claim to verify rather than accept — serving tools commonly check for their own updates, and the surrounding frameworks, middleware and libraries you add to the pipeline each bring their own assumptions.
Common culprits:
- Automatic update checks. Software that phones home to check for new versions, even if it does not install them automatically. This is usually harmless but violates air-gap policy.
- Crash reporting. Some frameworks send stack traces or usage telemetry to a cloud endpoint. This can leak information about your data or operations.
- Model card downloads. Hugging Face's transformers library can fetch metadata about models from the internet. If this fails silently, you might not notice.
- Package registration. Some Python packages contact a server on import, ostensibly for licensing or analytics.
None of these are intentionally malicious in popular open-source projects, but they exist. You need to find and disable them.
Verification: how to know nothing is calling home
Run the system on the air-gapped network and monitor network access. Set up logging or firewall rules that capture any attempt at outbound connection. Common tools:
tcpdump or Wireshark. Capture all network traffic. Run your inference workload and inspect what leaves the machine. Expect zero external connections if the air-gap is strict.
Firewall logging. If your air-gapped network is behind a stateful firewall, enable logging of rejected outbound connections. Run the system and check the logs for attempts to reach external hosts.
strace or system call monitoring. On Linux, trace system calls to see socket creation and DNS lookups. This is lower-level than network traffic and catches attempts even if they fail.
Run these checks before going live. Catch surprises in testing, not in production.
Offline model cards and documentation
Model cards describe the model's capabilities, training data, known limitations, and performance across different tasks. In a connected environment, these live on Hugging Face or the model publisher's site. In an air-gapped network, you lose access to them.
Before deploying, download model cards, papers, and documentation for every model you plan to use. Store them alongside the weights. This becomes the reference for understanding the model's behaviour and limitations.
Licensing and compliance
Most open-source models have permissive licences. But check the licence of every model and dependency you use. Some models require attribution; some prohibit commercial use; some have restrictions on data types or application domains. In an air-gapped deployment, you cannot rely on automated licence compliance tools, so do this review once and document it.
Putting it together
Air-gapped local inference is genuinely simpler than air-gapped deployment of cloud-based APIs. The model runs offline; nothing calls home by default. But it is not fire-and-forget.
Plan for weight transfer, establish an update strategy, audit dependencies for calls home, run network monitoring to verify silence, and store documentation offline. A deployment using Nodus Veritatis with Ollama is particularly well-suited to air-gapped environments: models run locally, the application runs on your own server, and no traffic reaches external services. Verify the setup before you deploy and you have a solid, offline-first workspace.