Notes on local inference, security, and the machinery underneath.
Attributing API costs to individuals enables enforcement of limits and detection of runaway workloads.
An internal tool with no advantage over free public alternatives will not be used, no matter how well it is built.
The choice between self-hosted and API pricing depends on utilisation, not per-token rates alone.
Security reviewers ask specific questions about AI tools — data flow, retention, sub-processors, access control, audit logging, key handling.
Internal AI tools aren't automatically exempt from regulation — risk classification depends on use case, and documentation matters.
The practical question of data flow when using AI assistants — what gets sent where and why self-hosted inference handles it differently than APIs.
Air-gapped LLM deployment requires planning for model weights, updates, telemetry, and verification that nothing calls home.
Ollama is built for single-box deployment and ease. vLLM prioritises throughput and concurrency. Each wins in different scenarios.
VRAM requirements depend on model size, quantisation, and context length. There's no single answer.