Ever caught yourself doing find-and-replace on a router config before pasting it into a chatbot? Scrubbing IPs, renaming hostnames, hoping you got every line? That ritual barely helps. A config is a map of your network, and the device types and services still leak through the syntax no matter how many octets you blank out. Depending on your world, CMMC, CJIS, PCI-DSS, HIPAA, pasting device data into a public AI tier can be a policy violation, not just bad hygiene.
A viewer asked me for the alternative by name, twice, with airtight reasoning: run the model locally, and there’s no redacting, nothing leaves the machine, and there’s no token bill. This post is that answer, with every claim measured on my own hardware. A free model on my laptop drafted real documentation from a 14,000-token router config, I fact-checked every line, and it invented exactly zero facts. Nothing left the machine.
One thing before we start: this is not an anti-cloud post. I use cloud AI every day. The honest trade-offs are further down, and they cut both ways.
The problem
Three things keep senior engineers from getting AI help on the work that actually needs it:
- The data you most want help with, configs, ACLs, log excerpts, is exactly the data you can’t paste into a free cloud tier. Free tiers train on inputs by default.
- Redaction doesn’t fix it. It’s tedious, error-prone, and the network’s shape leaks through syntax anyway.
- The local alternative has a reputation for being a toy. Slow, dumb, fiddly. Some of that reputation is earned, but most of it comes from silent misconfigurations nobody tells you about.
That last one is the real story. Local AI tooling fails quietly: no error, no warning, just confidently wrong output. I hit 4 of those silent failures on camera. Each one has a one-line fix.
The workflow
Everything below was measured on a MacBook Pro M3 Max with 64 GB of unified memory. You do not need this machine. The cheapest genuinely viable seat is a Mac Mini around $1,400, and the finale of the companion video runs entirely on a 9B model, which is 16 GB laptop territory. The runbook has both macOS and Windows blocks for every step.
Install Ollama and fix the defaults
The install is 60 seconds: download from ollama.com, launch the app, pull a model. One gotcha: the CLI appears after you launch the app once, not when you drag it to Applications. If ollama isn’t found, that’s why.
The defaults are the real trap. Out of the box, Ollama is tuned for chat, not for feeding it a 14,000-token config. Five settings fix that:
launchctl setenv OLLAMA_CONTEXT_LENGTH 65536
launchctl setenv OLLAMA_FLASH_ATTENTION 1
launchctl setenv OLLAMA_KV_CACHE_TYPE q8_0
launchctl setenv OLLAMA_NUM_PARALLEL 1
launchctl setenv OLLAMA_KEEP_ALIVE -1
Then restart the app. These are read at launch. Windows versions (setx instead of launchctl setenv) are in the runbook.
Pick a model by architecture, not size
Here’s the finding that surprised me most. I benchmarked 6 models on this machine, same prompt, machine quiet. The 35B model ran 4 times faster than the 27B model from the same family. Bigger model, same hardware, 4x the speed.
The reason is architecture. The 35B is mixture-of-experts, so only a few billion parameters wake up per token. The 27B is dense, and every parameter works on every token. On my whole ladder, from 75 tokens a second at the top to under 7 at the bottom, every fast model is MoE and every slow one is dense.
The model page won’t tell you which is which. This will:
ollama show qwen3.6:35b
Read the architecture field. That one check will save you a bad 40 GB download, and it’s the entire reason cheap hardware is viable for this at all.
Run the real test
Most local AI demos feed the model a 10-line toy config. I used a full show run all capture from my lab core router: 44 KB, about 14,000 tokens, every default, every line.
ollama run qwen3.6:35b --verbose "You are documenting a network device for an engineer handoff. From the following running configuration, produce: 1) an interface table (name, IP, mask, description, state), 2) a routing summary (protocols, router-id, networks, passive interfaces), 3) a list of configuration risks or oddities worth an engineer's attention. Config follows: $(cat core-rtr-01.cfg)"
Expect about 18 seconds of silence first. That’s the model ingesting 14,000 tokens before it can say a word, not a hang. Budget the pause into how you’d actually use it. Then the doc lands: an interface table, a routing summary, and a list of config risks it ranked without being asked.
Then the part that actually matters: verification. I checked every claim against the config, line by line. The interface table was 5 for 5, exact IPs, exact masks, descriptions word for word. Every flagged risk was real, at a real line number. Zero invented facts in the whole document. Full disclosure on the method: I didn’t eyeball 14,000 tokens by hand. I had Claude diff every claim against the raw config and reviewed the results. AI checking AI, with me as the judge.
The 4 silent failures
All 4 happened live on a well-configured machine, and none produced an error message.
1. The default context window eats your question. Ollama ships with a 4,096-token context. Feed it 14,301 tokens and it silently discards the front, which is where your question was. On my run, the model received the config’s tail with the first word of my question glued to an SSH line, diagnosed that remnant as a paste error in my config, and confidently summarized the half it saw. The receipt is in the --verbose stats: prompt eval count 2,050, out of 14,301 sent. One env var fixes it completely.
2. The window dies mid-answer. The context has to hold your prompt, the model’s thinking, and the answer. I pasted 28,391 tokens into a 32K window and generation stopped mid-table at 32,768 exactly. No error, just a table that ends. Thinking models spend the window twice. Rule: keep about a third of the window free for the response.
3. The empty prompt that never errors. Typo a filename inside $(cat ...) and the run does not stop. The prompt goes out empty and the model answers anyway. Mine politely asked for the config. A weaker model might have invented one.
4. Structured output returns nothing. Ask a thinking model for JSON with a format= schema and you can get an empty response with a success code. I hit it 3 straight times. The fix is one flag, think: false, on extraction calls. Then it’s flawless: valid JSON, passes Pydantic validation, matches the config exactly. Add temperature 0 and reruns come back token-for-token identical, which is what you want in a pipeline.
Prove the privacy claim
Don’t cite a privacy policy. Read the socket table, with wifi ON:
sudo lsof -i -P -n | grep -i -E 'ollama|lm ?studio'
Across a full inference lifecycle on my machine: 2 loopback listeners at idle, 3 more loopback-to-loopback connections during inference, back to 2 after. Every address on the screen was 127.0.0.1, my laptop talking to my laptop, with the internet fully available.
One honest exception: cloud-tagged models live in the same CLI. Ollama’s :cloud tags run remotely by definition, so check the tag. If you want the door locked, one line in ~/.ollama/server.json, {"disable_ollama_cloud": true}, survives reboots. After that, trying a cloud model on purpose returns an explicit refusal. Verify the default, name the exception, close it, prove it’s closed.
Local vs cloud, measured
I ran the same documentation job through Claude. Fresh chat, no saved context, same verbatim prompt, same config.
- Speed: Claude finished in about 25 seconds. Local took 2 minutes 24. Call it 6x faster in the cloud.
- Accuracy: the shocker. Line-by-line verification on both outputs found zero invented facts on either side. On this task class, the free local model matched Claude on trustworthiness.
- Judgment: the cloud’s real edge. Claude triaged findings by severity, reasoned across devices unprompted, and even sniffed the test itself, flagging my lab.local domain as a lab config wandering through a prompt that implied production.
- Exhaustiveness: local’s edge. It surfaced real findings Claude’s noise filter dropped, including an MD5 digest auth on the web interface, a legitimate security finding only the local model caught.
The verdict in trade-off language: for documentation and extraction work on your own configs, local is genuinely enough. Free, private, and on this job just as trustworthy. The cloud buys speed and judgment on the messy open-ended problems. Neither one removes you from the review chair.
Adapting for your network
The demo inputs are real Cisco IOS show run all captures from my lab, and they ship in the repo so you can replicate everything without owning a lab. For your own environment: any config export works as input, the prompts are vendor-neutral, and nothing about the workflow assumes Cisco. Size your model to your RAM (the runbook has an honest 16/32/64 GB breakdown), and check prompt eval count against what you sent until you trust your context settings. If you script it, use the structured-output pattern in the repo: schema, think: false, Pydantic, temperature 0.
Honest limitations
- Local is slower. 6x slower than the cloud on the measured job, and the 18-second time-to-first-token on big pastes is real.
- Quality degrades subtly as context grows. On a 2-device run, the local model confidently claimed a loopback was OSPF-active when the config says
passive-interface Loopback0explicitly. The same model got that fact right on the smaller run. Subtle and confident is the dangerous kind of wrong. A senior engineer catches it on review; a junior might not. - Thinking models occasionally garble a stretch inside their reasoning trace. Every final answer I checked was clean, but you’ll see weird text scroll by.
- The review step never goes away, local or cloud. That’s not a limitation of the tooling. That’s the job.
Get the artifact
The dual-platform quickstart runbook, the local AI prompt pack, the exact config captures from the demo, and the 3 working scripts (structured extraction, the sampling A/B harness, and a read-only MCP server) are all free on GitHub:
github.com/GTalksTech/netops-toolkit/tree/main/local-ai/local-ai-network-engineers