Trying a local AI at home: the one that gets it right and the one that lies with good handwriting

This week, twenty-five tech companies —Nvidia, Microsoft, Meta and IBM among them— signed a document arguing against restrictions on open-weight AI models. Reading it got me curious: opining about open models is one thing, actually having run one is another.

I also had unfinished business. Not long ago I dropped the idea of building an exam-question generator for Moodle because every API call cost money. A few cents, but per use and out of my pocket. So the question was obvious: could my own server do it for free?

I tried it. And along the way I got it wrong twice, which is usually the interesting part.

The setting: a server that’s already busy #

No clean lab here. A Mac mini M4 with 16 GB of RAM already running:

• this site and the blog
• a Moodle (aula.sergiocomeron.com)
• a Jitsi in Docker
• the monitoring stack (Prometheus, Grafana, Loki, Jaeger)
• PostgreSQL

With about 8 GB of RAM free and tight on disk. That rules out large models from the start: the guest has to fit without evicting anyone.

The install (boringly uneventful, which is the highest praise for anything in production) #

1brew install ollama

47 MB. Plus one small but deliberate decision: I didn’t leave it as a permanent service. On a production machine, what starts on its own at boot is a decision, not an oversight. So I bring it up by hand when I need it:

1OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q8_0 ollama serve

And I pull a small model, not the biggest one that fits:

1ollama pull qwen3:4b     # 2.3 GB on disk

First attempt: a wall of English text #

I gave it some notes on the OSI model and asked for 4 multiple-choice questions in GIFT format, the one Moodle can import. This is what came back:

Okay, I need to create exactly 4 multiple-choice questions in Moodle's GIFT
format based on the given text about the OSI model. Let me start by
understanding the text thoroughly.

The text describes the OSI model's seven layers...

Paragraph after paragraph, in English, until it ran out of room and cut off mid-sentence.

What “thinking out loud” means #

Today’s models come in two flavours. The classic ones: you ask, they answer. And the reasoning ones, which before answering write an internal monologue working through the problem. It’s not a quirk: on hard tasks, they get more right that way.

Normally that monologue travels on a separate channel —the API returns it in a thinking field— and you only show the final answer. Here that channel kept coming back empty and the whole monologue landed mixed into the response. I’ll come back to this, because the mistake was mine.

What the “token budget” is (and why it exists if it’s free) #

When you fire a request you set a limit on how much text it can generate (num_predict, or max_tokens in paid APIs). In the cloud that number is money. On my server it doesn’t cost a cent.

So why set it at all? Because a model doesn’t know how to stop on its own: it can ramble, or loop repeating itself indefinitely. It’s the handbrake.

But locally you also pay, just not in euros:

• In seconds: at ~28 tokens/s, 5,000 tokens is nearly 3 minutes of waiting.
• In RAM: everything generated accumulates in memory, on a machine already holding other things up.

In the cloud you pay per token. At home you pay per second.

I had set the limit at 1,500. The model spent about 1,400 thinking and had 100 left to answer. It’s like handing out two sheets for an exam and the student spending one and three quarters on “right, let me get organised”.

Second attempt: fine, I’ll get one that doesn’t think #

The conclusion seemed obvious: the problem is that this model rambles. So I tried a non-reasoning one, same family and similar size, so the only variable was that:

1ollama pull qwen2.5:3b   # 1.9 GB

7 seconds. Impeccable formatting. Four clean questions. And then I read them:

::Protocolo IP::Which protocol operates at the OSI network layer? {
=El protocolo TCP      ← marked as CORRECT
~El protocolo IP       ← the right answer, marked as wrong
~El protocolo UDP
~El protocolo HTTP
}

::TCP y UDP::Which protocol does NOT guarantee delivery...? {
=TCP                   ← it's UDP
~UDP
...
}

Two out of every four questions with the wrong answer marked as correct.

And before blaming the model: my first prompt used a template with placeholders (::Question title::) and the model copied it literally, returning things like ::Título::Capa física {=capa_fisica~capa_fisica}, with the correct and incorrect answers identical. I rewrote the prompt with a fully worked example and that fixed the formatting… but not the factual accuracy. The failure above is already from the good prompt.

Third attempt: what if the problem was me? #

This is where the experiment turned around. Someone asked me the obvious question: isn’t it a configuration thing? So I checked:

1ollama show qwen3:4b

Capabilities
  completion
  tools
  thinking      ← fully supported

The model does declare separate-reasoning capability. So I tried both configurations with the same request:

There was my mistake. I was using think: false believing it silenced the model. What it actually does is tell Ollama “don’t expect reasoning”: the model reasons anyway —it can’t help it, that’s how it was trained— and with nobody parsing for it, the whole monologue spills into the answer.

The correct setup is the opposite: let it think and read the other channel. With think: true and plenty of budget:

1body = {
2    "model": "qwen3:4b",
3    "messages": [{"role": "user", "content": PROMPT}],
4    "think": True,                          # ← the key
5    "options": {"num_predict": 12000},      # ← plenty of headroom
6}

The result:

• 180 seconds (3 minutes), 4,876 tokens
• 18,520 characters of reasoning, all of it in the thinking channel
• 1,029 characters in content: clean GIFT, not a word out of place

::Función de la capa física::What is the main function of the physical layer...? {
=Transmite bits por el medio
~Agrupa bits en tramas y detecta errores
~Realiza direccionamiento lógico y enrutamiento
~Garantiza la entrega extremo a extremo
}

Four out of four, correct. Nothing to rescue from the noise.

The real comparison #

That’s the real trade: 26 times slower in exchange for twice the accuracy. And while it was generating, I measured my services: the Moodle answered in 0.14 s and the site in 0.05 s. Neither noticed.

The lesson: the failure you can’t see #

I had a validator checking the output: balanced braces, exactly one correct answer, enough distractors. It passed all four of the fast model’s questions. Because my validator knows about braces and symbols, but it doesn’t know about networks.

Notice the difference between the two failures:

• The misconfigured reasoning model failed loudly: it filled the screen with English, impossible to miss.
• The fast model fails silently: it hands you something impeccable, well formatted, ready to import… with the answer swapped.

No automated validator catches the second one. Only somebody who knows the subject does. And if nobody catches it, it reaches the exam. And from the exam, the student.

My conclusion isn’t the one I expected going in. I was looking for whether a small model could replace a paid one, and I ended up learning something else: that monologue I took for noise was the actual work. Take away a model’s thinking time and you get text that’s perfect on the outside and empty on the inside. The three minutes aren’t the price of the formatting; they’re the price of the answers being right.

So, can it be done? #

Yes. Free in euros, it can. A 2.3 GB model, on a server already doing five other things, generates correct exam questions without sending anyone your notes. And for a school, keeping student data inside the building isn’t a minor detail: there, quality isn’t what competes — data protection law is.

But free isn’t the same as costless. You pay in minutes of waiting, in reading the documentation properly before declaring something impossible —lesson learned— and in a teacher reviewing the questions before setting them. Which, come to think of it, is exactly what a teacher does with any exam.

For now the models stay installed, because this has only just started. I want to try somewhat larger models to see how far the machine holds up, and give them uses beyond exam questions. I’ll keep you posted.

This experiment is also episode 20 of the podcast, Probando una IA local en casa, if you’d rather listen to it.