AIForum.

How to Run AI Coding Assistants Locally with Continue and Ollama

On this page

THE SHORT ANSWER

Continue and Ollama together let chat and tab autocomplete run against a model hosted on your own machine. The setup is short: install Ollama, start the service, pull a model with the exact tag, then add an Ollama provider entry to Continue's config.json. The harder part is choosing a model your hardware can actually sustain.
Quick facts

Running an AI coding assistant locally takes two components: a model server on your machine and an editor extension that talks to it. Ollama provides the server; Continue provides the editor integration for VS Code and JetBrains. Together they let chat and tab autocomplete work against a model you host yourself, without routing every request through a hosted API.

The path below follows the documented configuration in Continue’s Ollama guide and Ollama’s own Continue walkthrough. It is written for developers who are comfortable in a terminal but new to local inference. Model sizes in the examples are placeholders — the right choice depends on your hardware, not on a single recommended default.

Prerequisites

Continue’s documentation lists four requirements before you begin:

  • Operating system: macOS, Linux, or Windows
  • RAM: minimum 8 GB, with 16 GB or more recommended
  • Storage: at least 10 GB free
  • The Continue extension installed in your editor

Those figures are the extension’s baseline for running Ollama alongside an editor. They are not a model sizing table. The memory a model actually consumes depends on its parameter count, its quantization, the context length you configure, and runtime overhead such as the key/value cache. A 7B model at a short context and a 7B model at a long context are not equivalent loads, so treat the 8 GB minimum as a floor for getting started rather than a guarantee.

Step 1: Install and start the Ollama service

Install Ollama using the package manager or installer for your platform:

# macOS
brew install ollama

# Linux
curl -fsSL https://ollama.ai/install.sh | sh

# Windows
# Download from ollama.ai

Confirm the binary is available, then start the service:

ollama --version
ollama serve

ollama serve runs in the background and exposes a local HTTP API. Verify it with a single request:

curl http://localhost:11434

The documented response is Ollama is running. If you get a connection error instead, the service is not up — and that one check resolves a large share of “Continue can’t reach the model” reports later in the process. Leave the service running while you work.

Step 2: Download models with ollama pull

Continue’s guide is explicit about this point: use ollama pull, not ollama run. The run command opens an interactive session, which is not what an editor integration needs.

ollama pull qwen2.5-coder:1.5b
ollama pull deepseek-r1:32b
ollama pull mistral:latest

# List everything you have downloaded
ollama list

Tags are not decoration. They select which variant of a model you get:

  • :latest — the default version used when no tag is specified
  • :32b, :7b, :1.5b — parameter-count variants
  • :instruct, :base — model variants with different tuning

If a model page shows deepseek-r1:32b, pull that exact tag. Pulling deepseek-r1 alone resolves to :latest, which may be a different size and therefore a different memory footprint than the one you intended.

Step 3: Point Continue at Ollama

In Continue, click the gear icon to open config.json and add an Ollama provider entry. The shape documented by both Continue and Ollama is:

{
  "models": [
    {
      "title": "Codestral",
      "provider": "ollama",
      "model": "codestral"
    }
  ],
  "tabAutocompleteModel": {
    "title": "Codestral",
    "provider": "ollama",
    "model": "codestral"
  }
}

The model value must match a name from ollama list. Chat and tab autocomplete are configured as separate entries, which means you can assign a different model to each role.

Choosing a model for chat versus autocomplete

Ollama’s guest post from May 31, 2024 recommends Mistral AI’s Codestral 22B for both autocomplete and chat, describing it as capable at both tasks. The same post attaches two constraints: the model requires a substantial amount of VRAM, and it ships under a non-production license, so it is suited to research and testing rather than daily commercial use. Because that recommendation is dated, check the current model card before treating it as a default — model lineups and licenses change quickly.

For machines with less headroom, the same post suggests DeepSeek Coder 6.7B for autocomplete and Llama 3 8B for chat. It notes that Ollama can run multiple models and handle concurrent requests, which makes that split possible when the hardware allows it. If your machine cannot sustain both at once, the post’s advice is to try each configuration and decide whether local autocomplete or local chat is the more valuable half of the experience for your workflow.

Verifying the setup works

  1. Run ollama list and confirm the model name you configured appears exactly as written.
  2. Run curl http://localhost:11434 and confirm the service responds.
  3. Open Continue’s chat and ask a short question. The first response is usually slower because the model has to load into memory.
  4. In a source file, pause after typing and check whether ghost-text suggestions appear.
  5. Watch the terminal running ollama serve. Incoming requests confirm the editor is reaching the server rather than a different provider.

Troubleshooting common failures

Continue reports a connection error

Check that ollama serve is running and that the curl check passes. If the service responds but the editor still fails, confirm nothing else on the machine has taken port 11434.

“Model not found” or an empty response

This is almost always a tag mismatch. Compare the model string in config.json against the output of ollama list character for character, including the suffix.

Responses are extremely slow

The first request after a cold start includes model loading, so a slow initial reply is expected. If every request is slow, the model likely exceeds available VRAM and is spilling to CPU or system memory. Try a smaller parameter count or a more aggressively quantized variant, and reduce context length if you have raised it.

The machine becomes unresponsive with both features active

Two models loaded simultaneously can exceed available memory. Run only one, or pair a small autocomplete model with a separate chat model only after confirming both fit at the same time.

Suggestions appear to come from a cloud provider

Continue supports multiple providers. If the Ollama entry is malformed, another configured provider may be answering the request. Re-check the JSON syntax and the provider field before assuming the local model is at fault.

Limits and tradeoffs

Local inference is not automatically better; it is a different trade. A model small enough to run comfortably on a laptop is generally less capable at complex reasoning than a large hosted model, and CPU-only inference is slow enough to make tab autocomplete impractical. In exchange you get code that stays on your machine, operation without a network connection once models are pulled, and no per-token API charges — against capability and speed.

Licensing is a separate question from “open weights.” The Codestral non-production restriction cited above is a concrete example of a model whose weights are downloadable but whose permitted use is limited. Before using any model in a work context, read the license attached to that specific model rather than assuming that downloadable means commercially usable.

Where to go next

Start with a small model to confirm the whole pipeline works end to end, then increase size until latency becomes unacceptable and step back one notch. Record the exact tags you pulled and the configuration that worked; that record is what makes the setup reproducible on another machine or recoverable after an update.

If you hit an error the checks above do not explain, share your operating system, hardware, Ollama version, the exact model tag, and the relevant lines from config.json in the Ai-Forum community. That combination is usually enough for another reader to reproduce the problem and suggest a fix.

FAQ

Do I need a GPU to run Continue with Ollama?

The documented prerequisites list RAM and storage rather than a GPU, and Ollama can run on CPU. In practice, CPU-only inference is slow enough that tab autocomplete becomes frustrating, while a GPU with sufficient VRAM keeps latency usable. Treat the hardware guidance as a starting point and measure on your own machine.

Can I use Continue with Ollama offline?

Inference against a local Ollama model does not require a network connection once the model has been pulled. Note that Continue itself also supports hosted providers, so an offline workflow depends on every configured entry pointing at a local model.

Why does the first response take so long?

The first request after a cold start includes loading the model into memory. Subsequent requests to the same loaded model are typically faster. If every request is slow, the model is probably too large for available VRAM and is spilling to CPU or system memory.

Can I run one model for chat and another for autocomplete?

Yes. Continue configures chat models and the tab autocomplete model separately, and Ollama can run multiple models and handle concurrent requests. Whether both fit in memory at the same time depends on your hardware, so test before relying on it.

References

  1. https://docs.continue.dev/guides/ollama-guide
  2. https://www.ollama.ai/blog/continue-code-assistant

Related terms

Ollama
A local runtime that downloads and serves language models through a local HTTP API, by default on port 11434.
Continue
An open-source AI code assistant extension for VS Code and JetBrains that can connect to local or hosted model providers.
Tab autocomplete model
A separate Continue configuration entry that supplies inline ghost-text suggestions, which can point at a different model than chat.
Model tag
The suffix after a colon in a model name, such as :7b, :32b, or :latest, that selects a specific size or variant.
KV cache
Memory the model uses to store attention state during generation; it grows with context length and adds to the memory used by the model weights.

Leave a Reply

Your email address will not be published. Required fields are marked *