Using Ollama with Continue: A Developer's Guide

As developers, we want tools that respect our privacy, let us customize our experience, and integrate seamlessly into our workflows. Continue paired with Ollama offers exactly that—a way to run custom AI coding assistants locally.

The benefits of this include

1. Data privacy: your code stays on your machine, not sent to third-party servers
2. Full control: choose exactly which models you want to use for different coding tasks
3. Customization: configure rules, prompts, docs, etc. to tailor the assistant to your needs
4. No subscription costs: use powerful open-source models without recurring fees
5. Offline usage: work without an internet connection when needed

In this guide, I'll walk you through setting up Continue with Ollama, so you can build a development environment that respects your workflow.

What you'll need

• A computer capable of running LLMs locally (see hardware requirements below)
• A supported IDE with the Continue extension:
  • VS Code
  • JetBrains
• A Continue account (https://hub.continue.dev/signup)

Step 1: Install Ollama

Ollama lets you run powerful language models locally. Installation is straightforward:

MacOS/Linux

curl -fsSL https://ollama.ai/install.sh | sh

Windows

Download the installer from Ollama.

Step 2: Pull a code-optimized model

After installation, pull a model that works well for coding. I find that Qwen Coder 2.5 7B represents a sweet spot for many developers.

ollama pull qwen2.5-coder:7b

This will download the model to your local machine. You can verify it works by running:

ollama run qwen2.5-coder:7b "Write a function to calculate the factorial of 5"

Step 3: Set up your Continue account

Before creating your assistant, you'll need to:

1. Visit https://hub.continue.dev/signup to create your account
2. Verify your email address
3. Log in to access Continue Hub

Step 4: Create a new assistant

Once you have a Continue Hub account, you can create a new assistant:

1. From your Continue Hub dashboard, click the "+" button to create a new assistant
   

2. Give your assistant a name (e.g., "Llama Local")

3. By default, Continue will populate some useful model blocks for you. You can:

   • Delete these blocks if you don't need them
   • Navigate to https://hub.continue.dev/ollama/qwen2.5-coder-7b
   • Click to add this block to your assistant
     

4. Save your assistant configuration
   

5. Your assistant will now be available at a URL like: https://hub.continue.dev/chad/llama-local

Step 5: Install and log into Continue in your IDE

1. Install the Continue extension for your IDE:

   • For detailed instructions, see the official documentation
   • Follow the installation steps for VS Code or JetBrains

2. Once installed, log into your Continue account

3. Access all of your assistants in the Continue IDE extension
   

Next step is customizing all the things

After setting up your assistant, you can customize it further:

1. Rules: Define how your assistant should behave and what expertise it should have
2. Additional models: Set up different model roles for different types of tasks.
3. Other blocks: Explore all the different types of blocks on the Hub.

Conclusion

By combining Continue with Ollama, you've created a powerful, private, and customizable coding assistant that runs entirely on your machine. This setup gives you the benefits of AI assistance while maintaining control over your code and data.

As models continue to improve, you can easily upgrade your local setup by pulling newer models without changing your Continue configuration. Experiment with different models and settings to find the perfect balance between performance and capability for your specific hardware and workflow.

Happy coding!

Hardware requirements for different model sizes

Choosing the right model depends on your hardware capabilities. Here are general guidelines:

Additional Hardware Notes

• Small models (1.5B-3B) will work on modern CPUs with 16GB+ RAM, though your mileage may vary
• For CPU-only usage, expect significantly lower throughput (1-5 tokens/sec)
• CPUs with integrated GPU cores (Apple Silicon, AMD Ryzen™ AI Max 300, etc.) perform well with appropriate unified memory and bandwidth
• Medium and large models become impractical without dedicated GPU acceleration
• Using quantization (4-bit or 8-bit precision) can reduce VRAM requirements by 2-4x but with some loss of accuracy
• Consider running smaller models with higher quantization precision rather than larger models with aggressive quantization

Apple Silicon Specifications

For Mac users, here's a breakdown of Apple Silicon capabilities: