← All writing

Use MAX with Continue as a local Code Assistant

Serve a model locally with Modular's MAX Serve and connect it to the Continue extension for a private code assistant in VS Code.

This guide explores how to set up a powerful local code assistance solution by combining Modular’s MAX Serve with the Continue VS Code extension.

We’ll walk through the process of deploying a local language model using MAX Serve and integrating it with Continue, creating a seamless development experience that operates entirely on your local machine. This setup provides the benefits of AI-powered code assistance while maintaining full control over your development environment.

MAX Serve

MAX Serve is a Python-based inference server that provides an OpenAI-compatible REST endpoint for both local and cloud environments. At its core, MAX Serve uses MAX Engine, which features a high-performance graph compiler and runtime for model acceleration.

MAX Serve architecture: an HTTP client sends requests to the MAX container, where MAX Serve exposes an OpenAI REST API with scheduling, batching and model loading on top of MAX Engine, running on cloud providers and CPU, GPU or xPU hardware

source: https://docs.modular.com/max/serve

Continue

The open-source code assistant Continue can connect directly to MAX Serve’s OpenAI-compatible REST endpoint.

Continue

To get started, we’ll launch the MAX Serve endpoint and then configure it in Continue.

Deploy local Endpoint

The following is from the MAX documentation: Deploy a PyTorch model from Hugging Face | Modular

Prerequisites

Install the Hugging Face CLI: Command Line Interface (CLI)

Authenticate using huggingface-cli

huggingface-cli login

Setup MAX repository

Clone the MAX repo:

git clone git@github.com:modularml/max.git

After cloning, navigate to max/pipelines/python directory

cd max/pipelines/python/

Serve Endpoint

Use the magic CLI to start the server endpoint. If you don’t have the magic CLI yet, you can install it on macOS and Ubuntu Linux with this command:

curl -ssL https://magic.modular.com/deb1e9f0-447e-4b92-a807-29d62898936f | bash

Then run the source command that’s printed in your terminal.

The command we’ll run is magic run serve, which takes supports a
huggingface-repo-id parameter. This parameter allows you to specify an LLM
model hosted on Hugging Face.

magic run serve --huggingface-repo-id=Qwen/Qwen2.5-1.5B-Instruct

This command downloads the model and sets up MAX Serve to host a local endpoint.

The endpoint is ready when you see output similar to the following in your terminal:

uvicorn running on http://0.0.0.0:8000

Test the endpoint

once the server is up and running, we can test with this curl command:

curl -N http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "Qwen/Qwen2.5-1.5B-Instruct",
        "stream": true,
        "messages": [
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": "What is the capital of Mongolia?"}
        ]
    }' | grep -o '"content":"[^"]*"' | sed 's/"content":"//g' | sed 's/"//g' | tr -d '\n' | sed 's/\\n/\n/g'

Here is the expected output:

The capital city of Mongolia is Ulaanbaatar.

Configure MAX Serve with Continue

Now that we have our MAX Serve endpoint running locally, lets connect our endpoint to Continue in VSCode!

Setup VS Code

  1. Click Install on the Continue extension page in the Visual Studio Marketplace
  2. This will open the Continue extension page in VS Code, where you will need to click Install again
  3. The Continue logo will appear on the left sidebar. For a better experience, move Continue to the right sidebar

VS Code with the Continue chat panel moved to the right sidebar

Configure MAX Serve Endpoint

Next, we need to configure our MAX Serve endpoint.

  1. Open the VSCode command pallet (cmd+shift+p) and run Continue: Open Continue Config This will open the configuration settings for the Extension: ~./continue/config.json

The VS Code command palette showing the Continue: Open Continue Config command

  1. under the models section of the config file, add the following:
{
  "models": [
    {
      "model": "Qwen/Qwen2.5-1.5B-Instruct",
      "title": "MAX - Qwen/Qwen2.5-1.5B-Instruct",
      "apiBase": "http://localhost:8000/v1/",
      "apiKey": "",
      "provider": "openai"
    }
  ]
}

the apibase is the same of the server endpoint from before: http://0.0.0.0:8000

Once you save the config file, the model should be available from dropdown in the chat window

The Continue chat model dropdown with MAX - Qwen/Qwen2.5-1.5B-Instruct listed alongside other models

Continue Features

here are some Continue features using MAX Serve

Chat

The chat interface allows you to get AI assistance directly within your IDE environment

Edit

The Edit feature provides a convenient way to modify code directly within your current file

Key takeaways

Using MAX Serve with Continue provides a powerful local code assistant solution that combines the efficiency of local model inference with the user-friendly features of Continue. By leveraging MAX’s OpenAI-compatible REST endpoint, developers can access AI assistance directly within their IDE without relying on cloud services. The setup process involves deploying a local endpoint using MAX Serve and configuring Continue to communicate with this endpoint, resulting in a seamless development experience with features like contextual chat and direct code editing capabilities.

Sources