DEV Community

yukaty
yukaty

Posted on

Part 3: Implementing Vector Search with Ollama

Part 1 covered PostgreSQL with pgvector setup, and Part 2 implemented vector search using OpenAI embeddings. This final part demonstrates how to run vector search locally using Ollama! ✨


Contents


Why Ollama? πŸ¦™

Ollama allows you to run AI models locally with:

  • Offline operation for better data privacy
  • No API costs
  • Fast response times

We'll use the nomic-embed-text model in Ollama, which creates 768-dimensional vectors (compared to OpenAI's 1536 dimensions).

Setting Up Ollama with Docker 🐳

To add Ollama to your Docker setup, add this service to compose.yml:

services:
  db:
    # ... (existing db service)

  ollama:
    image: ollama/ollama
    container_name: ollama-service
    ports:
      - "11434:11434"
    volumes:
      - ollama_data:/root/.ollama

  data_loader:
    # ... (existing data_loader service)
    environment:
      - OLLAMA_HOST=ollama
    depends_on:
      - db
      - ollama

volumes:
  pgdata:
  ollama_data:
Enter fullscreen mode Exit fullscreen mode

Then, start the services and pull the model:

docker compose up -d

# Pull the embedding model
docker compose exec ollama ollama pull nomic-embed-text

# Test embedding generation
curl http://localhost:11434/api/embed -d '{
  "model": "nomic-embed-text",
  "input": "Hello World"
}'
Enter fullscreen mode Exit fullscreen mode

Database Updates 🐘

Update the database to store Ollama embeddings:

-- Connect to the database
docker compose exec db psql -U postgres -d example_db

-- Add a column for Ollama embeddings
ALTER TABLE items
ADD COLUMN embedding_ollama vector(768);
Enter fullscreen mode Exit fullscreen mode

For fresh installations, update postgres/schema.sql:

CREATE TABLE items (
    id SERIAL PRIMARY KEY,
    name VARCHAR(255) NOT NULL,
    item_data JSONB,
    embedding vector(1536),        # OpenAI
    embedding_ollama vector(768)   # Ollama
);
Enter fullscreen mode Exit fullscreen mode

Implementation πŸ‘Ύ

Update requirements.txt to install the Ollama Python library:

ollama==0.3.3
Enter fullscreen mode Exit fullscreen mode

Here’s an example update for load_data.py to add Ollama embeddings:

import ollama  # New import

def get_embedding_ollama(text: str):
    """Generate embedding using Ollama API"""
    response = ollama.embed(
        model='nomic-embed-text',
        input=text
    )
    return response["embeddings"][0]

def load_books_to_db():
    """Load books with embeddings into PostgreSQL"""
    books = fetch_books()

    for book in books:
        description = (
            f"Book titled '{book['title']}' by {', '.join(book['authors'])}. "
            f"Published in {book['first_publish_year']}. "
            f"This is a book about {book['subject']}."
        )

        # Generate embeddings with both OpenAI and Ollama
        embedding = get_embedding(description)                # OpenAI
        embedding_ollama = get_embedding_ollama(description)  # Ollama

        # Store in the database
        store_book(book["title"], json.dumps(book), embedding, embedding_ollama)
Enter fullscreen mode Exit fullscreen mode

Note that this is a simplified version for clarity. Full source code is here.

As you can see, the Ollama API structure is similar to OpenAI’s!

Search Queries πŸ”

Search query to retrieve similar items using Ollama embeddings:

-- View first 5 dimensions of an embedding
SELECT
    name,
    (replace(replace(embedding_ollama::text, '[', '{'), ']', '}')::float[])[1:5] as first_dimensions
FROM items;

-- Search for books about web development:
WITH web_book AS (
    SELECT embedding_ollama FROM items WHERE name LIKE '%Web%' LIMIT 1
)
SELECT
    item_data->>'title' as title,
    item_data->>'authors' as authors,
    embedding_ollama <=> (SELECT embedding_ollama FROM web_book) as similarity
FROM items
ORDER BY similarity
LIMIT 3;
Enter fullscreen mode Exit fullscreen mode

Performance Tips πŸ“Š

Add an Index

CREATE INDEX ON items
USING ivfflat (embedding_ollama vector_cosine_ops)
WITH (lists = 100);
Enter fullscreen mode Exit fullscreen mode

Resource Requirements

  • RAM: ~2GB for the model
  • First query: Expect slight delay for model loading
  • Subsequent queries: ~50ms response time

GPU Support

If processing large datasets, GPU support can greatly speed up embedding generation. For details, refer to the Ollama Docker image.

Troubleshooting πŸ”§

Connection Refused Error

The Ollama library needs to know where to find the Ollama service. Set the OLLAMA_HOST environment variable in data_loader service:

data_loader:
  environment:
    - OLLAMA_HOST=ollama
Enter fullscreen mode Exit fullscreen mode

Model Not Found Error

Pull the model manually:

docker compose exec ollama ollama pull nomic-embed-text
Enter fullscreen mode Exit fullscreen mode

Alternatively, you can add a script to automatically pull the model within your Python code using the ollama.pull() function.

High Memory Usage

  • Restart Ollama service
  • Consider using a smaller model

OpenAI vs. Ollama βš–οΈ

Feature OpenAI Ollama
Vector Dimensions 1536 768
Privacy Requires API calls Fully local
Cost Pay per API call Free
Speed Network dependent ~50ms/query
Setup API key needed Docker only

Wrap Up 🌯

This tutorial covered only how to set up a local vector search with Ollama. Real-world applications often include additional features like:

  • Query optimization and preprocessing
  • Hybrid search (combining with full-text search)
  • Integration with web interfaces
  • Security and performance considerations

The full source code, including a simple API built with FastAPI, is available on GitHub. PRs and feedback are welcome!

Resources:


Questions or feedback? Leave a comment below! πŸ’¬

Top comments (0)