AIForum.

Local Embeddings for RAG: How to Generate Vectors Without a Cloud API

On this page

THE SHORT ANSWER

Embeddings for a RAG pipeline can be generated entirely on local hardware. Two documented paths exist: single-file llamafiles from Mozilla and an Ollama-based workflow in MongoDB's local RAG tutorial. This guide covers the models, licenses, memory figures, and verification checks you need before committing to either.
Quick facts

You can generate the embeddings that power a retrieval-augmented generation (RAG) pipeline entirely on your own hardware. An embedding model — the component that converts text into a fixed-length numeric vector — can run as a single-file llamafile, as a model served locally through Ollama, or as a step inside a local pipeline, with no cloud API key and no per-call billing. Mozilla’s Builders blog describes llamafile as a good option for scalable RAG on local hardware, and MongoDB’s vector search documentation ships a tutorial that builds a complete local RAG implementation without API keys or credits.

This guide is for technically curious beginners and intermediate practitioners who already know roughly what RAG does and now want the embedding step to stop leaving their machine. It covers the two documented local patterns, the model choices and memory figures the sources actually publish, and the points where a local setup behaves differently from a hosted embedding endpoint.

What the embedding step does in a RAG pipeline

Retrieval-augmented generation works by handing a language model relevant text at question time instead of relying on its training data alone. The retrieval half depends on embeddings. Each document chunk is converted into a vector, those vectors are stored in a vector index, and at query time the user’s question is converted into a vector by the same model. The index returns the stored vectors closest to the query vector, and the matching text is passed to the generative model.

Two properties follow from this design, and they drive most local-embedding decisions:

  • The same model must embed both sides. Documents and queries have to be encoded by the same embedding model, because a vector produced by one model is not comparable to a vector produced by another.
  • The vector index is tied to the embedding dimension. A model that outputs 1,024-dimension vectors and a model that outputs 4,096-dimension vectors cannot share one index without re-embedding the corpus.

Neither property is cloud-specific. Both are satisfied by a model running locally, which is why the cloud API is optional rather than structural.

Two documented ways to run embeddings locally

Single-file llamafiles

Mozilla’s post describes releasing recommended embedding models as llamafiles to make local RAG easier to assemble. A llamafile bundles the model and its runtime into one executable file, so the embedding step becomes a matter of running that file rather than installing a serving stack. Mozilla frames the motivation as scale: indexing a large dataset into a vector store is computationally expensive, and doing it on local hardware avoids paying a hosted provider per token. The post also notes that llamafile’s ongoing performance optimizations are part of why it recommends this route.

Ollama plus a local vector store

The MongoDB vector search tutorial takes the second route. It walks through creating a local Atlas deployment with the atlas-local-lib-py library, which manages Docker containers and exposes connection strings programmatically. It then uses Nomic Embed Text as the local embedding model and Mistral 7B as the local generative model, both pulled through Ollama. The tutorial can also be followed with a MongoDB Community or Enterprise cluster that has Search and Vector Search installed.

The tutorial is explicit about one constraint: local Atlas deployments are intended for testing only, and production environments should use a deployed cluster.

Choosing a local embedding model

Mozilla’s post publishes a shortlist of embedding models it considers strong for RAG-relevant tasks, along with license and memory information. The figures below are reproduced from that source.

ModelLicenseMemory usage (fp32, as listed)Embedding size
Salesforce/SFR-Embedding-MistralCC-by-NC26.494,096
intfloat/e5-mistral-7b-instructMIT26.494,096
mixedbread-ai/mxbai-embed-large-v1Apache 21.251,024

Two cautions about reading this table. First, the source’s column is headed with an abbreviation and does not spell out units in the retrieved text, so treat the numbers as the source’s own sizing figures rather than a precise hardware budget. Second, the three licenses are not interchangeable: CC-by-NC is a non-commercial license, while MIT and Apache 2 are permissive. Mozilla labels e5-mistral-7b-instruct its best overall option with a commercial-friendly license, and mxbai-embed-large-v1 its best small model.

Why Mozilla picked them, according to the post:

  • Salesforce/SFR-Embedding-Mistral was produced by additional multi-task finetuning on top of intfloat/e5-mistral-7b-instruct using the training datasets of several tasks in the Massive Text Embedding Benchmark (MTEB).
  • intfloat/e5-mistral-7b-instruct was finetuned on synthetic text embedding datasets generated by another LLM.
  • mixedbread-ai/mxbai-embed-large-v1 is the smallest of the three and the one Mozilla recommends when memory is the constraint.

One scope limit matters if your content is not in English: Mozilla states that the post covers English-language models only, and that while some of the models may be multilingual, non-English performance was not taken into account when selecting them.

A documented local RAG workflow, step by step

MongoDB’s tutorial breaks the job into five stages. This is the documented sequence, not a test run performed for this article:

  1. Create a local Atlas deployment. The tutorial uses atlas-local-lib-py, which handles the Docker containers for you.
  2. Set up the environment. Install the dependencies and point your code at the local connection string the library returns.
  3. Generate vector embeddings with a local model. The tutorial uses Nomic Embed Text.
  4. Create a MongoDB Vector Search index on your data. The index definition has to match the embedding dimension the model produces.
  5. Answer questions with a local LLM. The tutorial uses Mistral 7B for the generation step.

Both models are obtained through Ollama. The general pattern looks like this; check the tutorial itself for the exact model tags it specifies.

# Illustrative pattern only — verify exact model tags in the tutorial
ollama pull <embedding-model>
ollama pull <generative-model>

A runnable Python notebook accompanies the tutorial, and MongoDB documents a LangChain variant of the same local RAG implementation for readers who prefer that framework.

Memory, hardware, and what the published numbers do not cover

The gap between 26.49 and 1.25 in the table above is the single largest practical difference between the recommended models, and it is the reason Mozilla distinguishes a best-overall tier from a best-small tier. But weight memory is not the whole footprint. A running process also needs room for the runtime itself, for input batches, and — if a generative model shares the same machine — for its own context and KV cache. The MongoDB workflow runs an embedding model and a 7B generative model side by side, so plan for both.

The sources do not publish quantization levels, throughput, latency, or benchmark scores for these models. That means the table is useful for shortlisting, not for predicting tokens per second on your hardware. If you need those numbers, measure them on your own machine with your own documents.

Verifying that your embeddings are genuinely local and working

These checks confirm the pipeline is behaving as intended. They are verification steps, not results from a test run.

  • Dimension match. Confirm the vector length returned by your embedding call equals the dimension declared in the vector search index. A mismatch usually surfaces as an indexing or query error rather than silent bad results.
  • Model identity on both sides. Confirm the ingestion script and the query path reference the same model identifier. This is the most common cause of retrieval that returns plausible but irrelevant chunks.
  • No credential is required. A truly local pipeline should complete ingestion and retrieval with no API key, no account, and no outbound request to a model provider.
  • Retrieval smoke test. Store a handful of chunks, then query with a close paraphrase of one of them. If that chunk does not appear near the top of the results, the problem is more likely chunking or index configuration than the embedding model itself.

Common failure points, ordered from simplest to hardest

Treat these as hypotheses to check in order rather than confirmed diagnoses.

  1. The local deployment will not start. Because atlas-local-lib-py manages Docker containers, a stopped or misconfigured Docker daemon is the first thing to rule out.
  2. The model is not available. If the embedding call fails immediately, verify the model was actually pulled and that the local server is running.
  3. The index rejects writes or returns nothing. Recheck the embedding dimension against the index definition, and confirm you did not rebuild the index for one model while querying with another.
  4. Retrieval quality is poor. Before blaming the embedding model, review chunk size, overlap, and whether your query text is being embedded the same way as your documents.
  5. Commercial use is blocked. This is a licensing question, not a technical one. Check the model card for the specific weights you downloaded.

Limitations and tradeoffs to weigh before you commit

  • Language coverage. Mozilla’s model list is explicitly English-only in its evaluation scope. If your corpus is not English, you are outside the evidence the source provides.
  • Production readiness. MongoDB states that local Atlas deployments are intended for testing only and directs production use to a deployed cluster.
  • No published local-versus-cloud quality comparison. Neither source benchmarks local embeddings against a hosted embedding API, so parity should not be assumed.
  • Operational ownership. Model files, runtimes, container lifecycle, and index rebuilds become your responsibility rather than a vendor’s.
  • License verification. Open weights, open source, and commercial-use permission are three different things. Read the applicable license for the exact model you intend to ship.

What to do next

Pick your model by license first and memory second: if you need commercial use, start from the MIT-licensed e5-mistral-7b-instruct or the Apache 2 mxbai-embed-large-v1 rather than the CC-by-NC option. Then follow the MongoDB tutorial end to end with a small sample of your own documents, verify the dimension and model-identity checks above, and only then scale the corpus. The real test of a local embedding setup is whether it retrieves the right chunk from your data — not whether the model runs.

If you run into a specific error while setting this up, sharing your operating system, container runtime, model identifier, and the exact message in the community is usually faster than guessing.

FAQ

Can I use a local embedding model in a commercial product?

It depends on the license of the specific model. In Mozilla's list, intfloat/e5-mistral-7b-instruct is MIT and mixedbread-ai/mxbai-embed-large-v1 is Apache 2, while Salesforce/SFR-Embedding-Mistral is CC-by-NC, which is a non-commercial license. Always check the model card for the exact weights you download.

Do these local embedding models work for non-English text?

Mozilla's post covers English-language models only. It notes that some of the listed models might be multilingual, but non-English performance was not taken into account when the models were selected, so you should verify quality on your own language data.

Is a local Atlas deployment suitable for production?

No. MongoDB states that local Atlas deployments are intended for testing only and that production environments should use a deployed cluster.

Do I have to re-embed my whole corpus if I switch embedding models?

Yes, in practice. Vectors from different models are not comparable to each other, and the models may also output different dimensions, so the stored vectors and the index generally have to be rebuilt with the new model.

References

  1. https://builders.mozilla.org/using-llamafiles-for-embeddings-in-local-rag-applications/
  2. https://www.mongodb.com/docs/vector-search/tutorials/local-rag

Related terms

Embedding
A fixed-length numeric vector that represents a piece of text, produced by an embedding model so that similar meanings land near each other in vector space.
Vector search index
A data structure that stores embeddings and returns the stored vectors closest to a query vector. Its definition is tied to the embedding dimension of the model that produced the vectors.
llamafile
A single executable file that bundles a model and its runtime, so the model can be run locally without installing a separate serving stack.
Ollama
A local model runner used by MongoDB's tutorial to download and serve both the embedding model and the generative model.
CC-by-NC
A Creative Commons license variant that permits non-commercial use. Mozilla lists Salesforce/SFR-Embedding-Mistral under this license, which makes it unsuitable for commercial products without separate permission.

Leave a Reply

Your email address will not be published. Required fields are marked *