Retrieval (RAG)

What changes when the answer comes from retrieved passages

In a plain chat request the model weights are the only source of facts. In RAG the application supplies them: it searches a corpus it controls, puts the best passages in the prompt and asks the model to answer from those. The ceiling on answer quality is set by what retrieval returned, so the embedding model and the reranker decide more about the final answer than one tier of generator does.

The split is visible in the API. Only the last stage is an OpenAI-compatible chat completion; embedding and reranking are their own calls with their own models. LLM EU catalogues the pattern as the task `rag`, and the router accepts it in the request body as `"task": "rag"` next to the standard chat fields.

The four stages, and the cards the catalogue recommends for them

Index once: split documents into chunks and embed each one. Query at runtime: embed the question, take the nearest vectors, reorder those candidates with a cross-encoder, and pass the survivors to the generator. Four stages, three of which can be a different model.

Two embedding cards are on the rag list. multilingual-e5-large is a 560M-parameter XLM-RoBERTa model under the MIT licence with around 100 declared languages; its card uses a 512-token maximum sequence length and has no instruction prefix, so documents and queries must be embedded the same way. embeddinggemma-300m is smaller at 308M parameters and takes 2048 tokens, with a 768-dimensional output truncatable to 512, 256 or 128 dimensions; it ships under Google’s Gemma terms of use, not an OSI licence.

Both rerankers are cross-encoders: they can neither embed an index nor generate text, only score a query against a passage, so they sit after the vector search. gte-multilingual-reranker-base has 305M parameters, an 8192-token input limit and 75 declared languages under Apache-2.0, and its card notes the examples truncate at 512 tokens unless configured otherwise. llama-nemotron-rerank-1b-v2 has 1.24B parameters under OpenMDW-1.1 and was evaluated on 26 languages, up to 8192 tokens, with no published benchmark score.

For the generator, command-r-08-2024 is the card built for this pattern: 32.3B parameters, a 128K-token context and answers that cite the retrieved documents — under CC-BY-NC-4.0, which excludes commercial serving. command-a-plus-05-2026 is the Apache-2.0 counterpart with 48 declared languages. Scanned paper goes through glm-ocr first, a small MIT document-OCR model tagged rag.

Chunk to the smallest limit in the chain

The chain’s three limits are 512, 2048 and 8192 tokens, and the smallest governs. Embed with multilingual-e5-large and a 900-token chunk is truncated during embedding — silently, however good the generator is. Set chunk length from the embedding card, not the generator’s context window.

A reranker sees query and passage in one input, so budget for both halves and decide the truncation deliberately. Reranking is a forward pass per candidate: feed it a shortlist from the vector search, not the whole index, and hand the generator only the passages that survived. Language lists matter when query and document differ: gte-multilingual-reranker-base declares 75 languages and llama-nemotron-rerank-1b-v2 was evaluated on 26, so a pair outside a card’s list is unverified for that model rather than wrong, and no language list replaces a test on your own corpus.

A request the router understands

This is the shape /docs/quickstart prints: one POST to /v1/chat/completions, a bearer key, and a namespaced llmeu object beside the standard fields. model `llmeu-auto` lets the router choose; residency narrows the candidate set and task tells the scorer what the request is for. Official SDKs ignore unknown fields.

The response adds an llmeu block: trace_id, model_used, model_version, endpoint_id, inference_region, sovereignty_class, route_reason, retention and cost_estimate_usd. route_reason is a sentence, not a code — for a coding request it reads “auto routing for task "coding" + residency eu-hosted + qwen3-coder-30b @ de-ber (EU-owned, EU-operated); 4 endpoint(s) excluded; partner endpoints not allowed.”

curl https://YOUR-API-ORIGIN/v1/chat/completions \
  -H "Authorization: Bearer $LLMEU_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llmeu-auto",
    "messages": [{"role": "user", "content": "Answer only from these passages and cite them.\n\n[1] …\n\nQuestion: …"}],
    "llmeu": {"residency": "eu-hosted", "task": "rag"}
  }'

Residency: one boundary for the whole pipeline

The rule for a retrieval pipeline is that retrieval, embedding, reranking and generation stay inside one boundary. It is easy to break one layer down: the query reaches the embedding model, the passages reach the reranker, the assembled prompt reaches the generator. If any of those runs under a different jurisdiction than you promised, the promise is already broken — even though the chat call itself was routed inside the EU.

The rule the router applies is narrower: residency is a filter applied before scoring, never a preference applied after. If no endpoint satisfies it the request returns 409 no_endpoint_for_policy, failover cannot step outside the filtered set, and catalogue_unknown is not an addressable residency at all, because a row that admits it does not know where it runs.

Two limits belong next to that. The classes sort ownership, operation and location — not a legal protection against disclosure — and LLM EU is a private company, so nothing here issues a compliance status. Hosting is platform state read from endpoint rows: the embedding route in this build answers with placeholder vectors and marks its response llmeu.mock. GET /v1/models lists only what a key may actually call, and a card with no live endpoint is labelled “catalogue only — not hosted on LLM EU yet”.

Answers you can check afterwards

Two habits make a deployment auditable. Make the generator cite: it should answer only from the supplied passages and say so when they lack the answer, which the command-r-08-2024 card names explicitly. Keep the trace: every response carries trace_id, and GET /platform/v1/traces/{id} returns the stored row.

Retention belongs to that record. Zero is the default and the only implemented mode: prompt and completion bodies are never written to the database and those columns stay null. A policy may store a 24h, 7d or 30d choice, which does not enable storage — a request above the limit returns 403 retention_exceeds_policy, and any allowed effective retention other than zero returns 501 retention_not_implemented. A trace is the platform’s own operational record, not geolocation and not a conformity assessment, and the cards above carry no retrieval benchmark run by LLM EU.