San Juan Riparian Watch
← back to the story
// the-agent · Quartzose RAG

A RAG agent that can't make things up

The chat at the end of the story runs on Quartzose, a production retrieval-augmented pipeline. Its whole design goal is a single rule — no source, no claim — enforced server-side, so every sentence it returns is pinned to a document you can open.

The request path

A question doesn't go straight to a language model. It runs a pipeline where each stage can refuse, correct, or short-circuit:

guard
QueryShield
prompt-injection screen
cache
semantic cache
vector hit → skip the LLM
retrieve
hybrid + rerank
dense + BM25
grade
CRAG
relevant? re-retrieve if not
generate
OLMo
grounded, cited
scrub
AnswerScrubber
PII out

Hybrid retrieval

Dense embeddings catch meaning; sparse keyword matching catches exact terms a manager actually types (species names, reach names, metrics). Using both, then reranking, beats either alone.

// dense

Arctic embeddings

Snowflake arctic-embed (1024-d) served over Ollama, indexed in Qdrant. Query and corpus share the exact embedder, so the index and the query can't silently disagree.

// sparse

BM25, then rerank

A FastEmbed BM25 sparse vector runs alongside the dense one; a cross-encoder reranker (ONNX, no GPU) orders the pooled candidates before anything reaches the model.


The guard pipeline

A public LLM endpoint is an attack surface. Three guards wrap generation:

GuardStops
QueryShieldDirect prompt injection in the user's question ("ignore your instructions…").
PassageShieldIndirect injection — malicious instructions hidden inside a retrieved document, the subtler attack.
AnswerScrubberPII leaking into the response — emails, keys, personal identifiers — scrubbed before it's returned.
Grounding is server-side The "no source, no claim" contract isn't a polite instruction in the prompt — it's enforced by the pipeline. If retrieval comes back empty or irrelevant, the honest non-answer is what ships.

Correction, memory, and cost

// CRAG

Self-correcting retrieval

Retrieved passages are graded for relevance. Weak retrieval triggers decomposed re-retrieval rather than answering from thin context — the model isn't handed junk and asked to sound confident.

// memory

Multi-turn, per session

A Redis-backed conversation window makes follow-ups work ("and on the river reaches?") without leaking one visitor's thread into another's.

// cache

Semantic cache

A Redis vector cache returns near-duplicate questions (the suggested chips, FAQs) with no LLM call at all — most real traffic is free, which is what keeps a public demo cheap.

// seam

Model-agnostic LLM

Generation goes through one OpenAI-compatible seam, so the model swaps by config — Ai2's OLMo when a host serves it, another open model otherwise — with a per-IP rate limit and a hard provider spend cap in front.

FastAPI · Haystack Qdrant hybrid Redis cache + sessions Ollama arctic embeddings OLMo / OpenAI-compatible rate limit + daily budget