RagLeap
Blog
2026-11-10

ragleap-rag 0.15.0: evaluate retrieval with ranking metrics, and what changed for Python 3.11

ragleap-rag is a Python RAG library: PostgreSQL with pgvector, your own API keys, nothing routed through us. Version 0.15.0 adds ranking metrics to its evaluation and requires Python 3.11 or newer. Source: its README and changelog at the 0.15.0 tag.

What is ragleap-rag?

The README describes hybrid dense and sparse retrieval, streaming, automatic provider fallback and reported token usage. The quickstart needs a PostgreSQL database with the pgvector extension and a Gemini API key for embeddings. Install it with pip install ragleap-rag.

How do I ingest text and ask a question?

This is the README quickstart:

from ragleap import RagLeap, ProviderConfig, EmbeddingConfig

rag = RagLeap(
    database_url="postgresql://user:pass@localhost/mydb",
    embedder=EmbeddingConfig(provider="gemini", model="models/gemini-embedding-001", dimensions=3072, api_key="your-gemini-key"),
    primary=ProviderConfig(provider="gemini", model="gemini-3.6-flash", api_key="your-gemini-key"),
)
rag.init_schema()  # one-time, idempotent — safe to call every run

rag.ingest_text("handbook.txt", "Employees get unlimited PTO and a $500/year learning budget.")

answer = rag.ask("How much PTO do employees get?")
print(answer["answer"])

How do I check retrieval quality?

rag.evaluate(test_cases) runs a labelled test set through ask() and reports deterministic signals: whether the expected document showed up in the sources, how many expected keywords appear in the answer, and how many of those also appear in the chunks the answer cited. It is not an LLM-as-judge framework. This is the README example; the numbers in its comments are an illustration, not a measurement:

result = rag.evaluate([
    {"query": "What's the refund policy?", "expected_document": "policy.pdf", "expected_keywords": ["30 days", "receipt"]},
    {"query": "How do I reset my password?", "expected_document": "faq.pdf", "expected_keywords": ["settings", "email link"]},
])

print(result["retrieval_hit_rate"])       # 1.0
print(result["keyword_coverage_rate"])    # 0.75
print(result["groundedness_rate"])        # 0.9
print(result["results"][0])               # per-case detail: query, answer, sources, hits

What did 0.15.0 add to evaluation?

  • Ranking metrics for cases that list relevant_documents: hit@k, recall@k, precision@k, reciprocal rank and nDCG@k, over the document-level ranking of the chunks sent to the model. metric_ks picks the cutoffs (default 1, 3, 5 and 10).
  • Per-case latency and token usage, and a summary with mean, min, p10, p50, p95 and a fail rate below fail_below.
  • An optional support_checker(answer_text, citations) callable for your own NLI model or LLM judge; its failures are recorded per case, not raised.
  • compare_runs(baseline, candidate, tolerance) to flag regressions between two runs.
  • Existing evaluate() keys are kept; the new keys are additions. No judge is bundled.

What else changed, and what about Python 3.10?

  • Python 3.10 is no longer supported: requires-python is now >=3.11. The changelog says Pillow 13, which fixes open Pillow security alerts, needs Python 3.11 or newer. Users on Python 3.10 keep working with ragleap-rag 0.14.0.
  • cheaperinference is in the provider base-URL table.
  • answer["sources"] now lists each document once, in the order its first chunk was sent to the model.

What is not verified?

The README says the Pinecone, Weaviate, Qdrant and Milvus backends are not live-verified: no account was available, so treat them as best-effort. Our Tests page counts 34 test files and 341 test functions on the main branch (a count, not a pass count).

More: the library page, the release notes, the RAG chatbot guide and the source.

Back to the blog