All numbers below come from a real, timed run against real infrastructure
— never projected or estimated. Full methodology, raw results, and the
runnable script live in
packages/ragleap-rag/benchmark/.
- Run date (UTC): 2026-08-03
- ragleap-rag version: 0.11.1
- Corpus: 40 synthetic documents (self-contained, no copyright concerns)
- Queries per config: 15
Ingestion throughput
| Config |
Docs ingested |
Time (s) |
Docs/sec |
gemini_pgvector |
40/40 |
36.3 |
1.10 |
gemini_faiss |
40/40 |
35.4 |
1.13 |
ollama_local |
40/40 |
112.7 |
0.36 |
Query latency
| Config |
Queries run |
P50 (ms) |
P95 (ms) |
Mean (ms) |
gemini_pgvector |
8 of 15 (partial — see caveats) |
3934 |
6530 |
4253 |
gemini_faiss |
0 of 15 (see caveats) |
— |
— |
— |
ollama_local |
15 of 15 |
67028 |
84883 |
47244 |
Cost per query
| Config |
Avg cost (USD) |
Pricing available? |
gemini_pgvector |
$0.0033 |
Yes |
gemini_faiss |
— |
No data (see caveats) |
ollama_local |
— |
No pricing table entry (local, effectively $0) |
Honest caveats — read before drawing conclusions
- Gemini free tier has two separate caps: 5 requests/minute and 20
generation requests/day. The daily cap was the binding constraint —
it was exhausted partway through
gemini_pgvector's queries, before
gemini_faiss could run any at all. The 8 gemini_pgvector numbers
above are real, but a partial sample (n=8, not 15).
gemini_faiss has zero completed queries — not because FAISS
failed (ingestion succeeded 40/40), but because the shared daily quota
ran out before it got a turn. A follow-up run is needed once quota
resets or a paid tier is used.- Grok/xAI was excluded entirely — the API key available during this
run failed authentication (confirmed independent of
ragleap-rag via a
raw API call).
- Ollama's latency (P50 ~67s) is real, not a bug — `qwen2.5:0.5b
running full retrieval + generation on CPU-only hardware, no GPU. This
is the genuine tradeoff of fully local, zero-cost inference on modest
hardware.
Reproduce this yourself
cd packages/ragleap-rag/benchmark
export GEMINI_API_KEY="..."
python3 run_benchmark.py
python3 render_benchmarks_md.py
Real numbers only get published here after a real run — see
Phase 6 of ROADMAP.md
for the standard this follows.
RagLeap