RagLeap
Tests

Tests

This page lists every automated test and benchmark in the project: its name, what it checks, what it uses and its real latest result. It is generated from the repository (main, commit 86a17cb, 2026-11-20) and its CI logs, never typed by hand. Results below come from the latest successful CI run on main: run 44686139027, 2026-11-20, commit 86a17cb.

Test files: 490. Test functions: 5492. Suites: 23. Functions described by their authors: 239. For the rest the page shows the test name in words and says so. Open a suite to see each test.

Test suites

SuiteFiles / testsUses (from ci.yml)Latest CI result
ragleap-core (the app)136 / 1581pytest, psql, service container pgvector/pgvector:pg16passed 2136 passed, 13 skipped, 0 failed
ragleap-agents11 / 176pytestpassed 251 passed, 0 skipped, 0 failed
ragleap-ansible1 / 1not stated in ci.ymlno CI job
ragleap-app-chart1 / 1not stated in ci.ymlno CI job
ragleap-graph11 / 190pytest, service container neo4j:5passed 240 passed, 14 skipped, 0 failed
ragleap-integrations9 / 303pytestpassed 626 passed, 0 skipped, 0 failed
ragleap-memory18 / 286pytestpassed 596 passed, 0 skipped, 0 failed
ragleap-observability1 / 1not stated in ci.ymlno CI job
ragleap-ops1 / 1not stated in ci.ymlno CI job
ragleap-optimizer7 / 120pytestpassed 217 passed, 0 skipped, 0 failed
ragleap-phone8 / 103pytestpassed 244 passed, 3 skipped, 0 failed
ragleap-physical40 / 410pytestpassed 787 passed, 16 skipped, 0 failed
ragleap-rag36 / 375pytest, service container pgvector/pgvector:pg16passed 486 passed, 18 skipped, 0 failed
ragleap-security10 / 169pytestpassed 679 passed, 1 skipped, 0 failed
ragleap-terraform1 / 1not stated in ci.ymlno CI job
ragleap-tools21 / 337pytestpassed 526 passed, 0 skipped, 0 failed
ragleap-trace2 / 47not stated in ci.ymlno CI job
ragleap-vectorstores10 / 109pytest, service container redis/redis-stack-server:latest, service container redis:7-alpinepassed 81 passed, 32 skipped, 0 failed
ragleap-voice33 / 365pytestpassed 535 passed, 3 skipped, 0 failed
ragleap-rag (Java port)62 / 534Maven, service container pgvector/pgvector:pg16passed 470 passed, 64 skipped, 0 failed
ragleap-graph (Java port)43 / 196Mavenpassed 144 passed, 52 skipped, 0 failed
ragleap-vectorstores (Java port)8 / 28Mavenpassed 18 passed, 10 skipped, 0 failed
@ragleap/rag (npm, TypeScript)20 / 158not stated in ci.ymlno CI job

End-to-end smoke tests (Docker)

The full stack is built and started with Docker Compose, then exercised. Result of each step in the latest run: CI job.

  • Run actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 success
  • Create .env for CI success
  • Build and start stack success
  • Wait for services to be healthy success
  • Wait for Neo4j to actually accept connections success
  • Smoke test - health check success
  • Smoke test - upload and chat (skipped if no CI API key) success
  • Smoke test - knowledge graph (skipped if no CI API key) success
  • Dump logs on failure skipped
  • Tear down success

Other CI workflows

These workflows are separate from the main test run above. Each description is copied from the workflow file’s own header comment, and the last column is the latest successful run on any branch. A green run means that workflow’s own steps finished, not that the whole project passes.

Workflow fileWhat its header saysLatest successful run
ansible-container-test.yml
ansible-container-test
Runs the ragleap-ansible roles twice inside a throwaway ubuntu:22.04 container with the package's own tests/container_test.sh: idempotence, the deploy user, opt-in SSH hardening (and the refusal without a key, and the rollback of a bad sshd config) and the opt-in firewall. It never touches the runner itself. Manual (workflow_dispatch) or when a PR changes this file or the Ansible package. Not a required check.2026-11-09, pull_request on test/fw-break-1, run
ansible-vm-firewall-test.yml
ansible-vm-firewall-test
Runs the ragleap-ansible FIREWALL role on a REAL host (a GitHub-hosted runner, a disposable VM) and tests enforcement from a container on the Docker bridge: an opened port must stay reachable, a port that was not opened must be blocked. The script refuses to run anywhere except a GitHub-hosted runner. Manual (workflow_dispatch) or when a PR changes this file, the script or the Ansible package. Not a required check.2026-11-09, workflow_dispatch on main, run
ansible-vm-test.yml
ansible-vm-test
Runs the ragleap-ansible roles on a REAL host: a GitHub-hosted runner, which is a disposable VM. Base role twice, opt-in SSH hardening twice (real reload handler, real systemd), then a real key login against the hardened sshd. The script refuses to run anywhere except a GitHub-hosted runner. Manual (workflow_dispatch) or when a PR changes this file, the script or the Ansible package. Not a required check.2026-11-09, pull_request on test/fw-break-1, run
chart-install-test.yml
chart-install-test
Install, upgrade and uninstall test for the three RagLeap Helm charts on a throwaway kind cluster. Manual (workflow_dispatch) or when a PR changes this file or the test script. Not a required check.2026-11-19, workflow_dispatch on test-rel-observability-0.8.6, run
chart-upgrade-test.yml
chart-upgrade-test
Upgrade test for the ragleap-ops chart: install a PUBLISHED version from PyPI on a throwaway kind cluster, write data, helm upgrade to the chart in this checkout, and check that the databases were not restarted and the data survived. Manual (workflow_dispatch, with a from_version input) or when a PR changes this file or the script. Not a required check.2026-10-26, pull_request on test/upgrade-k8s-matrix, run
java-ansible-test.yml
Java Ansible Resource Parity
no description in the file header2026-11-09, pull_request on test/fw-break-1, run
java-app-chart-test.yml
Java App Chart Parity
no description in the file header2026-10-28, pull_request on gpt-java-app-chart-parity-1028, run
java-observability-test.yml
Java Observability Resource Parity
no description in the file header2026-11-19, pull_request on obs-0.8.6-images, run
java-ops-test.yml
Java Ops Resource Parity
no description in the file header2026-11-11, pull_request on test/trivy-break-1, run
java-terraform-test.yml
java-terraform-test
Builds and tests the Java jar of ragleap-terraform and checks that it holds the same files as the PyPI wheel. Never publishes. Manual (workflow_dispatch) or when a PR touches the Java folder, the Python package or this check. Not a required check.2026-11-03, pull_request on docs/java-jars-terraform-ansible, run
obs-k8s-test.yml
obs-k8s-test
Kubernetes test of the ragleap-observability SLO alerts and AlertManager wiring on a throwaway kind cluster. Manual (workflow_dispatch) or when a PR changes this file or the test folder. Not a required check.2026-11-19, workflow_dispatch on test-rel-observability-0.8.6, run
ops-app-pods-test.yml
ops-app-pods-test
DIAGNOSTIC EXPERIMENT: can the ragleap-ops chart's app and voice pods pull the private ghcr.io/ragleap/ragleap-app image with the workflow token and become Ready on a throwaway kind cluster? The job is green whatever the outcome: read the VERDICT line in the log. Manual (workflow_dispatch) or when a PR changes this file or the script. Not a required check.2026-10-24, workflow_dispatch on main, run
ragleap-trace-test.yml
ragleap-trace-test
no description in the file header2026-11-06, pull_request on docs/ragleap-trace-0.1.0-release-prep, run
velero-test.yml
velero-test
Backup and restore test: Velero with a RustFS S3 stand-in on a throwaway kind cluster, using the ragleap-ops chart. Manual (workflow_dispatch, with version inputs) or when a PR changes this file or the test script. Not a required check. The velero-drill job runs all three cluster and storage combinations every Wednesday 03:30 UTC (schedule), or on demand with drill=true.2026-11-18, schedule on main, run

Benchmarks and evaluations

Java port: cross-backend retrieval benchmark and failure-mode matrix

Answers issue #565: do the Java vector backends return comparable results, and how do they behave on edge cases? Run on 2026-10-03. Every number below is generated by generate_report.py from the raw files in results/.

Result: Summary Retrieval is identical on FAISS, pgvector, Qdrant and Weaviate: the same top-10 documents for all 60 queries (overlap@10 = 1.000 against an exact float64 scan), and each finds the source paragraph exactly as often as the exact scan.…

Python ragleap-rag benchmark

All numbers below come from a real, timed run against real infrastructure — never projected or estimated. See run_benchmark.py for the exact methodology; re-run it yourself to reproduce.

Result: see the report

AI Employee role evaluations

Manual, on-demand eval harness for sensitive-domain AI Employee roles (legal_intake, healthcare_intake, insurance_agent, compliance_officer as of this first batch). Makes real /chat calls against a running instance to spot-check that a role's guardrails (never giving legal/medical/coverage/compliance rulings, always escalating) actually hold in practice -- not just that its personality prompt says it will.

Result: see the report

Other benchmark data

The benchmarks/ folder contains: THROUGHPUT.md, http_throughput.py, stress_office, stt-neutral-10s, voice. Open the folder.

Compiled by script from the public repository, its CI logs and the GitHub API. To correct something, open an issue.

All tests