Dependency-free tracing for RagLeap. Standard library only (dependencies = []).
Version 0.1.0: spans, a tracer, two exporters, and an adapter for ragleap-rag's
observability hooks (RagTraceAdapter).
from ragleap_trace import Tracer, JsonlExporter, traced
tracer = Tracer([JsonlExporter("traces.jsonl")])
with tracer.span("retrieve", top_k=5) as s:
s.set_attribute("hits", 4)
@traced(tracer)
async def answer(...): ...
Using it with ragleap-rag
from ragleap_trace import Tracer, JsonlExporter, RagTraceAdapter
tracer = Tracer([JsonlExporter("traces.jsonl")])
adapter = RagTraceAdapter(tracer)
hooks = adapter.hooks() # lists keyed on_ingest / on_query / on_answer
hooks() returns one-item lists keyed on_ingest / on_query / on_answer, because ragleap-rag
takes a list of callables for each (read from its RagLeap.__init__ and README). Pass them with
RagLeap(..., **adapter.hooks()), or merge them into your own lists. Spans recorded: rag.ingest,
rag.query, rag.answer.
- Only allow-listed values are copied: counts, booleans, provider name, token numbers and the query length. Query text, answer text and file names are never copied.
query_rewriteis recorded only ascontextual,hyde,multi_queryorother.- The events have no request id and no start event.
rag.answerduration_msis the time from the query event to the answer event: generation forask(), and forask_stream()also the consumer's time between tokens. Retrieval time is not measurable from events; wrap the call intracer.span("request")to time the whole request (the rag spans then nest under it). - Pairing is per thread and per asyncio task. Two streams interleaved in one thread or task can pair an answer with the wrong query. A stream stopped early fires no answer event.
- Handlers never raise; a malformed event is counted in
tracer.dropped.
Integration tests
integration/ runs the adapter against a real RagLeap with a real Postgres (pgvector), a fake
embedder and a fake generator (copied from ragleap-rag's own tests, so no paid API is called). It needs
ragleap-rag installed and RAGLEAP_TEST_DATABASE_URL. It covers ask(), ask_stream() run to the end,
a stream stopped early (no on_answer fires; the next query replaces the pending one), privacy through
real events, and a contract check that the event keys the adapter relies on still exist.
python scripts/integration_break.py breaks the adapter, the tracer and ragleap-rag itself in temp copies
and every break must make the tests fail. These tests are not part of the wheel.
What it guarantees (each has a test and a break proof)
- A span is recorded even when the body raises; the exception still propagates.
- Only the exception class name is recorded, never its message.
- Only
str,int,float,boolattribute values are kept; objects are dropped, neverrepr()'d. NaN/inf are dropped. - Limits: name 128, key 64, 32 attributes per span, strings cut at 256 characters. Dropped attributes are counted in
dropped_attributes. - Keys such as
prompt,answer,query,file_name,password,tokenare never recorded (case-insensitive,-equals_). - Exporter errors never reach your code; they are counted in
tracer.dropped. JsonlExportercreates files with mode 0600 and refuses an existing file readable by group/others.
What it does not do (honest limits)
- The key denylist is defense in depth, not a guarantee: a value under an innocent key is recorded as you gave it. Do not put secrets or personal data in attributes.
- No retention or rotation in 0.1.0: the JSONL file grows until you remove it.
- No opt-in content capture, no sampling, no network exporters, no multi-tenant tagging.
- Verified in CI (3.10, 3.12): the core and the adapter's unit tests. The adapter is also verified against a real RagLeap in CI (Python 3.11, Postgres 16 with pgvector, fake embedder and generator; see Integration tests). Not verified: real LLM providers (the usage keys in the tests come from a fake shaped like the real providers), vector backends other than pgvector, spans handed between threads, Windows file permissions.
Threat model (0.1.0)
What ragleap-trace protects, against what, and what it does not. It is a local library: no network, no server, no stored credentials.
Assets: the content that flows through your pipeline (prompts, answers, file names, user text), the availability of your application, and the trace file.
Trust boundary: your application code and ragleap-rag are trusted callers, but they can pass sensitive data by accident. Anyone who can read the trace file or the process memory is outside what this library defends.
| Threat | Mitigation in 0.1.0 | Evidence | Residual risk |
|---|---|---|---|
| Content recorded by accident | Denylisted keys are never recorded; the ragleap-rag adapter copies only an allow-list (counts, booleans, provider name, token numbers, lengths); only str, int, float and bool values are kept; exceptions are recorded by class name only |
Unit and integration tests, each with a break proof | A value under an innocent key is recorded as given |
| Tracing breaks your application | Exporter and handler errors are swallowed and counted in tracer.dropped; the caller's own exception always propagates unchanged |
Tests and break proofs; real RagLeap integration tests | Exporters run in the caller's thread: a slow exporter slows the caller (no timeout, no background queue) |
| Unbounded data per span | Name 128, key 64, 32 attributes, strings cut at 256 | Tests and break proofs | The JSONL file has no size limit or rotation |
| Trace file readable by others | Created with mode 0600; an existing group- or world-readable file is refused | Tests and break proofs on POSIX | Windows permissions are not verified; no encryption at rest |
| Events paired with the wrong query | Pairing is per thread and per asyncio task | Thread and asyncio tests; real RagLeap integration tests | Two streams interleaved in one thread or task can pair wrongly |
| Silent data loss when ragleap-rag changes an event | Contract tests fail if an event key the adapter relies on disappears | Integration contract tests; breaks simulate renamed keys | Only the keys the adapter uses are checked |
| Supply chain | No runtime dependencies (dependencies = []); the CI actions are pinned to commit SHAs; releases are published from the release workflow through PyPI trusted publishing (no stored token) |
pyproject.toml; the workflow files; the TestPyPI dry run published through trusted publishing, and its files carry a provenance record naming the repository, workflow and environment |
You still trust the PyPI account and the GitHub repository; there is no signature beyond that provenance record |
Out of scope in 0.1.0: hostile code in the same process, multi-tenant separation (no tenant tagging), retention policies, network exporters, sampling, and opt-in content capture (which would need ragleap-security and fail closed).
Development
pip install -e ".[test]"
pytest
python scripts/mutation_check.py # breaks each protection in a temp copy; every break must be caught