RagLeap
Blog
2026-11-09

ragleap-optimizer: measure token savings on your own cases before you trust them (alpha, not independently audited)

ragleap-optimizer 0.1.0 is an alpha Python library that reduces the token use of LLM requests, and reports what each step changed and whether it can lose information. It is not independently audited. Source: its README and changelog at the 0.1.0 tag.

What is ragleap-optimizer?

The idea in the README: “same quality with fewer tokens” is something one application can measure, not something a library can promise. So Optimizer runs steps over a request (a dict with a list of messages, and optionally tools, system and other fields) and returns the new request plus a Report with the tokens before and after, per step. A step whose result is longer than its input is discarded, and your input is never changed. Install it with pip install ragleap-optimizer; it needs Python 3.10 or newer and depends on ragleap-security.

How do I use it?

This is the example from the README:

from ragleap_optimizer import Optimizer, ExactCache, evaluate, steps
from ragleap_security import Scope

opt = Optimizer([steps.dedupe_context(), steps.minify_whitespace(), steps.trim_tool_output(max_tokens=800)])
out = opt.run(request)
print(out.report.tokens_before, out.report.tokens_after, out.report.lossy)

cache = ExactCache(ttl_seconds=3600)
scope = Scope("acme", subject="alice")
hit, answer = cache.get(scope, out.request)
if not hit:
    answer = my_llm_call(out.request)
    cache.put(scope, out.request, answer)

table = evaluate(cases, run=my_llm_call, optimizer=opt, judge=my_judge)
print(table.summary())

Which steps lose information?

  • dedupe_context() removes a long message that is an exact copy of a later one. System messages and short messages are never touched. Nothing is lost.
  • minify_whitespace() turns three or more newlines into two, and writes a message that is entirely JSON compactly. Only spacing changes.
  • stable_prefix() puts system messages first and tool JSON in sorted key order, so a provider’s prompt cache can recognise the start of the request.
  • trim_tool_output(max_tokens) keeps the start and end of an oversized tool message with an explicit marker. It loses the middle, so the report says lossy=True whenever it cuts.

What about the cache, retry and evaluate?

ExactCache is in memory and returns an answer only for the same scope and the same request: model, messages, tools and every other field are part of the key. It needs a Scope from ragleap-security, entries expire, and entries can be erased per scope. retry and with_fallback give bounded retries with doubling waits and a deadline, and try several callables in order. evaluate runs your cases with and without the optimizer, scores both answers with your judge, and reports the input tokens saved and the quality change with a 95% interval, listing every case that got worse. With fewer than 30 cases it gives no savings figure.

How was it tested?

Different measures, kept apart. Our Tests page counts 6 test files and 86 test functions on the main branch (a count, not a pass count). The changelog records 156 tests passing and all 84 mutations killed in the author’s sandbox (Python 3.12) and on the VPS (Python 3.10.12); CI on pull request #1183 passed all 52 checks, including the optimizer tests on Python 3.10, 3.11 and 3.12; and a lossless pipeline never made a request longer on 200 random requests.

What does it not do?

  • No speed or token-saving number has been measured, and the library makes no savings claim. Token counts are estimates (one token per 3 UTF-8 bytes, which over-counts English); pass counter= to use a real tokenizer.
  • The cache lives in one process and stores whatever you put in it. A wrong answer is served until it expires, so use a short ttl_seconds and never cache answers that depend on the current time or on private context that is not in the request.
  • evaluate is only as good as your judge and your cases, and counts input tokens only.
  • Not verified: other message shapes than the OpenAI-style messages list, several processes sharing one cache, and any real tokenizer. Not in 0.1.0 (planned, design issue #1164): model routing, prompt compression, a semantic cache, secret redaction and framework adapters.

More: the library page, the release notes and the source.

Back to the blog