token-compression

Token Compression

Instruction-only compression discipline. No proxy, MCP server, or library ships with this skill. If you adopt an external compressor, verify it on your data before trusting it.

Procedure

Measure first. Count characters or tokens for the raw result. If it is

already small enough to reason over, stop.

Classify the content. Structured JSON, logs, prose, code, or a mix. The

class determines the safe reduction.

Prefer lossless structure. Project JSON fields, filter log levels, keep

head and tail with an explicit marker, deduplicate repeated lines.

Summarize last. Semantic summaries lose detail; use them only when the

exact wording is not load-bearing, and keep a pointer to the raw artifact.

Keep decisive evidence verbatim. Error messages, stack traces, request

IDs, file paths, and the lines a decision depends on survive unchanged.

Record what was dropped. State the reduction method and the volume in

the output so a later reader can judge completeness.

Re-check the answer. Confirm the compressed form still supports every

claim you make from it.

Choosing a tool

If you use an external compressor, test it against a sample where you know the correct answer. Reject it if it drops error lines, breaks JSON keys, or cannot reproduce the raw input. Keep the raw output until the task is finished.

Good candidates

API responses where only a few fields matter
Logs where the error level and adjacent context suffice
Large diffs where file names and changed hunks matter more than full context
Retrieved chunks that repeat the same passage

Poor candidates

A short output you have not read yet
Content you cannot replay, such as a one-shot stream
Data where exact wording or formatting is the evidence

Worked example

A log compressed without losing the error line that drives the fix.

raw: 8,200 lines
kept: timestamp, ERROR lines, request ids, final summary
dropped: INFO/DEBUG lines, duplicate stack frames
compressed: 74 lines; pointer: artifacts/logs/run-42.gz

Anti-patterns

Compressing before reading the error.
Using compression instead of backend pagination or a narrower query.
Summarizing streaming output that cannot be retrieved again.
Reporting a reduction percentage without checking answer quality.
Discarding the raw artifact while the task still depends on it.

Verification Checklist

[ ] The raw size and the reduction method are recorded. (step 1)
[ ] Every error line and identifier survives the compressed form. (step 5)
[ ] The compressed output reproduces the answer you relied on. (step 4)
[ ] The raw artifact remains available until the task closes. (step 4)

Install and discovery

Contents