Open source project / 2026
LogShrink
A streaming ANSI-aware log compressor for CI/CD runs that collapses noisy repetitive lines while preserving error stack traces and exit codes.
Context
Modern CI/CD pipelines generate tens of thousands of log lines: build steps, download progress bars, repeated polling messages, and compiler noise overwhelm debugging.
The problem
Massive CI logs cause browser slowdowns, exhaust artifact storage limits, and bury root causes under Megabytes of redundant status updates.
Approach
LogShrink processes piped stdout/stderr streams in constant memory, strips transient ANSI escapes, deduplicates continuous identical patterns, and aggregates consecutive lines.
System architecture
Pure Python standard library streaming pipeline with regex pattern collapse, zero intermediate disk buffering, and passthrough signal handling.
Key decisions
Failsafe transparency: always preserves panic stacks, tracebacks, and non-zero exit codes unmodified even under aggressive compression heuristics.
Outcome
Available on PyPI as logshrink. Reduces CI log sizes by 70–90% on verbosity-heavy test runners without sacrificing diagnostic fidelity.
Learnings
Treating log streams as data pipes rather than text blobs enables real-time deduplication with predictable CPU and memory overhead.