Profiling a Python code analysis pipeline: 157s → 12s by chasing bottlenecks

I recently spent some time profiling a Python code analysis pipeline that processes large repositories into a structured knowledge bundle. Rather than trying random optimizations, I profiled each stage, fixed the biggest bottleneck, and then profiled again. What surprised me was that every optimization exposed a completely different bottleneck. The benchmark was run on a real workspace containing: 23 GB source tree ~18,000 source files ~41,000 extracted concepts Timeline Stage Before After ---------------------------------------- Filesystem Walk 58.0s 0.65s Parsing 16.7s 3.23s Cross-reference Link 9.5s 0.61s Bundle Write 98.0s 4.10s ---------------------------------------- Total 157.0s 12.0s The biggest improvements came from removing unnecessary work rather than simply adding more parallelism: Replaced Path.rglob() with os.walk() and directory pruning (avoiding hundreds of thousands of unnecessary filesystem visits). Parallelized parsing with ProcessPoolExecutor because parsing was genuinely CPU-bound. Replaced repeated list scans with precomputed indexes, caching, and set-based lookups in the linker. Profiled inside the write stage and discovered that yaml.safe_dump() accounted for nearly 95% of render time for a fixed front matter schema. Replacing it with a schema-specific serializer became one of the biggest remaining wins. One takeaway that surprised me is how misleading optimization can be without profiling. I initially assumed parsing would dominate, but after each fix, a completely different stage became the bottleneck. I documented the entire optimization journey—including benchmark methodology, implementation details, trade-offs, and the optimizations that didn't matter—in this write-up: Performance write-up: performance The implementation is part of an open-source project called OKF Generator , but I tried to make the article useful even if you're not interested in the project itself: okf-generator I'd love feedback from anyone who has worked on compiler