How I Processed 666K Pages of Flattened PDFs into a Full Text Search Engine
In 2017 the National Archives and Records Administration (NARA) released the JFK files in an unsearchable manner 🔍. I tried doing manual research 🕵🏻. I relied on their provided CSV file of metadata to look for relevant documents to discover something - but I was looking for a needle in the haystack. I didn't know where to begin - but at the very least, I wanted to be able to search the contents therein. At least the National Archives allowed me to bulk download the PDFs. From that, I was able to birth the Apario Writer . In 2020, I began with rails new phoenixvault 🐦🔥 and I proceeded on a Zoom call with DJ Nicke - a former animator at Disney - to watch me build the proof of concept of the crowd sourcing declass utility that I envisioned. You see, when I was 7 years old, I had a dream after watching a space focused science program on TV that involved me sitting at the home computer, but interacting with an advanced interface that would help me uncover the mysteries of the day and time of the era. In Stargate SG-1, this concept was explored with the Tolan where Nareem was shocked to discover what Teal'c found in the records buried within a full text interface. Connecting it back to the JFK files released by NARA, they were unsearchable. Agenda on why aside, what could I do about it? This proof of concept grew into a SaaS platform that cost me $7,000 per month to operate over 12 bare meta servers in a private cloud using ESXi. This interface worked, but it was going to be replaced by a cost saving solution architected from the ground up in Go to reduce the dependency graph of the SaaS solution down to a single binary . In order to do this, I needed to create a pipeline. Looking at the SaaS model, I had a series of sidekiq jobs that compiled the assets. In order to improve the performance of that process, running off from Ruby code, I needed to build a new binary from the ground up using Go. I took the course on YouTube from Matt Holiday called Programming In Go and wa