8051: Building a Custom Disassembler

Industrializing the disassembly of an undocumented processor from a raw binary is a complex task that can be broken down into four key steps: Verify that the binary does not belong to a known processor. Verify that the binary is not obfuscated, compressed, or encrypted code for a known processor. Build an undocumented processor generator. Create the analysis pipeline and custom disassembler generation process. For the first phase of this project, the goal is to build dedicated, lightweight disassemblers—since, for bare-metal binaries, tools like Ghidra require manual processor target selection before analysis can begin. 1. Why Build a Custom Disassembler? To determine whether a binary was compiled for a specific architecture, the strategy consists of disassembling the binary (both statically and dynamically) against candidate instruction sets until: One or more bytes fail to match any valid instruction for that architecture, allowing us to rule it out. The disassembly succeeds completely. (Note: a successful disassembly does not guarantee that the binary was originally intended for that CPU; control flow validity must also be verified). Static disassembly is the first line of defense. However, if it fails due to obfuscation, compression, or encryption, we must escalate to dynamic execution and analysis. Only after systematically eliminating all known architectures can we confidently conclude that we are dealing with a custom or undocumented processor . 2. How to Build Your Custom Disassembler Before deploying heavy machinery for undocumented processors, the logical first step was to check against known architectures. Approach 1: Ghidra and SLAgh Ghidra relies on the SLAgh specification language and maintains an extensive library of processor definitions. The original plan was to leverage its API to extract a normalized opcode mapping table. However, after several attempts, Ghidra proved unsuitable for this specific pipeline for two reasons: Operand Type Loss: Detail