Reverse engineering — the process of analysing compiled software or binaries to understand their structure, behaviour, and intent — is one of the most powerful skills in an offensive and defensive security toolkit. In an era of rapidly evolving malware, supply chain attacks, and AI-generated code, the ability to analyse unknown binaries at a low level has become an increasingly differentiating capability for incident responders, malware analysts, threat intelligence practitioners, and penetration testers.
This post explores the core concepts, toolchain, and use cases of reverse engineering, with a focus on how it applies to the daily work of security professionals in enterprise environments.
What Is Reverse Engineering in a Security Context?
Reverse engineering in security involves analysing binaries, firmware, protocols, or code without access to the original source, with the goal of understanding what a piece of software does — particularly when it may be malicious. It is applied in several key security scenarios:
- Malware analysis: Determining the capabilities, persistence mechanisms, C2 communication patterns, and evasion techniques of suspected malware samples.
- Vulnerability research: Identifying exploitable conditions in software by analysing compiled binaries when source code is unavailable.
- Threat intelligence: Attributing malware campaigns to known threat actor groups by identifying code reuse, tooling signatures, and behavioural patterns.
- Incident response: Understanding exactly what a threat actor’s tooling did on a compromised system when log data is insufficient.
- Supply chain verification: Validating that compiled software matches expected behaviour and has not been tampered with.
The Core Toolchain
Proficiency in reverse engineering requires familiarity with a specific set of tools that operate at different levels of abstraction:
Static analysis tools examine binaries without executing them. IDA Pro and Ghidra (NSA’s open-source disassembler and decompiler) are the primary platforms for static analysis. Ghidra, in particular, has become the reference tool for malware analysis training because it is freely available and produces high-quality decompiled C-like pseudocode from x86, ARM, and other architectures.
Dynamic analysis tools examine binaries during execution, in a controlled environment. x64dbg (Windows debugger), OllyDbg, and GDB (Linux/macOS) allow analysts to step through code, set breakpoints, and observe the runtime behaviour of binaries — including memory manipulation, registry writes, and network connections.
Sandboxes such as Cuckoo Sandbox, Any.Run, and VirusTotal provide automated dynamic analysis environments that capture behavioural artefacts — file writes, network indicators, process trees — without requiring manual debugging. They are the first-line analysis tool for most SOC workflows.
String and entropy analysis tools identify suspicious patterns in binaries: encoded payloads, hardcoded IP addresses and domains, API call sequences, and encryption routines. Tools like FLOSS (FLARE Obfuscated String Solver) automate the extraction of obfuscated strings from malware samples.
Reverse Engineering and the CISSP CBK
For CISSP practitioners, reverse engineering intersects with several CBK domains:
- Domain 6 (Security Assessment and Testing): Malware analysis and binary review are assessment techniques within the software security testing domain.
- Domain 7 (Security Operations): Incident response increasingly requires reverse engineering capability to fully characterise threat actor tooling and determine the scope of compromise.
- Domain 8 (Software Development Security): Understanding how compiled code differs from source code — and how obfuscation, packing, and anti-analysis techniques work — is relevant to secure software development governance.
A Note on Legal and Ethical Boundaries
Reverse engineering in Australia is governed by the Copyright Act 1968, which contains specific provisions regarding decompilation for interoperability purposes under Section 47D. Defensive reverse engineering of malware samples captured in the course of incident response is generally outside the scope of these concerns, but any reverse engineering of commercial software requires careful legal review. Certifications such as GREM (GIAC Reverse Engineering Malware) provide structured, legally scoped training frameworks for practitioners.
Getting Started: Recommended Learning Path
- Develop foundational x86/x64 assembly language knowledge — “Computer Organisation and Architecture” by Stallings is a solid starting reference.
- Install and learn Ghidra — the NSA CISA training materials are freely available and practical.
- Work through malware samples in controlled VM environments — Malware Traffic Analysis (malware-traffic-analysis.net) and VirusTotal provide real samples in a legal, sandboxed context.
- Study the FLARE-On challenge archive — FireEye/Mandiant’s annual reverse engineering competition with detailed writeups is one of the best learning resources available.
- Consider pursuing GREM (GIAC Reverse Engineering Malware) certification for a structured, recognised credential in this domain.
References and Further Reading
- NSA/CISA — Ghidra Reverse Engineering Framework — ghidra-sre.org
- SANS GREM — GIAC Reverse Engineering Malware Certification
- FireEye/Mandiant — FLARE-On Challenge Archive
- Sikorski, M. & Honig, A. — Practical Malware Analysis (No Starch Press)
- OWASP Mobile Security Testing Guide — Static and Dynamic Analysis Sections
- Malware Traffic Analysis — malware-traffic-analysis.net
CISSP Made Easy 