Lesson 19.1: A safe malware analysis workflow

Malware analysis is risky mostly because one small mistake can infect your real machine, leak the sample onto the company network, or tell the attacker you’re investigating. Beginners tend to rush into opening the sample in IDA. Professionals build the workflow first and run the sample later. This lesson is that workflow. Follow it in order, don’t skip steps, and you won’t have to clean up a machine at midnight.

Everything here is for defense, meaning understanding samples to detect, block, and respond to incidents. Don’t distribute samples, and don’t reuse them to cause harm.

Before touching the sample: build the lab

See Lesson 0.3 again, because this is where a mistake costs the most. Use an isolated VM, never a real machine. Take a clean snapshot before running anything, so you can go back in seconds. Use a host-only or simulated network (INetSim/FakeNet-NG), not one connected straight to the internet. When malware calls home to its C2, you’ve told the attacker that their sample is being dissected, and it may even download more payloads. Turn off shared folders and the shared clipboard when running a real sample.

Build the lab once and reuse it. The half day it takes is worth it.

Four layers of analysis

Not every sample goes through all four layers. You stop as soon as you’ve answered the question you set out to answer. The order is almost always from light to heavy, because the light layers are cheaper and safer.

Layer 1: Basic static (don’t run, look at the surface)

Cheap, fast, and safe because the sample doesn’t execute. Start with a hash (MD5/SHA-256), which identifies the sample, and look it up on VirusTotal to see whether anyone has seen it and how they classify it. VirusTotal is public, so looking up by hash is safe, but uploading the sample shares it with a third party.

Then run strings / FLOSS to get URLs, domains, paths, error messages, mutex names, and commands. FLOSS also decodes stack strings and encoded strings. After that, look at imports and DIE. The import table tells a story (file/registry/network/crypto APIs, see Lesson 1.10), and DIE tells you the packer/compiler and entropy. High entropy suggests packing, and then the surface static view is misleading, so you have to unpack first (Part 14).

At the end of layer 1 you should have a one-line summary covering the file type, whether it is packed, and what clues leaked.

Layer 2: Advanced static (reading code)

Open a disassembler/decompiler (Ghidra, IDA), go from strings and imports back to the logic, following the workflow in Lesson 0.4. This layer tells you what the sample does without running it. If the sample is packed, you deal with it here (Part 14), because the code on disk isn’t the real code yet.

Layer 3: Basic dynamic (run and observe behavior)

Only now do you let the sample run, in the isolated VM, after taking a snapshot. Don’t rush to debug, just observe from the outside using monitoring tools (Lesson 2.8). With Procmon, filter by process name and see which files it touches, which registry keys it writes, which child processes it creates, and where it connects. With Process Hacker / System Informer, view child processes, threads, memory regions (decrypted code shows up here), handles, and mutexes. With network monitoring (Wireshark + a simulated network), see where the sample tries to call home and with what protocol. Even if the real server is dead, INetSim answering with fakes can make the sample reveal its behavior.

When you’re done, go back to the clean snapshot. Try one thing, observe, roll back.

Layer 4: Advanced dynamic (debugging)

When observing from the outside isn’t enough, attach a debugger (x64dbg) to see values at runtime, catch self-decrypting code, follow a specific branch. This is the most labor-intensive layer, so save it for the part that needs it.

Automated sandboxes

You don’t always have to click through by hand. A sandbox runs the sample itself and outputs a behavior report. CAPE / CAPEv2 is a very strong open source sandbox that automatically unpacks and dumps the config of many malware families (pulling out C2, keys, campaign id without manual unpacking). Cuckoo3 and DRAKVUF are dynamic sandboxes with detailed behavior reports.

Online services include ANY.RUN, Triage (tria.ge), Joe Sandbox and Hybrid Analysis. They’re convenient and have interactive environments, but submitting a sample makes it public. Don’t upload sensitive samples (targeting your organization, containing internal data), because attackers also watch these platforms to learn that their sample has been detected.

An automated sandbox gives you a quick picture in a few minutes, and then you decide whether you need to dissect by hand.

Record IOCs as you go

If you don’t write things down, you start over. Record Indicators of Compromise as you work (see Lesson 19.2), including the hashes of the sample and the files it drops, the domains, IPs and URLs of the C2, the mutexes, registry keys, file paths and scheduled tasks (signs of persistence), and the characteristic behaviors you’ll write YARA/Sigma from later.

Ethics and legality

See Lesson 0.2 again. Analyzing malware for defense, research, and incident response is legal and necessary. But don’t distribute samples to people with no legitimate need, and don’t reuse malicious code to attack. Be careful when uploading samples to public services, especially targeted samples. Get samples from decent sources (MalwareBazaar, vx-underground) and always work in an isolated lab.

Lab

The goal is to practice running the four-layer workflow properly and using the monitoring tools, without touching any real malware. Use a harmless test sample or a benign program you write yourself. You need a VM built as in Lesson 0.3 with a clean snapshot, plus Procmon, Process Hacker (or System Informer), DIE, and optionally Wireshark with INetSim or FakeNet.

Start by writing your own workflow checklist for a hypothetical sample, following the four layers (basic static, advanced static, basic dynamic, advanced dynamic). At each layer note which tool you use, what you look for and when you move to the next layer.

Next, make the EICAR safe test sample by creating a text file containing exactly the industry-standard test string (look up “EICAR test string”, a harmless string used to test antivirus, not a virus) and watch how your AV reacts. This is an exercise in recognition, not in running malicious code. Then write (or use) a harmless program that creates a temp file, reads a registry key and writes a log line. Run it in the VM and use Procmon filtered by process name to see exactly those file and registry operations. This is a risk-free way to practice reading Procmon output, and you can reuse watchme.c from Lesson 2.8.

For the basic static layer on a clean file, take any system exe, compute its SHA-256, look it up on VirusTotal by hash (without uploading), run DIE to see the compiler and packer and run strings to see what leaks. When you finish, roll back to the clean snapshot and confirm the VM is back in exactly its original state.

Some questions to think about. Why is looking up VirusTotal by hash safer than uploading the file? When do you decide to move from basic dynamic analysis to debugging (layer 4)? And if a sample targets your own company, should you send it to ANY.RUN, and why?

Show solution

Try it yourself before reading. A sample workflow checklist looks like this:

[ ] PREPARATION
    [ ] Isolated VM, host-only or simulated network (INetSim/FakeNet)
    [ ] "clean-base" snapshot
    [ ] Shared folders and shared clipboard turned off
    [ ] Procmon, Process Hacker, Wireshark opened ahead of time

[ ] LAYER 1 - BASIC STATIC
    [ ] SHA-256, look it up on VirusTotal by hash (no upload)
    [ ] DIE: compiler, packer, entropy, 32/64-bit
    [ ] strings / FLOSS: URLs, domains, paths, mutexes, commands
    [ ] Imports: API groups (file, registry, net, crypto, process)
    [ ] Conclusion: packed or not? what are the first leads?

[ ] LAYER 2 - ADVANCED STATIC
    [ ] If packed: unpack (Part 14)
    [ ] Ghidra/IDA: go from strings and imports to the logic
    [ ] Rename and comment as soon as you understand something

[ ] LAYER 3 - BASIC DYNAMIC
    [ ] Take a snapshot again
    [ ] Run the sample, Procmon filtered by process name
    [ ] Process Hacker: child processes, RWX regions, mutexes, handles
    [ ] Wireshark: where do the first packets go
    [ ] Roll back

[ ] LAYER 4 - ADVANCED DYNAMIC
    [ ] x64dbg: breakpoints at key APIs and instructions
    [ ] Catch self-decrypting code, dump if needed

[ ] RECORD IOCs throughout
    [ ] Hash, C2 (domain/IP/URL), mutex, registry key, persistence
    [ ] Distinctive behavior for writing YARA/Sigma

For EICAR, the test string is a 68-character industry-standard string (look up the name to get it) that every AV recognizes as a “test virus” but is completely harmless and contains no malicious executable code. Save it to a .txt or .com file and the AV warns right away. It’s a safe way to check whether your AV or EDR is working, and to see what a detection looks like, without needing a real sample.

For reading Procmon on the benign program, run the program that creates a temp file and reads the registry (for example watchme.c from Lesson 2.8), then set the filter Process Name is <name>.exe in Procmon. You’ll see CreateFile to the temp file in %TEMP% with the result SUCCESS, RegOpenKey and RegQueryValue to the key being read, and WriteFile when it writes the log. Filter by Operation (only RegSetValue, WriteFile, TCP Connect) to cut the noise. This is how you read the behavior of real malware too, except here the sample is harmless so you can practice freely.

For basic static on a clean file, use certutil -hashfile file.exe SHA256 (Windows) or sha256sum (Linux) to get the hash and paste it into VirusTotal’s search box (search, don’t upload). DIE reports the compiler (for example MSVC), no packer and low entropy, and strings reveals paths, version strings and the names of imported DLLs.

For the rollback, choose the clean-base snapshot and restore it. The VM returns to exactly the state before the run, and every file or registry entry the program created disappears. Rolling back after every run is your insurance.

On the questions. A hash lookup only asks “has anyone seen this sample before” and doesn’t expose its content, while an upload hands the whole sample to a third party and anyone can download it again, including the attacker. You move to layer 4 when outside observation (layer 3) can’t explain a step, for example when the C2 string appears only after decryption in memory, or when you need to follow one particular conditional branch. Debugging costs effort, so save it for exactly where it’s needed. A sample that targets your own company shouldn’t be uploaded to a public platform, because the upload tells the attacker that their campaign has been detected and can leak internal information embedded in the sample. Use an internal sandbox (self-hosted CAPE or Cuckoo) instead of an online one.

Key takeaways

Build an isolated lab + snapshot + simulated network before touching the sample. The four layers go in order, basic static, advanced static, basic dynamic, advanced dynamic, and you stop when you’ve answered the question. Looking up VirusTotal by hash is safe, while uploading the sample makes it public.

An automated sandbox (CAPE dumps configs automatically) gives a quick picture before dissecting by hand. Record IOCs while you work, not later. Keep the analysis defensive, with no distributing and no re-weaponizing.