wWNLtdh1

1
2 
3Lab 9: Basic Malware Classification
4Do at home:
5Complete the exercises from Ch 9 of the lab text. Make notes and upload a report.
6In-class portion
7from:
8The Malware Analyst's Cookbook:
9Tools and Techniques for Fighting Malicious Code
10by Michael Hale Ligh, Steven Adair, Blake Hartstein, and Matthew Richard
11published by Wiley, 2011
12ISBN: 978-0-470-61303-0
13
14ebook at Seneca Library
15
16The iso of the DVD used in this book is available on the IFS lab fileserver
17Resource:
18
19    Read the SANS:Malware101 - Viruses document (link to pdf on Moodle)
20    For a brief overview of the current state of affairs check out this report.
21
22Checkpoints:
23to achieve checkpoints in the lab, you must demonstrate your understanding of the material leading up to that checkpoint. You may do so by elaborating on the examples from the cookbook. For example, when making your own signature, use something other than the example in the cookbook.
24
25NOTE: It would not be wise to perform this lab in Kali Linux. Kali is an offensive tool, and ClamAV is defensive. If you make a mistake running ClamAV, it may disable some of the offensive tools on Kali - not the intended result.
26Best environment for malware analysis:
27Remnux, a customized Linux distro maintained by Lenny Zelster and David Westcott. Lenny Zeltser is one of those people I highly recommend you put on your watch list. Remnux contains many useful analysis tools,including many used in the Malware Analyst's Cookbook, such as ClamAV and Yara from this lab.
28Get the REMnux Distro.
29Note: If you are using kvm for virtualization, and using the vmdk provided on the file server, you need to use qemu-img to convert vmdk to qcow2.
30
31Chapter 3: Malware Classification
32
33One of the most common tasks malware analysts perform is initial triage, or classification of unknown content. Classification ranges from the simple, as in detecting the type of file, to the more complex, such as detecting the percent similarity with other samples in the wild and determining which behaviors are shared between variants of the same malware.
34
35This chapter shows how to use various free and open source tools such as ClamAV and YARA to quickly identify and classify malware. There are a number of companion Python scripts in this chapter for converting from one signature format to another, scanning files with multiple antivirus products, creating your own heuristic-based malicious file detector, and so on.
36Classification with ClamAV
37
38ClamAV is an open source antivirus engine owned by Sourcefire, the makers of the Snort intrusion-detection engine. ClamAV offers a fast and flexible framework for detecting malicious code and artifacts. The uses for ClamAV include incident response, forensics, and general malware protection or malware discovery. You can also use ClamAV to supplement or replace existing antivirus scanners on desktops, file servers, mail servers, and other places you might use an antivirus scanner.
39
40ClamAV has a number of built-in scanning capabilities for handling archive files, packed executables, HTML, mail, and other data types. This functionality allows you to write signatures and scan a broad range of content without writing specific parsers. Additionally, the ClamAV package includes the libclamav library as well as the command-line executables that interface with it. To keep signatures updated, you can invoke the command-line tool called freshclam manually or install it as a cron job.
41
42The most recent production-quality version of ClamAV is available from http://www.clamav.net/download/sources/, but you can also use a package manager to install it. On your Ubuntu machine, type the following commands:
43
44 $ apt-get install clamav clamav-freshclam
45
46Alternatively, if you'd like to use a more cutting-edge snapshot, you can download the latest development release using git, like this:
47
48 $ git clone http://git.clamav.net/clamav-devel.git
49
50Sourcefire maintains the latest documentation for ClamAV at http://www.clamav.net/doc/latest/. This documentation provides an excellent reference for writing ClamAV signatures. Additionally, the next few recipes discuss real-world scenarios where modifying ClamAV signatures allows you to detect samples not already included in the ClamAV database.
51
52The primary detection databases in ClamAV include:
53
54    MD5 hashes of known malicious binaries (stored in .hdb)
55
56    MD5 hashes of PE sections (stored in .mdb)
57
58    Hexadecimal signatures (stored in .ndb)
59
60    Archive metadata signatures (stored in .zmd or .rmd)
61
62    White list database of known good files (stored in .fp)
63
64Starting with ClamAV version 0.96, archive metadata signatures are deprecated. However, the developers added the following new features:
65
66    Matching signatures (stored in .ldb)
67
68    Icon signatures (stored in .ldb)
69
70    PE metadata strings (stored in .ldb or .ndb)
71
72    Container metadata (stored in .cdb)
73
74These detection capabilities provide a strong framework for you to build new signatures and detect specific characteristics in a collection of unknown, potentially malicious files.
75
76Checkpoint 1: ClamAV Installed
77
78Recipe 3-1: Examining Existing ClamAV Signatures
79
80The ClamAV signatures by default exist in compressed, binary files. You may want to see the criteria for an existing rule so that you can confirm or deny a false positive, or build a modified version of an existing signature. Luckily, ClamAV comes with a tool that allows you to decompress and inspect the signatures in its database.
81
82Typically, the ClamAV signatures exist in /usr/local/share/clamav or /usr/lib/clamav on Linux systems. You should expect to find main.cld and daily.cld (alternately they may have .cvd extensions). The main.cld file contains the primary base of signatures and daily.cld contains incremental daily updates.
83
84To unpack the signature files, use sigtool, which is provided with the ClamAV source package.
85
86    $ sigtool -u /var/lib/clamav/main.cld 
87
88    $ sigtool –u /var/lib/clamav/daily.cld
89
90These commands should result in the creation of the following files:
91
92 $ ls –Al 
93
94    total 61684 
95
96    -rw-r--r-- 1 root root 17992 Jul 7 20:49 COPYING 
97
98    -rw-r--r-- 1 root root 288 Jul 7 20:49 daily.cfg 
99
100    -rw-r--r-- 1 root root 25622 Jul 7 20:49 daily.db 
101
102    -rw-r--r-- 1 root root 16556 Jul 7 20:49 daily.fp 
103
104    -rw-r--r-- 1 root root 6891 Jul 7 20:49 daily.ftm 
105
106    -rw-r--r-- 1 root root 967678 Jul 7 20:49 daily.hdb 
107
108    -rw-r--r-- 1 root root 1425 Jul 7 20:49 daily.hdu 
109
110    -rw-r--r-- 1 root root 12542 Jul 7 20:49 daily.idb 
111
112    -rw-r--r-- 1 root root 686 Jul 7 20:49 daily.ign 
113
114    -rw-r--r-- 1 root root 397 Jul 7 20:49 daily.ign2 
115
116    -rw-r--r-- 1 root root 1790 Jul 7 20:49 daily.info 
117
118    -rw-r--r-- 1 root root 7249 Jul 7 20:49 daily.ldb 
119
120    -rw-r--r-- 1 root root 4908268 Jul 7 20:49 daily.mdb 
121
122    -rw-r--r-- 1 root root 37626 Jul 7 20:49 daily.mdu 
123
124    -rw-r--r-- 1 root root 317426 Jul 7 20:49 daily.ndb 
125
126    -rw-r--r-- 1 root root 13229 Jul 7 20:49 daily.ndu 
127
128    -rw-r--r-- 1 root root 4064 Jul 7 20:49 daily.pdb 
129
130    -rw-r--r-- 1 root root 3687 Jul 7 20:49 daily.wdb 
131
132    -rw-r--r-- 1 root root 8689 Jul 7 20:49 daily.zmd 
133
134    -rw-r--r-- 1 root root 4731085 Jul 7 20:49 main.db 
135
136    -rw-r--r-- 1 root root 13533 Jul 7 20:49 main.fp 
137
138    -rw-r--r-- 1 root root 1502569 Jul 7 20:49 main.hdb 
139
140    -rw-r--r-- 1 root root 901 Jul 7 20:49 main.info 
141
142    -rw-r--r-- 1 root root 34403973 Jul 7 20:49 main.mdb 
143
144    -rw-r--r-- 1 root root 15994685 Jul 7 20:49 main.ndb 
145
146    -rw-r--r-- 1 root root 217 Jul 7 20:49 main.zmd
147
148Now, when you scan a file and ClamAV detects it, you can search the uncompressed signature file to see the byte pattern that produced the alert.
149
150NOTE: This will not work as outlined in the book because ClamAV has evolved. The next step will enable you to find where the signature is. Because of the constantly changing nature of Anti-Virus software, we will always have to trouble shoot our way through this lab.
151
152 $ clamscan 76ed99f6a94c542f81bf6af35d829744
153
154 76ed99f6a94c542f81bf6af35d829744: XF.Sic.E FOUND 
155
156    ----------- SCAN SUMMARY ----------- 
157
158    Known viruses: 726064 
159
160    Engine version: 0.96 
161
162    Scanned directories: 0 
163
164    Scanned files: 1 
165
166    Infected files: 1 
167
168    Data scanned: 2.72 MB 
169
170    Data read: 1.36 MB (ratio 2.00:1) 
171
172    Time: 3.680 sec (0 m 3 s) 
173
174NOTE: The file name in the following output may not be exactly the same depending on your ClamAV version. You may have to spend time finding where the file exists
175
176    $ grep "XF.Sic.E" * 
177
178    daily.ndb:XF.Sic.E:2:*:2a2a536574204f75722056616c75657320616e642050\
179
180     617468732a2a??00002a2a416464204e657720576f726b626f6f6b\
181
182     2c20496e666563742049742c205361766520497420417320426f6f\
183
184     6b312e
185
186If you convert this hexadecimal signature into ASCII (there's an online conversion tool here: http://www.dolcevie.com/js/converter.html), you'll find the signature is looking for the following content.
187
188 **Set Our Values and Paths**???**Add New Workbook, Infect It, Save It As Book1.
189
190You could modify this signature to detect similar variations of the string, such as one that ends with Book2 instead of Book1. However, you cannot include your modified signatures in the default signature database. Any signature that you modify and save must go into a new database file that we'll discuss more in the next recipe.
191Checkpoint 2: Show ClamAV Signature
192show that you know what is in a signature
193
194Recipe 3-2: Creating a Custom ClamAV Database
195
196Image from book You can find supporting material for this recipe on the companion DVD.
197
198Writing new signatures for a custom ClamAV database allows you to scan for patterns that the default signatures do not currently detect. This recipe shows how ClamAV's flexible syntax for writing signatures allows you to write anything from simple hexadecimal signatures to complex logical signatures.
199
200ASCII Signatures (Hello World)
201
202To create a simple ASCII-based signature, you can use sigtool to convert the text to hexadecimal. To use sigtool for this purpose, you execute it with the --hex-dump flag. sigtool expects you to provide your text via STDIN and it outputs the hexadecimal version to STDOUT. One common mistake when entering text via STDIN is failing to remove the trailing line feed character, which is appended when you hit the enter key.
203
204The example that follows shows how you can use sigtool to generate the hexadecimal output of hello world. Note the trailing 0a that must be removed to match the original pattern.
205
206 $ sigtool --hex-dump
207
208 hello world
209
210 68656c6c6f20776f726c640a
211
212To convert this into a usable signature, you need to format it according to the ClamAV signature syntax. Starting with ClamAV version 0.96, the basic signature format is deprecated in favor of an extended signature format. This recipe focuses only on the extended signature format, which consists of the following four fields separated by colons:
213
214    SigName:Target:Offset:HexadecimalSignature
215
216The SigName field is a unique, descriptive name for your signature. The Target parameter can be any of the following values.
217
218    0 = Any file type
219
220    1 = Windows PE
221
222    2 = OLE (e.g. Office, VBA)
223
224    3 = Normalized HTML
225
226    4 = E-mail file (e.g. RFC822 message, TNEF)
227
228    5 = Image files (e.g. jpeg, png)
229
230    6 = ELF
231
232    7 = Normalized ASCII file
233
234    8 = Unused
235
236    9 = Mach-O binaries (new in v0.96)
237
238Assuming you want to detect any file containing the hello world string, you would create the following signature:
239
240 TestHelloWorld:0:*:68656c6c6f20776f726c64
241
242This is a simple example using text, but you can create more complex signatures using wildcards. For example, let's say you want to detect hello and world but not necessarily with a space between them. You can do that with the following signature, which uses a wildcard (??) to match any byte value between 0 and FF.
243
244 TestHelloWorldAnySeparator:0:*:68656c6c6f??776f726c64
245
246You can also specify that hello world occur at a fixed offset within a file.
247
248 TestHelloWorldOffset45:0:45:68656c6c6f20776f726c64
249
250And you can also specify a range of offsets. The following signature will only trigger if ClamAV detects hello world between offsets 200 and 250 of a file.
251
252 TestHelloWorldBetween200And250:0:200,50:68656c6c6f20776f726c64
253
254Finally, you can specify that hello and world occur in that order at any offset in the file.
255
256 TestHelloWorldAnyDistance:0:*:68656c6c6f*776f726c64
257
258To use these signatures, you need to place them into a file with a .ndb extension. For convenience, we've added the signatures to a file named clam_helloworld.ndb on the book's DVD. For testing purposes, we created a file with the following content:
259
260 "This is the data I'd like to scan looking for 'hello' and 'world'.
261
262 I'm not picky how close these words are together."
263
264When using the custom signature database, you need to specify its location on the command line for clamscan using the -d flag.
265
266 $ clamscan -d clam_helloworld.ndb test.txt
267
268 test.txt: TestHelloWorldAnyDistance.UNOFFICIAL FOUND
269
270 ----------- SCAN SUMMARY -----------
271
272 Known viruses: 5
273
274 Engine version: 0.96
275
276  Scanned directories: 0
277
278 Scanned files: 1
279
280 Infected files: 1
281
282 Data scanned: 0.00 MB
283
284 Data read: 0.00 MB (ratio 0.00:1)
285
286 Time: 0.015 sec (0 m 0 s)
287
288Note that any time you create and use a signature that is not in the project signature base, it will display with the extension .UNOFFICIAL. ClamAV adds this extension to any signatures that are not in the default project signature set. If you have multiple custom databases, you can place all of the .ndb files into a directory and call clamscan with the –d DIRNAME argument.
289
290Checkpoint 3: Custom Signature Demo
291write your own signature and show it working
292
293Binary Signatures (Shellcode)
294Resources
295
296    shellcode
297    alphanumeric shellcode
298    how shellcodes work
299    exploit-db shellcode database
300    Generating Payloads in Metasploit
301    The Shellcoder's Handbook: Discovering and Exploiting Security Holes, 2nd Edition
302
303With the basic building blocks that we've discussed thus far, you can detect more complicated malicious artifacts, such as shellcode. For example, consider the following disassembly of shellcode from a malicious Microsoft Office document:
304
305 Offset Instruction Byte codes 
306
307 00000000 xor ecx,ecx 33c9
308
309 00000002 mov cx,0x147 66b94701
310
311 00000006 xor byte [edx+ecx],0xe9 80340ae9
312
313 0000000A loop 0xfffffffc e2fa
314
315 0000000C jmp 0xc eb0a
316
317You can use the byte code values to create a binary signature, like this:
318
319 ShellcodeXOR:0:*:33c966b9470180340ae9e2faeb0a
320
321This signature detects the specific shellcode block but fails to detect shellcode with different length values in CX, or different XOR mask values. You can broaden your signature by inserting wildcards for the length value, XOR mask, and jump length. Here is the final signature:
322
323 shellcode_xor:0:*:33c966b9????80340a??e2??eb
324
325This signature detects shellcode that performs the following list of actions:
326
327    zeroes-out the CX register (33c9)
328
329    moves a length into CX (66b9????)
330
331    uses XOR to modify the data located at [edx+ecx] (80340a??)
332
333    loops back to start (e2??)
334
335    executes a jump to the resultant data (eb) when the loop is complete
336
337Effectively, this signature detects the following pseudocode, which matches any pattern of activity without regard to specific values.
338
339 xor ecx, ecx
340
341 mov cx, ??
342
343 xor byte [edx+ecx], ??
344
345 loop ??
346
347 jmp ??
348
349
350Checkpoint 4: Binary Signature Demo
351
352To use this signature, you can simply add it to your custom signature database (.ndb file) and use the -d parameter with clamscan.
353
354
355Logical Signatures (New in v0.96)
356
357One of the most powerful new features in recent versions of ClamAV is the capability to understand complex signatures based on logical expressions. This capability allows you to write signatures where you need to include optional values or only trigger alerts when multiple conditions are met. The format for logical signatures is:
358
359    SigName;Target;Expression;Sig0;Sig1;..;SigN
360
361The SigName and Target fields have the same meaning as we described when discussing the extended signature format. The Expression field consists of a logical expression where each signature is represented by its index value. Thus, the number 0 refers to Sig0 and the number 1 refers to Sig1 and so on. Each signature can be combined with the logical operators OR (|) and AND (&). Further, by using the =, <, and > operators, you can control the number of occurrences of each signature that must be found in a file before producing an alert. For example, the expression (0>5)&(1=3) will trigger an alert when signature 0 occurs more than five times and signature 1 matches exactly three times.
362
363Using the original hello world example, you can write a signature to detect the presence of both hello and world without regard to their ordering or position.
364
365 HelloWorldLogic;Target:0;0&1;68656c6c6f;776f726c64
366
367This signature has two sub-signatures, 68656c6c6f (hello) and 776f726c64 (world), and a logical expression, 0&1. The values 0 and 1 represent the indices of the sub-signatures. You should also specify a file type target value of 0 that results in the scanning of any file type.
368
369For a more realistic example, consider malware that uses code injection to execute within another process. One common way malware performs code injection is detectable using the following criteria:
370
371    The WriteProcessMemory and CreateRemoteThread strings: These are names of API functions used to perform the injection.
372
373    The SeDebugPrivilege string: The name of the debug system privilege, which a process must enable before calling either of the above API functions.
374
375    A string such as iexplore.exe or explorer.exe: The name of the target process.
376
377Logically, you can express this scenario by looking for any executable with either the string iexplore.exe or explorer.exe, both WriteProcessMemory and CreateRemoteThread strings, and the string SeDebugPrivilege. In other words, you want to match:
378
379 ("iexplore" | "explorer.exe") & \
380
381 ("WriteProcessMemory" & "CreateRemoteThread" & "SeDebugPrivilege")
382
383Using the logical signature syntax, you could express that as the following rule.
384
385 ProcessInjector;Target:1;(0|1)&(2&3&4);696578706c6f72652e6578\
386
387 65;6578706c6f7265722e657865;53654465627567507\
388
389 26976696c656765;43726561746552656d6f746554687\
390
391 2656164;577269746550726f636573734d656d6f7279
392
393This signature is named clam_inject.ldb and it is included on the DVD that accompanies this book. If you want an alert for malware that injects a different target process, then you'll need to modify the signature. Also, keep in mind this is just a simple example. If the malware is packed, the strings we're using for detection may not be visible to ClamAV.
394  	
395
396Note
397	
398
399Also see http://www.clamav.net/doc/webinars/Webinar-Alain-2009-03-04.ppt for additional examples of writing ClamAV signatures.
400
401
402Checkpoint 5: Logical Signature Demo
403
404Classification with YARA
405NOTE: for this section, run the conversion script on your previous signatures and try them with yara
406
407YARA (http://libaccess.senecacollege.ca:2138/p/yara-project/) is an extremely flexible identification and classification engine written by Victor Manuel Alvarez of Hipasec Sistemas. Using YARA, you can create rules that detect strings, instruction sequences, regular expressions, byte patterns, and so on. Then you can scan files using the command-line yara utility or integrate the scanning engine into your own C or Python tools with YARA's API. In the next few recipes, we'll show you how to get started with YARA and we'll introduce you to other usage scenarios throughout the book.
408
409
410
411Recipe 3-3: Converting ClamAV Signatures to YARA
412
413Image from book You can find supporting material for this recipe on the companion DVD.
414
415You will need to install yara and iibpcre3.
416
417In Ubuntu:
418
419sudo apt-get install yara
420
421sudo apt-get install libpcre3
422
423
424This recipe provides a script for converting ClamAV signatures to YARA format. Generally, ClamAV is able to perform scans quicker than YARA, so it is not useful to convert all ClamAV signatures. However, it is also not useful to "reinvent the wheel" and manually convert signatures if you need to use them with YARA.
425
426The clamav_to_yara.py script included on the book's DVD handles the conversion process for you by modifying ClamAV signatures to meet the requirements of YARA. In particular, ClamAV jumps of more than 255 bytes, or where the end of the jump is more than 255 bytes, require special handling. For example, the following ClamAV signature uses {100000-} to indicate that there must be 100,000 or greater bytes between the first sequence of hex bytes and the second sequence of hex bytes.
427
428 Trojan.Dropper-554:0:33107:4d5a80000100000004001000ffff0000\
429
430 400100000000000040{100000-}646c6c00446c6c43616\
431
432 e556e6c6f61644e6f7700446c6c476574436c
433
434In order to convert this signature to YARA format, you must change the {100000-} tag to comply with YARA's rules. YARA allows a maximum jump of 255 bytes, thus the ClamAV signature must be split into two sequences of hex bytes joined with an AND clause. In addition, the script automatically converts rule names to a YARA-compatible syntax. YARA does not allow non-alphanumeric characters, except the underscore (_), in rule names.
435
436 rule Trojan_Dropper_554
437
438 {
439
440 strings:
441
442 $a0 = { 4d5a80000100000004001000ffff0000400100000000000040 }
443
444 $a1 = { 646c6c00446c6c43616e556e6c6f61644e6f7700446c6c476574436c }
445
446 condition:
447
448 $a0 and $a1
449
450 }
451
452This rule is less specific than the original ClamAV rule, because the second string could theoretically occur within fewer than 100,000 bytes. Furthermore, the second string could exist after the first string and still trigger a hit. One method of fine-tuning the conversion process involves using YARA's first occurrence operator (@) in the condition field. If you precede the name of a string with the @ operator, you can get the offset of the first occurrence of the string. For example, @a0 stores the first occurrence of $a0 and @a1 stores the first occurrence of$a1. By using a condition of @a0 < @a1 you can ensure that $a0 exists first. You could also use (@a1 - @a0) >= 10000 to ensure that at least 10,000 bytes are between the two strings.
453
454ClamAV and YARA use the same syntax for wildcards (?? for byte wildcards and (aa|bb|cc) for explicit selection). In these cases, the conversion script does not perform any modifications. The only exception is that YARA does not allow a signature to start with any type of wildcard so the script skips any signature that starts with a wildcard.
455
456The clamav_to_yara.py script requires two parameters, -f for the input file name that must be a ClamAV-formatted signature file and -o for the output file name. Optionally, the script accepts a -s flag to filter the results only to those that match the specified string. Using -s is the recommended use; otherwise, the script will create over 60,000 signatures from the standard ClamAV database. The following command shows how to convert all signatures that contain the term "Agent":
457
458 $ python clamav_to_yara.py -f main.ndb -o clamav.yara -s Agent
459
460 [+] Read 61123 lines from main.ndb
461
462 [+] Wrote 3894 rules to test
463
464Scanning files using the new clamav.yara rules shows that YARA can properly interpret the converted ClamAV signatures. In the output below, we scanned a directory of files recursively with YARA and started getting hits:
465
466 $ yara -r clamav.yara /data/malcode
467
468 Trojan_Agent_13844 /data/malcode/mft.exe
469
470 Trojan_Agent_78 /data/malcode/file.php
471
472 Trojan_Agent_130266 /data/malcode/payload.exe
473
474
475Checkpoint 6: Yara Demo
476
477Recipe 3-4: Identifying Packers with YARA and PEiD
478
479Image from book You can find supporting material for this recipe on the companion DVD.
480
481PEiD[1] is a GUI tool for Windows that you can use to detect packers. The PEiD signatures are stored in a plain-text file that you can extend with new signatures and/or parse with your own tools. The syntax for PEiD signatures is very similar to YARA, allowing you to easily use PEiD signatures within YARA. Identifying packers in YARA allows you to leverage the detection from PEiD in a more flexible way. For example, when using YARA as part of a Python script, you could automatically take additional actions if you detect a particular packer.
482
483The YARA project's wiki[2] provides a handful of sample packer rules based on the PEiD database. You can download the default PEiD database from the PEiD website (look for UserDB.zip). Each PEiD rule is in the following format:
484
485 [signature name]
486
487 signature = hex_signature
488
489 ep_only=(true|false)
490
491Here is an example signature:
492
493 [$PIRIT v1.5]
494
495 signature = B4 4D CD 21 E8 ?? ?? FD E8 ?? ?? B4 51 CD 21
496
497 ep_only = true
498
499According to its name, the signature detects files packed with v1.5 of the $PIRIT packer. Setting ep_only to true means that PEiD should only check for the signature at the program's entry point. Otherwise, PEiD should check for the signature in the entire file. Using the peid_to_yara.py script on the book's DVD, you can convert the entire PEiD ruleset into a YARA-compatible rule file. Here is an example of using the script:
500
501 $ python peid_to_yara.py -f UserDB.TXT -o packer.yara
502
503The resulting signatures in the packer.yara file will look like the following:
504
505 rule PIRITv15
506
507 {
508
509 strings:
510
511 $a0 = { B4 4D CD 21 E8 ?? ?? FD E8 ?? ?? B4 51 CD 21 }
512
513 condition:
514
515 $a0 at entrypoint
516
517 }
518
519Here are some key points about the conversion process:
520
521    The at entrypoint keywords in the condition of a YARA rule have the same effect as setting ep_only to true.
522
523    Some PEiD rules leverage wildcards at the beginning of the rule, which YARA does not support; therefore those rules are not converted.
524
525    In some cases, the name of the YARA rule may be different from the PEiD rule name (for example, $PIRIT v1.5 versus PIRITv15). This is because YARA does not allow non-alphanumeric rule names.
526
527You can use the new packer.rules file in the same manner as any other YARA ruleset. This gives you a cross-platform (Windows, Linux, Mac OS X, etc.) method of detecting packed files on command line.
528
529 $ yara -r packer.yara /data/malcode
530
531 UPXv20MarkusLaszloReiser bad_file.exe
532
533 WinUpackv030betaByDwing e1.exe
534
535 WiseInstallerStub NoWinDvdUpdate.EXE
536
537In the output, we found files that triggered UPX, WinUpack, and WiseInstallerStub signatures. For demonstration purposes, we wrote a script with YARA's Python API that automatically unpacks files if they're packed with UPX. You'll need the UPX utility, which you can get by typing apt-get install upx-ucl on your Ubuntu machine. Here is the code and example usage:
538
539 $ cat sample_script.py
540
541 #!/usr/bin/python
542
543 import sys, yara, commands
544
545 rules = yara.compile(sys.argv[1])
546
547 data = open(sys.argv[2], 'rb').read()
548
549 matches = rules.match(data=data)
550
551 isupx = [m for m in matches if m.rule.startswith("UPX")]
552
553 if isupx:
554
555 outp = commands.getoutput("upx -d %s" % sys.argv[2])
556
557 print outp
558
559 $ python sample_script.py packer.yara /data/malcode/bad_file.exe
560
561 Ultimate Packer for eXecutables
562
563 Copyright (C) 1996 - 2009
564
565 UPX 3.04 Markus Oberhumer, Laszlo Molnar & John Reiser Sep 27th 2009
566
567 File size Ratio Format Name
568
569 -------------------- ------ ----------- -----------
570
571 422400 <- 176128 41.70% win32/pe bad_file.exe
572
573As you can see, the Python script calls upx –d (for decompress) after bad_file.exe triggered the UPX packer signature. To extend this into a more useful script, you would need to add handlers for any packers on which you want to conduct further analysis.
574
575
576Checkpoint 7: Packer ID Demo (SKIP THIS FOR NOW)
577
578Recipe 3-5: Detecting Malware Capabilities with YARA
579
580Image from book You can find supporting material for this recipe on the companion DVD.
581
582This recipe shows how you can use YARA to design rules for detecting malware capabilities. The common argument against using signature- or pattern-based detection is that packers and encryption can evade your efforts. While this is true, the number of malware samples that you can detect with creative YARA signatures will far exceed the few samples that slip through the cracks. The capabilities.yara file on the book's DVD contains the rules presented in this recipe.
583
584The following rule detects embedded PE files, which is a common characteristic of droppers and installers. It produces an alert only if the string is found at an offset greater than 1024 in the file, which is outside of the typical PE header (otherwise it would produce an alert on every PE file). The filesize keyword represents the total number of bytes in the file or data buffer being scanned.
585
586 rule embedded_exe
587
588 {
589
590 meta:
591
592 description = "Detects embedded executables"
593
594 strings:
595
596 $a = "This program cannot be run in DOS mode"
597
598 condition:
599
600 $a in (1024..filesize)
601
602 }
603
604The following rule detects several attempts to identify virtual machines, emulators, sandboxes, or behavior-monitoring applications. The nocase keyword indicates a case-insensitive string.
605
606 rule vmdetect
607
608 {
609
610 meta:
611
612 description = "Detects VMs/EMUs/Mons"
613
614 strings:
615
616 $vm0 = "VIRTUAL HD" nocase
617
618 $vm1 = "VMWARE VIRTUAL IDE HARD DRIVE" nocase
619
620 $vm2 = "QEMU HARDDISK" nocase
621
622 $vm3 = "VBOX HARDDRIVE" nocase
623
624 $vm4 = "The Wireshark Network Analyzer"
625
626 $vm5 = "C:\sample.exe"
627
628 $vm6 = "C:\windows\system32\sample_1.exe"
629
630 $vm7 = "Process Monitor - Sysinternals: www.sysinternals.com"
631
632 $vm8 = "File Monitor - Sysinternals: www.sysinternals.com"
633
634 $vm9 = "Registry Monitor - Sysinternals: www.sysinternals.com"
635
636 condition:
637
638 any of them
639
640 }
641
642The following rule detects malware that is static-linked with Zlib or OpenSSL libraries. If you get positive hits with this rule, it's highly likely that the malware uses encoding and/or encryption to obfuscate its network communications. Instead of specifying $zlib0 and $zlib1and $zlib2[…] in the condition, you can specify all of $zlib*, which has the same effect.
643
644 rule encoding
645
646 {
647
648 meta:
649
650 description = "Indicates encryption/compression"
651
652 strings:
653
654 $zlib0 = "deflate" fullword
655
656 $zlib1 = "Jean-loup Gailly"
657
658 $zlib2 = "inflate" fullword
659
660 $zlib3 = "Mark Adler"
661
662 $ssl0 = "OpenSSL" fullword
663
664 $ssl1 = "SSLeay" fullword
665
666 condition:
667
668 (all of ($zlib*)) or (all of ($ssl*))
669
670 }
671
672The following rule detects malware that utilizes IRC. Because the strings may exist frequently in files that do not utilize IRC, this rule produces an alert only if any file contains at least four of the strings.
673
674 rule irc
675
676 {
677
678 meta:
679
680 description = "Indicates use of IRC"
681
682 strings:
683
684 $irc0 = "join" nocase fullword
685
686 $irc1 = "msg" nocase fullword
687
688 $irc2 = "nick" nocase fullword
689
690 $irc3 = "notice" nocase fullword
691
692 $irc4 = "part" nocase fullword
693
694 $irc5 = "ping" nocase fullword
695
696 $irc6 = "quit" nocase fullword
697
698 $irc7 = "chat" nocase fullword
699
700 $irc8 = "privmsg" nocase fullword
701
702 condition: 4 of ($irc*)
703
704 }
705
706The following rule detects attempts to sniff network traffic based on the existence of "sniffer" in the file (believe it or not, this yields a good number of positive hits). It also detects the names of WinPcap API functions, since many malware families drop or download WinPcap DLLs for sniffing packets.
707
708 rule sniffer
709
710 {
711
712 meta:
713
714 description = "Indicates network sniffer"
715
716 strings:
717
718 $sniff0 = "sniffer" nocase fullword
719
720 $sniff1 = "rpcap:////" nocase
721
722 $sniff2 = "wpcap.dll" nocase fullword
723
724 $sniff3 = "pcap_findalldevs" nocase
725
726 $sniff4 = "pcap_open" nocase
727
728 $sniff5 = "pcap_loop" nocase
729
730 $sniff6 = "pcap_compile" nocase
731
732 $sniff7 = "pcap_close" nocase
733
734 condition:
735
736 any of them
737
738 }
739
740The following rule detects malware that attempts to spread through autorun functionality. The rule includes strings necessary for building an autorun.inf file that uses the open action to execute a program.
741
742 rule autorun
743
744 {
745
746 meta:
747
748 description = "Indicates attempt to spread through autorun"
749
750 strings:
751
752 $a = "[autorun]"
753
754 $b = "open="
755
756 condition:
757
758 all of them
759
760 }
761
762The following rule detects attempts to send spam e-mails (or just e-mails in general based on SMTP commands). The number of required matches can be increased to detect spam or other strings that won't be found in normal SMTP communication.
763
764 rule spam
765
766 {
767
768 meta:
769
770 description = "Indicates spam-related activity"
771
772 strings:
773
774 $spam1 = "e-cards@hallmark.com" nocase
775
776 $spam2 = "hallmark e-card" nocase
777
778 $spam3 = "rcpt to:" nocase
779
780 $spam4 = "mail from:" nocase
781
782 $spam5 = "smtp server" nocase
783
784 $spam6 = "cialis" nocase fullword 
785
786 $spam7 = "pharma" nocase fullword
787
788 $spam8 = "casino" nocase fullword
789
790 $spam9 = "ehlo " nocase fullword
791
792 $spama = "from: " nocase fullword
793
794 $spamb = "subject: " nocase fullword
795
796 $spamc = "Content-Disposition: attachment;" nocase
797
798 condition:
799
800 3 of ($spam*)
801
802 }
803
804The following rule detects malware that uses the wrmsr instruction to patch the SYSENTER_EIP_MSR register. The operands for wrmsr are placed in EAX, ECX, and EDX, but they can be initialized in any order and using any source (a 32-bit immediate constant or a stack variable). Therefore, the rule uses wildcards to detect many possible variations of the behavior.
805
806 rule write_msr
807
808 {
809
810 meta:
811
812 description = "Writing MSR"
813
814 strings:
815
816 /*
817
818 mov ecx, [ebp+??]
819
820 mov eax, [ebp+??]
821
822 mov edx, [ebp+??]
823
824 wrmsr
825
826 */
827
828 $wr0 = {8B 4D ?? 8B 55 ?? 8B 45 ?? 0F 30}
829
830 $wr1 = {8B 4D ?? 8B 45 ?? 8B 55 ?? 0F 30}
831
832 $wr2 = {8B 55 ?? 8B 4D ?? 8B 45 ?? 0F 30}
833
834 $wr3 = {8B 55 ?? 8B 45 ?? 8B 4D ?? 0F 30}
835
836 $wr4 = {8B 45 ?? 8B 55 ?? 8B 4D ?? 0F 30}
837
838 $wr5 = {8B 45 ?? 8B 4D ?? 8B 55 ?? 0F 30}
839
840 /*
841
842 mov ecx, imm32
843
844 mov eax, imm32
845
846 mov edx, imm32
847
848 wrmsr
849
850 */
851
852 $wr6 = {B8 ?? ?? ?? BA ?? ?? ?? B9 ?? ?? ?? 0F 30}
853
854 $wr7 = {B8 ?? ?? ?? B9 ?? ?? ?? BA ?? ?? ?? 0F 30}
855
856 $wr8 = {B9 ?? ?? ?? B8 ?? ?? ?? BA ?? ?? ?? 0F 30}
857
858 $wr9 = {B9 ?? ?? ?? BA ?? ?? ?? B8 ?? ?? ?? 0F 30} 
859
860 $wra = {BA ?? ?? ?? B8 ?? ?? ?? B9 ?? ?? ?? 0F 30}
861
862 $wrb = {BA ?? ?? ?? B9 ?? ?? ?? B8 ?? ?? ?? 0F 30}
863
864 condition:
865
866 any of them
867
868 }
869
870Here are a few additional ways you can use YARA signatures:
871
872    Create a rules file with common passwords to catch malware that attempts to brute force accounts and logins.
873
874    Create a rules file with login strings, URL fields, or bank domains to catch malware that targets financial institutions.
875
876    Create a rules file with names of antivirus processes, services, and domains to catch malware that attempts to terminate or disable A/V products.
877
878
879Putting it All Together
880
881The best part about all of the tools described in this chapter thus far is that you can incorporate them into tools that automate several actions at once. You can use a single script to scan files with ClamAV, scan files with YARA, determine file type, detect packers, compute checksums, and various other tasks. The next few recipes show how to combine some of the aforementioned functionality and build your own multi-AV scanner and PE file scanner.
882
883
884Checkpoint 8: Malware Detection Demo (SKIP THIS FOR NOW)
885
886Recipe 3-6: File Type Identification and Hashing in Python
887
888Image from book You can find supporting material for this recipe on the companion DVD.
889
890This recipe shows how to determine file type and calculate cryptographic hashes in Python. A common way to organize malware collections is in a directory structure based on file type and/or hash value. For example, you might have a layout like this:
891
892
893    malware/639ff32e13aa789324c112d9cfad31b9
894    malware/69e46a1967b4dacce63fa9fa6f342209
895    malware/be72b15fa85a65ce9fa12c97d60b14a3
896
897Or you may have a layout like this:
898
899
900    malware/dll/639ff32e13aa789324c112d9cfad31b9
901    malware/pdf/69e46a1967b4dacce63fa9fa6f342209
902    malware/exe/be72b15fa85a65ce9fa12c97d60b14a3
903
904When you get new malware samples, you can process them automatically and save them to the proper directory. Of course, if you plan to store samples in a database, you can also use similar techniques.
905
906Determining File Type
907
908On a Linux system, you can use the file command to determine a file's type. The output of the following command shows that the ack388 file is a PE executable despite its missing file extension.
909
910 $ file ack388
911
912 ack388: MS-DOS executable PE for MS Windows (GUI) Intel 80386 32-bit
913
914In Python, you can determine file type using the python-magic package (apt-get install python-magic). Once installed, you can use the following commands in a Python script:
915
916 >>> import magic
917
918 >>> ms = magic.open(magic.MAGIC_NONE)
919
920 >>> ms.load()
921
922 >>> data = open("ack388", "rb").read()
923
924 >>> print ms.buffer(data)
925
926 MS-DOS executable PE for MS Windows (GUI) Intel 80386 32-bit
927
928As an alternate method, you can also write YARA signatures for detecting file types. On the book's DVD, you can find a file named magic.yara, which contains signatures such as the following:
929
930 rule pdf_document
931
932 { 
933
934 strings:
935
936 $a = "%PDF-"
937
938 condition:
939
940 $a at 0
941
942 }
943
944 rule zip_file
945
946 {
947
948 strings:
949
950 $magic1 = { 50 4b 03 04 }
951
952 $magic2 = { 50 4b 05 06 }
953
954 $magic3 = { 50 4b 07 08 }
955
956 condition:
957
958 ($magic1 at 0) or ($magic2 at 0) or ($magic3 at 0)
959
960 }
961
962 rule mz_executable // from YARA user's manual
963
964 {
965
966 condition:
967
968 // MZ signature at offset 0 and ...
969
970 uint16(0) == 0x5A4D and
971
972 // ... PE signature at offset stored in MZ header at 0x3C
973
974 uint32(uint32(0x3C)) == 0x00004550
975
976 }
977
978Here is an example of using the YARA rules for file type detection:
979
980 $ yara –r magic.yara ack388
981
982 mz_execuTable 388
983
984Calculating Hashes
985
986On a Linux system, you can use commands such as md5sum, sha1sum, sha256sum, and sha512sum to generate hashes for files.
987
988 $ md5sum ack388
989
990 69e46a1967b4dacce63fa9fa6f342209 ack388
991
992 $ sha1sum ack388
993
994 4c570b44c8dac70af742af446d8a475be702dc97 ack388
995
996In Python, you can use the built-in hashlib module or the PyCrypto module (see Chapter 12 for more details). Here is an example:
997
998 >>> import hashlib
999
1000 >>> data = open("ack388", "rb").read()
1001
1002 >>> print hashlib.md5(data).hexdigest()
1003
1004 69e46a1967b4dacce63fa9fa6f342209
1005
1006 >>> print hashlib.sha1(data).hexdigest()
1007
1008 4c570b44c8dac70af742af446d8a475be702dc97
1009
1010Calculating Fuzzy Hashes
1011
1012Fuzzy hashes can help you determine similarity among files. We present various usage scenarios in Recipe 3-9, so for now we'll just show how to calculate the hashes. You can use the ssdeep command (apt-get install ssdeep) in the following manner:
1013
1014 $ ssdeep ack388
1015
1016 ssdeep,1.0--blocksize:hash:hash,filename
1017
1018 6144:DrIx6zNhlY7zJc3VesoteSAV/EfjAyGXElheAt[REMOVED],"ack388"
1019
1020If you install the pyssdeep[3] module (Python bindings for ssdeep), you can also generate fuzzy hashes in your Python scripts, as shown in the following commands:
1021
1022 >>> from ssdeep import ssdeep
1023
1024 >>> s = ssdeep()
1025
1026 >>> print s.hash_file("ack388")
1027
1028 6144:DrIx6zNhlY7zJc3VesoteSAV/EfjAyGXElheAt[REMOVED]
1029
1030This recipe summarized a few of the ways you can identify files for organization and determine if they already exist in your collection. In the next few recipes, you'll learn how to start gathering more detailed information on the samples.
1031
1032
1033Checkpoint 9a: Filetype Demo
1034Checkpoint 9b: Hash Demo, including ssdeep
1035
1036Recipe 3-7: Writing a Multiple-AV Scanner in Python
1037
1038Image from book You can find supporting material for this recipe on the companion DVD.
1039
1040Many antivirus products include a command-line utility that you can execute from your own scripts to scan files. If you install several of these antivirus products, you can leverage the signatures and detection capabilities of the multiple vendors without the potential privacy issues associated with public online services. All you need to do is create a script that invokes each of the command-line utilities sequentially, captures the results, and produces a report in the format of your choice.
1041
1042
1043  	
1044
1045Important Note
1046	
1047
1048Scanning malware samples has inherent risks. A file could be specially crafted to exploit an antivirus engine and thus compromise your system. For example, Alex Wheeler and Neel Mehta showed how to get remote, unauthenticated system-level access to a machine running ClamAV due to a flaw in the scanner's file format parsers (see www.blackhat.com/presentations/bh-usa-05/bh-us-05-wheeler.pdf). We highly recommend you perform all scanning of malware in a controlled environment that can be monitored for suspicious activity.
1049
1050
1051Choosing the Scanners
1052
1053Selecting antivirus products for your multi-scanner typically depends on several factors including the availability of a command-line version, supported platforms, and licensing. When deciding which scanners to use, make sure that you properly license any scanners according to their acceptable use policies. Often, antivirus products have different licenses for research, home, and corporate use. Table 3-1 shows a few antivirus vendors that provide free personal or research command-line scanners.
1054
1055
1056Table 3-1: Available AV Vendors with Free, Personal Command-Line Scanners
1057Open table as spreadsheet
1058
1059Vendor
1060	
1061
1062Description
1063	
1064
1065Web Site
1066
1067ClamAV
1068	
1069
1070An open source, free version
1071	
1072
1073http://www.clamav.net
1074
1075AntiVir
1076	
1077
1078A free Windows personal edition
1079	
1080
1081http://www.free-av.com/en/products/index.html
1082
1083AVG
1084	
1085
1086A free Linux/FreeBSD edition
1087	
1088
1089http://free.avg.com/us-en/download?prd=afl
1090
1091BitDefender
1092	
1093
1094A free Windows personal version
1095	
1096
1097http://www.bitdefender.com/PRODUCT-14-en--BitDefender-Free-Edition.html
1098
1099Panda
1100	
1101
1102A free research and academic command-line scanner for Windows
1103	
1104
1105http://research.pandasecurity.com/free-commandline-scanner/
1106
1107F-Prot
1108	
1109
1110A free Linux/FreeBSD for personal use
1111	
1112
1113http://www.f-prot.com/products/home_use/linux/
1114
1115Many other vendors, such as Sophos and McAfee, provide 30-day free trials of their antivirus products. If you are interested in testing this type of script, a 30-day trial can allow you to tweak your parameters and reports before you decide to buy.
1116
1117Choosing an OS
1118
1119The operating system on which you want to run your multi-scanner may also limit your choices. Virtually all vendors support Windows, a few support Linux, and very few support Mac OS X. In some cases, you may be able to use Wine to run some scanners on Linux or Mac OS X. Wine emulates Windows API calls, and we'll show you how to use it in this recipe.
1120
1121The Book's Example Multi-Scanner
1122
1123On the book's DVD, you can find an example multi-scanner Python script named av_multiscan.py. This version of the script is not a comprehensive scanner; rather, it provides you with a starting point to add your own antivirus products. The version on the DVD allows you to use the following:
1124
1125    ClamAV with default signatures
1126
1127    ClamAV with custom signatures
1128
1129    YARA
1130
1131    f-prot using default signatures
1132
1133    OfficeMalScanner
1134
1135    Team CYMRU MHR4[4] (Malware Hash Registry) score
1136
1137The most important part of the multi-scanner is the execution of the command-line utilities and the interpretation of their results. This is handled by using the Python subprocess[5] module, which allows you to spawn a new process, specify command-line parameters, and redirect STDIN, STDOUT, and STDERR. In the multi-scanner, we launch the various command-line scanners with the appropriate options and capture STDOUT. After execution, you need to parse STDOUT to find the results from the scan.
1138
1139Scanning with ClamAV
1140
1141If a file triggers a signature in the ClamAV database, clamscan prints a line of output with the name of the file and the name of the signature, separated by a colon, like this:
1142
1143 $ clamscan 5728c58b8f21678a2317abcf7fdffe6b
1144
1145 5728c58b8f21678a2317abcf7fdffe6b: Exploit.PDF-1880 FOUND
1146
1147The following function demonstrates how av_multiscan.py processes results from the ClamAV engine.
1148
1149 clam_conf_file = "clam_shellcode.ndb"
1150
1151 path_to_clamscan = "/usr/local/bin/clamscan"
1152
1153 def clam_custom(fname):
1154
1155 # check to see if the right path for the scanner and
1156
1157 # the custom configuration file exist
1158
1159 if os.path.isfile(path_to_clamscan) and \
1160
1161 os.path.isfile(clam_conf_file):
1162
1163 output = subprocess.Popen([path_to_clamscan, \
1164
1165 "-d", clam_conf_file, fname], \
1166
1167 stdout = subprocess.PIPE).communicate()[0]
1168
1169 result = output.split('\n')[0].split(': ')[1]
1170
1171 else:
1172
1173 result = 'ERROR - %s not found' % path_to_clamscan
1174
1175 return ({'name': 'clam_custom', 'result': result })
1176
1177Make sure you configure the path_to_clamscan (location of the clamscan binary) and clam_conf_file (location of your custom signature database) variables by modifying the av_multiscan.py script before using it.
1178
1179Scanning with OfficeMalScanner
1180
1181If you install Wine[6] (apt-get install wine) you can run many Windows command-line antivirus scanners directly on Linux or Max OS X. For example, if you're developing your multi-scanner on a non-Windows platform, you can still integrate Windows executables such as OfficeMalScanner.exe by using Wine. The following function demonstrates how to use Wine.
1182
1183 path_to_officemalscanner = "/data/OfficeMalScanner/OfficeMalScanner.exe"
1184
1185 def officemalscanner(fname):
1186
1187 if os.path.isfile(path_to_officemalscanner):
1188
1189 env = os.environ.copy()
1190
1191 env['WINEDEBUG'] = '-all'
1192
1193 output = subprocess.Popen(["wine", path_to_officemalscanner,
1194
1195 fname, "scan", "brute"],
1196
1197 stdout = subprocess.PIPE,
1198
1199 stderr = None, env=env).communicate()[0]
1200
1201 if "Analysis finished" in output:
1202
1203  output = output.split('\r\n')
1204
1205 while "Analysis finished" not in output[0]:
1206
1207 output = output[1:]
1208
1209 result = output[3]
1210
1211 else:
1212
1213 result = "Not an MS Office file" 
1214
1215 else:
1216
1217 result = 'ERROR - %s not found' % path_to_officemalscanner
1218
1219 return ({'name': 'officemalscanner', 'result': result})
1220
1221To suppress the standard Wine debug messages, the code creates a new environment variable named WINEDEBUG with the value -all. This way, the output of the command only contains the OfficeMalScanner.exe results. In particular, the code extracts the malicious index value calculated by OfficeMalScanner (a numerical value that represents how malicious a file is). For more information about OfficeMalScanner and its scoring system, see Recipe 6-11.
1222
1223Using the Multi-Scanner
1224
1225The av_multiscan.py script requires one parameter, -f, which specifies the file you would like to scan. You can use it in the following manner:
1226
1227 $ python av_multiscan.py -f sample.exe
1228
1229 filename: sample.exe
1230
1231 filesize: 22016
1232
1233 md5: 66a736c5f37d1769db3a2028e7a1c5b4
1234
1235 ssdeep: 384:OG7iQzd6Iw+wyMHtwMF/x4GTTIpABkG[...]
1236
1237 clamav: OK
1238
1239 clam_custom: OK
1240
1241 yara: 'mz_executable'
1242
1243 yara_packer: 'ASPackv1061bAlexeySolodovnikov'
1244
1245 officemalscanner: Not an MS Office file
1246
1247 cymru_hash_db: Sat, 12 Dec 2009 11:32:50 - 60
1248
1249As you can see, sample.exe is packed with AsPack. The file didn't trigger any ClamAV signatures, but Team Cymru's MHR score is 60 (which indicates 60 percent detection across antivirus scanners that they use).
1250
1251The -v flag to av_multiscan.py produces more verbose output. The example that follows shows how to scan a Microsoft Word document using the verbose flag.
1252
1253 $ python av_multiscan.py -v -f bad.doc
1254
1255 [+] Using YARA signatures magic.yara
1256
1257 [+] Using ClamAV signatures clam_shellcode.ndb
1258
1259 filename bad.doc
1260
1261 filesize 568832
1262
1263 md5 a5f8f82d2e5ad953bb986bb2bbcd20ee
1264
1265 ssdeep 6144:L4Rz0Q/DMtI+XDpiUxchygVNFGGsOkxh:mz0Q/F4
1266
1267 clamav OK
1268
1269 clam_custom shellcode_xor.UNOFFICIAL FOUND
1270
1271 yara 'office_magic_bytes' 'word_document'
1272
1273 yara_packer officemalscanner bad.doc seems to be malicious! Malicious Index = 31
1274
1275 cymru_hash_db Sun, 14 Mar 2010 14:13:28 - NO_DATA
1276
1277The results show that bad.doc did not trigger any signatures in the default ClamAV database and the file's hash isn't recognized by MHR. However, it did trigger the custom ClamAV signature we presented in Recipe 3-2 and OfficeMalScanner assigned a malicious index value of 31 (which is quite high). Here are some ideas you may find useful to implement in your multi-scanner:
1278
1279    Write a plug-in that stores the output in a database for easy searching and retrieval.
1280
1281    Add additional antivirus products to the scanning engine.
1282
1283    Perform extra actions based on file type (for example, scan executables with the PE file scanner presented in Recipe 3-8).
1284
1285
1286Checkpoint 10: Multi-AV Scanner Demo
1287
1288Recipe 3-8: Detecting Malicious PE Files in Python
1289
1290Image from book You can find supporting material for this recipe on the companion DVD.
1291
1292Executables on Windows must conform to the PE/COFF (Portable Executable/Common Object File Format) specification. This includes, but is not limited to, console and GUI applications (.exe), Dynamic Link Libraries (.dll), kernel drivers (.sys), and ActiveX controls (.ocx). We don't cover the PE file basics, because you can find that in many other books and online articles. For a good introduction, see Matt Pietrek's two-part series: Peering Inside the PE[7] and An In-Depth Look into the Win32 Portable Executable File Format.[8]
1293
1294In this recipe, the authors show you several ways to detect suspicious files based on values in the PE header. Thus, independent of any antivirus scanners, you can use heuristics to quickly determine which files exhibit suspicious attributes. The code for this recipe uses Ero Carrera's pefile,[9] which is a Python module for parsing PE headers. You can find the script, named pescanner.py, on the book's DVD. It currently detects the following criteria:
1295
1296    Files with TLS entries: TLS entries are functions that execute before the program's main thread, thus before the initial breakpoint set by debuggers. Malware typically uses TLS entries to run code before your debugger gets control. The pescanner.py script prints the addresses of all TLS callback functions.
1297
1298    Files with resource directories: Resource directories can contain arbitrary data types such as icons, cursors, and configurations. If you're scanning an entire system32 directory, then you will likely find many false positives because resource directories are legitimate. However, if you're scanning a folder full of malware, the presence of a resource directory likely indicates that the file drops another executable at run-time. The pescanner.py script extracts all resources from the PE file and runs them through the file type identification process described in Recipe 3-6.
1299
1300    Suspicious IAT entries: Imported functions can indicate how a program behaves at run-time. You can create a list of API functions that are suspicious and then produce an alert whenever you find a malware sample that imports a function from your list. The pescanner.py script has a default list of about 15 APIs, but it's up to you to add additional ones.
1301
1302    Suspicious entry point sections: An entry point section is the name of the PE section that contains the AddressOfEntryPoint. The AddressOfEntryPoint value for legitimate, or non-packed, files typically resides in a section named .code or .text for user mode programs, and PAGE or INIT for kernel drivers. Therefore, you can detect potentially packed files if the entry point resides in a section that is not in your list of known-good sections.
1303
1304    Sections with zero-length raw sizes: The raw size is the amount of bytes that a section requires in the file on disk (as opposed to bytes required when the section is mapped into memory). The most common reason a raw size would be zero on disk but greater than zero in memory is because packers copy decrypted instructions or data into the section at run-time.
1305
1306    Sections with extremely low or high entropy: Entropy is a value between 0 and 8 that describes the randomness of data. Encrypted or compressed data typically has high entropy, whereas a long string of the same character has low entropy. By calculating entropy, you can get a good idea of which sections in a PE file contain packed or abnormal code.
1307
1308    Invalid timestamps: The TimeDateStamp field is a 32-bit value (the number of seconds since December 31st, 1969, 4 P.M.) that indicates when the linker or compiler produced the PE file. Malware authors (and packers) obscure this value to hide the true build date. If pescanner.py detects an invalid date, it produces an alert.
1309
1310    File version information: A PE file's version information may contain the name of the person or company who created the file, a description of the file, a version and/or build number, the original file name, and other comments. This type of information is not available in all PE files, but many times malware authors will accidentally leave it in or intentionally forge the values. In both cases, the information yields interesting forensic evidence.
1311
1312Example 1: UPX
1313
1314The command that follows shows example output from a malware sample packed with UPX. The entry point (EP) is 0x4292e0, which lands in the section named UPX1. Therefore, pescanner.py adds the [SUSPICIOUS] tag on that line. The PEiD signatures can report the exact version of UPX (2.90). Under the sections header, UPX0 and UPX1 are tagged as suspicious, but for different reasons. UPX0 is suspicious because its raw size is zero. UPX1 is suspicious because its entropy score is very high (7.91 out of 8.00).
1315
1316 $ python pescanner.py /samples/22a9c61c71fa5cef552a94e479dfe41e
1317
1318 Meta-data ============================================================
1319
1320 File: /samples/22a9c61c71fa5cef552a94e479dfe41e
1321
1322 Size: 72704 bytes
1323
1324 Type: MS-DOS executable PE for MS Windows (GUI) Intel 80386 32-bit
1325
1326 MD5: 22a9c61c71fa5cef552a94e479dfe41e
1327
1328 SHA1: 14ac258df52d0131c5984b00dc14960ee94e6aad
1329
1330 ssdeep: 1536:JxXOg1j5jBWSNzrpGhDZuiq3AC+wcnG4Pqvtuz+[REMOVED]
1331
1332 Date: 0x49277573 [Sat Nov 22 02:58:59 2008 UTC]
1333
1334 EP: 0x4292e0 (UPX1) [SUSPICIOUS]
1335
1336 Packers: UPX 2.90 [LZMA] -> Markus Oberhumer, Laszlo Molnar & John Reiser
1337
1338 Sections
1339
1340 ============================================================
1341
1342 Name VirtAddr VirtSize RawSize Entropy
1343
1344 ------------------------------------------------------------
1345
1346 UPX0 0x1000 0x17000 0x0 0.000000 [SUSPICIOUS]
1347
1348 UPX1 0x18000 0x12000 0x11600 7.912755 [SUSPICIOUS]
1349
1350 UPX2 0x2a000 0x1000 0x200 2.71365
1351
1352Example 2: Trojan Droppers
1353
1354The command that follows shows the pescanner.py output for a trojan dropper. The file triggered our YARA rule for embedded PE files. The information in the resource section validates this finding—there is a resource named BIN at RVA 0x3580 with an executable file type. You can expect that this malware would drop a 0x4200 byte file when executed on a system.
1355
1356 $ python pescanner.py /samples/01C96CD0699DD2C0_Winlr66_sys.PE
1357
1358 Meta-data
1359
1360 ============================================================
1361
1362 File: /samples/01C96CD0699DD2C0_Winlr66_sys.PE
1363
1364 Size: 31616 bytes
1365
1366 Type: MS-DOS executable PE for MS Windows (native) Intel 80386 32-bit
1367
1368 MD5: d884094437fe2d8fac33da75de2e96be
1369
1370 SHA1: 8b57624f954b0baefd4941bf44ad8ef7cad3b463
1371
1372 ssdeep: 768:oxQK0HWA4bci5neO8NCxpW2ghFHTVMgscZ4Rw:oxQVUci5eO8ExY2grzVTsx
1373
1374 Date: 0x48B531A2 [Wed Aug 27 10:51:14 2008 UTC]
1375
1376 EP: 0x10b90 (.text)
1377
1378 Signature scans
1379
1380 ============================================================
1381
1382 YARA: embedded_exe
1383
1384 0x35ce => This program cannot be run in DOS mode
1385
1386 Resource entries
1387
1388 ============================================================
1389
1390  Name RVA Size Type 
1391
1392 ------------------------------------------------------------
1393
1394 BIN 0x3580 0x4200 MS-DOS executable PE
1395
1396 Sections
1397
1398 ============================================================
1399
1400 Name VirtAddr VirtSize RawSize Entropy
1401
1402 ------------------------------------------------------------
1403
1404 .text 0x480 0x26f4 0x2700 5.705293
1405
1406 .rdata 0x2b80 0x180 0x180 3.830066
1407
1408 .data 0x2d00 0x2d5 0x300 0.316915 [SUSPICIOUS]
1409
1410 INIT 0x3000 0x4d8 0x500 5.202389
1411
1412 .rsrc 0x3500 0x4280 0x4280 7.088351 [SUSPICIOUS]
1413
1414 .reloc 0x7780 0x394 0x400 4.373185
1415
1416The names of resource entries are similar to names of PE sections in the sense that they can easily be forged. Just because a section is named .rdata doesn't mean it contains read-only data. Likewise, attackers can load an executable into a resource with one of the standard names such as RT_ICON, RT_STRING, or RT_CURSOR. This is why we scan the entire file with YARA signatures and also perform individual file type identification on each resource entry.
1417
1418Example 3: IAT and Version Information
1419
1420The following command shows the output for a 2007 Zeus sample (date based on the timestamp). You can see that the file imports API functions related to code injection (WriteProcessMemory) and launching processes (CreateProcess, WinExec). The version information has clearly been obscured or randomized. For the sake of brevity, we've removed the PE sections and resources.
1421
1422 $ python pescanner.py /samples/sdra64.exe
1423
1424 Meta-data
1425
1426 ============================================================
1427
1428 File: /samples/sdra64.exe
1429
1430 Size: 124416 bytes
1431
1432 Type: MS-DOS executable PE for MS Windows (GUI) Intel 80386 32-bit
1433
1434 MD5: a99889e994e8e2248f5779b54505aa81
1435
1436 SHA1: 93437058ddfdd2c97b3ff07e3c7853bd0441065c
1437
1438 ssdeep: 3072:CNIl9M0O6M6PYpfaUmhylsDXczSYilhnJ+toJ+T0nW1paaM[REMOVED]
1439
1440 Date: 0x471FB71B [Wed Oct 24 21:20:27 2007 UTC]
1441
1442 EP: 0x416c33 (.text)
1443
1444 Suspicious IAT alerts
1445
1446 ============================================================
1447
1448 ReadProcessMemory
1449
1450 WriteProcessMemory
1451
1452 CreateProcessW
1453
1454 VirtualAllocEx
1455
1456 CreateProcessA
1457
1458 WinExec
1459
1460 Version info
1461
1462 ============================================================
1463
1464 LegalCopyright: Gaaqnewicyvee
1465
1466 InternalName: Maamduas
1467
1468 CompanyName: Leepcaseuzevwee
1469
1470 LegalTrademarks: Eludpuuhcaidgyv
1471
1472 ProductName: Toxiwoewikaxoq
1473
1474 FileDescription: Kunuwihycuap
1475
1476 OriginalFilename: Calyi
1477
1478 Translation: 0x0409 0x04b0
1479
1480Here are some additional facts about pescanner.py and malicious PE attributes that you may find useful:
1481
1482    You can pass pescanner.py a directory instead of an individual file name. The script will recursively parse all PE files found in the directory and sub-directories.
1483
1484    The main code for pescanner.py is implemented as a Python class named PEScanner. Therefore, instead of using it on command-line, you can import the module from your own Python scripts. Recipe 8-7 shows how to import PEScanner into an automated sandbox.
1485
1486    You can use several additional heuristics to detect malicious PE files. For other ideas, reference the Parsing Malicious and Malformed Executables[10] document by researchers at Sunbelt Software.
1487
1488Checkpoint 11a: UPX Detection Demo (SKIP THIS FOR NOW) 
1489Checkpoint 11b: Trojan Dropper Detection Demo  (SKIP THIS FOR NOW)
1490Checkpoint 11c: IAT and Version Information Demo  (SKIP THIS FOR NOW)
1491
1492Recipe 3-9: Finding Similar Malware with ssdeep
1493
1494Ssdeep[11] is an application by Jesse Kornblum that calculates context-triggered piecewise hashes, also known as fuzzy hashes. Using the ssdeep command, you can determine the percent similarity between two or more files. For example, you could perform the following tasks:
1495
1496    Detecting source code reuse: Given a file containing several functions, you could search through archives looking for any files that may contain the same functions.
1497
1498    Finding related malware: Given the ssdeep hash of a malware sample, you could find variants of the same family.
1499
1500    Finding forensic artifacts on disk: Given all or part of an image, document, or e-mail, you could scan a raw disk looking for sectors that contain similar content. This could reveal content on suspect machines even if the original files were deleted.
1501
1502    Detecting infections across computers on a network: Given a memory dump of a machine infected with malware, you could extract the memory segments of all machines in the network and detect if the same or similar malware has infected other systems.
1503
1504    Detecting self-modifying code: Given the ssdeep hash of a file on disk, you could compare it to the ssdeep hash of the file running in memory. If the two hashes are less than 75–80 percent similar, then the file is probably packed or self-modifying.
1505
1506Finding Similar Malware
1507
1508The following commands show how to use ssdeep for comparing two arbitrary binary files. As you can see, although the MD5 checksum is different, the files are 49 percent similar.
1509
1510 $ md5sum INSTALL.COM Attach.exe
1511
1512 MD5 (INSTALL.COM) = a85bd266f431cf2a4bcc466f8bfa5b01
1513
1514 MD5 (Attach.exe) = 9f922a71356c177202a7b88538c234ef
1515
1516 $ ssdeep -b INSTALL.COM > hash.txt
1517
1518 $ ssdeep -bm hash.txt Attach.exe
1519
1520 Attach.exe matches INSTALL.COM (49)
1521
1522The following example shows how to use ssdeep to find related malware in an archive of samples. The first command shows that there are just over 6,000 files in the directory, and the second command generates the similarity output.
1523
1524 $ ls Malware | wc –l
1525
1526 6346
1527
1528 $ ssdeep -brd Malware/
1529
1530 01C84D3BB350E080_ap2_exe.PE matches 01C84D3BB34F5950_002[1]_gif.PE (100)
1531
1532 01C84D3BBDBB5EB0_ap1_exe.PE matches 01C84D3BBDA2EBB0_003[1]_gif.PE (100)
1533
1534 726769232.exe matches 01C72E743C20AE50_944983008_exe.PE (100)
1535
1536 944983008.exe matches 01C96CD01D196A30_csrssc_exe.PE (100)
1537
1538 944983008.exe matches 01C96CD1C6F237D0_3239120928_exe.PE (100)
1539
1540 _812.COM matches _737.COM (79)
1541
1542 api32.dll matches 01C96CCF695F44C0_ld_exe.PE (75)
1543
1544 api32.dll matches 01C96CCF6980E2E0_api32_dll.PE (100)
1545
1546 api32.dll matches 01C96CCFA48FAC00_ld_exe.PE (75)
1547
1548 Backdoor.IRC.Cloner.j matches Backdoor.IRC.Cloner (69)
1549
1550 Backdoor.IRC.Cloner.k matches Backdoor.IRC.Cloner.g (47)
1551
1552 Backdoor.IRC.Cloner.r matches Backdoor.IRC.Cloner.o (44)
1553
1554 Backdoor.IRC.Cloner.x matches Backdoor.IRC.Cloner.o (99)
1555
1556 Backdoor.IRC.Cloner.x matches Backdoor.IRC.Cloner.r (44)
1557
1558Finding Similar Malware (in Memory)
1559
1560The following example shows you how to extract suspicious memory segments using the malfind Volatility plug-in (see Recipe 16-6) and then compare them with ssdeep. The first command dumps suspicious memory segments to the samples directory. The second command lists the contents of the samples directory, and shows (based on the file name) that the plug-in identified suspicious content in memory range 1f00000–1f27fff in process with PID 1064, and so on. The third command shows that most of the memory segments from one process are at least 50 percent similar to the segments extracted from all other processes. This is indicative of malware that injects the same body of code into multiple processes.
1561
1562 $ python volatility.py malfind -d samples -f memory.dmp > /dev/null
1563
1564 $ ls -Al samples/
1565
1566 total 6160
1567
1568 163840 Mar 31 11:14 1064.1f00000-1f27fff.dmp
1569
1570 163840 Mar 31 11:14 1112.880000-8a7fff.dmp
1571
1572 163840 Mar 31 11:14 1156.9c0000-9e7fff.dmp
1573
1574 163840 Mar 31 11:14 1320.6b0000-6d7fff.dmp 
1575
1576 163840 Mar 31 11:14 1488.ec0000-ee7fff.dmp
1577
1578 4096 Mar 31 11:14 1624.1b50000-1b50fff.dmp
1579
1580 28672 Mar 31 11:14 1624.1d80000-1e7ffff.dmp
1581
1582 163840 Mar 31 11:14 1624.ac0000-ae7fff.dmp
1583
1584 163840 Mar 31 11:14 1740.800000-827fff.dmp
1585
1586 163840 Mar 31 11:14 1760.3c0000-3e7fff.dmp
1587
1588 163840 Mar 31 11:14 1768.b00000-b27fff.dmp [REMOVED]
1589
1590 $ ssdeep -brd samples/
1591
1592 1112.880000-8a7fff.dmp matches 1064.1f00000-1f27fff.dmp (54)
1593
1594 1156.9c0000-9e7fff.dmp matches 1064.1f00000-1f27fff.dmp (58)
1595
1596 1156.9c0000-9e7fff.dmp matches 1112.880000-8a7fff.dmp (57)
1597
1598 1320.6b0000-6d7fff.dmp matches 1064.1f00000-1f27fff.dmp (54)
1599
1600 1320.6b0000-6d7fff.dmp matches 1112.880000-8a7fff.dmp (57)
1601
1602 1320.6b0000-6d7fff.dmp matches 1156.9c0000-9e7fff.dmp (58)
1603
1604 1488.ec0000-ee7fff.dmp matches 1064.1f00000-1f27fff.dmp (58)
1605
1606 1488.ec0000-ee7fff.dmp matches 1112.880000-8a7fff.dmp (54)
1607
1608 1488.ec0000-ee7fff.dmp matches 1156.9c0000-9e7fff.dmp (57)
1609
1610 1488.ec0000-ee7fff.dmp matches 1320.6b0000-6d7fff.dmp (50)
1611
1612 1624.ac0000-ae7fff.dmp matches 1064.1f00000-1f27fff.dmp (50) [REMOVED]
1613
1614When you use ssdeep, you can pass it a parameter such as –t 60 to only display matches above a given threshold. If 60 percent isn't what you need, you'll have to adjust it depending on your objectives.
1615
1616Checkpoint 12: SSDeep - Similar Malware Demo
1617
1618Recipe 3-10: Detecting Self-modifying Code with ssdeep
1619
1620Image from book You can find supporting material for this recipe on the companion DVD.
1621
1622This recipe shows how you can use ssdeep to compare processes in memory with their corresponding files on disk. It is normal for processes to change slightly at run-time—for example, when the program modifies global variables. However, code that is packed or that self-mutates (such as polymorphic viruses) will change significantly at run-time. Therefore, the copy of the code in memory will be much different from the code on disk.
1623
1624Using ssdeep_procs.py
1625
1626To use the ssdeep_procs.py script on the book's DVD, you need to install the ctypes and pywin32[12] modules for Python on the target system. pywin32 provides wrappers around Windows API functions so you can call them from Python. If you want to run the script from a USB drive, you can convert ssdeep_procs.py to an executable with py2exe.[13]
1627
1628The following command demonstrates how to use the ssdeep_procs.py script. The test bed consisted of an XP system running processes packed with VMProtect, FSG, Neolite, and UPX. Notice how the four packed processes are 55 percent, 72 percent, 75 percent, and 0 percent similar, respectively, to their files on disk. All other processes are between 83 percent and 99 percent similar to their files on disk.
1629
1630 C:\> python ssdeep_procs.py
1631
1632 Process Pid Matched
1633
1634 smss.exe 588 96%
1635
1636 csrss.exe 660 96%
1637
1638 winlogon.exe 692 97%
1639
1640 services.exe 736 94%
1641
1642 lsass.exe 748 96%
1643
1644 vmacthlp.exe 904 96%
1645
1646 svchost.exe 928 91%
1647
1648 svchost.exe 1000 91%
1649
1650 Explorer.EXE 1584 97% 
1651
1652 spoolsv.exe 1724 99% 
1653
1654 wscntfy.exe 1276 91%
1655
1656 alg.exe 2076 94%
1657
1658 wuauclt.exe 3724 86%
1659
1660 TSCHelp.exe 3168 83%
1661
1662 IEXPLORE.EXE 3664 97%
1663
1664 cmd.exe 1036 94%
1665
1666 p-vmprotect.exe 372 55% possible packed exe
1667
1668 p-fsg.exe 3200 72% possible packed exe
1669
1670 p-neolite.exe 4084 75% possible packed exe
1671
1672 p-upx.exe 3860 0% possible packed exe
1673
1674 python.exe 4044 96%
1675
1676The ssdeep_procs.py script can detect another malicious behavior called "hollow processes" (which we discuss more in Recipe 15-8). Hollow processes are legitimate programs (such as notepad.exe) started by malware. Once the program is running, the malware replaces the body, or executable instructions, of the legitimate program with malicious instructions. This is a form of code injection that you can detect using ssdeep, because the notepad.exe file on disk will differ significantly from the one in memory.
1677
1678
1679Checkpoint 13: SSDeep - Self-Modifying Code Demo
1680
1681Recipe 3-11: Comparing Binaries with IDA and BinDiff
1682
1683Image from book You can find supporting material for this recipe on the companion DVD.
1684
1685Binary diffing is a fundamental technique used in reverse engineering. It is especially popular in the vulnerability research realm (for analyzing vendor patches). However, it also has a place in malware research. While ssdeep can help you identify variants of the same malware family, it cannot tell you exactly what changed. If you have two files that are 75 percent similar, you still have some work to do before your analysis is complete. For example, did the attackers remove the brute-force password guessing code? Did they add a rootkit component to hide files on disk? Perhaps both files exhibit all of the same behaviors, but the attackers just used a different packer. This recipe shows you how to address these types of questions using BinDiff,[14] which is an IDA Pro plug-in for binary diffing.
1686
1687BinDiff examines files after you load them into IDA Pro. It determines which functions exist in both files based on attributes such as the function's CRC or hash value, the number of instructions in each basic block of a function, the number of cross-references to and from a function, and a variety of other algorithms (see the online BinDiff manual[15] for more details). Once you know which functions exist in both binaries, you can use BinDiff's color-coded GUI to zoom-in and examine the changes at the instruction-level.
1688
1689Good Old Zeus …
1690
1691The following summary describes the context and objective for the demonstration that we present in this recipe.
1692
1693In November 2006, the authors wrote a research paper[16] on one of the first Zeus variants seen in the wild. During the reverse engineering phase, we loaded the Zeus binary in IDA Pro and named as many functions as possible based on their behavior. Zeus stole information from victim computers, compressed it, encrypted it, and sent it over the network to the attackers. Based on the algorithm we saw in the Zeus binary, we wrote a decryption tool to recover the stolen data. However, after a while, the tool stopped working. Clearly, the Zeus authors had updated the code in some way that prevented our old decryption algorithm from working, and we needed to figure out how to fix it.
1694
1695Using BinDiff
1696
1697The following steps describe how to use BinDiff to quickly locate the decryption function and determine exactly how it changed.
1698
1699    Create an IDA database (IDB) for both of the files that you plan to diff. Designate one as the primary and one as the secondary. In our case, we'll use new_zeus.idb (a sample from December 2008) as the primary and old_zeus.idb (the original sample from November 2006) as the secondary.
1700
1701    With the primary IDB open in IDA and the secondary IDB closed, click Edit è Plugins è zynamics BinDiff 3.0 (or use the keyboard shortcut Shift+D).
1702
1703    When you see the prompt shown in Figure 3-1, click Diff Database and select your secondary IDB.
1704
1705    Image from book
1706    Figure 3-1: BinDiff's main selection menu
1707
1708    When the diff is complete, you'll have the following new tabs in IDA:
1709
1710        Statistics: A summary of the overall similarity between the two files
1711
1712        Matched functions: Functions that exist in both files. This tab shows the degree of similarity (from 0 to 1.00), the degree of confidence (0 to 1.00), the address and names of the functions in both files, the algorithm BinDiff used to match, and statistics regarding the exact number of basic blocks, instructions, and edges that matched.
1713
1714        Primary unmatched: Functions in the primary file that cannot be matched with any functions in the secondary.
1715
1716        Secondary unmatched: Functions in the secondary file that cannot be matched with any functions in the primary.
1717
1718    Examine the matched functions tab. As you can see in Figure 3-2, the functions in the "name secondary" column (from old_zeus.idb) are labeled according to their functionality. BinDiff found a possible match for the function we labeled as DecodeData in 2006. The similarity score is .70/1.00 and the confidence level is .98/1.00.
1719
1720    Image from book
1721    Figure 3-2: Invoking a Visual Diff from the Matched Functions tab
1722
1723    To take a closer look at the two functions, right-click the line and select Visual Diff. This brings up the BinDiff GUI, as shown in Figure 3-3. The window is split into two parts. On the left, you see the potential match for the DecodeData function. On the right, you see the DecodeData function.
1724
1725    Image from book
1726    Figure 3-3: A Side-by-side flow graph of both functions in BinDiff
1727
1728    You can zoom and pan the graph as necessary in order to see exactly which instructions were added, removed, or modified. Remember, we're dealing with samples that were created more than two years apart, so some of the differences that you see may be due to the attackers using a new compiler version or operating system to develop the malware.
1729
1730    You can view the two functions from a different perspective by clicking the Assembler tab in the BinDiff GUI, as shown in Figure 3-4. Then use the scrollbar in the middle for navigation.
1731
1732    Image from book
1733    Figure 3-4: The secondary function uses compression, but the primary does not
1734
1735In Figure 3-4, you can see that the function in the secondary IDB calls RtlDecompressBuffer, whereas the function in the primary IDB does not. Both functions exhibit a similar algorithm that involves adding 5 to a number and subtracting 0xF9 from a number. Despite using different registers for temporary storage, the algorithms perform the same tasks. Thus, the only apparent difference between these two functions is the removal of RtlDecompressBuffer. In fact, the Zeus sample from 2006 uses compression and the sample from 2008 doesn't. This was the key to fixing our decryption tool.
1736
1737
1738Checkpoint 14 (optional): BinDiff & IDA Demo
1739
1740  	
1741
1742Note
1743	
1744
1745See the following resources for more information on determining relationships among binary files:
1746
1747    Zynamics VxClass: http://www.zynamics.com/vxclass.html
1748
1749    The State of Malware Family Ties by Ero Carerra and Peter Silberman: http://blog.mandiant.com/archives/934
1750
1751    DarunGrim: http://www.darungrim.org/
1752
1753    Tenable Security's PatchDiff2: http://cgi.tenablesecurity.com/tenable/patchdiff.php
1754
1755    CoreLabs' turbodiff: http://corelabs.coresecurity.com/index.php?module=Wiki&action=view&type=tool&name=turbodiff
1756
1757[3]http://libaccess.senecacollege.ca:2138/p/pyssdeep/
1758
1759[4]http://www.team-cymru.org/Services/MHR/
1760
1761[5]http://docs.python.org/library/subprocess.html
1762
1763[6]http://www.winehq.org/
1764
1765[7]http://msdn.microsoft.com/en-us/magazine/ms809762.aspx
1766
1767[8]http://msdn.microsoft.com/en-us/magazine/cc301805.aspx
1768
1769[9]http://libaccess.senecacollege.ca:2138/p/pefile/
1770
1771[10]http://www.sunbelt-software.com/ihs/alex/vb07_paper.pdf
1772
1773[11]http://ssdeep.sourceforge.net
1774
1775[12]http://sourceforge.net/projects/pywin32/
1776
1777[13]http://www.py2exe.org/
1778
1779[14]Zynamics VxClass: http://www.zynamics.com/bindiff.html,
1780
1781[15]http://www.zynamics.com/downloads/bindiff30-manual.zip
1782
1783[16]http://www.mnin.org/write/ZeusMalware.pdf