Metasm, the Ruby assembly manipulation suite

  • sample scripts in samples/ – read comments at the beginning of the files

  • all files are licensed under the terms of the LGPL

Author: Yoann Guillot <john at ofjj.net>

Basic overview:

Metasm allows you to interact with executables formats (ExeFormat): PE, ELF, Mach-O, Shellcode, etc. There are three approaches to an ExeFormat:

- compiling one up, from scratch
- decompiling an existing format
- manipulating the file structure

Ready-to-use scripts can be found in the samples/ subdirectory, check the comments in the scripts headers. You can also try the –help argument if you're feeling lucky.

For more information, check the doc/ subdirectory. The text files can be compiled to html using the misc/txt2html.rb script.

Here is a short overview of the Metasm internals.

Assembly:

When compiling, you start from a source text (ruby String, consisting mostly in a sequence of instructions/data/padding directive), which is parsed.

The string is handed to a Preprocessor instance (which handles #if, #ifdef, #include, #define, /* */ etc, should be 100% compatible with gcc -E), which is encapsulated in an AsmPreprocessor for assembler sources (to handles asm macro definitions, 'equ' and asm ';' comments). The interface to do that is ExeFormat#parse(text[, filename, lineno]) or ExeFormat.assemble (which calls .new, #parse and #assemble).

The (Asm)Preprocessor returns tokens to the ExeFormat, which parses them as Data, Padding, Labels or parser directives. Parser directives always start with a dot. They can be generic (.pad, .offset…) or ExeFormat-specific (.section, .import, .entrypoint…). They are handled by #parse_parser_instruction(). If the ExeFormat does not recognize a word, it is handed to its CPU instance, which is responsible for parsing Instructions (or raise an exception). All those tokens are stored in one or more arrays in the @source attribute of the ExeFormat (Shellcode's @source is an Array, for PE/ELF it is a hash

section name

> [Array of parsed data])

Every immediate value can be an arbitrary Expression (see later).

You can then assemble the source to binary sections using ExeFormat#assemble.

Once the section binaries are available, the whole binary executable can be written to disk using ExeFormat#encode_file(filename[, format]).

PE and ELF include an autoimport feature that allows automatic creation of import-related data for known OS-specific functions (e.g. unresolved calls to 'strcpy' will generate data so that the binary is linked against the libc library at runtime).

The samples/exe,pe,elfencode.rb can take an asm source file as argument and compile it to a working executable.

The CPU classes are responsible for parsing and encoding individual instructions. The current Ia32 parser uses the Intel syntax (e.g. mov eax, 42). The generic parser recognizes labels as a string at the beginning of a line followed by a colon (e.g. 'some_label:'). GCC-style local labels may be used (e.g. '1:', refered to using '1b' (backward) or '1f' (forward) ; may be redefined as many times as needed.) Data are specified using 'db'-style notation (e.g. 'dd 42h', 'db “blabla”, 0') See samples/asmsyntax.rb

EncodedData:

In Metasm all binary data is stored as an EncodedData. EncodedData has 3 main attributes:

- #data which holds the raw binary data (generally a ruby String, but see

VirtualString)

- #export which is a hash associating an export name (label name) to an offset

within #data

- #reloc which is a hash whose keys are offsets within #data, and whose values

are Relocation objects. A Relocation object has an endianness (:little/:big), a type (:u32 for unsigned 32bits) and a target (the intended value stored here). The target is an arbitrary arithmetic/logic Expression.

EncodedData also has a #virtsize (for e.g. .bss sections), and a #ptr (internal offset used when decoding things)

You can fixup an EncodedData, with a Hash variable name => value (value should be an Expression or a numeric value). When you do that, each relocation's target is bound using the binding, and if the result is calculable (no external variable name used in the Expression), the result is encoded using the relocation's size/sign/endianness information. If it overflows (try to store 128 in an 8bit signed relocation), an EncodeError exception is raised. Use the :a32 type to allow silent overflow truncating. If the relocation's target is not numeric, the target is unchanged if you use EncodedData#fixup, or it is replaced with the bound target with #fixup! .

Disassembly:

This code is found in the metasm/decode.rb source file, which defines the Disassembler class.

The disassembler needs a decoded ExeFormat (to be able to say what data is at which virtual address) and an entrypoint (a virtual address or export name). It can then start to disassemble instructions. When it encounters an Opcode marked as :setip, it asks the CPU for the jump destination (an Expression that may involve register values, for e.g. jmp eax), and backtraces instructions until it finds the numeric value.

On decoding, the Disassembler maintains a #decoded hash associating addresses (expressions/integer #normalize()d) to DecodedInstructions.

The disassembly generates an InstructionBlock graph. Each block holds a list of DecodedInstruction, and pointers to the next/previous block (by address).

The disassembler also traces data accesses by instructions, and stores Xrefs for them. The backtrace parameters can be tweaked, and the maximum depth to consider can be specifically changed for :r/:w backtraces (instruction memory xrefs) using #backtrace_maxblocks_data. When an Expression is backtracked, each walked block is marked so that loops are detected, and so that if a new code path is found to an existing block, backtraces can be resumed using this new path.

The disassembler makes very few assumptions, and in particular does not suppose that functions will return ; they will only if the backtrace of the 'ret' instructions is conclusive. This is quite powerful, but also implies that any error in the backtracking process can lead to a full stop ; and also means that the disassembler is quite slow.

The special method #disassemble_fast can be used to work around this when the code is known to be well-formed (ie it assumes that all calls returns)

When a subfunction is found, a special DecodedFunction is created, which holds a summary of the function's effects (like a DecodedInstruction on steroids). This allows the backtracker to 'step over' subfunctions, which greatly improves speed. The DecodedFunctions may be callback-based, to allow a very dynamic behaviour. External function calls create dedicated DecodedFunctions, which holds some API information (e.g. stack fixup information, basic parameter accesses…) This information may be derived from a C header parsed beforehand. If no C function prototype is available, a special 'default' entry is used, which assumes that the function has a standard ABI.

Ia32 implements a specific :default entry, which handles automatic stack fixup resolution, by assuming that the last 'call' instruction returns. This may lead to unexpected results ; for maximum accuracy a C header holding information for all external functions is recommanded (see samples/factorize-headers-peimports for a script to generate such a header from a full Visual Studio installation and the target binary).

Ia32 also implements a specific GetProcAddress/dlsym callback, that will yield the correct return value if the parameters can be backtraced.

The scripts implementing a full disassembler are samples/disassemble-gui.rb See the comments for the GUI key bindings.

ExeFormat manipulation:

You can encode/decode an ExeFormat (ie decode sections, imports, headers etc)

Constructor: ExeFormat.decode_file(str), ExeFormat.decode_file_header(str) Methods: ExeFormat#encode_file(filename), ExeFormat#encode_string

PE and ELF files have a LoadedPE/LoadedELF counterpart, that is able to work with memory-mmaped versions of those formats (e.g. to debugging running processes)

VirtualString:

A VirtualString is a String-like object: you can read and may rewrite slices of it. It can be used as EncodedData#data, and thus allows virtualization of most Metasm algorithms. You cannot change a VirtualString length. Taking a slice of a VirtualString will return either a String (for small sizes) or another VirtualString (a 'window' into the other). You can force getting a small VirtualString using the #dup(offset, length) method. Any unimplemented method called on it is forwarded to a frozen String which is a full copy of the VirtualString (should be avoided if possible, the underlying string may be very big & slow to access).

There are currently 3 VirtualStrings implemented:

  • VirtualFile, whichs loads a file by page-sized chunks on demand,

  • WindowsRemoteString, which maps another process' virtual memory (uses the

windows debug api through WinDbgAPI)

  • LinuxRemoteString, which maps another process' virtual memory (need ptrace

rights, memory reading is done using /proc/pid/mem)

The Win/Lin version are quite powerful, and allow things like live process disassembly/patching easily (using LoadedPE/LoadedELF as ExeFormat)

Debugging:

Metasm includes a few interfaces to allow live debugging. The WinOS and LinOS classes offer access to the underlying OS processes (e.g. OS.current.find_process('foobar') will retrieve a running process with foobar in its filename ; then process.mem can be used to access its memory.)

The Windows and Linux debugging APIs (x86 only) have a basic ruby interface (PTrace32, extended in samples/rubstop.rb ; and WinDBG, a simple mapping of the windows debugging API) ; those will be more worked on/integrated in the future.

A linux console debugging interface is available in samples/lindebug.rb ; it uses a SoftICE-like look and feel. This interface can talk to a gdb-server through samples/gdbclient.rb ; use [udp:]<host:port> as target.

The disassembler scripts allow live process interaction by using as target 'live:<pid or part of filename>'.

A generic debugging interface is available, it is defined in metasm/os/main.rb It may be accessed using the Metasm::OS.current.create_debugger('foo')

It can be viewed in action using the GUI and 'open live' target.

C Parser:

Metasm includes a hand-written C Parser. It handles all the constructs i am aware of, except hex floats:

- static const L"bla"
- variable arguments
- incomplete types
- __attributes__(()), __declspec()
- #pragma once
- #pragma pack()
- C99 declarators - type bla = { [ 2 ... 14 ].toto = 28 };
- Nested functions
- __int8 etc native types
- Label addresses (&&label)

Also note that all those things are parsed, but most of them will fail to compile on the Ia32 backend (the only one implemented so far.)

When you parse a C String using C::Parser.parse(text), you receive a Parser object. It holds a #toplevel field, which is a C::Block, which holds #structs, #symbols and #statements. The top-level functions are found in the #symbol hash whose keys are the symbol names, associated to a C::Variable object holding the functions. The function parameter/attributes are accessible through func.type, and the code is in func.initializer, which is itself a C::Block. Under it you'll find a tree-like structure of C::Statements (If, While, Asm, CExpressions…)

A C::Parser may be #precompiled to transform it into a simplified version that is easier to compile: typedefs are removed, control sequences are transformed in if () goto ; etc.

To compile a C program, use PE/ELF.compile_c, that will create a C::Parser with exe-specific macros defined (eg __PE__ or __ELF__).

The prefered way to create a C::Parser is to initialize it with a CPU and the desired ExeFormat, so that it is correctly initialized (eg type sizes: is long 4 or 8 bytes? etc) ; and may define preprocessor macros needed to correctly parse standard headers. Vendor-specific headers may need to use either #pragma prepare_visualstudio (to parse the Microsoft Visual Studio headers) or prepare_gcc (for gcc), the latter may be auto-detected (or may not). Vendor headers tested are VS2003 (incl. DDK) and gcc4 ; ymmv.

Currently the CPU#compilation of a C code will generate an asm source (text), which may then be parsed & assembled to binary code.

See ExeFormat#compile_c, and samples/exeencode.rb