Dorothy2

A malware/botnet analysis framework written in Ruby.

Introduction

Dorothy2 is a framework created for mass malware analysis. Currently, it is mainly based on analyzing the network behavior of a virtual machine where a suspicious executable was executed. However, static binary analysis and system behavior analysis will be shortly introduced in the next version.

Dorothy2 is a continuation of my degree's final project (Dorothy: inside the Storm ) that I presented on Feb 2009. The main framework's structure remained almost the same, and it has been fully detailed in my degree's final project or in this short paper. More information about the whole project can be found on the Italian Honeyproject website.

The framework is manly composed by four big elements that can be even executed separately:

  • The Dorothy analysis engine (this gem)
  • The Network analysis module (included in this module, but still not working perfectly at the time of this writing)
  • The Webgui (Coded in Rails by Andrea Valerio, and not yet included in this gem)
  • The Java Dorothy Drone (Mainly coded by Patrizia Martemucci and Domenico Chiarito, but not part of this gem and not publicly available.)

The first three modules are (or will be soon) publicly released under GPL 2/3 license as tribute to the the Honeynet Project Alliance. All the information generated by the framework - i.e. binary info, timestamps, dissected network analysis - are stored into a postgres DB (Dorothive) in order to be used for further analysis. A no-SQL database (CouchDB) is also used to mass strore all the traffic dumps thanks to the pcapr/xtractr technology.

I started to code this project in late 2009 while learning Ruby at the same time. Since then, I´ve been changing/improving it as long as my Ruby coding skills were improving. Because of that, you may find some parts of code not-really-tidy :)

Requirements

WARNING: The current version of Dorothy only utilizes VMWare ESX5 as its Virtual Sandbox Module (VSM). Thus, the free version of ESXi is not supported due to its limitations in using the vSphere 5 API. However, the overall framework could be easily customized in order to use another virtualization engine. Dorothy2 is very modular,and any customization or modification is very welcome.

Dorothy needs the following software (not expressly in the same host) in order to be executed:

  • VMWare ESX >= 5.0 (tip: if you download ESXi, you can evaluate ESX for 30 days)
  • Ruby 1.8.7
  • Postgres >= 9.0
  • At least one Windows virtual machine
  • One unix-like machine dedicated to the Network Analysis Engine(NAM) (tcpdump/ssh needed)
  • pcapr-local (only used by doroParser)
  • MaxMind libraries (only used by doroParser)

Installation

It is recommended to follow this step2step process:

  1. Set your ESX environment
    • Sample setup
  2. Install the required software
  3. Install Dorothy and libmagic libraries
  4. Start Dorothy, and configure it
  5. Use Dorothy

1. Set your ESX environment

  1. Basic configuration (ssh)

    • From vSphere:

      Configuration->Security Profile->Services->Proprieties->SSH->Options->Start and Stop with host->Start->OK
      
  2. Configure two separate virtual networks, one dedicated exclusively to the SandBoxes (See Sample Setups)

  3. Configure the Windows VMs used for sandboxing

  • Disable Windows firewall (preferred)
  • VMWare Tools must be installed in the Windows guest system.
  • Configure a static IP
  • After configuring everything on the Guest OS, create a snapshot of the sandbox VM from vSphere console. Dorothy will use it when reverting the VM after a binary execution.
  1. Configure the unix VM dedicated to the NAM
  2. Configure the NIC on the virtual machine that will be used for the network sniffing purpose (NAM).

    The vSwitch where the vNIC resides must allow the promisc mode, to enable it from vSphere:

    Configuration->Networking->Proprieties on the vistualSwitch used for the analysis->Double click on the virtual network used for the analysis->Securiry->Tick "Promiscuous Mode", then select "Accept" from the list menu.

  • Install tcpdump and sudo

        #apt-get install tcpdump sudo
    
  • Create a dedicated user for dorothy (e.g. "dorothy")

          #useradd dorothy
    
  • Add dorothy's user permission to execute/kill tcpdump to the sudoers file:

           #visudo
            add the following line:
            dorothy  ALL = NOPASSWD: /usr/sbin/tcpdump, /bin/kill
    

* Sample Setups

  1. Basic setup

    In the following example, the Dorothy gem is installed in the same host where Dorothive (the DB) resides. This setup is strongly recommended

    dorothy.basicsetup

  2. Advanced setup

    This setup is recommended if Dorothy is going to be installed in a Corporate environment. By leveraging a private VPN, all the sandbox traffics exits from the Corporate network with an external IP addresses.

dorothy.advancedsetup

2. Install the required software

  1. Install postgres

    $sudo apt-get install postgresql-9.1
    

    or

    http://www.postgresql.org/download/
    
  2. Configure a dedicated postgres user for Dorothy (or use root user instead, up to you :)

  3. Install the following packages

    $sudo apt-get install ruby1.8 rubygems postgresql-server-dev-9.1 libxml2-dev  libxslt1-dev libmagic-dev
    

For OSX users: all the above software are available through mac ports. A tip for libmagic: use brew instead:

    $ brew install libmagic
    $ brew link libmagic

Add a user dedicated to dorothy (or use the root one, up to you :)

3. Install Dorothy gem

*Install Dorothy gem

    $ sudo gem install dorothy2

4. Start Dorothy, and configure it!

  1. Install MaxMind libraries

  2. Start Dorothy

    $ dorothy_start -v
    

    The following message should appear

    [WARNING] It seems that the Dorothy configuration file is not present,
    please answer to the following question in order to create it now.
    
  3. Follow the instruction to configure

    • The environment variables (db, esx server, etc)
    • The Dorothy sources (where to get new binaries)
    • The ESX Virtual machines used for the analysis

The first time you execute Dorothy, it will ask you to fill those information in order to create the required configuration files into the etc/ folder. However, you are free to modify/create such files directly - configuration example files can be found there too.

5. Use Dorothy

  1. Copy a .exe or .bat file into $yourdorothyhome/opt/bins/manual/
  2. Execute dorothy with the malwarefolder source type (if you left the default one)

    $ dorothy_start -v -s malwarefolder

Usage

Usage:
$./dorothy_start [options]
where [options] are:
   --verbose, -v:   Enable verbose mode
  --infoflow, -i:   Print the analysis flow
--source, -s <s>:   Choose a source (from the ones defined in etc/sources.yml)
    --daemon, -d:   Stay in the background, by constantly pooling datasources

--SandboxUpdate, -S: Update Dorothive with the new Sandbox file --DorothiveInit, -D: (RE)Install the Dorothy Database (Dorothive) --help, -h: Show this message

Example

$dorothy_start -v -s malwarefolder $dorothy_stop

6. Debugging problems

I recognize that setting up Dorothy is not the easiest task of the world. By considering that the whole framework consists in the union of several 3rd pats, it is very likely that one of them will fail during the process. Below there are some tips about how understand the root-cause of your crash.

  1. Execute the Dorothy UnitTest (tc_dorothy_full.rb) that resides in its gem home directory

Example

$cd /opt/local/lib/ruby/gems/1.8/gems/dorothy2-0.0.1/test/
$ruby tc_dorothy_full.rb
  1. Set the verbose flag (-v) while executing dorothy

$dorothy_start -v -s malwarefolder

  1. Drop an email to info at honeynet.it with the output of your errors :)

Contributing

  1. Fork it
  2. Create your feature branch (git checkout -b my-new-feature)
  3. Commit your changes (git commit -am 'Add some feature')
  4. Push to the branch (git push origin my-new-feature)
  5. Create new Pull Request

Every contribution is more than welcome! For any help, please don't hesitate in contacting us at : info at honeynet.it

License

Dorothy is copyrighted by Marco Riccardi and is licensed under the following GNU General Public License version 3.

                GNU GENERAL PUBLIC LICENSE
                   Version 3, 29 June 2007

Copyright (C) 2007 Free Software Foundation, Inc. http://fsf.org/ Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed.