Class: Mu::Xtractr

Inherits:
Object
  • Object
show all
Defined in:
lib/mu/xtractr.rb,
lib/mu/xtractr/flow.rb,
lib/mu/xtractr/host.rb,
lib/mu/xtractr/term.rb,
lib/mu/xtractr/about.rb,
lib/mu/xtractr/field.rb,
lib/mu/xtractr/flows.rb,
lib/mu/xtractr/views.rb,
lib/mu/xtractr/packet.rb,
lib/mu/xtractr/stream.rb,
lib/mu/xtractr/content.rb,
lib/mu/xtractr/packets.rb,
lib/mu/xtractr/service.rb,
lib/mu/xtractr/stream/http.rb,
lib/mu/xtractr/test/tc_flow.rb,
lib/mu/xtractr/test/tc_host.rb,
lib/mu/xtractr/test/tc_term.rb,
lib/mu/xtractr/test/tc_field.rb,
lib/mu/xtractr/test/tc_flows.rb,
lib/mu/xtractr/test/tc_views.rb,
lib/mu/xtractr/test/tc_packet.rb,
lib/mu/xtractr/test/tc_stream.rb,
lib/mu/xtractr/test/tc_packets.rb,
lib/mu/xtractr/test/tc_service.rb,
lib/mu/xtractr/test/tc_xtractr.rb,
lib/mu/xtractr/test/stream/tc_http.rb

Overview

Mu::Xtractr

:main: Mu::Xtractr

This gem is Ruby front-end to the RESTful API that xtractr[http://www.pcapr.net/xtractr] provides. We primarily use this for unit testing xtractr’s API, but on its own this gem provides for a powerful programmable interface into xtractr and is a super fast way to extract information out of large pcaps.

Getting Started

First download xtractr from www.pcapr.net/xtractr. Follow the instructions to index your pcap. Finally run xtractr in browse mode and then you can hang out in IRB poking around flows and packets.

Examples

You can run the xtractr-gem from within IRB which makes it a fun interactive network forensics tool. Make sure you are running the xtractr binary in browse mode. Turning on auto-completion in IRB also makes it easier to try out different things and interactively experiment with the API:

$ irb -rirb/completion -rmu/xtractr

All of the examples below work off the test/test.pcap bundled with the gem. We’ll also assume that you’ve done this at the start of the IRB session:

irb> xtractr = Mu::Xtractr.new

Top DNS query names

We first pull out all DNS flows and then map/reduce the unique values of the dns.qry.name.

irb> xtractr.flows('flow.service:DNS').values('dns.qry.name')

Services used by the top talker (based on bytes sent/received)

We first sum the total number of bytes using the src address as the key. The sum function returns the matches sorted by the #bytes. We then use the first object (the top talker) to in turn map/reduce the unique list of services supported by it.

irb> xtractr.flows.sum('flow.src', 'flow.bytes').first.count('flow.service')

Generating #new pcaps based on search criteria

We first get a list of the unique HTTP methods in the index and then for each of methods, query for all the packets and then save them into a new pcap.

irb> xtractr.packets.count('http.request.method').each { |c| c.packets.save("#{c.value}.pcap") }

– rdoc –exclude test –force-update –inline-source –op mu/xtractr/doc –main Mu::Xtractr

Direct Known Subclasses

DoroParser::Doroxtractr

Defined Under Namespace

Classes: About, Content, Field, Flow, Flows, Host, Packet, Packets, Service, Stream, Term, Test, Views

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(address = 'localhost', port = 8080, relurl = nil) ⇒ Xtractr

Create a new instance to connect to the xtractr binary running in browse mode.

Xtractr.new
Xtractr.new 'localhost', 8080



99
100
101
102
103
104
105
106
107
108
109
110
 
# File 'lib/mu/xtractr.rb', line 99

def initialize address='localhost', port=8080, relurl=nil
    @address = address
    @port = port
    @relurl = relurl || '/'
    @relurl << '/' if @relurl[-1,1] != '/'            
    #unless about.version =~ /^4\.5\.(svn|41604)$/
    #    puts "boh"
    #    puts "xtractr version #{about.version} out of date!"
    #    puts "please download a new one from http://www.pcapr.net/xtractr"
    #    raise
    #end
end

Instance Attribute Details

#address ⇒ Object (readonly)

Return the IP address of the xtractr instance



79
80
81
 
# File 'lib/mu/xtractr.rb', line 79

def address
  @address
end

#port ⇒ Object (readonly)

Return the listening port of the xtractr instance



82
83
84
 
# File 'lib/mu/xtractr.rb', line 82

def port
  @port
end

#relurl ⇒ Object (readonly)

Relative URL



85
86
87
 
# File 'lib/mu/xtractr.rb', line 85

def relurl
  @relurl
end

Class Method Details

.create(url) ⇒ Object

Create a new instance to connect to the xtractr binary using a url.

Xtractr.create 'http://some.host:8080/'



90
91
92
93
 
# File 'lib/mu/xtractr.rb', line 90

def self.create url
    uri = URI.parse url
    self.new uri.host, uri.port, uri.path
end

Instance Method Details

#about ⇒ Object

Fetch the meta data about the index. This includes information about the total number of packets, flows as well as the duration of the entire set of pcaps in the index.

xtractr.about



116
117
118
 
# File 'lib/mu/xtractr.rb', line 116

def about
    @about ||= About.new json('api/about')
end

#field(name) ⇒ Object

Fetch a field of the given name.

xtractr.field 'http.server'

Raises:

  • (ArgumentError) 


134
135
136
137
138
 
# File 'lib/mu/xtractr.rb', line 134

def field name
    obj = fields.find { |f| f.name == name }
    raise ArgumentError, "Unknown field #{name}" if not obj
    return obj
end

#fields(regex = nil) ⇒ Object

Fetch the list of fields in the index. The fields are only available if the –mode forensics was used during the indexing process.

xtractr.fields
xtractr.fields /^http/
xtractr.fields 'http.server'



125
126
127
128
129
130
 
# File 'lib/mu/xtractr.rb', line 125

def fields regex=nil
    regex = Regexp.new(regex, Regexp::IGNORECASE) if regex.is_a? String
    result = (@fields ||= json 'api/fields')
    result = result.select { |name| name =~ regex } if regex
    return result.map { |name| Field.new self, name }
end

#flow(id) ⇒ Object

Return the id’th flow from the index.

xtractr.flow 1

Raises:

  • (ArgumentError) 


200
201
202
203
204
205
 
# File 'lib/mu/xtractr.rb', line 200

def flow id
    result = json "api/flows", :start => id, :limit => id
    rows = result['rows']
    raise ArgumentError, "Unknown flow #{id}" if rows.empty?
    return Flow.new(self, rows[0])
end

#flows(q = '*') ⇒ Object

Return an iterator that can yield each flow that matched the query. If q is a Range, then it’s used to extract the set of flows that match all of those ids.

xtractr.flows.each { |flow| ... }
xtractr.flows(1..10).each { |flow| ... }
xtractr.flows("flow.src:192.168.1.1").each { |flow| ... }



188
189
190
191
192
193
194
195
196
 
# File 'lib/mu/xtractr.rb', line 188

def flows(q='*') # :yields: flow
    if q.is_a? Range
        first = q.first
        last  = q.last
        last -= 1 if q.exclude_end?
        q = "flow.id:[#{first} #{last}]"
    end
    return Flows.new(self, :q => q)
end

#get(url, opts = {}) ⇒ Object

Fetch the URL and return the response body, as is



242
243
244
245
246
247
248
249
250
251
 
# File 'lib/mu/xtractr.rb', line 242

def get url, opts={} # :nodoc:
    _url = relurl + url
    if opts.size
        _url << '?'
        _url << opts.keys.map { |key| key.to_s + '=' + opts[key].to_s }.join('&')
    end
    
    _url = "http://#{address}:#{port}" + URI.escape(_url)
    return Net::HTTP.get(URI.parse(_url))
end

#host(address) ⇒ Object

Fetch a host of the given address.

xtractr.host '192.168.1.1'

Raises:

  • (ArgumentError) 


155
156
157
158
159
 
# File 'lib/mu/xtractr.rb', line 155

def host address
    obj = hosts.find { |h| h.address == address }
    raise ArgumentError, "Unknown host #{address}" if not obj
    return obj
end

#hosts(regex = nil) ⇒ Object

Fetch the list of hosts in the index. The optional regex (or String) can be used to filter the hosts list.

xtractr.hosts
xtractr.hosts /192.168/
xtractr.hosts '10.10'



145
146
147
148
149
150
151
 
# File 'lib/mu/xtractr.rb', line 145

def hosts regex=nil
    regex = Regexp.new(regex, Regexp::IGNORECASE) if regex.is_a? String
    result = (@hosts ||= json 'api/hosts')
    rows = result['rows']
    rows = rows.select { |row| row['name'] =~ regex } if regex
    return rows.map { |row| Host.new self, row['name'] }
end

#inspect ⇒ Object

:nodoc:



253
254
255
 
# File 'lib/mu/xtractr.rb', line 253

def inspect # :nodoc:
    "#<xtractr #{address}:#{port}>"
end

#json(url, opts = {}) ⇒ Object

Fetch the URL with the GET parameters and interpret the response body as a JSON object

Raises:

  • (ArgumentError) 


234
235
236
237
238
239
 
# File 'lib/mu/xtractr.rb', line 234

def json url, opts={} # :nodoc:
    res = get url, opts
    js = JSON.parse(res)
    raise ArgumentError, js['reason'] if js.is_a?(Hash) and js['error']
    return js
end

#packet(id) ⇒ Object

Return the id’th packet from the index.

xtractr.packet 1

Raises:

  • (ArgumentError) 


225
226
227
228
229
230
 
# File 'lib/mu/xtractr.rb', line 225

def packet id
    result = json "api/packets", :start => id, :limit => id
    rows = result['rows']
    raise ArgumentError, "Unknown packet #{id}" if rows.empty?
    return Packet.new(self, rows[0])
end

#packets(q = '*') ⇒ Object

Return an iterator that can yield each packet that matched the query. If q is a Range, then it’s used to extract the set of packets that match all of those ids.

xtractr.packets.each { |pkt| ... }
xtractr.packets(5..32).each { |pkt| ... }
xtractr.packets("http.user.agent:mozilla").each { |pkt| ... }



213
214
215
216
217
218
219
220
221
 
# File 'lib/mu/xtractr.rb', line 213

def packets(q='*') # :yields: packet
    if q.is_a? Range
        first = q.first
        last  = q.last
        last -= 1 if q.exclude_end?
        q = "pkt.id:[#{first} #{last}]"
    end
    return Packets.new(self, :q => q)
end

#service(name) ⇒ Object

Fetch a service of the given name.

xtractr.service 'dns'

Raises:

  • (ArgumentError) 


176
177
178
179
180
 
# File 'lib/mu/xtractr.rb', line 176

def service name
    obj = services.find { |s| s.name.downcase == name.downcase }
    raise ArgumentError, "Unknown service #{name}" if not obj
    return obj
end

#services(regex = nil) ⇒ Object

Fetch the list of services in the index. The optional regex (or String) can be used to filter the services lists.

xtractr.services
xtractr.services /http/
xtractr.services 'sip'



166
167
168
169
170
171
172
 
# File 'lib/mu/xtractr.rb', line 166

def services regex=nil
    regex = Regexp.new(regex, Regexp::IGNORECASE) if regex.is_a? String
    result = (@services ||= json 'api/services')
    rows = result['rows']
    rows = rows.select { |row| row['name'] =~ regex } if regex
    return rows.map { |row| Service.new self, row['name'] }
end