VectorEmbed

Vector embedding of strings, booleans, numerics, and arrays into LIBSVM / LIBLINEAR format.

Inspired by Sally, except VectorEmbed is meant to handle categorical and continuous data at the same time.

csv2libsvm

$ cat houses.csv 
label,household_income,zip_code_id,year
+1,35893,53703,1904
-1,43708,53711,1977
+1,103214,53719,NULL
-1,49250,53704,1950

$ csv2libsvm houses.csv 
1 1243483:35893 6439848:53703 8227451:1904
-1 1243483:43708 6439848:53711 8227451:1977
1 1243483:103214 6439848:53719 8227451:0
-1 1243483:49250 6439848:53704 8227451:1950

Usage

Create a VectorEmbed instance, which auto-detects and then remembers what kind of data goes into each feature:

>> require 'vector_embed'
=> true
>> v = VectorEmbed.new
=> #<VectorEmbed:0x007fd605815208 [...]>

Output a line with a label and arbitrary features:

>> label = 1
=> 1
>> features = { color: 'red', year: 1995, weight: 5.4e9 }
=> {:color=>"red", :year=>1995, :weight=>5400000000.0}
>> v.line(label, features)
=> "1 1997960:1 5556418:5400000000.0 8227451:1995"

Output another line:

>> label = 0
=> 0
>> features = { color: 'blue', year: 1821, weight: 3.3 }
=> {:color=>"blue", :year=>1821, :weight=>3.3}
>> v.line(label, features)
=> "0 1089740:1 5556418:3.3 8227451:1821"

Note that color: 'red' and color: 'blue' are being translated into categories:

1997960:1 # murmur3("color\x00red"):1
1089740:1 # murmur3("color\x00blue"):1

A similar thing happens with true/false:

>> v.line(1, yes: true, no: false)
=> "1 1559987:1 3324244:1"

i.e.

1559987:1 # murmur3("yes\x00true"):1
3324244:1 # murmur3("no\x00false"):1

N-grams

Currently uses same parameter names as Sally.

Word ngrams

>> v = VectorEmbed.new ngram_len: 1, ngram_delim: /\s+/, stop_words: %w{the and or}
=> #<VectorEmbed:0x007fd6033b77f8 [...]>
>> v.line(1, notes: 'the quick brown fox')
=> "1 1512788:1 3426202:1 5079692:1"

You get the idea: ("the" has been filtered out by stop words)

1512788:1 # murmur3("notes\x00quick"):1
3426202:1 # murmur3("notes\x00brown"):1
5079692:1 # murmur3("notes\x00fox"):1

Byte n-grams

>> v = VectorEmbed.new ngram_len: 2, ngram_delim: ''
=> #<VectorEmbed:0x007fd60337ea20 [...]>
>> v.line(1, notes: 'foobar')
=> "1 2148745:1 2878919:1 3600333:1 3621715:1 5885921:1"

So therefore:

2148745:1 # murmur3("notes\x00fo"):1
2878919:1 # murmur3("notes\x00oo"):1
3600333:1 # murmur3("notes\x00ob"):1
3621715:1 # murmur3("notes\x00ba"):1
5885921:1 # murmur3("notes\x00ar"):1

Debugging

VectorEmbed tries to do the right thing, but if it's not, try turning on debugging:

>> v = VectorEmbed.new
=> #<VectorEmbed:0x007fd6034020a0 [...]>
>> v.logger.level = Logger::DEBUG
=> 0
>> v.line(1, '3' => 7, foo: 'bar', truthy: false, nullity: nil)
D, [2013-02-20T16:55:00.139299 #21595] DEBUG -- : Interpreting "3" as Number given first value 7
D, [2013-02-20T16:55:00.139561 #21595] DEBUG -- : Interpreting :foo as Phrase given first value "bar"
D, [2013-02-20T16:55:00.139671 #21595] DEBUG -- : Interpreting :truthy as Boolean given first value false
D, [2013-02-20T16:55:00.139755 #21595] DEBUG -- : Interpreting :nullity as Boolean given first value nil
D, [2013-02-20T16:55:00.139872 #21595] DEBUG -- : Interpreting "label" as Number given first value 1
=> "1 2647413:7 4091306:1 7123386:1 9259635:1"

One thing it doesn't like: (assuming you have already performed the lines above)

>> v.line(1, '3' => 'bar')
ArgumentError: Can't embed "bar" in number feature "3".

It's saying that, given you first passed it 7, it thought "3" was a feature that held numbers.

Gotchas

Following Sally, it only uses the first 22 bits of the murmur hash for feature indices... more and LIBSVM seems to choke.
Stop words are currently filtered out of feature indices... probably shouldn't be.