Clusta

Clusta is a Ruby gem for network analysis built on top of Wukong.

Wukong lets you write Ruby scripts that run on your laptop as well as on a Hadoop cluster.

Clusta is:

• classes that make describing the geometry of networks easy

• network algorithms written with these classes to use Wukong

• a shim command-line program for running these algorithms

Start with a file containing edges:

Edge  1  2
Edge  2  3
Edge  1  4
Edge  4  5
Edge  5  6
Edge  5  7
Edge  6  8
Edge  7  8
Edge  8  9

Run this through a transformation named edges_to_degrees:

$ clusta --transform=edges_to_degrees /local/edges.tsv -
Degree  1  2
Degree  2  2
Degree  3  1
Degree  4  2
Degree  5  3
Degree  6  2
Degree  7  2
Degree  8  3
Degree  9  1

Chain transformations together:

$ clusta --transform=edges_to_neighborhoods /local/edges.tsv - | clusta --transform=neighborhoods_to_degree_pairs - - | clusta --transform=degree_pairs_to_assortativities - -
Assortativity 1 2 1
Assortativity 1 3 1
Assortativity 2 1 1
Assortativity 2 2 4
Assortativity 2 3 5
Assortativity 3 1 1
Assortativity 3 2 5

And then leverage Wukong when you’re ready:

$ clusta --run=hadoop --transform=edges_to_neighborhoods /hdfs/edges.tsv /hdfs/neighborhoods.tsv
I, [2012-03-03T21:00:39.992750 #25835]  INFO -- :   Launching hadoop!
I, [2012-03-03T21:00:39.992979 #25835]  INFO -- : Running

/usr/lib/hadoop/bin/hadoop  \
  jar /usr/lib/hadoop/contrib/streaming/hadoop-*streaming*.jar  \
  -D mapred.job.name='clusta---spec/data/edges/undirected.unweighted.tsv----'   \
  -mapper  '/usr/bin/ruby1.9.1 clusta --map --log_interval=10000 --log_seconds=30 --transform=edges_to_degrees'   \
  -reducer '/usr/bin/ruby1.9.1 clusta --reduce --log_interval=10000 --log_seconds=30 --transform=edges_to_degrees'  \
  -input   'spec/data/edges/undirected.unweighted.tsv'  \
  -output  '-'  \
  -file    '/home/user/projects/networks/clusta/bin/clusta'
...