scylla

Scylla is a language categorizing gem that allows you to guess the language of a given text. Scylla is a Ruby port of TextCat (www.let.rug.nl/~vannoord/TextCat) and is based on the text categorization algorithm presented in Cavnar, W. B. and J. M. Trenkle, “N-Gram-Based Text Categorization” In Proceedings of Third Annual Symposium on Document Analysis and Information Retrieval, Las Vegas, NV, UNLV Publications/Reprographics, pp. 161-175, 11-13 April 1994.

Installation

gem install scylla

Usage

require 'scylla'

"this is english text".language
=> "english" 

"Este es un texto español".language
=> "spanish"

Multiple results for other possible languages:

"isso poderia ser confundido com espanhol, bem".language
=> "portuguese"

"isso poderia ser confundido com espanhol, bem".guess
=> ["portuguese", "spanish"]

Training

You can train scylla in new languages by providing sample texts in different languages. The default set is located in the ‘source_texts’ folder in the gem directory. Add new .txt files to this directory named according to the language i.e. a text file full of Hebrew text should be called ‘hebrew.txt’. At least 500 lines of text recommended. Then, in the gem folder, run this:

rake scylla:train

If you want to store texts in your own folder, you can specify that to the rake task. WARNING: specifying a different folder deletes all language support for files located in the default directory if they are not copied over.

rake scylla:train[/Users/hash/mytextdir]
"Creating language map for /Users/hash/mytextdir/english.txt"
"Creating language map for /Users/hash/mytextdir/kannada.txt"
.
.
etc

Contributing to scylla

  • Check out the latest master to make sure the feature hasn’t been implemented or the bug hasn’t been fixed yet

  • Check out the issue tracker to make sure someone already hasn’t requested it and/or contributed it

  • Fork the project

  • Start a feature/bugfix branch

  • Commit and push until you are happy with your contribution

  • Make sure to add tests for it. This is important so I don’t break it in a future version unintentionally.

  • Please try not to mess with the Rakefile, version, or history. If you want to have your own version, or is otherwise necessary, that is fine, but please isolate to its own commit so I can cherry-pick around it.

Copyright © 2011 Ashwin Hegde. See LICENSE.txt for further details.