Class: Linguist::Classifier
- Inherits:
-
Object
- Object
- Linguist::Classifier
- Defined in:
- lib/linguist/classifier.rb
Overview
Language bayesian classifier.
Class Method Summary collapse
-
.classify(db, tokens, languages = nil) ⇒ Object
Public: Guess language of data.
-
.train!(db, language, data) ⇒ Object
Public: Train classifier that data is a certain language.
Instance Method Summary collapse
-
#classify(tokens, languages) ⇒ Object
Internal: Guess language of data.
-
#initialize(db = {}) ⇒ Classifier
constructor
Internal: Initialize a Classifier.
-
#language_probability(language) ⇒ Object
Internal: Probably of a language occurring - P©.
-
#token_probability(token, language) ⇒ Object
Internal: Probably of token in language occurring - P(F | C).
-
#tokens_probability(tokens, language) ⇒ Object
Internal: Probably of set of tokens in a language occurring - P(D | C).
Constructor Details
#initialize(db = {}) ⇒ Classifier
Internal: Initialize a Classifier.
63 64 65 66 67 68 69 |
# File 'lib/linguist/classifier.rb', line 63 def initialize(db = {}) @tokens_total = db['tokens_total'] @languages_total = db['languages_total'] @tokens = db['tokens'] @language_tokens = db['language_tokens'] @languages = db['languages'] end |
Class Method Details
.classify(db, tokens, languages = nil) ⇒ Object
Public: Guess language of data.
db - Hash of classifier tokens database. data - Array of tokens or String data to analyze. languages - Array of language name Strings to restrict to.
Examples
Classifier.classify(db, "def hello; end")
# => [ 'Ruby', 0.90], ['Python', 0.2], ... ]
Returns sorted Array of result pairs. Each pair contains the String language name and a Float score.
57 58 59 60 |
# File 'lib/linguist/classifier.rb', line 57 def self.classify(db, tokens, languages = nil) languages ||= db['languages'].keys new(db).classify(tokens, languages) end |
.train!(db, language, data) ⇒ Object
Public: Train classifier that data is a certain language.
db - Hash classifier database object language - String language of data data - String contents of file
Examples
Classifier.train(db, 'Ruby', "def hello; end")
Returns nothing.
Set LINGUIST_DEBUG=1 or =2 to see probabilities per-token, per-language. See also dump_all_tokens, below.
20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 |
# File 'lib/linguist/classifier.rb', line 20 def self.train!(db, language, data) tokens = Tokenizer.tokenize(data) db['tokens_total'] ||= 0 db['languages_total'] ||= 0 db['tokens'] ||= {} db['language_tokens'] ||= {} db['languages'] ||= {} tokens.each do |token| db['tokens'][language] ||= {} db['tokens'][language][token] ||= 0 db['tokens'][language][token] += 1 db['language_tokens'][language] ||= 0 db['language_tokens'][language] += 1 db['tokens_total'] += 1 end db['languages'][language] ||= 0 db['languages'][language] += 1 db['languages_total'] += 1 nil end |
Instance Method Details
#classify(tokens, languages) ⇒ Object
Internal: Guess language of data
data - Array of tokens or String data to analyze. languages - Array of language name Strings to restrict to.
Returns sorted Array of result pairs. Each pair contains the String language name and a Float score.
78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 |
# File 'lib/linguist/classifier.rb', line 78 def classify(tokens, languages) return [] if tokens.nil? tokens = Tokenizer.tokenize(tokens) if tokens.is_a?(String) scores = {} if verbosity >= 2 dump_all_tokens(tokens, languages) end languages.each do |language| scores[language] = tokens_probability(tokens, language) + language_probability(language) if verbosity >= 1 printf "%10s = %10.3f + %7.3f = %10.3f\n", language, tokens_probability(tokens, language), language_probability(language), scores[language] end end scores.sort { |a, b| b[1] <=> a[1] }.map { |score| [score[0], score[1]] } end |
#language_probability(language) ⇒ Object
Internal: Probably of a language occurring - P©
language - Language to check.
Returns Float between 0.0 and 1.0.
129 130 131 |
# File 'lib/linguist/classifier.rb', line 129 def language_probability(language) Math.log(@languages[language].to_f / @languages_total.to_f) end |
#token_probability(token, language) ⇒ Object
Internal: Probably of token in language occurring - P(F | C)
token - String token. language - Language to check.
Returns Float between 0.0 and 1.0.
116 117 118 119 120 121 122 |
# File 'lib/linguist/classifier.rb', line 116 def token_probability(token, language) if @tokens[language][token].to_f == 0.0 1 / @tokens_total.to_f else @tokens[language][token].to_f / @language_tokens[language].to_f end end |
#tokens_probability(tokens, language) ⇒ Object
Internal: Probably of set of tokens in a language occurring - P(D | C)
tokens - Array of String tokens. language - Language to check.
Returns Float between 0.0 and 1.0.
104 105 106 107 108 |
# File 'lib/linguist/classifier.rb', line 104 def tokens_probability(tokens, language) tokens.inject(0.0) do |sum, token| sum += Math.log(token_probability(token, language)) end end |