Module: Zabon

Defined in:
lib/zabon.rb,
lib/zabon/helper.rb,
lib/zabon/railtie.rb,
lib/zabon/segment.rb,
lib/zabon/version.rb,
lib/zabon/analyzer.rb,
lib/zabon/constants.rb,
lib/zabon/configuration.rb

Defined Under Namespace

Modules: Helper Classes: Analyzer, Configuration, Railtie, Segment

Constant Summary collapse

VERSION = 
"0.2.0"
JOSHI =

Joshi (助詞), Japanese particles written in Hiragana, are suffixes or short words that follow a modified noun, verb, adjective, or sentence. Some particals can appear in two types. They give pretty reliable cues depending on the following character, whether a line break is allowed or not.

/
  (でなければ|について|かしら|くらい|けれど|なのか|ばかり|ながら|ことよ|こそ|こと|さえ|しか|した|たり|だけ|だに|だの|つつ|ても|てよ|でも|
  とも|から|など|なりので|ので|のに|ほど|まで|もの|やら|より|って|で|と|な|に|ね|の|も|は|ば|へ|や|わ|を|か|が|さ|し|ぞ|て)
/x
KEYWORDS =

A simple way to find word segementations in Japanese is to tokenise by grouping characters continuously by script (Hiragana, Katakana, Kanji, Romaji)

The following regular expression matches in this order:

  • non breaking space
  • domains
  • any Japanese Kanji or Chinese character
  • Hirgana (+ chisai kana)
  • Katakana (+ chisai kana)
  • Latin
  • Latin (double width)
/
  (\ |
  [a-zA-Z0-9]+\.[a-z]{2,}|
  [一-龠々〆ヵヶゝ]+|
  [ぁ-んゝ]+|
  [ァ-ヴー]+|
  [a-zA-Z0-9]+|
  [ａ-ｚＡ-Ｚ０-９]+)
/x
BRACKETS_BEGIN =

Brackets & Quotations

/([〈《「『｢（(\[【〔〚〖〘❮❬❪❨(<{❲❰｛❴])/
BRACKETS_END = 
/([〉》」』｣)）\]】〕〗〙〛}>\)❩❫❭❯❱❳❵｝])/
PERIODS = 
/([\.\,。、！\!？\?]+)$/
HIRAGANA = 
/[ぁ-んゝ]+/

Class Method Summary collapse

Class Method Details

.config ⇒ Object 



17
18
19
 
# File 'lib/zabon.rb', line 17

def config
  @config ||= Configuration.new
end

.configure {|config| ... } ⇒ Object

Yields:



21
22
23
 
# File 'lib/zabon.rb', line 21

def configure
  yield config
end

.reset_config! ⇒ Object 



25
26
27
 
# File 'lib/zabon.rb', line 25

def reset_config!
  @config = Configuration.new
end

.split(text) ⇒ Object 



13
14
15
 
# File 'lib/zabon.rb', line 13

def split(text)
  Analyzer.segments(text)
end