Fluent::Plugin::Mongokpi

Description

Fluent BufferedOutput plugin: counting chunk, inserting counts to MongoDB version 2.4 or newer MongoDB is required (using setOnInsert command)

Installation

$ gem install fluent-plugin-mongokpi

Configuration

config default info
address localhost:27017 address of mongodb (for replicaset, plz set like localhost:27017,localhost:27018,localhost:27019)
db kpidb db name of mongodb
collection kpiColyyyymmdd collection name of mongodb (yyyymmdd will be replaced based on time)
write_concern mongodb connection option
name mongodb connection option
read mongodb connection option
refresh_mode mongodb connection option
refresh_interval mongodb connection option
kpi_type use 'access' for activating access count
kpi_unit min time unit of count [min,hour,day]
time_key this time_key field or time of record is used
time_format (with time_key) the value of time_key will be parsed with this format
count_key used like group by (for multiple keys, plz set like site,host,instance)
count_name count the count result will be set to this field
response_threshold 1000000 (with kpi_type = access) used as threshold of response time
f_code code (with kpi_type = access) field name of response code in the log records
f_response_time response_time (with kpi_type = access) field name of response time in the log records

MongoDB document

field notification info
_id time + value of $count_key
yyyymmdd
hh (with kpi_unit = min or hour)
mm (with kpi_unit = min)
$count_key
$count_name count
count1xx (with kpi_type = access) count of $f_code = 1XX
count2xx (with kpi_type = access) count of $f_code = 2XX
count3xx (with kpi_type = access) count of $f_code = 3XX
count4xx (with kpi_type = access) count of $f_code = 4XX
count5xx (with kpi_type = access) count of $f_code = 5XX
responseTimeAve m(_ _)m (with kpi_type = access) average of $f_response_time
responseTimeMax m(_ _)m (with kpi_type = access) maximum of $f_response_time
responseTimeMin m(_ _)m (with kpi_type = access) minimum of $f_response_time
responseTimeSum (with kpi_type = access) sum of $f_response_time
countOver (with kpi_type = access) count of the access: $f_response_time >= $response_threshold
okRatio m(_ _)m (with kpi_type = access) (count - countOver) / count
counter[60] (with kpi_type = access & kpy_unit = min) qps for each second
qpsAve m(_ _)m (with kpi_type = access & kpy_unit = min) average of qps
qpsMax m(_ _)m (with kpi_type = access & kpy_unit = min) maximum of qps
qpsMin m(_ _)m (with kpi_type = access & kpy_unit = min) minimum of qps

m(_ _)m : these parameters are updated by non-atmic queries, then not reliable in multi processing

Samples

sample: access log (simple)

config

<match ***>
  type mongokpi
  kpi_type access
  kpi_unit hourly
  count_key site
  time_key request_time
  time_format %d/%b/%Y:%H:%M:%S %Z
</match>

log

{"request_time":"25/Nov/2013:20:03:02 +0700","code":"100","response_time":"1000","site":"git"}
{"request_time":"25/Nov/2013:20:03:03 +0700","code":"200","response_time":"2000","site":"git"}
{"request_time":"25/Nov/2013:20:03:03 +0700","code":"300","response_time":"3000","site":"git"}
{"request_time":"25/Nov/2013:20:03:03 +0700","code":"404","response_time":"4000","site":"blog"}
{"request_time":"25/Nov/2013:20:03:03 +0700","code":"500","response_time":"5000","site":"blog"}

MongoDB

db.kpiCol20131125.find()
{ "_id" : "git2013112520",   "count" : 3, "count1xx" : 1, "count2xx" : 1, "count3xx" : 1, "count4xx" : 0, "count5xx" : 0, "countOver" : 0,
  "hh" : 20, "okRatio" : 1, "responseTimeAve" : 2000, "responseTimeMax" : 3000, "responseTimeMin" : 1000, "responseTimeSum" : 6000, "site" : "git", "yyyymmdd" : 20131125 }
{ "_id" : "blog2013112520",  "count" : 2, "count1xx" : 0, "count2xx" : 0, "count3xx" : 0, "count4xx" : 4, "count5xx" : 1, "countOver" : 0,
  "hh" : 20, "okRatio" : 1, "responseTimeAve" : 4500, "responseTimeMax" : 5000, "responseTimeMin" : 4000, "responseTimeSum" : 9000, "site" : "blog", "yyyymmdd" : 20131125 }
{ "_id" : "total2013112520", "count" : 5, "count1xx" : 0, "count2xx" : 0, "count3xx" : 0, "count4xx" : 0, "count5xx" : 0, "countOver" : 0,
  "hh" : 20, "okRatio" : 1, "responseTimeAve" : 3000, "responseTimeMax" : 5000, "responseTimeMin" : 1000, "responseTimeSum" : 15000, "site" : "total", "yyyymmdd" : 20131125 }

sample: access log (detail)

config

<match ***>
  type mongokpi
  kpi_type access
  count_key site
  time_key request_time
  time_format %d/%b/%Y:%H:%M:%S %Z
</match>

log

{"request_time":"25/Nov/2013:20:03:02 +0700","code":"100","response_time":"1000","site":"git"}
{"request_time":"25/Nov/2013:20:03:03 +0700","code":"200","response_time":"2000","site":"git"}
{"request_time":"25/Nov/2013:20:03:03 +0700","code":"300","response_time":"3000","site":"git"}
{"request_time":"25/Nov/2013:20:03:03 +0700","code":"404","response_time":"4000","site":"blog"}
{"request_time":"25/Nov/2013:20:03:03 +0700","code":"500","response_time":"5000","site":"blog"}

MongoDB

db.kpiCol20131125.find()
{ "_id" : "git201311252003",   "count" : 3, "count1xx" : 1, "count2xx" : 1, "count3xx" : 1, "count4xx" : 0, "count5xx" : 0, "countOver" : 0,
  "counter" : [ 0,0,1,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
  "hh" : 20, "mm" : 3, "okRatio" : 1, "qpsAve" : 0, "qpsMax" : 2, "qpsMin" : 0, "responseTimeAve" : 2000, "responseTimeMax" : 3000, "responseTimeMin" : 1000, "responseTimeSum" : 6000, "site" : "git", "yyyymmdd" : 20131125 }
{ "_id" : "blog201311252003",  "count" : 2, "count1xx" : 0, "count2xx" : 0, "count3xx" : 0, "count4xx" : 4, "count5xx" : 1, "countOver" : 0,
  "counter" : [ 0,0,0,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
  "hh" : 20, "mm" : 3, "okRatio" : 1, "qpsAve" : 0, "qpsMax" : 2, "qpsMin" : 0, "responseTimeAve" : 4500, "responseTimeMax" : 5000, "responseTimeMin" : 4000, "responseTimeSum" : 9000, "site" : "blog", "yyyymmdd" : 20131125 }
{ "_id" : "total201311252003", "count" : 5, "count1xx" : 0, "count2xx" : 0, "count3xx" : 0, "count4xx" : 0, "count5xx" : 0, "countOver" : 0,
  "counter" : [ 0,0,1,4,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
  "hh" : 20, "mm" : 3, "okRatio" : 1, "qpsAve" : 0, "qpsMax" : 4, "qpsMin" : 0, "responseTimeAve" : 3000, "responseTimeMax" : 5000, "responseTimeMin" : 1000, "responseTimeSum" : 15000, "site" : "total", "yyyymmdd" : 20131125 }

sample: app log

config

<match ***>
  type mongokpi
  count_key site
</match>

log

{"gerogero":"Let's get Rocking!","site":"git"}
{"gerogero":"Let's get Rocking!","site":"git"}
{"gerogero":"Let's get Rocking!","site":"git"}
{"gerogero":"Let's get Rocking!","site":"blog"}
{"gerogero":"Let's get Rocking!","site":"blog"}

MongoDB

db.kpiCol20131125.find()
{ "_id" : "git201311252248",  "count" : 3, "hh" : 22, "mm" : 48, "site" : "git", "yyyymmdd" : 20131125 }
{ "_id" : "blog201311252248", "count" : 2, "hh" : 22, "mm" : 48, "site" : "blog", "yyyymmdd" : 20131125 }
{ "_id" : "total201311252248","count" : 5, "hh" : 22, "mm" : 48, "site" : "total", "yyyymmdd" : 20131125 }

Contributing

  1. Fork it
  2. Create your feature branch (git checkout -b my-new-feature)
  3. Commit your changes (git commit -am 'Add some feature')
  4. Push to the branch (git push origin my-new-feature)
  5. Create new Pull Request