Fluent::Plugin::Mongokpi

Description

Fluent BufferedOutput plugin: counting chunk, inserting counts to MongoDB

Installation

$ gem install fluent-plugin-mongokpi

Configuration

config | default | info ——————-|—————–|—– address | localhost:27017 | address of mongodb (for replicaset, plz set like localhost:27017,localhost:27018,localhost:27019) db | kpidb | db name of mongodb collection | kpiColyyyymmdd | collection name of mongodb (yyyymmdd will be replaced based on time) write_concern | | mongodb connection option name | | mongodb connection option read | | mongodb connection option refresh_mode | | mongodb connection option refresh_interval | | mongodb connection option kpi_type | | use ‘access’ for activating access count kpi_unit | min | time unit of count [min,hour,day] time_key | | this time_key field or time of record is used time_format | | (with time_key) the value of time_key will be parsed with this format count_key | | used like group by (for multiple keys, plz set like site,host,instance) count_name | count | the count result will be set to this field response_threshold | 1000000 | (with kpi_type = access) used as threshold of response time f_code | code | (with kpi_type = access) field name of response code in the log records f_response_time | response_time | (with kpi_type = access) field name of response time in the log records

MongoDB document

field | notification | info —————-|————–|—– _id | | time + value of $count_key yyyymmdd | | hh | | (with kpi_unit = min or hour) mm | | (with kpi_unit = min) $count_key | | $count_name | | count count1xx | | (with kpi_type = access) count of $f_code = 1XX count2xx | | (with kpi_type = access) count of $f_code = 2XX count3xx | | (with kpi_type = access) count of $f_code = 3XX count4xx | | (with kpi_type = access) count of $f_code = 4XX count5xx | | (with kpi_type = access) count of $f_code = 5XX responseTimeAve | m(_ _)m | (with kpi_type = access) average of $f_response_time responseTimeMax | m(_ _)m | (with kpi_type = access) maximum of $f_response_time responseTimeMin | m(_ _)m | (with kpi_type = access) minimum of $f_response_time responseTimeSum | | (with kpi_type = access) sum of $f_response_time countOver | | (with kpi_type = access) count of the access: $f_response_time >= $response_threshold okRatio | m(_ _)m | (with kpi_type = access) (count - countOver) / count counter[60] | | (with kpi_type = access & kpy_unit = min) qps for each second qpsAve | m(_ _)m | (with kpi_type = access & kpy_unit = min) average of qps qpsMax | m(_ _)m | (with kpi_type = access & kpy_unit = min) maximum of qps qpsMin | m(_ _)m | (with kpi_type = access & kpy_unit = min) minimum of qps

m(_ _)m : these parameters are updated by non-atmic queries, then not reliable in multi processing

Samples

### sample: access log (simple) config

<match ***>
  type mongokpi
  kpi_type access
  kpi_unit hourly
  count_key site
  time_key request_time
  time_format %d/%b/%Y:%H:%M:%S %Z
</match>

log

{"request_time":"25/Nov/2013:20:03:02 +0700","code":"100","response_time":"1000","site":"git"}
{"request_time":"25/Nov/2013:20:03:03 +0700","code":"200","response_time":"2000","site":"git"}
{"request_time":"25/Nov/2013:20:03:03 +0700","code":"300","response_time":"3000","site":"git"}
{"request_time":"25/Nov/2013:20:03:03 +0700","code":"404","response_time":"4000","site":"blog"}
{"request_time":"25/Nov/2013:20:03:03 +0700","code":"500","response_time":"5000","site":"blog"}

MongoDB

db.kpiCol20131125.find()
{ "_id" : "git201311252003",   "count" : 3, "count1xx" : 1, "count2xx" : 1, "count3xx" : 1, "count4xx" : 0, "count5xx" : 0, "countOver" : 0,
  "hh" : "20", "okRatio" : 1, "responseTimeAve" : 2000, "responseTimeMax" : 3000, "responseTimeMin" : 1000, "responseTimeSum" : 6000, "site" : "git", "yyyymmdd" : "20131125"}
{ "_id" : "blog201311252003",  "count" : 2, "count1xx" : 0, "count2xx" : 0, "count3xx" : 0, "count4xx" : 4, "count5xx" : 1, "countOver" : 0,
  "hh" : "20", "okRatio" : 1, "responseTimeAve" : 4500, "responseTimeMax" : 5000, "responseTimeMin" : 4000, "responseTimeSum" : 9000, "site" : "blog", "yyyymmdd" : "20131125"}
{ "_id" : "total201311252003", "count" : 5, "count1xx" : 0, "count2xx" : 0, "count3xx" : 0, "count4xx" : 0, "count5xx" : 0, "countOver" : 0,
  "hh" : "20", "okRatio" : 1, "responseTimeAve" : 3000, "responseTimeMax" : 5000, "responseTimeMin" : 1000, "responseTimeSum" : 15000, "site" : "total", "yyyymmdd" : "20131125" }

sample: access log (detail)

config

<match ***>
  type mongokpi
  kpi_type access
  count_key site
  time_key request_time
  time_format %d/%b/%Y:%H:%M:%S %Z
</match>

log

{"request_time":"25/Nov/2013:20:03:02 +0700","code":"100","response_time":"1000","site":"git"}
{"request_time":"25/Nov/2013:20:03:03 +0700","code":"200","response_time":"2000","site":"git"}
{"request_time":"25/Nov/2013:20:03:03 +0700","code":"300","response_time":"3000","site":"git"}
{"request_time":"25/Nov/2013:20:03:03 +0700","code":"404","response_time":"4000","site":"blog"}
{"request_time":"25/Nov/2013:20:03:03 +0700","code":"500","response_time":"5000","site":"blog"}

MongoDB

db.kpiCol20131125.find()
{ "_id" : "git201311252003",   "count" : 3, "count1xx" : 1, "count2xx" : 1, "count3xx" : 1, "count4xx" : 0, "count5xx" : 0, "countOver" : 0,
  "counter" : [ 0,0,1,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
  "hh" : "20", "mm" : "03", "okRatio" : 1, "qpsAve" : 0, "qpsMax" : 2, "qpsMin" : 0, "responseTimeAve" : 2000, "responseTimeMax" : 3000, "responseTimeMin" : 1000, "responseTimeSum" : 6000, "site" : "git", "yyyymmdd" : "20131125"}
{ "_id" : "blog201311252003",  "count" : 2, "count1xx" : 0, "count2xx" : 0, "count3xx" : 0, "count4xx" : 4, "count5xx" : 1, "countOver" : 0,
  "counter" : [ 0,0,0,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
  "hh" : "20", "mm" : "03", "okRatio" : 1, "qpsAve" : 0, "qpsMax" : 2, "qpsMin" : 0, "responseTimeAve" : 4500, "responseTimeMax" : 5000, "responseTimeMin" : 4000, "responseTimeSum" : 9000, "site" : "blog", "yyyymmdd" : "20131125"}
{ "_id" : "total201311252003", "count" : 5, "count1xx" : 0, "count2xx" : 0, "count3xx" : 0, "count4xx" : 0, "count5xx" : 0, "countOver" : 0,
  "counter" : [ 0,0,1,4,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
  "hh" : "20", "mm" : "03", "okRatio" : 1, "qpsAve" : 0, "qpsMax" : 4, "qpsMin" : 0, "responseTimeAve" : 3000, "responseTimeMax" : 5000, "responseTimeMin" : 1000, "responseTimeSum" : 15000, "site" : "total", "yyyymmdd" : "20131125" }

sample: app log

config

<match ***>
  type mongokpi
  count_key site
</match>

log

{"gerogero":"Let's get Rocking!","site":"git"}
{"gerogero":"Let's get Rocking!","site":"git"}
{"gerogero":"Let's get Rocking!","site":"git"}
{"gerogero":"Let's get Rocking!","site":"blog"}
{"gerogero":"Let's get Rocking!","site":"blog"}

MongoDB

db.kpiCol20131125.find()
{ "_id" : "git201311252248",  "count" : 3, "hh" : "22", "mm" : "48", "site" : "git", "yyyymmdd" : "20131125" }
{ "_id" : "blog201311252248", "count" : 2, "hh" : "22", "mm" : "48", "site" : "blog", "yyyymmdd" : "20131125" }
{ "_id" : "total201311252248","count" : 5, "hh" : "22", "mm" : "48", "site" : "total", "yyyymmdd" : "20131125" }

Contributing

  1. Fork it
  2. Create your feature branch (git checkout -b my-new-feature)
  3. Commit your changes (git commit -am 'Add some feature')
  4. Push to the branch (git push origin my-new-feature)
  5. Create new Pull Request