Fluent::Plugin::Mongokpi
Description
Fluent BufferedOutput plugin: counting chunk, inserting counts to MongoDB version 2.4 or newer MongoDB is required (using setOnInsert command)
Installation
$ gem install fluent-plugin-mongokpi
Configuration
config | default | info |
---|---|---|
address | localhost:27017 | address of mongodb (for replicaset, plz set like localhost:27017,localhost:27018,localhost:27019) |
db | kpidb | db name of mongodb |
collection | kpiColyyyymmdd | collection name of mongodb (yyyymmdd will be replaced based on time) |
write_concern | mongodb connection option | |
name | mongodb connection option | |
read | mongodb connection option | |
refresh_mode | mongodb connection option | |
refresh_interval | mongodb connection option | |
kpi_type | use 'access' for activating access count | |
kpi_unit | min | time unit of count [min,hour,day] |
time_key | this time_key field or time of record is used | |
time_format | (with time_key) the value of time_key will be parsed with this format | |
count_key | used like group by (for multiple keys, plz set like site,host,instance) | |
count_name | count | the count result will be set to this field |
response_threshold | 1000000 | (with kpi_type = access) used as threshold of response time |
f_code | code | (with kpi_type = access) field name of response code in the log records |
f_response_time | response_time | (with kpi_type = access) field name of response time in the log records |
MongoDB document
field | notification | info |
---|---|---|
_id | time + value of $count_key | |
yyyymmdd | ||
hh | (with kpi_unit = min or hour) | |
mm | (with kpi_unit = min) | |
$count_key | ||
$count_name | count | |
count1xx | (with kpi_type = access) count of $f_code = 1XX | |
count2xx | (with kpi_type = access) count of $f_code = 2XX | |
count3xx | (with kpi_type = access) count of $f_code = 3XX | |
count4xx | (with kpi_type = access) count of $f_code = 4XX | |
count5xx | (with kpi_type = access) count of $f_code = 5XX | |
responseTimeAve | m(_ _)m |
(with kpi_type = access) average of $f_response_time |
responseTimeMax | m(_ _)m |
(with kpi_type = access) maximum of $f_response_time |
responseTimeMin | m(_ _)m |
(with kpi_type = access) minimum of $f_response_time |
responseTimeSum | (with kpi_type = access) sum of $f_response_time | |
countOver | (with kpi_type = access) count of the access: $f_response_time >= $response_threshold | |
okRatio | m(_ _)m |
(with kpi_type = access) (count - countOver) / count |
counter[60] | (with kpi_type = access & kpy_unit = min) qps for each second | |
qpsAve | m(_ _)m |
(with kpi_type = access & kpy_unit = min) average of qps |
qpsMax | m(_ _)m |
(with kpi_type = access & kpy_unit = min) maximum of qps |
qpsMin | m(_ _)m |
(with kpi_type = access & kpy_unit = min) minimum of qps |
m(_ _)m
: these parameters are updated by non-atmic queries, then not reliable in multi processing
Samples
sample: access log (simple)
config
<match ***>
type mongokpi
kpi_type access
kpi_unit hourly
count_key site
time_key request_time
time_format %d/%b/%Y:%H:%M:%S %Z
</match>
log
{"request_time":"25/Nov/2013:20:03:02 +0700","code":"100","response_time":"1000","site":"git"}
{"request_time":"25/Nov/2013:20:03:03 +0700","code":"200","response_time":"2000","site":"git"}
{"request_time":"25/Nov/2013:20:03:03 +0700","code":"300","response_time":"3000","site":"git"}
{"request_time":"25/Nov/2013:20:03:03 +0700","code":"404","response_time":"4000","site":"blog"}
{"request_time":"25/Nov/2013:20:03:03 +0700","code":"500","response_time":"5000","site":"blog"}
MongoDB
db.kpiCol20131125.find()
{ "_id" : "git2013112520", "count" : 3, "count1xx" : 1, "count2xx" : 1, "count3xx" : 1, "count4xx" : 0, "count5xx" : 0, "countOver" : 0,
"hh" : 20, "okRatio" : 1, "responseTimeAve" : 2000, "responseTimeMax" : 3000, "responseTimeMin" : 1000, "responseTimeSum" : 6000, "site" : "git", "yyyymmdd" : 20131125 }
{ "_id" : "blog2013112520", "count" : 2, "count1xx" : 0, "count2xx" : 0, "count3xx" : 0, "count4xx" : 4, "count5xx" : 1, "countOver" : 0,
"hh" : 20, "okRatio" : 1, "responseTimeAve" : 4500, "responseTimeMax" : 5000, "responseTimeMin" : 4000, "responseTimeSum" : 9000, "site" : "blog", "yyyymmdd" : 20131125 }
{ "_id" : "total2013112520", "count" : 5, "count1xx" : 0, "count2xx" : 0, "count3xx" : 0, "count4xx" : 0, "count5xx" : 0, "countOver" : 0,
"hh" : 20, "okRatio" : 1, "responseTimeAve" : 3000, "responseTimeMax" : 5000, "responseTimeMin" : 1000, "responseTimeSum" : 15000, "site" : "total", "yyyymmdd" : 20131125 }
sample: access log (detail)
config
<match ***>
type mongokpi
kpi_type access
count_key site
time_key request_time
time_format %d/%b/%Y:%H:%M:%S %Z
</match>
log
{"request_time":"25/Nov/2013:20:03:02 +0700","code":"100","response_time":"1000","site":"git"}
{"request_time":"25/Nov/2013:20:03:03 +0700","code":"200","response_time":"2000","site":"git"}
{"request_time":"25/Nov/2013:20:03:03 +0700","code":"300","response_time":"3000","site":"git"}
{"request_time":"25/Nov/2013:20:03:03 +0700","code":"404","response_time":"4000","site":"blog"}
{"request_time":"25/Nov/2013:20:03:03 +0700","code":"500","response_time":"5000","site":"blog"}
MongoDB
db.kpiCol20131125.find()
{ "_id" : "git201311252003", "count" : 3, "count1xx" : 1, "count2xx" : 1, "count3xx" : 1, "count4xx" : 0, "count5xx" : 0, "countOver" : 0,
"counter" : [ 0,0,1,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
"hh" : 20, "mm" : 3, "okRatio" : 1, "qpsAve" : 0, "qpsMax" : 2, "qpsMin" : 0, "responseTimeAve" : 2000, "responseTimeMax" : 3000, "responseTimeMin" : 1000, "responseTimeSum" : 6000, "site" : "git", "yyyymmdd" : 20131125 }
{ "_id" : "blog201311252003", "count" : 2, "count1xx" : 0, "count2xx" : 0, "count3xx" : 0, "count4xx" : 4, "count5xx" : 1, "countOver" : 0,
"counter" : [ 0,0,0,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
"hh" : 20, "mm" : 3, "okRatio" : 1, "qpsAve" : 0, "qpsMax" : 2, "qpsMin" : 0, "responseTimeAve" : 4500, "responseTimeMax" : 5000, "responseTimeMin" : 4000, "responseTimeSum" : 9000, "site" : "blog", "yyyymmdd" : 20131125 }
{ "_id" : "total201311252003", "count" : 5, "count1xx" : 0, "count2xx" : 0, "count3xx" : 0, "count4xx" : 0, "count5xx" : 0, "countOver" : 0,
"counter" : [ 0,0,1,4,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
"hh" : 20, "mm" : 3, "okRatio" : 1, "qpsAve" : 0, "qpsMax" : 4, "qpsMin" : 0, "responseTimeAve" : 3000, "responseTimeMax" : 5000, "responseTimeMin" : 1000, "responseTimeSum" : 15000, "site" : "total", "yyyymmdd" : 20131125 }
sample: app log
config
<match ***>
type mongokpi
count_key site
</match>
log
{"gerogero":"Let's get Rocking!","site":"git"}
{"gerogero":"Let's get Rocking!","site":"git"}
{"gerogero":"Let's get Rocking!","site":"git"}
{"gerogero":"Let's get Rocking!","site":"blog"}
{"gerogero":"Let's get Rocking!","site":"blog"}
MongoDB
db.kpiCol20131125.find()
{ "_id" : "git201311252248", "count" : 3, "hh" : 22, "mm" : 48, "site" : "git", "yyyymmdd" : 20131125 }
{ "_id" : "blog201311252248", "count" : 2, "hh" : 22, "mm" : 48, "site" : "blog", "yyyymmdd" : 20131125 }
{ "_id" : "total201311252248","count" : 5, "hh" : 22, "mm" : 48, "site" : "total", "yyyymmdd" : 20131125 }
Contributing
- Fork it
- Create your feature branch (
git checkout -b my-new-feature
) - Commit your changes (
git commit -am 'Add some feature'
) - Push to the branch (
git push origin my-new-feature
) - Create new Pull Request