fluent-plugin-mysql-load, a plugin for Fluentd Build Status

Overview

BufferedOutput plugin to mysql import. Internal processing uses the "LOAD DATA LOCAL INFILE".

Buffer Plugin Overview is here

MySQL Manual of LOAD DATA is here

Configuration

Ex1.

<match loaddata.**>
  type mysql_load
  host localhost
  port 3306
  username taro
  password abcdefg
  database fluentd
  column_names id,txt,txt2,txt3,created_at

  buffer_type file
  buffer_path /var/log/fluent/test.*.buffer
  flush_interval 60s
</match>

In.

loaddata.test: {"id":111,"txt":"hoge","txt2":"foo","txt3":"bar","created_at":"2014-01-01 00:00:00"}
loaddata.test: {"id":112,"txt":"hoge2","txt2":"foo2","txt3":"bar2","created_at":"2014-01-01 00:00:01"}
loaddata.test: {"id":123,"txt":"hoge3","txt2":"foo3","txt3":"bar3","created_at":"2014-01-01 00:00:02"}

Out.

mysql> desc test;
+------------+----------+------+-----+---------+----------------+
| Field      | Type     | Null | Key | Default | Extra          |
+------------+----------+------+-----+---------+----------------+
| id         | int(11)  | NO   | PRI | NULL    | auto_increment |
| txt        | text     | YES  |     | NULL    |                |
| txt2       | text     | YES  |     | NULL    |                |
| txt3       | text     | YES  |     | NULL    |                |
| created_at | datetime | YES  |     | NULL    |                |
+------------+----------+------+-----+---------+----------------+
mysql> select * from test;
+---------------+------------------+---------------------+---------------------+---------------------+
| id            | txt              | txt2                | txt3                |created_at           |
+---------------+------------------+---------------------+---------------------+---------------------+
| 111           | hoge             | foo                 | bar                 |2014-01-01 00:00:00  |
| 112           | hoge2            | foo2                | bar2                |2014-01-01 00:00:01  |
| 113           | hoge3            | foo3                | bar3                |2014-01-01 00:00:02  |
+---------------+------------------+---------------------+---------------------+---------------------+

Ex2.

<match loaddata.**>
  type mysql_load
  host localhost
  port 3306
  username taro
  password abcdefg
  database fluentd
  key_names dummy1,dummy2,dummy3,create_d
  column_names txt,txt2,txt3,created_at

  buffer_type file
  buffer_path /var/log/fluent/test.*.buffer
  flush_interval 60s
</match>

In.

loaddata.test: {"dummy1":"hoge","dummy2":"foo","dummy3":"bar","create_d":"2014-01-01 00:00:00"}
loaddata.test: {"dummy1":"hoge2","dummy2":"foo2","dummy3":"bar2","create_d":"2014-01-01 00:00:01"}
loaddata.test: {"dummy1":"hoge3","dummy2":"foo3","dummy3":"bar3","create_d":"2014-01-01 00:00:02"}

Out.

mysql> desc test;
+------------+----------+------+-----+---------+----------------+
| Field      | Type     | Null | Key | Default | Extra          |
+------------+----------+------+-----+---------+----------------+
| id         | int(11)  | NO   | PRI | NULL    | auto_increment |
| txt        | text     | YES  |     | NULL    |                |
| txt2       | text     | YES  |     | NULL    |                |
| txt3       | text     | YES  |     | NULL    |                |
| created_at | datetime | YES  |     | NULL    |                |
+------------+----------+------+-----+---------+----------------+
mysql> select * from test;
+---------------+------------------+---------------------+---------------------+---------------------+
| id            | txt              | txt2                | txt3                |created_at           |
+---------------+------------------+---------------------+---------------------+---------------------+
| 1             | hoge             | foo                 | bar                 |2014-01-01 00:00:00  |
| 2             | hoge2            | foo2                | bar2                |2014-01-01 00:00:01  |
| 3             | hoge3            | foo3                | bar3                |2014-01-01 00:00:02  |
+---------------+------------------+---------------------+---------------------+---------------------+

Ex3.

<match loaddata.**>
  type mysql_load
  host localhost
  port 3306
  username taro
  password abcdefg
  database fluentd
  key_names dummy1,dummy2,dummy3,${time}
  column_names txt,txt2,txt3,created_at

  buffer_type file
  buffer_path /var/log/fluent/test.*.buffer
  flush_interval 60s
</match>

In.

loaddata.test: {"dummy1":"hoge","dummy2":"foo","dummy3":"bar"}
loaddata.test: {"dummy1":"hoge2","dummy2":"foo2","dummy3":"bar2"}
loaddata.test: {"dummy1":"hoge3","dummy2":"foo3","dummy3":"bar3"}

Out.

mysql> desc test;
+------------+----------+------+-----+---------+----------------+
| Field      | Type     | Null | Key | Default | Extra          |
+------------+----------+------+-----+---------+----------------+
| id         | int(11)  | NO   | PRI | NULL    | auto_increment |
| txt        | text     | YES  |     | NULL    |                |
| txt2       | text     | YES  |     | NULL    |                |
| txt3       | text     | YES  |     | NULL    |                |
| created_at | datetime | YES  |     | NULL    |                |
+------------+----------+------+-----+---------+----------------+
mysql> select * from test;
+---------------+------------------+---------------------+---------------------+---------------------+
| id            | txt              | txt2                | txt3                |created_at           |
+---------------+------------------+---------------------+---------------------+---------------------+
| 1             | hoge             | foo                 | bar                 |2014-01-01 00:00:00  |
| 2             | hoge2            | foo2                | bar2                |2014-01-01 00:00:01  |
| 3             | hoge3            | foo3                | bar3                |2014-01-01 00:00:02  |
+---------------+------------------+---------------------+---------------------+---------------------+

Parameters

  • host (option)

Set the host name of MySQL Server.The default is "localhost".

  • port (option)

Set the port of MySQL Server.The default is "3306".

  • username (option)

Set the user of MySQL Server.The default is "root".

  • password (option)

Set the password of MySQL Server.The default is ""(blank).

  • database (required)

Set the database of MySQL Server.

  • tablename (required)

Set the table name of the import data.

  • key_names (option)

It is possible to specify the key for JSON input.

Will be converted to a format in this embedding $time. (Format:'%Y-%m-%d %H:%M:%S')

  • column_names (required)

Set if you want to specify the column of the table of data import destination.

If you do not specify a "key_names" parameters, to be used as a key input value of JSON the parameters specified in the "column_names".

  • encoding (option)

Set the encode of MySQL Server.The default is "utf8".

And Buffer Plugin Parameters...

And Logging of Fluentd Parameters...(>=v0.10.43)

ChangeLog

See CHANGELOG.md for details.

Copyright

Copyright:: Copyright (c) 2014- Fukui ReTu License:: Apache License, Version 2.0