Class: MysqlBinlog::BinlogFieldParser
- Inherits:
-
Object
- Object
- MysqlBinlog::BinlogFieldParser
- Defined in:
- lib/mysql_binlog/binlog_field_parser.rb
Overview
Parse various types of standard and non-standard data types from a provided binary log using its reader to read data.
Instance Attribute Summary collapse
-
#binlog ⇒ Object
Returns the value of attribute binlog.
-
#reader ⇒ Object
Returns the value of attribute reader.
Instance Method Summary collapse
-
#convert_mysql_type_date(value) ⇒ Object
Convert a packed
DATE
from a uint24 into a string representing the date. -
#convert_mysql_type_datetime(value) ⇒ Object
Convert a packed
DATETIME
from a uint64 into a string representing the date and time. -
#convert_mysql_type_time(value) ⇒ Object
Convert a packed
TIME
from a uint24 into a string representing the time. -
#extract_bits(value, bits, offset) ⇒ Object
Extract a number of sequential bits at a given offset within an integer.
-
#initialize(binlog_instance) ⇒ BinlogFieldParser
constructor
A new instance of BinlogFieldParser.
-
#read_bit_array(length) ⇒ Object
Read an arbitrary-length bitmap, provided its length.
-
#read_double ⇒ Object
Read a double-precision (8-byte) floating point number.
-
#read_float ⇒ Object
Read a single-precision (4-byte) floating point number.
-
#read_int16_be ⇒ Object
Read a signed 16-bit (2-byte) big-endian integer.
-
#read_int24_be ⇒ Object
Read a signed 24-bit (3-byte) big-endian integer.
-
#read_int32_be ⇒ Object
Read a signed 32-bit (4-byte) big-endian integer.
-
#read_int8 ⇒ Object
Read a signed 8-bit (1-byte) integer.
- #read_int_be_by_size(size) ⇒ Object
-
#read_lpstring(size = 1) ⇒ Object
Read a (Pascal-style) length-prefixed string.
-
#read_lpstringz(size = 1) ⇒ Object
Read an lpstring (as above) which is also terminated with a null byte.
-
#read_mysql_type(type, metadata = nil) ⇒ Object
Read a single field, provided the MySQL column type as a symbol.
-
#read_newdecimal(precision, scale) ⇒ Object
Read a (new) decimal value.
-
#read_nstring(length) ⇒ Object
Read a non-terminated string, provided its length.
-
#read_nstringz(length) ⇒ Object
Read a null-terminated string, provided its length (with the null).
-
#read_uint16 ⇒ Object
Read an unsigned 16-bit (2-byte) integer.
-
#read_uint24 ⇒ Object
Read an unsigned 24-bit (3-byte) integer.
-
#read_uint32 ⇒ Object
Read an unsigned 32-bit (4-byte) integer.
-
#read_uint40 ⇒ Object
Read an unsigned 40-bit (5-byte) integer.
-
#read_uint48 ⇒ Object
Read an unsigned 48-bit (6-byte) integer.
-
#read_uint56 ⇒ Object
Read an unsigned 56-bit (7-byte) integer.
-
#read_uint64 ⇒ Object
Read an unsigned 64-bit (8-byte) integer.
-
#read_uint8 ⇒ Object
Read an unsigned 8-bit (1-byte) integer.
-
#read_uint8_array(length) ⇒ Object
Read an array of unsigned 8-bit (1-byte) integers.
-
#read_uint_bitmap_by_size_and_name(size, bit_names) ⇒ Object
Read a uint value using the provided size, and convert it to an array of symbols derived from a mapping table provided.
- #read_uint_by_size(size) ⇒ Object
-
#read_varint ⇒ Object
Read a variable-length “Length Coded Binary” integer.
-
#read_varstring ⇒ Object
Read a MySQL-style varint length-prefixed string.
Constructor Details
#initialize(binlog_instance) ⇒ BinlogFieldParser
Returns a new instance of BinlogFieldParser.
45 46 47 48 49 |
# File 'lib/mysql_binlog/binlog_field_parser.rb', line 45 def initialize(binlog_instance) @format_cache = {} @binlog = binlog_instance @reader = binlog_instance.reader end |
Instance Attribute Details
#binlog ⇒ Object
Returns the value of attribute binlog.
42 43 44 |
# File 'lib/mysql_binlog/binlog_field_parser.rb', line 42 def binlog @binlog end |
#reader ⇒ Object
Returns the value of attribute reader.
43 44 45 |
# File 'lib/mysql_binlog/binlog_field_parser.rb', line 43 def reader @reader end |
Instance Method Details
#convert_mysql_type_date(value) ⇒ Object
Convert a packed DATE
from a uint24 into a string representing the date.
328 329 330 331 332 333 334 |
# File 'lib/mysql_binlog/binlog_field_parser.rb', line 328 def convert_mysql_type_date(value) "%04i-%02i-%02i" % [ extract_bits(value, 15, 9), extract_bits(value, 4, 5), extract_bits(value, 5, 0), ] end |
#convert_mysql_type_datetime(value) ⇒ Object
Convert a packed DATETIME
from a uint64 into a string representing the date and time.
348 349 350 351 352 353 354 355 356 357 358 359 360 |
# File 'lib/mysql_binlog/binlog_field_parser.rb', line 348 def convert_mysql_type_datetime(value) date = value / 1000000 time = value % 1000000 "%04i-%02i-%02i %02i:%02i:%02i" % [ date / 10000, (date % 10000) / 100, date % 100, time / 10000, (time % 10000) / 100, time % 100, ] end |
#convert_mysql_type_time(value) ⇒ Object
Convert a packed TIME
from a uint24 into a string representing the time.
338 339 340 341 342 343 344 |
# File 'lib/mysql_binlog/binlog_field_parser.rb', line 338 def convert_mysql_type_time(value) "%02i:%02i:%02i" % [ value / 10000, (value % 10000) / 100, value % 100, ] end |
#extract_bits(value, bits, offset) ⇒ Object
Extract a number of sequential bits at a given offset within an integer. This is used to unpack bit-packed fields.
322 323 324 |
# File 'lib/mysql_binlog/binlog_field_parser.rb', line 322 def extract_bits(value, bits, offset) (value & ((1 << bits) - 1) << offset) >> offset end |
#read_bit_array(length) ⇒ Object
Read an arbitrary-length bitmap, provided its length. Returns an array of true/false values. This is used both for internal usage in RBR events that need bitmaps, as well as for the BIT type.
286 287 288 289 290 |
# File 'lib/mysql_binlog/binlog_field_parser.rb', line 286 def read_bit_array(length) data = reader.read((length+7)/8) data.unpack("b*").first. # Unpack into a string of "10101" split("").map { |c| c == "1" }.shift(length) # Return true/false array end |
#read_double ⇒ Object
Read a double-precision (8-byte) floating point number.
162 163 164 |
# File 'lib/mysql_binlog/binlog_field_parser.rb', line 162 def read_double reader.read(8).unpack("E").first end |
#read_float ⇒ Object
Read a single-precision (4-byte) floating point number.
157 158 159 |
# File 'lib/mysql_binlog/binlog_field_parser.rb', line 157 def read_float reader.read(4).unpack("e").first end |
#read_int16_be ⇒ Object
Read a signed 16-bit (2-byte) big-endian integer.
101 102 103 |
# File 'lib/mysql_binlog/binlog_field_parser.rb', line 101 def read_int16_be reader.read(2).reverse.unpack('s').first end |
#read_int24_be ⇒ Object
Read a signed 24-bit (3-byte) big-endian integer.
106 107 108 109 110 111 112 113 |
# File 'lib/mysql_binlog/binlog_field_parser.rb', line 106 def read_int24_be a, b, c = reader.read(3).unpack('CCC') if a & 128 (a << 16) | (b << 8) | c else (-1 << 24) | (a << 16) | (b << 8) | c end end |
#read_int32_be ⇒ Object
Read a signed 32-bit (4-byte) big-endian integer.
116 117 118 |
# File 'lib/mysql_binlog/binlog_field_parser.rb', line 116 def read_int32_be reader.read(4).reverse.unpack('l').first end |
#read_int8 ⇒ Object
Read a signed 8-bit (1-byte) integer.
96 97 98 |
# File 'lib/mysql_binlog/binlog_field_parser.rb', line 96 def read_int8 reader.read(1).unpack("c").first end |
#read_int_be_by_size(size) ⇒ Object
141 142 143 144 145 146 147 148 149 150 151 152 153 154 |
# File 'lib/mysql_binlog/binlog_field_parser.rb', line 141 def read_int_be_by_size(size) case size when 1 read_int8 when 2 read_int16_be when 3 read_int24_be when 4 read_int32_be else raise "read_int#{size*8}_be not implemented" end end |
#read_lpstring(size = 1) ⇒ Object
Read a (Pascal-style) length-prefixed string. The length is stored as a 8-bit (1-byte) to 32-bit (4-byte) unsigned integer, depending on the optional size parameter (default 1), followed by the string itself with no termination character.
207 208 209 210 |
# File 'lib/mysql_binlog/binlog_field_parser.rb', line 207 def read_lpstring(size=1) length = read_uint_by_size(size) read_nstring(length) end |
#read_lpstringz(size = 1) ⇒ Object
Read an lpstring (as above) which is also terminated with a null byte.
213 214 215 216 217 |
# File 'lib/mysql_binlog/binlog_field_parser.rb', line 213 def read_lpstringz(size=1) string = read_lpstring(size) reader.read(1) # null string end |
#read_mysql_type(type, metadata = nil) ⇒ Object
Read a single field, provided the MySQL column type as a symbol. Not all types are currently supported.
364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 |
# File 'lib/mysql_binlog/binlog_field_parser.rb', line 364 def read_mysql_type(type, =nil) case type when :tiny read_uint8 when :short read_uint16 when :int24 read_uint24 when :long read_uint32 when :longlong read_uint64 when :float read_float when :double read_double when :var_string read_varstring when :varchar, :string prefix_size = ([:max_length] > 255) ? 2 : 1 read_lpstring(prefix_size) when :blob, :geometry read_lpstring([:length_size]) when :timestamp read_uint32 when :year read_uint8 + 1900 when :date convert_mysql_type_date(read_uint24) when :time convert_mysql_type_time(read_uint24) when :datetime convert_mysql_type_datetime(read_uint64) when :enum, :set read_uint_by_size([:size]) when :bit byte_length = ([:bits]+7)/8 read_uint_by_size(byte_length) when :newdecimal precision = [:precision] scale = [:decimals] read_newdecimal(precision, scale) else raise UnsupportedTypeException.new("Type #{type} is not supported.") end end |
#read_newdecimal(precision, scale) ⇒ Object
Read a (new) decimal value. The value is stored as a sequence of signed big-endian integers, each representing up to 9 digits of the integral and fractional parts. The first integer of the integral part and/or the last integer of the fractional part might be compressed (or packed) and are of variable length. The remaining integers (if any) are uncompressed and 32 bits wide.
233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 |
# File 'lib/mysql_binlog/binlog_field_parser.rb', line 233 def read_newdecimal(precision, scale) digits_per_integer = 9 compressed_bytes = [0, 1, 1, 2, 2, 3, 3, 4, 4, 4] integral = (precision - scale) uncomp_integral = integral / digits_per_integer uncomp_fractional = scale / digits_per_integer comp_integral = integral - (uncomp_integral * digits_per_integer) comp_fractional = scale - (uncomp_fractional * digits_per_integer) # The sign is encoded in the high bit of the first byte/digit. The byte # might be part of a larger integer, so apply the optional bit-flipper # and push back the byte into the input stream. value = read_uint8 str, mask = (value & 0x80 != 0) ? ["", 0] : ["-", -1] reader.unget(value ^ 0x80) size = compressed_bytes[comp_integral] if size > 0 value = read_int_be_by_size(size) ^ mask str << value.to_s end (1..uncomp_integral).each do value = read_int32_be ^ mask str << value.to_s end str << "." (1..uncomp_fractional).each do value = read_int32_be ^ mask str << value.to_s end size = compressed_bytes[comp_fractional] if size > 0 value = read_int_be_by_size(size) ^ mask str << value.to_s end BigDecimal.new(str) end |
#read_nstring(length) ⇒ Object
Read a non-terminated string, provided its length.
194 195 196 |
# File 'lib/mysql_binlog/binlog_field_parser.rb', line 194 def read_nstring(length) reader.read(length) end |
#read_nstringz(length) ⇒ Object
Read a null-terminated string, provided its length (with the null).
199 200 201 |
# File 'lib/mysql_binlog/binlog_field_parser.rb', line 199 def read_nstringz(length) reader.read(length).unpack("A*").first end |
#read_uint16 ⇒ Object
Read an unsigned 16-bit (2-byte) integer.
57 58 59 |
# File 'lib/mysql_binlog/binlog_field_parser.rb', line 57 def read_uint16 reader.read(2).unpack("v").first end |
#read_uint24 ⇒ Object
Read an unsigned 24-bit (3-byte) integer.
62 63 64 65 |
# File 'lib/mysql_binlog/binlog_field_parser.rb', line 62 def read_uint24 a, b, c = reader.read(3).unpack("CCC") a + (b << 8) + (c << 16) end |
#read_uint32 ⇒ Object
Read an unsigned 32-bit (4-byte) integer.
68 69 70 |
# File 'lib/mysql_binlog/binlog_field_parser.rb', line 68 def read_uint32 reader.read(4).unpack("V").first end |
#read_uint40 ⇒ Object
Read an unsigned 40-bit (5-byte) integer.
73 74 75 76 |
# File 'lib/mysql_binlog/binlog_field_parser.rb', line 73 def read_uint40 a, b = reader.read(5).unpack("CV") a + (b << 8) end |
#read_uint48 ⇒ Object
Read an unsigned 48-bit (6-byte) integer.
79 80 81 82 |
# File 'lib/mysql_binlog/binlog_field_parser.rb', line 79 def read_uint48 a, b, c = reader.read(6).unpack("vvv") a + (b << 16) + (c << 32) end |
#read_uint56 ⇒ Object
Read an unsigned 56-bit (7-byte) integer.
85 86 87 88 |
# File 'lib/mysql_binlog/binlog_field_parser.rb', line 85 def read_uint56 a, b, c = reader.read(7).unpack("CvV") a + (b << 8) + (c << 24) end |
#read_uint64 ⇒ Object
Read an unsigned 64-bit (8-byte) integer.
91 92 93 |
# File 'lib/mysql_binlog/binlog_field_parser.rb', line 91 def read_uint64 reader.read(8).unpack("Q").first end |
#read_uint8 ⇒ Object
Read an unsigned 8-bit (1-byte) integer.
52 53 54 |
# File 'lib/mysql_binlog/binlog_field_parser.rb', line 52 def read_uint8 reader.read(1).unpack("C").first end |
#read_uint8_array(length) ⇒ Object
Read an array of unsigned 8-bit (1-byte) integers.
279 280 281 |
# File 'lib/mysql_binlog/binlog_field_parser.rb', line 279 def read_uint8_array(length) reader.read(length).bytes.to_a end |
#read_uint_bitmap_by_size_and_name(size, bit_names) ⇒ Object
Read a uint value using the provided size, and convert it to an array of symbols derived from a mapping table provided.
294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 |
# File 'lib/mysql_binlog/binlog_field_parser.rb', line 294 def read_uint_bitmap_by_size_and_name(size, bit_names) value = read_uint_by_size(size) named_bits = [] # Do an efficient scan for the named bits we know about using the hash # provided. bit_names.each do |(name, bit_value)| if (value & bit_value) != 0 value -= bit_value named_bits << name end end # If anything is left over in +value+, add "unknown" names to the result # so that they can be identified and corrected. if value > 0 0.upto(size * 8).map { |n| 1 << n }.each do |bit_value| if (value & bit_value) != 0 named_bits << "unknown_#{bit_value}".to_sym end end end named_bits end |
#read_uint_by_size(size) ⇒ Object
120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 |
# File 'lib/mysql_binlog/binlog_field_parser.rb', line 120 def read_uint_by_size(size) case size when 1 read_uint8 when 2 read_uint16 when 3 read_uint24 when 4 read_uint32 when 5 read_uint40 when 6 read_uint48 when 7 read_uint56 when 8 read_uint64 end end |
#read_varint ⇒ Object
Read a variable-length “Length Coded Binary” integer. This is derived from the MySQL protocol, and re-used in the binary log format. This format uses the first byte to alternately store the actual value for integer values <= 250, or to encode the number of following bytes used to store the actual value, which can be 2, 3, or 8. It also includes support for SQL NULL as a special case.
See: forge.mysql.com/wiki/MySQL_Internals_ClientServer_Protocol#Elements
174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 |
# File 'lib/mysql_binlog/binlog_field_parser.rb', line 174 def read_varint first_byte = read_uint8 case when first_byte <= 250 first_byte when first_byte == 251 nil when first_byte == 252 read_uint16 when first_byte == 253 read_uint24 when first_byte == 254 read_uint64 when first_byte == 255 raise "Invalid variable-length integer" end end |
#read_varstring ⇒ Object
Read a MySQL-style varint length-prefixed string. The length is stored as a variable-length “Length Coded Binary” value (see read_varint) which is followed by the string content itself. No termination is included.
222 223 224 225 |
# File 'lib/mysql_binlog/binlog_field_parser.rb', line 222 def read_varstring length = read_varint read_nstring(length) end |