Module: LZ4

Defined in:
lib/extlz4.rb,
lib/extlz4/compat.rb,
lib/extlz4/version.rb,
lib/extlz4/oldstream.rb,
lib/extlz4/fix-0.1bug.rb,
ext/extlz4.c

Overview

LZ4 data and streaming data processor.

Defined Under Namespace

Modules: BasicStream Classes: BlockDecoder, BlockEncoder, Decoder, Encoder, Error, StreamDecoder, StreamEncoder, StreamFixerForBug_0_1

Constant Summary collapse

LZ4 = 
self
RawStreamEncoder = 
BlockEncoder
RawStreamDecoder = 
BlockDecoder
VERSION = 
"0.2.4.2"
LIBVERSION = 
ver
LZ4HC_CLEVEL_MIN = 
INT2FIX(LZ4HC_CLEVEL_MIN)
LZ4HC_CLEVEL_DEFAULT = 
INT2FIX(LZ4HC_CLEVEL_DEFAULT)
LZ4HC_CLEVEL_OPT_MIN = 
INT2FIX(LZ4HC_CLEVEL_OPT_MIN)
LZ4HC_CLEVEL_MAX = 
INT2FIX(LZ4HC_CLEVEL_MAX)
HC_CLEVEL_MIN = 
INT2FIX(LZ4HC_CLEVEL_MIN)
HC_CLEVEL_DEFAULT = 
INT2FIX(LZ4HC_CLEVEL_DEFAULT)
HC_CLEVEL_OPT_MIN = 
INT2FIX(LZ4HC_CLEVEL_OPT_MIN)
HC_CLEVEL_MAX = 
INT2FIX(LZ4HC_CLEVEL_MAX)

Class Method Summary collapse

Class Method Details

.block_decode(*args) ⇒ Object Also known as: block_decompress, block_uncompress, raw_decode 



282
283
284
 
# File 'lib/extlz4.rb', line 282

def self.block_decode(*args)
  BlockDecoder.decode(*args)
end

.block_encode(*args) ⇒ Object Also known as: block_compress, raw_encode 



278
279
280
 
# File 'lib/extlz4.rb', line 278

def self.block_encode(*args)
  BlockEncoder.encode(*args)
end

.block_stream_decode(*args) ⇒ Object Also known as: block_stream_decompress, block_stream_uncompress, raw_stream_decode

Call LZ4::BlockDecoder.new.



303
304
305
306
307
308
309
310
311
312
 
# File 'lib/extlz4.rb', line 303

def self.block_stream_decode(*args)
  lz4 = BlockDecoder.new(*args)
  return lz4 unless block_given?

  begin
    yield(lz4)
  rescue
    lz4.release rescue nil
  end
end

.block_stream_encode(*args) ⇒ Object Also known as: block_stream_compress, raw_stream_encode

Call LZ4::BlockEncoder.new.



289
290
291
292
293
294
295
296
297
298
 
# File 'lib/extlz4.rb', line 289

def self.block_stream_encode(*args)
  lz4 = BlockEncoder.new(*args)
  return lz4 unless block_given?

  begin
    yield(lz4)
  ensure
    lz4.release rescue nil
  end
end

.decode(obj, *args) ⇒ Object Also known as: decompress, uncompress

call-seq:

decode(encoded_data_string) -> decoded data
decode(input_io) -> decoder
decode(input_io) { |decoder| ... } -> yield_status

Decode LZ4 Frame data. This is available streaming process.

decode(encoded_data_string)

RETURN (String)

decoded_data

decode(input_io)

RETURN (LZ4::Decoder)

ストリーム展開オブジェクトです。簡素な機能の読み込み専用IOオブジェクトとして扱うことが出来ます。

decoder は GC によって開放処理が行われますが、利用しなくなった時点で利用者が明示的に close を呼び出すことが望まれます。

input_io (IO)

This is IO like object. Need read method. ‘extlz4’ is call as read(size, buf) style.

decode(input_io) { |decoder| … }

RETURN

returned value from given block

YIELD (decoder)

ブロックなしで与えた場合の戻り値と等価です。

ただしこちらはブロックを抜けたらすぐに開放処理が実施されます。利用者が明示的に close を呼んだり、GC されるのを待ったりせずに行われると言うことです。

example: directly decode

LZ4.decode(lz4_encoded_string) # => decoded binary string

example: streaming decode

File.open("sample.lz4", "rb") do |fd|
  LZ4.decode(fd) do |lz4dec|
    lz4dec.read(16)  # string with read 16 bytes
    lz4dec.getbyte   # integer with a byte
    lz4dec.read      # string with rest data
  end
end



260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
 
# File 'lib/extlz4.rb', line 260

def self.decode(obj, *args)
  if obj.kind_of?(String)
    lz4 = Decoder.new(StringIO.new(obj), *args)
    dest = lz4.read
    lz4.close
    return (dest || "".b)
  end

  lz4 = Decoder.new(obj, *args)
  return lz4 unless block_given?

  begin
    yield(lz4)
  ensure
    lz4.close
  end
end

.decode_file(inpath, outpath) ⇒ Object

call-seq:

decode_file(inpath, outpath) -> nil

Decode lz4 file to regular file.

RETURN

Return nil always.

inpath

Give input file path, or input IO (liked) object its has “read” method.

outpath

Give output file path, or output IO (liked) object its has “<<” method.



32
33
34
35
36
37
38
39
40
41
42
43
44
 
# File 'lib/extlz4.rb', line 32

def self.decode_file(inpath, outpath)
  open_file(inpath, "rb") do |infile|
    decode(infile) do |lz4|
      open_file(outpath, "wb") do |outfile|
        inbuf = ""
        slicesize = 1 << 20
        outfile << inbuf while lz4.read(slicesize, inbuf)
      end
    end
  end

  nil
end

.decode_old(io, &block) ⇒ Object 



77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
 
# File 'lib/extlz4/oldstream.rb', line 77

def self.decode_old(io, &block)
  if io.kind_of?(String)
    lz4 = StreamDecoder.new(StringIO.new(io))
    dest = lz4.read
    lz4.close
    return dest
  end

  dec = StreamDecoder.new(io)
  return dec unless block_given?

  begin
    yield(dec)
  ensure
    dec.close
  end
end

.encode(*args, **opts) ⇒ Object Also known as: compress

call-seq:

encode(source_string, level = 1, opts = {}) -> lz4 frame'd data
encode(output_io, level = 1, opts = {}) -> stream encoder
encode(output_io, level = 1, opts = {}) { |stream_encoder| ... } -> yield_status

Encode to LZ4 Frame data. This is available streaming process.

Created data is decodable by lz4-cli.

Common parameters

level = 1 (Integer)

圧縮レベルを指定します。0 から 9 までの整数値が指定出来ます。

現時点では lz4cli (lz4io) に倣って、3以下が標準圧縮、4以上が高圧縮となります。

4以上の値は、高効率圧縮器の圧縮レベルとして渡されます。

blocklink: false (true or false)

Enable or disable block dependency funcion. Default is false.

真を与えた場合、ストリームの圧縮効率が向上します。

checksum: true (true or false)

ストリーム全体のチェックサム (XXhash32) の有効・無効を切り替えます。

encode(source_string, level = 1, opts = {}) -> encoded_data

Basic encode method.

RETURN (String)

Encoded data as LZ4 stream

source_string (String)

LZ4 ストリームとして圧縮したいバイナリデータ列としての文字列です。

文字符号情報は無視されて純粋なバイナリデータ列として処理されます。

encode(output_io, level = 1, opts = {}) -> encoder

Available streaming LZ4 Frame encode.

Write to encoder for data encoding.

After finished encode process, you must call Encoder#close.

Return stream encoder if given an IO (liked) object.

この圧縮器に『書き込む』ことでデータは圧縮されます。 圧縮処理を完了するときには #close を呼び出す必要があります。

RETURN (LZ4::Encoder)
output_io (IO)

LZ4 ストリームの出力先を指定します。IO#<< と同等の機能を持つオブジェクトである必要があります。

一例を挙げると、IO、StringIO、Array などのインスタンスが当てはまります。

encode(output_io, level = 1, opts = {}) { |encoder| … } -> yield_status

IO オブジェクトとともにブロックを渡した場合、ブロック引数として圧縮器が渡されます。この場合は #close を呼び出す必要がありません。

RETURN

return value of given block

YIELD (encoder)
YIELDRETURN

return as method return value

example: directly encode

LZ4.encode("abcdefghijklmn") # => Encoded LZ4 stream data (string object)

example: streaming encode with block

この用例は、encode_file の実装とほぼ同じです。丸写しで利用するよりは encode_file の利用を推奨します。

File.open("hello.txt", "rb") do |src|
  File.open("hello.txt.lz4", "wb") do |dest|
    srcbuf = ""
    LZ4.encode(dest) do |lz4encoder|
      nil while lz4encoder << src.read(4096, srcbuf)
    end
  end
end



193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
 
# File 'lib/extlz4.rb', line 193

def self.encode(*args, **opts)
  if args.empty? || !args[0].kind_of?(String)
    lz4 = LZ4::Encoder.new(*args, **opts)
    return lz4 unless block_given?
    begin
      yield(lz4)
      lz4.outport
    ensure
      lz4.close
    end
  else
    obj = args.shift
    outport = "".force_encoding(Encoding::BINARY)
    lz4 = LZ4::Encoder.new(outport, *args, **opts)
    lz4 << obj
    lz4.close
    outport
  end
end

.encode_file(inpath, outpath, *args, **opts) ⇒ Object

call-seq:

encode_file(inpath, outpath, level = 1, opts = {}) -> nil

Encode regular file to lz4 file.

RETURN

Return nil always.

inpath

Give input file path, or input IO (liked) object its has “read” method.

outpath

Give output file path, or output IO (liked) object its has “<<” method.

level = 1 (Integer)

See LZ4.encode method.

opts = {} (Hash)

See LZ4.encode method.



67
68
69
70
71
72
73
74
75
76
77
78
79
 
# File 'lib/extlz4.rb', line 67

def self.encode_file(inpath, outpath, *args, **opts)
  open_file(inpath, "rb") do |infile|
    open_file(outpath, "wb") do |outfile|
      encode(outfile, *args, **opts) do |lz4|
        inbuf = ""
        slicesize = 1 << 20
        lz4 << inbuf while infile.read(slicesize, inbuf)
      end
    end
  end

  nil
end

.encode_old(first, *args) ⇒ Object 



19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
 
# File 'lib/extlz4/oldstream.rb', line 19

def self.encode_old(first, *args)
  case args.size
  when 0
    level = nil
    opts = StreamEncoder::OPTIONS
  when 1
    level = args[0]
    if level.respond_to?(:to_hash)
      opts = StreamEncoder::OPTIONS.merge(level)
      level = nil
    else
      level = level.to_i
      opts = StreamEncoder::OPTIONS
    end
  when 2
    level = args[0].to_i
    opts = StreamEncoder::OPTIONS.merge(args[1])
  else
    raise ArgumentError, "wrong number of arguments (#{args.size + 1} for 1 .. 3)"
  end

  left = opts.keys - StreamEncoder::OPTIONS.keys
  unless left.empty?
    if left.size > 10
      raise ArgumentError, "unknown key - #{left[0]} (for #{StreamEncoder::OPTIONS.keys.slice(0, 10).join(", ")} and more...)"
    else
      raise ArgumentError, "unknown key - #{left[0]} (for #{StreamEncoder::OPTIONS.keys.join(", ")})"
    end
  end

  if first.kind_of?(String)
    src = first
    dest = StringIO.new("".b)
  else
    src = nil
    dest = first
  end

  lz4 = StreamEncoder.new(dest, level || 1,
                          opts[:blocksize], opts[:block_dependency],
                          opts[:block_checksum], opts[:stream_checksum])

  case
  when src
    lz4 << src
    lz4.close
    dest.string
  when block_given?
    begin
      yield(lz4)
    ensure
      lz4.close
    end
  else
    lz4
  end
end

.fix_extlz4_0_1_bug(inpath, outpath, &block) ⇒ Object 



86
87
88
89
90
91
92
93
94
95
 
# File 'lib/extlz4/fix-0.1bug.rb', line 86

def self.fix_extlz4_0_1_bug(inpath, outpath, &block)
  open_file(inpath, "rb") do |infile|
    open_file(outpath, "wb") do |outfile|
      fixer = LZ4::StreamFixerForBug_0_1.new(infile)
      fixer.fix(outfile, &block)
    end
  end

  nil
end

.open_file(file, mode) ⇒ Object 



93
94
95
96
97
98
99
100
101
102
103
 
# File 'lib/extlz4.rb', line 93

def self.open_file(file, mode)
  case
  when file.kind_of?(String)
    File.open(file, mode, &proc)
  when file.respond_to?(:binmode)
    file.binmode rescue nil
    yield(file)
  else
    yield(file)
  end
end

.test_file(inpath) ⇒ Object 



81
82
83
84
85
86
87
88
89
90
91
 
# File 'lib/extlz4.rb', line 81

def self.test_file(inpath)
  open_file(inpath, "rb") do |infile|
    decode(infile) do |lz4|
      inbuf = ""
      slicesize = 1 << 20
      nil while lz4.read(slicesize, inbuf)
    end
  end

  nil
end