Class: Aws::Firehose::Types::OrcSerDe

Inherits:

Struct

• Object
• Struct
• Aws::Firehose::Types::OrcSerDe

Includes:

Structure

Defined in:

lib/aws-sdk-firehose/types.rb

Overview

Note:

When making an API call, you may pass OrcSerDe data as a hash:

{
  stripe_size_bytes: 1,
  block_size_bytes: 1,
  row_index_stride: 1,
  enable_padding: false,
  padding_tolerance: 1.0,
  compression: "NONE", # accepts NONE, ZLIB, SNAPPY
  bloom_filter_columns: ["NonEmptyStringWithoutWhitespace"],
  bloom_filter_false_positive_probability: 1.0,
  dictionary_key_threshold: 1.0,
  format_version: "V0_11", # accepts V0_11, V0_12
}

A serializer to use for converting data to the ORC format before storing it in Amazon S3. For more information, see [Apache ORC].

[1]: orc.apache.org/docs/

See Also:

• AWS API Documentation

Instance Attribute Summary

• #block_size_bytes ⇒ Integer

  The Hadoop Distributed File System (HDFS) block size.

• #bloom_filter_columns ⇒ Array<String>

  The column names for which you want Kinesis Data Firehose to create bloom filters.

• #bloom_filter_false_positive_probability ⇒ Float

  The Bloom filter false positive probability (FPP).

• #compression ⇒ String

  The compression code to use over data blocks.

• #dictionary_key_threshold ⇒ Float

  Represents the fraction of the total number of non-null rows.

• #enable_padding ⇒ Boolean

  Set this to ‘true` to indicate that you want stripes to be padded to the HDFS block boundaries.

• #format_version ⇒ String

  The version of the file to write.

• #padding_tolerance ⇒ Float

  A number between 0 and 1 that defines the tolerance for block padding as a decimal fraction of stripe size.

• #row_index_stride ⇒ Integer

  The number of rows between index entries.

• #stripe_size_bytes ⇒ Integer

  The number of bytes in each stripe.

Instance Attribute Details

#block_size_bytes ⇒ Integer

The Hadoop Distributed File System (HDFS) block size. This is useful if you intend to copy the data from Amazon S3 to HDFS before querying. The default is 256 MiB and the minimum is 64 MiB. Kinesis Data Firehose uses this value for padding calculations.

Returns:

• (Integer)

# File 'lib/aws-sdk-firehose/types.rb', line 2402

class OrcSerDe < Struct.new(
  :stripe_size_bytes,
  :block_size_bytes,
  :row_index_stride,
  :enable_padding,
  :padding_tolerance,
  :compression,
  :bloom_filter_columns,
  :bloom_filter_false_positive_probability,
  :dictionary_key_threshold,
  :format_version)
  include Aws::Structure
end

#bloom_filter_columns ⇒ Array<String>

The column names for which you want Kinesis Data Firehose to create bloom filters. The default is ‘null`.

Returns:

• (Array<String>)

# File 'lib/aws-sdk-firehose/types.rb', line 2402

class OrcSerDe < Struct.new(
  :stripe_size_bytes,
  :block_size_bytes,
  :row_index_stride,
  :enable_padding,
  :padding_tolerance,
  :compression,
  :bloom_filter_columns,
  :bloom_filter_false_positive_probability,
  :dictionary_key_threshold,
  :format_version)
  include Aws::Structure
end

#bloom_filter_false_positive_probability ⇒ Float

The Bloom filter false positive probability (FPP). The lower the FPP, the bigger the Bloom filter. The default value is 0.05, the minimum is 0, and the maximum is 1.

Returns:

• (Float)

# File 'lib/aws-sdk-firehose/types.rb', line 2402

class OrcSerDe < Struct.new(
  :stripe_size_bytes,
  :block_size_bytes,
  :row_index_stride,
  :enable_padding,
  :padding_tolerance,
  :compression,
  :bloom_filter_columns,
  :bloom_filter_false_positive_probability,
  :dictionary_key_threshold,
  :format_version)
  include Aws::Structure
end

#compression ⇒ String

The compression code to use over data blocks. The default is ‘SNAPPY`.

Returns:

• (String)

# File 'lib/aws-sdk-firehose/types.rb', line 2402

class OrcSerDe < Struct.new(
  :stripe_size_bytes,
  :block_size_bytes,
  :row_index_stride,
  :enable_padding,
  :padding_tolerance,
  :compression,
  :bloom_filter_columns,
  :bloom_filter_false_positive_probability,
  :dictionary_key_threshold,
  :format_version)
  include Aws::Structure
end

#dictionary_key_threshold ⇒ Float

Represents the fraction of the total number of non-null rows. To turn off dictionary encoding, set this fraction to a number that is less than the number of distinct keys in a dictionary. To always use dictionary encoding, set this threshold to 1.

Returns:

• (Float)

# File 'lib/aws-sdk-firehose/types.rb', line 2402

class OrcSerDe < Struct.new(
  :stripe_size_bytes,
  :block_size_bytes,
  :row_index_stride,
  :enable_padding,
  :padding_tolerance,
  :compression,
  :bloom_filter_columns,
  :bloom_filter_false_positive_probability,
  :dictionary_key_threshold,
  :format_version)
  include Aws::Structure
end

#enable_padding ⇒ Boolean

Set this to ‘true` to indicate that you want stripes to be padded to the HDFS block boundaries. This is useful if you intend to copy the data from Amazon S3 to HDFS before querying. The default is `false`.

Returns:

• (Boolean)

# File 'lib/aws-sdk-firehose/types.rb', line 2402

class OrcSerDe < Struct.new(
  :stripe_size_bytes,
  :block_size_bytes,
  :row_index_stride,
  :enable_padding,
  :padding_tolerance,
  :compression,
  :bloom_filter_columns,
  :bloom_filter_false_positive_probability,
  :dictionary_key_threshold,
  :format_version)
  include Aws::Structure
end

#format_version ⇒ String

The version of the file to write. The possible values are ‘V0_11` and `V0_12`. The default is `V0_12`.

Returns:

• (String)

# File 'lib/aws-sdk-firehose/types.rb', line 2402

class OrcSerDe < Struct.new(
  :stripe_size_bytes,
  :block_size_bytes,
  :row_index_stride,
  :enable_padding,
  :padding_tolerance,
  :compression,
  :bloom_filter_columns,
  :bloom_filter_false_positive_probability,
  :dictionary_key_threshold,
  :format_version)
  include Aws::Structure
end

#padding_tolerance ⇒ Float

A number between 0 and 1 that defines the tolerance for block padding as a decimal fraction of stripe size. The default value is 0.05, which means 5 percent of stripe size.

For the default values of 64 MiB ORC stripes and 256 MiB HDFS blocks, the default block padding tolerance of 5 percent reserves a maximum of 3.2 MiB for padding within the 256 MiB block. In such a case, if the available size within the block is more than 3.2 MiB, a new, smaller stripe is inserted to fit within that space. This ensures that no stripe crosses block boundaries and causes remote reads within a node-local task.

Kinesis Data Firehose ignores this parameter when OrcSerDe$EnablePadding is ‘false`.

Returns:

• (Float)

# File 'lib/aws-sdk-firehose/types.rb', line 2402

class OrcSerDe < Struct.new(
  :stripe_size_bytes,
  :block_size_bytes,
  :row_index_stride,
  :enable_padding,
  :padding_tolerance,
  :compression,
  :bloom_filter_columns,
  :bloom_filter_false_positive_probability,
  :dictionary_key_threshold,
  :format_version)
  include Aws::Structure
end

#row_index_stride ⇒ Integer

The number of rows between index entries. The default is 10,000 and the minimum is 1,000.

Returns:

• (Integer)

# File 'lib/aws-sdk-firehose/types.rb', line 2402

class OrcSerDe < Struct.new(
  :stripe_size_bytes,
  :block_size_bytes,
  :row_index_stride,
  :enable_padding,
  :padding_tolerance,
  :compression,
  :bloom_filter_columns,
  :bloom_filter_false_positive_probability,
  :dictionary_key_threshold,
  :format_version)
  include Aws::Structure
end

#stripe_size_bytes ⇒ Integer

The number of bytes in each stripe. The default is 64 MiB and the minimum is 8 MiB.

Returns:

• (Integer)

# File 'lib/aws-sdk-firehose/types.rb', line 2402

class OrcSerDe < Struct.new(
  :stripe_size_bytes,
  :block_size_bytes,
  :row_index_stride,
  :enable_padding,
  :padding_tolerance,
  :compression,
  :bloom_filter_columns,
  :bloom_filter_false_positive_probability,
  :dictionary_key_threshold,
  :format_version)
  include Aws::Structure
end