Module: Zvec::DataTypes

Included in:: Zvec

Defined in:: lib/zvec/data_types.rb

Overview

Data type constants, coercion utilities, and dispatch tables for mapping between Ruby types and the underlying C++ zvec engine types.

Scalar Types

BINARY -- Raw binary data
STRING -- UTF-8 string
BOOL -- Boolean (true/false)
INT32 -- 32-bit signed integer
INT64 -- 64-bit signed integer
UINT32 -- 32-bit unsigned integer
UINT64 -- 64-bit unsigned integer
FLOAT -- 32-bit IEEE 754 float
DOUBLE -- 64-bit IEEE 754 double

Dense Vector Types

Dense vectors store a fixed-length array of numeric values. Choose the precision that balances accuracy vs. memory:

VECTOR_FP32 -- 32-bit float vector (default, best accuracy)
VECTOR_FP64 -- 64-bit double vector (highest accuracy, 2x memory)
VECTOR_FP16 -- 16-bit half-precision vector (half the memory of FP32)
VECTOR_INT8 -- 8-bit integer vector (smallest, for quantized models)

Sparse Vector Types

Sparse vectors store only non-zero elements, ideal for high-dimensional data where most values are zero (e.g., BM25 or TF-IDF features):

SPARSE_VECTOR_FP32 -- Sparse vector with 32-bit float values
SPARSE_VECTOR_FP16 -- Sparse vector with 16-bit float values

Binary Vectors

Binary vectors use the BINARY type and store bit-packed data, useful for binary hash codes or Hamming distance searches.

Array Types

ARRAY_STRING -- Array of strings (e.g., tags)
ARRAY_INT32 -- Array of 32-bit integers
ARRAY_INT64 -- Array of 64-bit integers
ARRAY_FLOAT -- Array of 32-bit floats
ARRAY_DOUBLE -- Array of 64-bit doubles
ARRAY_BOOL -- Array of booleans

Quantization Types

Quantization reduces memory usage and speeds up search at the cost of some accuracy. Specify a quantization type when creating an index:

Ext::HnswIndexParams.new(metric, quantize_type: Ext::QuantizeType::INT8)

Available quantization types (via Ext::QuantizeType):

FP16 -- Half-precision (16-bit) quantization. Good balance of speed and accuracy. Halves memory vs. FP32.
INT8 -- 8-bit integer quantization. ~4x memory reduction vs. FP32. Slight accuracy loss.
INT4 -- 4-bit integer quantization. ~8x memory reduction vs. FP32. Larger accuracy loss, best for large-scale approximate search.

Metric Types

L2 -- Euclidean (L2) distance. Lower is more similar.
IP -- Inner product. Higher is more similar.
COSINE -- Cosine similarity. Higher is more similar. Vectors are normalized internally.

Constant Summary collapse

BINARY = Returns Raw binary data type. Returns: (Symbol) — Raw binary data type

Ext::DataType::BINARY

STRING = Returns UTF-8 string data type. Returns: (Symbol) — UTF-8 string data type

Ext::DataType::STRING

BOOL = Returns Boolean data type. Returns: (Symbol) — Boolean data type

Ext::DataType::BOOL

INT32 = Returns 32-bit signed integer data type. Returns: (Symbol) — 32-bit signed integer data type

Ext::DataType::INT32

INT64 = Returns 64-bit signed integer data type. Returns: (Symbol) — 64-bit signed integer data type

Ext::DataType::INT64

UINT32 = Returns 32-bit unsigned integer data type. Returns: (Symbol) — 32-bit unsigned integer data type

Ext::DataType::UINT32

UINT64 = Returns 64-bit unsigned integer data type. Returns: (Symbol) — 64-bit unsigned integer data type

Ext::DataType::UINT64

FLOAT = Returns 32-bit float data type. Returns: (Symbol) — 32-bit float data type

Ext::DataType::FLOAT

DOUBLE = Returns 64-bit double data type. Returns: (Symbol) — 64-bit double data type

Ext::DataType::DOUBLE

VECTOR_FP32 = Returns 32-bit float dense vector. Returns: (Symbol) — 32-bit float dense vector

Ext::DataType::VECTOR_FP32

VECTOR_FP64 = Returns 64-bit double dense vector. Returns: (Symbol) — 64-bit double dense vector

Ext::DataType::VECTOR_FP64

VECTOR_FP16 = Returns 16-bit half-precision dense vector. Returns: (Symbol) — 16-bit half-precision dense vector

Ext::DataType::VECTOR_FP16

VECTOR_INT8 = Returns 8-bit integer dense vector (quantized). Returns: (Symbol) — 8-bit integer dense vector (quantized)

Ext::DataType::VECTOR_INT8

SPARSE_VECTOR_FP32 = Returns 32-bit float sparse vector. Returns: (Symbol) — 32-bit float sparse vector

Ext::DataType::SPARSE_VECTOR_FP32

SPARSE_VECTOR_FP16 = Returns 16-bit float sparse vector. Returns: (Symbol) — 16-bit float sparse vector

Ext::DataType::SPARSE_VECTOR_FP16

ARRAY_STRING = Returns Array of strings. Returns: (Symbol) — Array of strings

Ext::DataType::ARRAY_STRING

ARRAY_INT32 = Returns Array of 32-bit integers. Returns: (Symbol) — Array of 32-bit integers

Ext::DataType::ARRAY_INT32

ARRAY_INT64 = Returns Array of 64-bit integers. Returns: (Symbol) — Array of 64-bit integers

Ext::DataType::ARRAY_INT64

ARRAY_FLOAT = Returns Array of 32-bit floats. Returns: (Symbol) — Array of 32-bit floats

Ext::DataType::ARRAY_FLOAT

ARRAY_DOUBLE = Returns Array of 64-bit doubles. Returns: (Symbol) — Array of 64-bit doubles

Ext::DataType::ARRAY_DOUBLE

ARRAY_BOOL = Returns Array of booleans. Returns: (Symbol) — Array of booleans

Ext::DataType::ARRAY_BOOL

L2 = Returns Euclidean (L2) distance metric. Returns: (Symbol) — Euclidean (L2) distance metric

Ext::MetricType::L2

IP = Returns Inner product metric. Returns: (Symbol) — Inner product metric

Ext::MetricType::IP

COSINE = Returns Cosine similarity metric. Returns: (Symbol) — Cosine similarity metric

Ext::MetricType::COSINE

VECTOR_TYPES = Vector data types for dimension validation Returns: (Array<Symbol>) — All dense vector data type constants

[
  Ext::DataType::VECTOR_FP32,
  Ext::DataType::VECTOR_FP64,
  Ext::DataType::VECTOR_FP16,
  Ext::DataType::VECTOR_INT8,
].freeze

SETTER_FOR = Setter dispatch table: DataType -> Doc setter method name Returns: (Hash{Symbol => Symbol})

{
  Ext::DataType::STRING => :set_string,
  Ext::DataType::BOOL   => :set_bool,
  Ext::DataType::INT32  => :set_int32,
  Ext::DataType::INT64  => :set_int64,
  Ext::DataType::UINT32 => :set_uint32,
  Ext::DataType::UINT64 => :set_uint64,
  Ext::DataType::FLOAT  => :set_float,
  Ext::DataType::DOUBLE => :set_double,
  Ext::DataType::VECTOR_FP32 => :set_float_vector,
  Ext::DataType::VECTOR_FP64 => :set_double_vector,
  Ext::DataType::ARRAY_STRING => :set_string_array,
}.freeze

GETTER_FOR = Getter dispatch table: DataType -> Doc getter method name Returns: (Hash{Symbol => Symbol})

{
  Ext::DataType::STRING => :get_string,
  Ext::DataType::BOOL   => :get_bool,
  Ext::DataType::INT32  => :get_int32,
  Ext::DataType::INT64  => :get_int64,
  Ext::DataType::UINT32 => :get_int32,
  Ext::DataType::UINT64 => :get_int64,
  Ext::DataType::FLOAT  => :get_float,
  Ext::DataType::DOUBLE => :get_double,
  Ext::DataType::VECTOR_FP32 => :get_float_vector,
  Ext::DataType::VECTOR_FP64 => :get_double_vector,
  Ext::DataType::ARRAY_STRING => :get_string_array,
}.freeze

Class Method Summary collapse

.coerce_value(value, target_type, field_name: nil) ⇒ Object
Coerce a Ruby value into a form suitable for the given zvec data type.
.detect_type(value) ⇒ Symbol^?
Detect the zvec data type for a Ruby value.

Class Method Details

.coerce_value(value, target_type, field_name: nil) ⇒ `Object`

Coerce a Ruby value into a form suitable for the given zvec data type.

Examples:

DataTypes.coerce_value(42, Ext::DataType::STRING)  #=> "42"
DataTypes.coerce_value("3.14", Ext::DataType::DOUBLE)  #=> 3.14
DataTypes.coerce_value([1, 2], Ext::DataType::VECTOR_FP32)  #=> [1.0, 2.0]

Parameters:

value (Object) —
the value to coerce
target_type (Symbol) —
the target zvec data type constant
field_name (String, nil) (defaults to: nil) —
optional field name for error messages

Returns:

(Object) —
the coerced value

Raises:

(ArgumentError) —
if the value cannot be coerced to the target type

# File 'lib/zvec/data_types.rb', line 206

def self.coerce_value(value, target_type, field_name: nil)
  return value if value.nil?

  ctx = field_name ? " for field '#{field_name}'" : ""

  case target_type
  when Ext::DataType::STRING
    value.to_s
  when Ext::DataType::BOOL
    coerce_bool(value, ctx)
  when Ext::DataType::INT32, Ext::DataType::INT64,
       Ext::DataType::UINT32, Ext::DataType::UINT64
    coerce_integer(value, ctx)
  when Ext::DataType::FLOAT, Ext::DataType::DOUBLE
    coerce_float(value, ctx)
  when Ext::DataType::VECTOR_FP32, Ext::DataType::VECTOR_FP64
    coerce_float_vector(value, ctx)
  when Ext::DataType::ARRAY_STRING
    coerce_string_array(value, ctx)
  else
    value
  end
end

.detect_type(value) ⇒ `Symbol`^?

Detect the zvec data type for a Ruby value.

Handles edge cases: Integer vs Float, String booleans, nil, empty arrays.

Examples:

DataTypes.detect_type("hello")  #=> Ext::DataType::STRING
DataTypes.detect_type(42)       #=> Ext::DataType::INT64
DataTypes.detect_type([1.0])    #=> Ext::DataType::VECTOR_FP32
DataTypes.detect_type(nil)      #=> nil

Parameters:

value (Object) —
the Ruby value to inspect

Returns:

(Symbol, nil) —
the zvec data type constant, or nil for nil input

# File 'lib/zvec/data_types.rb', line 183

def self.detect_type(value)
  case value
  when NilClass              then nil
  when String                then Ext::DataType::STRING
  when Integer               then Ext::DataType::INT64
  when Float                 then Ext::DataType::DOUBLE
  when TrueClass, FalseClass then Ext::DataType::BOOL
  when Array                 then detect_array_type(value)
  end
end

Module: Zvec::DataTypes

Overview

Scalar Types

Dense Vector Types

Sparse Vector Types

Binary Vectors

Array Types

Quantization Types

Metric Types

Constant Summary collapse

Class Method Summary collapse

Class Method Details

.coerce_value(value, target_type, field_name: nil) ⇒ Object

Examples:

.detect_type(value) ⇒ Symbol?

Examples:

.coerce_value(value, target_type, field_name: nil) ⇒ `Object`

.detect_type(value) ⇒ `Symbol`^?