Class: Rimless::AvroUtils

Inherits:
Object
  • Object
show all
Defined in:
lib/rimless/avro_utils.rb

Overview

Due to dynamic constrains on the Apache Avro schemas we need to compile our schema templates to actual ready-to-consume schemas. The namespace part of the schemas and cross-references to other schemas must be rendered according to the dynamic namespace prefix which reflects the application environment. Unfortunately we need to mess around with actual files to support the Avro and AvroTurf gems.

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize ⇒ AvroUtil

Create a new instance of the AvroUtil class.



16
17
18
19
20
 
# File 'lib/rimless/avro_utils.rb', line 16

def initialize
  @namespace = ENV.fetch('KAFKA_SCHEMA_SUBJECT_PREFIX',
                         Rimless.topic_prefix).tr('-', '_').gsub(/\.$/, '')
  @env = @namespace.split('.').first
end

Instance Attribute Details

#env ⇒ Object (readonly)

Returns the value of attribute env.



11
12
13
 
# File 'lib/rimless/avro_utils.rb', line 11

def env
  @env
end

#namespace ⇒ Object (readonly)

Returns the value of attribute namespace.



11
12
13
 
# File 'lib/rimless/avro_utils.rb', line 11

def namespace
  @namespace
end

Instance Method Details

#base_path ⇒ Pathname

Return the base path of the Avro schemas on our project.

Returns:

  • (Pathname) —

    the Avro schemas base path



89
90
91
 
# File 'lib/rimless/avro_utils.rb', line 89

def base_path
  Rimless.configuration.avro_schema_path
end

#clear ⇒ Object

Clear previous compiled Avro schema files to provide a clean rebuild.



63
64
65
66
67
68
69
70
71
 
# File 'lib/rimless/avro_utils.rb', line 63

def clear
  # In a test environment, especially with parallel test execution the
  # recompiling of Avro schemas is error prone due to the deletion of the
  # configuration (output) directory. This leads to random failures due to
  # file read calls to temporary not existing files. So we just keep the
  # files and just overwrite them in place while testing.
  FileUtils.rm_rf(output_path) unless Rimless.env.test?
  FileUtils.mkdir_p(output_path)
end

#output_path ⇒ Pathname

Return the path to the compiled Avro schemas. This path must be consumed by the AvroTurf::Messaging constructor.

Returns:

  • (Pathname) —

    the compiled Avro schemas path



97
98
99
 
# File 'lib/rimless/avro_utils.rb', line 97

def output_path
  Rimless.configuration.compiled_avro_schema_path
end

#recompile_schemas ⇒ Object

Clean and recompile all templated Avro schema files to their respective output path.



24
25
26
27
 
# File 'lib/rimless/avro_utils.rb', line 24

def recompile_schemas
  clear
  Dir[base_path.join('**', '*.erb')].each { |src| render_file(src) }
end

#render_file(src) ⇒ Object

Render (compile) a single Avro schema template. The given source file path will serve to calculate the destination path. So even deep path’ed templates will keep their hierarchy.

Parameters:

  • src (String) —

    the Avro schema template file path



34
35
36
37
38
39
40
41
42
43
 
# File 'lib/rimless/avro_utils.rb', line 34

def render_file(src)
  # Convert the template path to the destination path
  dest = schema_path(src)
  # Create the deep path when not yet existing
  FileUtils.mkdir_p(File.dirname(dest))
  # Write the rendered file contents to the destination
  File.write(dest, ERB.new(File.read(src)).result(binding))
  # Check the written file for correct JSON
  validate_file(dest)
end

#schema_path(src) ⇒ Pathname

Return the compiled Avro schema file path for the given Avro schema template.

Parameters:

  • src (String) —

    the Avro schema template file path

Returns:

  • (Pathname) —

    the resulting schema file path



78
79
80
81
82
83
84
 
# File 'lib/rimless/avro_utils.rb', line 78

def schema_path(src)
  # No trailing dot on the prefix namespace directory
  prefix = env.remove(/\.$/)
  # Calculate the destination path based on the source file
  Pathname.new(src.gsub(/^#{base_path}/, output_path.join(prefix).to_s)
                  .gsub(/\.erb$/, ''))
end

#validate_file(dest) ⇒ Object

Check the given file for valid JSON.

rubocop:disable Security/JSONLoad because we wrote the file contents

Parameters:

  • dest (Pathname, File, IO) —

    the file to check

Raises:

  • (JSON::ParserError) —

    when invalid



51
52
53
54
55
56
57
58
59
 
# File 'lib/rimless/avro_utils.rb', line 51

def validate_file(dest)
  JSON.load(dest)
rescue JSON::ParserError => e
  path = File.expand_path(dest.is_a?(File) ? dest.path : dest.to_s)
  prefix = "Invalid JSON detected: #{path}"
  Rimless.logger.fatal("#{prefix}\n#{e.message}")
  e.message.prepend("#{prefix} - ")
  raise e
end