Module: Bxtjson

Defined in:
lib/bxtjson.rb

Overview

dependencies

  • gem: json_schema

  • gem: multi_json

  • JSON standard library or other (e.g. oj) json parsers

  • You implement a model in Sequel or ActiveRecord if you want.

Constants (examples)

cleaner_proc = ->(str) {str.gsub(/\W+/, " ").lstrip
    .gsub(" ", "_")
    .gsub(/PPPO_|PPCO_/, "")
    .downcase
}
json_filename = "../../../fsms-tmp/PP_PROPOSAL_2015.JSON"
schema_filename = "./docs/schema.json"
authorizinng_pointer '#/departments/primary_dept'

Class Method Summary collapse

Class Method Details

.compact_hash!(hash) ⇒ Hash

Recursively remove falsey values from hash Falsey values are those that return true from respond_to(:empty?) or :nil?

Parameters:

  • hash (Hash)

Returns:

  • (Hash) 


108
109
110
111
112
113
114
 
# File 'lib/bxtjson.rb', line 108

def self.compact_hash!(hash)
  p = proc do |_, v|
    v.delete_if(&p) if v.respond_to? :delete_if
    v.respond_to?(:empty?) && v.empty? || v.nil?
  end
  hash.delete_if(&p)
end

.compact_values!(hash) ⇒ Object 



115
116
117
118
119
120
121
122
123
124
125
126
127
128
 
# File 'lib/bxtjson.rb', line 115

def self.compact_values!(hash)
  Hash[hash.map do |key, value|
         [key,
          if value.is_a?(Array)
            value.map {|item| Bxtjson.compact_hash!(item) }
          elsif value.respond_to?( :delete_if)
            Bxtjson.compact_hash!(value)
          else
            value
          end
         ]
       end
      ]
end

.muscle(json_filename:, schema_filename:, clean_proc: ->(str){str}, model: nil, schema_entity: nil, authorizing_pointer:, data_attr: :data) ⇒ Object

Parse json-schema file and map contents of json file into initialized schema

Mapping of contents will search for the first key in the source, that match the schema (recursively). So, the source should be flat for clarity, and the schema can be nested.

If it cannot find a key, it will look for “top/next/final” path key in the source data.

For example, in the skeleton

{key: {nest: "this"} }

Will be filled with “muscle” if the source has a ket

{"key/nest": "data muscle"}

TODO: design interface from csv to json that fits these principles.

Parameters:

  • json_filename (String) —

    filename for jsonl source

  • schema_filename (String) —

    filename for json-schema

  • clean_proc (Proc) (defaults to: ->(str){str}) —

    a function to clean up keys

  • model (String, #create) (defaults to: nil) —

    the name of a model to call :create on

  • authorizing_pointer (String) —

    json-pointer that fills in the key “authorized_by”



75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
 
# File 'lib/bxtjson.rb', line 75

def self.muscle(json_filename:,
                schema_filename:,
                clean_proc: ->(str){str},
                model: nil,
                schema_entity: nil,
                authorizing_pointer:,
                data_attr: :data)     
  skeleton = Bxtjson.skeleton(schema_data: MultiJson.load(File.read(schema_filename)),
                              entity: schema_entity)
  if model
    model = constantize(model.to_s.capitalize)

    text_to_lazy_json(json_filename: json_filename, clean_proc: clean_proc )
      .map {|data| 
      data = fillin(source_hash: _map_onto_skeleton_of_schema( data, 
                                                               skeleton: skeleton ),
                    skeleton: skeleton) 
      result = model.create( data_attr => data)
    }
  else
    out = []
    text_to_lazy_json(json_filename: json_filename, clean_proc: clean_proc )
      .map {|data| out << fillin(source_hash: _map_onto_skeleton_of_schema( data, 
                                                                            skeleton: skeleton ),
                                 skeleton: skeleton)
    }
  end
end

.skeleton(schema_data:, entity: nil) ⇒ Hash

Initialize an empty hashmap given a json-schema

Parameters:

  • schema_data (Hash) —

    the data from json-schema

  • entity (String, nil) (defaults to: nil) —

    the resource name to enfocus

Returns:

  • (Hash) —

    the an empty “initialized” json-schema



28
29
30
31
32
33
34
35
36
37
38
 
# File 'lib/bxtjson.rb', line 28

def self.skeleton(schema_data:,
                  entity: nil)
  schema = JsonSchema.parse!(schema_data)
  schema.expand_references!
  if entity.nil?
    entity_schema = schema
  else 
    entity_schema = schema.properties[entity]
  end
  return _skeleton(entity_schema, acc: {})
end

.text_to_lazy_json(json_filename:, clean_proc:) ⇒ Hash

process a file of jsonl (linefeed) and clean keys with proc

Parameters:

  • json_filename (String) —

    source of json objects in jsonl format

  • a (Proc) —

    function to clean up keys

Returns:

  • (Hash) —

    hash of arrays ‘[{ }]`



44
45
46
47
48
49
50
51
 
# File 'lib/bxtjson.rb', line 44

def self.text_to_lazy_json(json_filename:,
                           clean_proc:)
  File.foreach(json_filename)
    .lazy
    .map do |line|
    _key_cleaner(data: MultiJson.load(line), clean_proc: clean_proc)
  end
end