Class: Binpacker::ShardCheck

Inherits:
Object
  • Object
show all
Defined in:
lib/binpacker/shard_check.rb,
sig/binpacker/shard_check.rbs

Overview

Fan-in audit for a sharded matrix: did the shards, between them, actually run the whole suite?

Shards never coordinate. Each cuts the same N-way partition from the timing data it loaded and takes its own bin, which is correct exactly as long as they all loaded the SAME data. In CI that means every shard job restoring the same timing cache — and the failure mode when one does not is the bad one: it partitions differently, some tests land in no shard, and every job still reports success. Nothing inside a single shard can notice, because a shard cannot tell "not mine" from "does not exist".

So the check belongs after the matrix, over the run reports it produced. Point it at every shard's report and it fails unless the reports describe one coherent split of one suite.

Defined Under Namespace

Classes: Result

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(paths) ⇒ ShardCheck 



23
24
25
 
# File 'lib/binpacker/shard_check.rb', line 23

def initialize(paths)
  @paths = Array(paths)
end

Class Method Details

.call(paths) ⇒ Binpacker::ShardCheck::Result 



21
 
# File 'lib/binpacker/shard_check.rb', line 21

def self.call(paths) = new(paths).call

Instance Method Details

#agreement_problems(shards, field, label) ⇒ Array[String]

Every report must describe the same matrix. Disagreement here means the reports were not produced by one run, and no coverage conclusion drawn from them would mean anything.



80
81
82
83
84
85
 
# File 'lib/binpacker/shard_check.rb', line 80

def agreement_problems(shards, field, label)
  values = shards.map { |s| s[field] }.uniq
  return [] if values.size <= 1

  ["shards disagree on #{label}: #{values.sort_by(&:to_s).inspect}"]
end

#call ⇒ Binpacker::ShardCheck::Result 



27
28
29
30
31
32
33
34
35
36
37
38
39
 
# File 'lib/binpacker/shard_check.rb', line 27

def call
  return failure(['no run reports given']) if @paths.empty?

  reports, unreadable = load_reports
  return failure(unreadable) unless unreadable.empty?

  shards = reports.filter_map { |path, data| shard_of(path, data) }
  missing = reports.map(&:first) - shards.map { |s| s[:path] }
  return failure(missing.map { |p| "#{p}: no `shard` section — was it run with --shard?" }) unless missing.empty?

  problems = check(shards)
  problems.empty? ? success(shards) : failure(problems)
end

#check(shards) ⇒ Array[String] 



69
70
71
72
73
74
75
76
 
# File 'lib/binpacker/shard_check.rb', line 69

def check(shards)
  problems = []
  problems.concat(agreement_problems(shards, :total, 'shard count'))
  problems.concat(agreement_problems(shards, :discovered, 'discovered test count'))
  problems.concat(completeness_problems(shards))
  problems.concat(coverage_problems(shards))
  problems
end

#completeness_problems(shards) ⇒ Array[String]

A matrix that lost a job silently drops that job's slice, which looks exactly like a smaller suite.



88
89
90
91
92
93
94
95
96
97
98
99
100
 
# File 'lib/binpacker/shard_check.rb', line 88

def completeness_problems(shards)
  total = shards.first[:total]
  return [] if total.nil?

  seen = shards.map { |s| s[:index] }
  duplicates = seen.tally.select { |_, n| n > 1 }.keys.sort
  problems = []
  problems << "shard #{duplicates.join(', ')} reported more than once" unless duplicates.empty?

  absent = (1..total).to_a - seen
  problems << "no report for shard #{absent.join(', ')} of #{total}" unless absent.empty?
  problems
end

#coverage_problems(shards) ⇒ Array[String]

The check this class exists for: the slices must add up to the suite.



103
104
105
106
107
108
109
110
111
112
113
 
# File 'lib/binpacker/shard_check.rb', line 103

def coverage_problems(shards)
  discovered = shards.first[:discovered]
  return [] if discovered.nil?

  selected = shards.sum { |s| s[:selected].to_i }
  return [] if selected == discovered

  verb = selected < discovered ? 'ran no shard' : 'ran in more than one shard'
  ["shards cover #{selected} of #{discovered} tests — #{(discovered - selected).abs} #{verb}. " \
   'The shards partitioned different timing data; make every shard load the same timing file.']
end

#failure(problems) ⇒ Binpacker::ShardCheck::Result 



123
124
125
 
# File 'lib/binpacker/shard_check.rb', line 123

def failure(problems)
  Result.new(ok: false, problems: problems, summary: 'shard coverage check failed')
end

#load_reports ⇒ [Array[[String, untyped]], Array[String]] 



43
44
45
46
47
48
49
50
51
52
53
54
 
# File 'lib/binpacker/shard_check.rb', line 43

def load_reports
  reports = []
  unreadable = []
  @paths.each do |path|
    reports << [path, JSON.parse(File.read(path))]
  rescue Errno::ENOENT
    unreadable << "#{path}: no such file"
  rescue JSON::ParserError => e
    unreadable << "#{path}: not valid JSON (#{e.message})"
  end
  [reports, unreadable]
end

#shard_of(path, data) ⇒ Hash[Symbol, untyped]? 



56
57
58
59
60
61
62
63
64
65
66
67
 
# File 'lib/binpacker/shard_check.rb', line 56

def shard_of(path, data)
  section = data['shard']
  return nil unless section.is_a?(Hash)

  {
    path: path,
    index: section['index'],
    total: section['total'],
    discovered: section['discovered_tests'],
    selected: section['selected_tests']
  }
end

#success(shards) ⇒ Binpacker::ShardCheck::Result 



115
116
117
118
119
120
121
 
# File 'lib/binpacker/shard_check.rb', line 115

def success(shards)
  Result.new(
    ok: true,
    problems: [],
    summary: "#{shards.size} shards cover all #{shards.first[:discovered]} tests"
  )
end