Class: MesaTestSubmitter
- Inherits:
-
Object
- Object
- MesaTestSubmitter
- Defined in:
- lib/mesa_test.rb
Constant Summary collapse
- DEFAULT_URI =
'https://mesa-test-hub.herokuapp.com'.freeze
Instance Attribute Summary collapse
-
#base_uri ⇒ Object
Returns the value of attribute base_uri.
-
#computer_name ⇒ Object
Returns the value of attribute computer_name.
-
#config_file ⇒ Object
Returns the value of attribute config_file.
-
#email ⇒ Object
Returns the value of attribute email.
-
#github_protocol ⇒ Object
Returns the value of attribute github_protocol.
-
#logs_token ⇒ Object
Returns the value of attribute logs_token.
-
#mesa_mirror ⇒ Object
Returns the value of attribute mesa_mirror.
-
#mesa_work ⇒ Object
Returns the value of attribute mesa_work.
-
#password ⇒ Object
Returns the value of attribute password.
-
#platform ⇒ Object
Returns the value of attribute platform.
-
#platform_version ⇒ Object
Returns the value of attribute platform_version.
-
#processor ⇒ Object
Returns the value of attribute processor.
-
#shell ⇒ Object
readonly
Returns the value of attribute shell.
-
#user_name ⇒ Object
Returns the value of attribute user_name.
Class Method Summary collapse
Instance Method Summary collapse
-
#build_log_params(mesa) ⇒ Object
Parameters for reporting a failed compilation to the logs server.
-
#commit_params(mesa, entire: true, empty: false) ⇒ Object
Parameters to be submitted in JSON format for reporting information about the overall commit being tested; used even if only submitting an entire test.
-
#confirm_computer ⇒ Object
Phone home to testhub and confirm that computer and user are valid.
- #confirm_computer_data ⇒ Object
-
#initialize(computer_name: nil, user_name: nil, email: nil, github_protocol: nil, mesa_mirror: nil, platform: nil, platform_version: nil, processor: nil, config_file: nil, base_uri: nil, logs_token: nil) {|_self| ... } ⇒ MesaTestSubmitter
constructor
many defaults are set in body.
-
#instance_params(mesa) ⇒ Object
Given a valid
Mesa
object, create an array of hashes that describe the test cases and the test results. - #load_computer_data ⇒ Object
-
#save_computer_data ⇒ Object
For one “computer” on the web server, and for [subjective] consistency reasons, the platform, processor, and RAM cannot be changed! If you change platforms (i.e. switch from mac to linux, or change between linux flavors), you should create a new computer account.
-
#setup ⇒ Object
set up config file for computer.
-
#single_instance_params(test_case) ⇒ Object
Parameters for a single test case.
-
#submit_build_log(mesa) ⇒ Object
send build log to the logs server.
-
#submit_commit(mesa, empty: false, force_logs: false) ⇒ Object
submit entire commit’s worth of test cases, OR submit compilation status and NO test cases.
-
#submit_instance(mesa, test_case, force_logs: false) ⇒ Object
submit results for a single test case instance.
-
#submit_logs(params) ⇒ Object
make generic request to LOGS server
params
is a hash of data to be encoded as JSON and sent off. -
#submit_test_log(test_case, skip_passing: true) ⇒ Object
send build log to the logs server.
-
#submitter_params ⇒ Object
Parameters to be submitted in JSON format for reporting information about the submitting user and computer.
-
#test_log_params(test_case) ⇒ Object
Parameters for reporting a failed test to the logs server.
-
#update {|_self| ... } ⇒ Object
ease setup of a blank/default submitter.
Constructor Details
#initialize(computer_name: nil, user_name: nil, email: nil, github_protocol: nil, mesa_mirror: nil, platform: nil, platform_version: nil, processor: nil, config_file: nil, base_uri: nil, logs_token: nil) {|_self| ... } ⇒ MesaTestSubmitter
many defaults are set in body
125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 |
# File 'lib/mesa_test.rb', line 125 def initialize( computer_name: nil, user_name: nil, email: nil, github_protocol: nil, mesa_mirror: nil, platform: nil, platform_version: nil, processor: nil, config_file: nil, base_uri: nil, logs_token: nil ) @computer_name = computer_name || Socket.gethostname.scan(/^[^\.]+\.?/)[0] @computer_name.chomp!('.') if @computer_name @user_name = user_name || (ENV['USER'] || ENV['USERNAME']) @email = email || '' @password = password || '' @github_protocol = github_protocol || :ssh @mesa_mirror = mesa_mirror || File.join(ENV['HOME'], '.mesa_test', 'mirror') @mesa_work = mesa_work || File.join(ENV['HOME'], '.mesa_test', 'work') @platform = platform if @platform.nil? @platform = if OS.osx? 'macOS' elsif OS.linux? 'Linux' else '' end end @platform_version = platform_version || '' @processor = processor || '' @config_file = config_file || File.join(ENV['HOME'], '.mesa_test', 'config.yml') @base_uri = base_uri @logs_token = logs_token || ENV['MESA_LOGS_TOKEN'] # set up thor-proof way to get responses from user. Thor hijacks the # gets command, so we have to use its built-in "ask" method, which is # actually more useful @shell = Thor::Shell::Color.new yield self if block_given? end |
Instance Attribute Details
#base_uri ⇒ Object
Returns the value of attribute base_uri.
118 119 120 |
# File 'lib/mesa_test.rb', line 118 def base_uri @base_uri end |
#computer_name ⇒ Object
Returns the value of attribute computer_name.
118 119 120 |
# File 'lib/mesa_test.rb', line 118 def computer_name @computer_name end |
#config_file ⇒ Object
Returns the value of attribute config_file.
118 119 120 |
# File 'lib/mesa_test.rb', line 118 def config_file @config_file end |
#email ⇒ Object
Returns the value of attribute email.
118 119 120 |
# File 'lib/mesa_test.rb', line 118 def email @email end |
#github_protocol ⇒ Object
Returns the value of attribute github_protocol.
118 119 120 |
# File 'lib/mesa_test.rb', line 118 def github_protocol @github_protocol end |
#logs_token ⇒ Object
Returns the value of attribute logs_token.
118 119 120 |
# File 'lib/mesa_test.rb', line 118 def logs_token @logs_token end |
#mesa_mirror ⇒ Object
Returns the value of attribute mesa_mirror.
118 119 120 |
# File 'lib/mesa_test.rb', line 118 def mesa_mirror @mesa_mirror end |
#mesa_work ⇒ Object
Returns the value of attribute mesa_work.
118 119 120 |
# File 'lib/mesa_test.rb', line 118 def mesa_work @mesa_work end |
#password ⇒ Object
Returns the value of attribute password.
118 119 120 |
# File 'lib/mesa_test.rb', line 118 def password @password end |
#platform ⇒ Object
Returns the value of attribute platform.
118 119 120 |
# File 'lib/mesa_test.rb', line 118 def platform @platform end |
#platform_version ⇒ Object
Returns the value of attribute platform_version.
118 119 120 |
# File 'lib/mesa_test.rb', line 118 def platform_version @platform_version end |
#processor ⇒ Object
Returns the value of attribute processor.
118 119 120 |
# File 'lib/mesa_test.rb', line 118 def processor @processor end |
#shell ⇒ Object (readonly)
Returns the value of attribute shell.
122 123 124 |
# File 'lib/mesa_test.rb', line 122 def shell @shell end |
#user_name ⇒ Object
Returns the value of attribute user_name.
118 119 120 |
# File 'lib/mesa_test.rb', line 118 def user_name @user_name end |
Class Method Details
.new_from_config(config_file: File.join(ENV['HOME'], '.mesa_test', 'config.yml'), force_setup: false, base_uri: DEFAULT_URI) ⇒ Object
102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 |
# File 'lib/mesa_test.rb', line 102 def self.new_from_config( config_file: File.join(ENV['HOME'], '.mesa_test', 'config.yml'), force_setup: false, base_uri: DEFAULT_URI ) new_submitter = new(config_file: config_file, base_uri: base_uri) if force_setup new_submitter.setup elsif not File.exist? config_file puts "No such config file #{config_file}. Starting setup wizard." new_submitter.setup end new_submitter.load_computer_data return new_submitter end |
Instance Method Details
#build_log_params(mesa) ⇒ Object
Parameters for reporting a failed compilation to the logs server
288 289 290 291 292 293 294 |
# File 'lib/mesa_test.rb', line 288 def build_log_params(mesa) { 'computer_name' => computer_name, 'commit' => mesa.sha, 'build.log' => mesa.build_log_64 } end |
#commit_params(mesa, entire: true, empty: false) ⇒ Object
Parameters to be submitted in JSON format for reporting information about the overall commit being tested; used even if only submitting an entire test. This also determines if the submission is for an entire commit (compilation information and every test), an empty commit (just compilation information), or a non-empty, but also non-entire submission (results for a single test without compilation information)
244 245 246 247 248 249 250 251 252 253 254 255 256 257 |
# File 'lib/mesa_test.rb', line 244 def commit_params(mesa, entire: true, empty: false) # the compiler data should be able to be used as-is, but right now the # names don't match with what the database expects, so we do some renaming # shenanigans. # #################################### # THIS SHOULD GO BEFORE PRODUCTION # { sha: mesa.sha, compiled: mesa.installed?, entire: entire, empty: empty, }.merge(mesa.compiler_hash) end |
#confirm_computer ⇒ Object
Phone home to testhub and confirm that computer and user are valid. Useful for confirming that submissions will be accepted before wasting time on a test later.
319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 |
# File 'lib/mesa_test.rb', line 319 def confirm_computer uri = URI.parse(base_uri + '/check_computer.json') https = Net::HTTP.new(uri.hostname, uri.port) https.use_ssl = base_uri.include? 'https' request = Net::HTTP::Post.new( uri, initheader = { 'Accept' => 'application/json', 'Content-Type' => 'application/json' } ) request.body = { email: email, password: password, computer_name: computer_name }.to_json JSON.parse(https.request(request).body).to_hash end |
#confirm_computer_data ⇒ Object
171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 |
# File 'lib/mesa_test.rb', line 171 def confirm_computer_data puts 'Ready to submit the following data:' puts '-------------------------------------------------------' puts "Computer Name #{computer_name}" puts "User email #{email}" puts 'Password ***********' puts "logs API token #{logs_token}" puts "GitHub Protocol #{github_protocol}" puts "MESA Mirror Location #{mesa_mirror}" puts "MESA Work Location #{mesa_work}" puts "Platform #{platform} #{platform_version}" # puts "Config location #{config_file}" puts '-------------------------------------------------------' puts '' response = shell.ask 'Is this correct? (y/Y = Yes, anything else = No):' response.strip.casecmp('y').zero? end |
#instance_params(mesa) ⇒ Object
Given a valid Mesa
object, create an array of hashes that describe the test cases and the test results. These will be encoded as an array of JSON objects.
262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 |
# File 'lib/mesa_test.rb', line 262 def instance_params(mesa) has_errors = [] res = [] mesa.test_case_names.each do |mod, names| names.each do |test_name| begin test_case = mesa.test_cases[mod][test_name] res << test_case.results_hash rescue TestCaseDirError # shell.say "It appears that #{test_case.test_name} has not been "\ # 'run yet. Unable to submit data for this test.', :red has_errors << test_case end end end unless has_errors.empty? shell.say "The following test cases could NOT be read for submission:", :red has_errors.each do |test_case| shell.say "- #{test_case.test_name}", :red end end res end |
#load_computer_data ⇒ Object
214 215 216 217 218 219 220 221 222 223 224 225 |
# File 'lib/mesa_test.rb', line 214 def load_computer_data data_hash = YAML.load(File.read(config_file)) @computer_name = data_hash['computer_name'] @email = data_hash['email'] @password = data_hash['password'] @logs_token = data_hash['logs_token'] @github_protocol = data_hash['github_protocol'].to_sym @mesa_mirror = data_hash['mesa_mirror'] @mesa_work = data_hash['mesa_work'] @platform = data_hash['platform'] @platform_version = data_hash['platform_version'] end |
#save_computer_data ⇒ Object
For one “computer” on the web server, and for [subjective] consistency reasons, the platform, processor, and RAM cannot be changed! If you change platforms (i.e. switch from mac to linux, or change between linux flavors), you should create a new computer account. Similarly, create new computer accounts if you change your RAM or processor. You do not need to change computers if you upgrade your platform (macOS 10.12 -> 10.13
195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 |
# File 'lib/mesa_test.rb', line 195 def save_computer_data data_hash = { 'computer_name' => computer_name, 'email' => email, 'password' => password, 'logs_token' => logs_token, 'github_protocol' => github_protocol, 'mesa_mirror' => mesa_mirror, 'mesa_work' => mesa_work, 'platform' => platform, 'platform_version' => platform_version, } # make sure there's a directory to write to unless dir_or_symlink_exists? File.dirname(config_file) FileUtils.mkdir_p File.dirname(config_file) end File.open(config_file, 'w') { |f| f.write(YAML.dump(data_hash)) } end |
#setup ⇒ Object
set up config file for computer
24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 |
# File 'lib/mesa_test.rb', line 24 def setup update do |s| shell.say 'This wizard will guide you through setting up a computer profile and default data for test case submissions to MESATestHub. You will be able to confirm entries at the end. Default/current values are always shown in parentheses at the end of a prompt. Pressing enter will accept the default values. To submit to MESATestHub, a valid computer name, email address, and password are all required. To actually run a test, you need to specify a location for your base MESA git repository. All other data are useful, but optional. Any data transferred to MESATestHub will be encrypted via HTTPS, but be warned that your e-mail and password will be stored in plain text.' # Get computer name response = shell.ask('What is the name of this computer (required)? ' \ "(#{s.computer_name}):", :blue) s.computer_name = response unless response.empty? # Get user e-mail response = shell.ask 'What is the email you can be reached ' \ "at (required)? (#{s.email}):", :blue s.email = response unless response.empty? # Get user password response = shell.ask 'What is the password associated with the email ' \ "#{s.email} (required)? (#{s.password})", :blue s.password = response unless response.empty? # Get API key for submitting failure logs response = shell.ask 'What is the logs submission API token associated '\ "with the email #{s.email} (required; contact Josiah Schwab if you "\ "need a key)? (#{s.logs_token})", :blue s.logs_token = response unless response.empty? # Determine if we'll use ssh or https to access github response = shell.ask 'When accessing GitHub, which protocol do you '\ 'want to use?', :blue, limited_to: %w[ssh https] s.github_protocol = response.strip.downcase.to_sym # Get location of source MESA repo (the mirror) response = shell.ask "Where is/should your mirrored MESA repository " \ "located? This is where a mirror will be stored from which test " \ "repos will be generated. You won't touch this in regular operation. " \ "(#{s.mesa_mirror}):", :blue s.mesa_mirror = response unless response.empty? # Get location of source MESA work (where testing happens) response = shell.ask "Where is/should your working directory for "\ "testing be located? This is where testing actually occurs, but all "\ "files it uses are cached in the mirror repo to save time later. " \ "(#{s.mesa_work}):", :blue s.mesa_work = response unless response.empty? # Get platform information response = shell.ask 'What is the platform of this computer (eg. ' \ "macOS, Ubuntu)? (#{s.platform}):", :blue s.platform = response unless response.empty? response = shell.ask 'What is the version of the platform (eg. 10.15.5, ' \ "Ubuntu 16.04)? (#{s.platform_version}):", :blue s.platform_version = response unless response.empty? # we are powerless to do change the location for now, so stop asking # about it # # Confirm save location # response = shell.ask "This will be saved in #{s.config_file}. Press " \ # 'enter to accept or enter a new location:', :blue, path: true # s.config_file = response unless response.empty? end # Confirm data. If not confirmed, restart whole wizard. if confirm_computer_data save_computer_data else shell.say "Restarting wizard.\n" setup end end |
#single_instance_params(test_case) ⇒ Object
Parameters for a single test case. mesa
is an instance of Mesa
, and test_case
is an instance of MesaTestCase representing the test case to be submitted
312 313 314 |
# File 'lib/mesa_test.rb', line 312 def single_instance_params(test_case) [test_case.results_hash] end |
#submit_build_log(mesa) ⇒ Object
send build log to the logs server
459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 |
# File 'lib/mesa_test.rb', line 459 def submit_build_log(mesa) # intercept and don't send if mesa was properly installed return true if mesa.installed? # don't even try unless we have a logs token set unless logs_token shell.say 'Cannot submit to logs server; need to set mesa_logs_token '\ 'in the mesa_test config file.' return false end # do submission res = submit_logs(build_log_params(mesa)) # report out results if !res.is_a? Net::HTTPOK shell.say "\nFailed to submit build.log to the LOGS server for commit "\ "#{mesa.sha}.", :red false else shell.say "\nSuccessfully submitted build.log to the LOGS server for "\ "#{mesa.sha}.", :green true end end |
#submit_commit(mesa, empty: false, force_logs: false) ⇒ Object
submit entire commit’s worth of test cases, OR submit compilation status and NO test cases
337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 |
# File 'lib/mesa_test.rb', line 337 def submit_commit(mesa, empty: false, force_logs: false) unless mesa.install_attempted? raise MesaDirError, 'No testhub.yml file found in installation; '\ 'must attempt to install before subitting.' end uri = URI.parse(base_uri + '/submissions/create.json') https = Net::HTTP.new(uri.hostname, uri.port) https.use_ssl = true if base_uri.include? 'https' request = Net::HTTP::Post.new( uri, initheader = { 'Accept' => 'application/json', 'Content-Type' => 'application/json' } ) # create the request body for submission to the submissions API # # if we have an empty submission, then it is necessarily not entire. # Similarly, a non-empty submission is necessarily entire (otherwise one # would use +submit_instance+). Also, make a "nonempty" submission be # empty if there was an overall build error empty ||= !mesa.installed? request_data = {submitter: submitter_params, commit: commit_params(mesa, empty: empty, entire: !empty)} # don't need test instances if it's an empty submission or if compilation # failed if !empty && request_data[:commit][:compiled] request_data[:instances] = instance_params(mesa) end request.body = request_data.to_json # actually do the submission response = https.request request if !response.is_a? Net::HTTPCreated shell.say "\nFailed to submit some or all test case instances and/or "\ 'commit data.', :red false else shell.say "\nSuccessfully submitted commit #{mesa.sha}.", :green # commit submitted to testhub, now submit build log if compilation failed # and exit unless mesa.installed? return submit_build_log(mesa) end # compilation succeded, so submit any logs for failing tests res = true unless empty mesa.test_cases.each do |mod, test_case_hash| test_case_hash.each do |tc_name, test_case| # get at each individual test case, see if it failed, and if it # did, submit its log files if !test_case.passed? || force_logs res &&= submit_test_log(test_case, skip_passing: !force_logs) end end end end # a true return value means that any and all log submission were # successful res end end |
#submit_instance(mesa, test_case, force_logs: false) ⇒ Object
submit results for a single test case instance. Does not report overall compilation status to testhub. Use an empty commit submission for that
404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 |
# File 'lib/mesa_test.rb', line 404 def submit_instance(mesa, test_case, force_logs: false) unless mesa.install_attempted? raise MesaDirError, 'No testhub.yml file found in installation; '\ 'must attempt to install before subitting.' end uri = URI.parse(base_uri + '/submissions/create.json') https = Net::HTTP.new(uri.hostname, uri.port) https.use_ssl = true if base_uri.include? 'https' request = Net::HTTP::Post.new( uri, initheader = { 'Accept' => 'application/json', 'Content-Type' => 'application/json' } ) # create the request body for submission to the submissions API # # submission is not empty (there is one test case), and it is also not # entire (... there is only test case) request_data = {submitter: submitter_params, commit: commit_params(mesa, empty: false, entire: false), instances: single_instance_params(test_case)} request.body = request_data.to_json # actually do the submission response = https.request request if !response.is_a? Net::HTTPCreated shell.say "\nFailed to submit #{test_case.test_name} for commit "\ "#{mesa.sha}", :red false else shell.say "\nSuccessfully submitted instance of #{test_case.test_name} "\ "for commit #{mesa.sha}.", :green # submit logs if test failed if !test_case.passed? || force_logs return submit_test_log(test_case, skip_passing: !force_logs) end true end end |
#submit_logs(params) ⇒ Object
make generic request to LOGS server params
is a hash of data to be encoded as JSON and sent off
448 449 450 451 452 453 454 455 456 |
# File 'lib/mesa_test.rb', line 448 def submit_logs(params) uri = URI('https://logs.mesastar.org/uploads') https = Net::HTTP.new(uri.host, uri.port) https.use_ssl = true req = Net::HTTP::Post.new(uri.path, 'Content-Type' => 'application/json', 'X-Api-Key' => logs_token) req.body = params.to_json https.request(req) end |
#submit_test_log(test_case, skip_passing: true) ⇒ Object
send build log to the logs server
486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 |
# File 'lib/mesa_test.rb', line 486 def submit_test_log(test_case, skip_passing: true) # skip submission if mesa was never installed, test was never run, or if # the test passed if !test_case.mesa.installed? || !test_case.ran? || (test_case.passed? && skip_passing) return true end # don't even try unless we have a logs token set unless logs_token shell.say 'Cannot submit to logs server; need to set mesa_logs_token '\ 'in the mesa_test config file..' return false end # do submission res = submit_logs(test_log_params(test_case)) # report out results if !res.is_a? Net::HTTPOK shell.say "Failed to submit logs for test case #{test_case.test_name} "\ "in commit #{test_case.mesa.sha}.", :red false else shell.say "Successfully submitted logs for test case "\ "#{test_case.test_name} in commit #{test_case.mesa.sha}.", :green true end end |
#submitter_params ⇒ Object
Parameters to be submitted in JSON format for reporting information about the submitting user and computer
229 230 231 232 233 234 235 236 |
# File 'lib/mesa_test.rb', line 229 def submitter_params { email: email, password: password, computer: computer_name, platform_version: platform_version } end |
#test_log_params(test_case) ⇒ Object
Parameters for reporting a failed test to the logs server
297 298 299 300 301 302 303 304 305 306 307 |
# File 'lib/mesa_test.rb', line 297 def test_log_params(test_case) res = { 'computer_name' => computer_name, 'commit' => test_case.mesa.sha, 'test_case' => test_case.test_name } res['mk.txt'] = test_case.mk_64 unless test_case.mk_64.empty? res['out.txt'] = test_case.out_64 unless test_case.out_64.empty? res['err.txt'] = test_case.err_64 unless test_case.err_64.empty? res end |
#update {|_self| ... } ⇒ Object
ease setup of a blank/default submitter
167 168 169 |
# File 'lib/mesa_test.rb', line 167 def update yield self if block_given? end |