Class: Subprocess::Process

Inherits:
Object
  • Object
show all
Defined in:
lib/subprocess.rb

Overview

A child process. The preferred way of spawning a subprocess is through the functions on Subprocess (especially check_call and check_output).

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(cmd, opts = {}) {|process| ... } ⇒ Process

Create a new process.

Parameters:

  • cmd (Array<String>) —

    The command to run and its arguments (in the style of an argv array). Unlike Python's subprocess module, cmd cannot be a String.

  • opts (Hash) (defaults to: {}) —

    a customizable set of options

Options Hash (opts):

  • :stdin (IO, Integer, String, Subprocess::PIPE, nil) —

    The IO, file descriptor number, or file name to use for the process's standard input. If the magic value Subprocess::PIPE is passed, a new pipe will be opened.

  • :stdout (IO, Integer, String, Subprocess::PIPE, nil) —

    The IO, file descriptor number, or file name to use for the process's standard output. If the magic value Subprocess::PIPE is passed, a pipe will be opened and attached to the process.

  • :stderr (IO, Integer, String, Subprocess::PIPE, Subprocess::STDOUT, nil) —

    The IO, file descriptor number, or file name to use for the process's standard error. If the special value Subprocess::PIPE is passed, a pipe will be opened and attached to the process. If the special value STDOUT is passed, the process's stderr will be redirected to its stdout (much like bash's 2>&1).

  • :cwd (String) —

    The directory to change to before executing the child process.

  • :env (Hash<String, String>) —

    The environment to use in the child process.

  • :retain_fds (Array<Integer>) —

    An array of file descriptor numbers that should not be closed before executing the child process. Note that, unlike Python (which has :close_fds defaulting to false), all file descriptors not specified here will be closed.

  • :exec_opts (Hash) —

    A hash that will be merged into the options hash of the call to Kernel#exec.

  • :preexec_fn (Proc) —

    A function that will be called in the child process immediately before executing cmd.

Yields:

  • (process) —

    Yields the just-spawned Subprocess::Process to the optional block. This occurs after all of Subprocess::Process's error handling has been completed, and is a great place to call #communicate, especially when used in conjunction with check_call.

Yield Parameters:

  • process (Process) —

    The process that was just spawned.

Raises:

  • (ArgumentError) 


268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
 
# File 'lib/subprocess.rb', line 268

def initialize(cmd, opts={}, &blk)
  raise ArgumentError, "cmd must be an Array of strings" unless Array === cmd
  raise ArgumentError, "cmd cannot be empty" if cmd.empty?

  @command = cmd

  # Figure out what file descriptors we should pass on to the child (and
  # make externally visible ourselves)
  @child_stdin, @stdin = parse_fd(opts[:stdin], 'r')
  @child_stdout, @stdout = parse_fd(opts[:stdout], 'w')
  unless opts[:stderr] == STDOUT
    @child_stderr, @stderr = parse_fd(opts[:stderr], 'w')
  end

  retained_fds = Set.new(opts[:retain_fds] || [])

  # A control pipe for ferrying errors back from the child
  control_r, control_w = IO.pipe

  @pid = fork do
    begin
      ::STDIN.reopen(@child_stdin) if @child_stdin
      ::STDOUT.reopen(@child_stdout) if @child_stdout
      if opts[:stderr] == STDOUT
        ::STDERR.reopen(::STDOUT)
      else
        ::STDERR.reopen(@child_stderr) if @child_stderr
      end

      # Set up a new environment if we're requested to do so.
      if opts[:env]
        ENV.clear
        begin
          ENV.update(opts[:env])
        rescue TypeError => e
          raise ArgumentError, "`env` option must be a hash where all keys and values are strings (#{e})"
        end
      end

      # Call the user back, maybe?
      if opts[:preexec_fn]
        if opts[:cwd]
          Dir.chdir(opts[:cwd], &opts[:preexec_fn])
        else
          opts[:preexec_fn].call
        end
      end

      options = {close_others: true}.merge(opts.fetch(:exec_opts, {}))
      if opts[:retain_fds]
        retained_fds.each { |fd| options[fd] = fd }
      end
      if opts[:cwd]
        # We use the chdir option to `exec` since wrapping the
        # `exec` in a Dir.chdir block caused these sporadic errors on macOS:
        # Too many open files - getcwd (Errno::EMFILE)
        options[:chdir] = opts[:cwd]
      end

      begin
        # Ruby's Kernel#exec will call an exec(3) variant if called with two
        # or more arguments, but when called with just a single argument will
        # spawn a subshell with that argument as the command. Since we always
        # want to call exec(3), we use the third exec form, which passes a
        # [cmdname, argv0] array as its first argument and never invokes a
        # subshell.
        exec([cmd[0], cmd[0]], *cmd[1..-1], options)
      rescue TypeError => e
        raise ArgumentError, "cmd must be an Array of strings (#{e})"
      end

    rescue Exception => e
      # Dump all errors up to the parent through the control socket
      Marshal.dump(e, control_w)
      control_w.flush
    end

    # Something has gone terribly, terribly wrong if we're hitting this :(
    exit!(1)
  end

  # Meanwhile, in the parent process...

  # First, let's close some things we shouldn't have access to
  @child_stdin.close if our_fd?(opts[:stdin])
  @child_stdout.close if our_fd?(opts[:stdout])
  @child_stderr.close if our_fd?(opts[:stderr])
  control_w.close

  # Any errors during the spawn process? We'll get past this point when the
  # child execs and the OS closes control_w
  begin
    e = Marshal.load(control_r)
    e = "Unknown Failure" unless e.is_a?(Exception) || e.is_a?(String)
    # Because we're throwing an exception and not returning a
    # Process, we need to make sure the child gets reaped
    wait
    raise e
  rescue EOFError # Nothing to read? Great!
  ensure
    control_r.close
  end

  # Everything is okay. Good job, team!
  blk.call(self) if blk
end

Instance Attribute Details

#command ⇒ Array<String> (readonly)

Returns The command this process was invoked with.

Returns:

  • (Array<String>) —

    The command this process was invoked with.



219
220
221
 
# File 'lib/subprocess.rb', line 219

def command
  @command
end

#pid ⇒ Integer (readonly)

Returns The process ID of the spawned process.

Returns:

  • (Integer) —

    The process ID of the spawned process.



222
223
224
 
# File 'lib/subprocess.rb', line 222

def pid
  @pid
end

#status ⇒ ::Process::Status? (readonly)

Returns The exit status code of the process. Only set after the process has exited.

Returns:

  • (::Process::Status, nil) —

    The exit status code of the process. Only set after the process has exited.



226
227
228
 
# File 'lib/subprocess.rb', line 226

def status
  @status
end

#stderr ⇒ IO (readonly)

Returns The IO that is connected to this process's stderr.

Returns:

  • (IO) —

    The IO that is connected to this process's stderr.



216
217
218
 
# File 'lib/subprocess.rb', line 216

def stderr
  @stderr
end

#stdin ⇒ IO (readonly)

Returns The IO that is connected to this process's stdin.

Returns:

  • (IO) —

    The IO that is connected to this process's stdin.



210
211
212
 
# File 'lib/subprocess.rb', line 210

def stdin
  @stdin
end

#stdout ⇒ IO (readonly)

Returns The IO that is connected to this process's stdout.

Returns:

  • (IO) —

    The IO that is connected to this process's stdout.



213
214
215
 
# File 'lib/subprocess.rb', line 213

def stdout
  @stdout
end

Instance Method Details

#communicate(input = nil, timeout_s = nil) {|stdout, stderr| ... } ⇒ Array(String, String)?

Write the (optional) input to the process's stdin and read the contents of stdout and stderr. If a block is provided, stdout and stderr are yielded as they are read. Otherwise they are buffered in memory and returned when the process exits. Do this all using IO::select, so we don't deadlock due to full pipe buffers.

This is only really useful if you set some of :stdin, :stdout, and :stderr to Subprocess::PIPE.

Parameters:

  • input (String) (defaults to: nil) —

    A string to feed to the child's standard input.

  • timeout_s (Numeric) (defaults to: nil) —

    Raise CommunicateTimeout if communicate does not finish after timeout_s seconds.

Yield Parameters:

  • stdout (String) —

    Data read from stdout since the last yield

  • stderr (String) —

    Data read from stderr since the last yield

Returns:

  • (Array(String, String), nil) —

    An array of two elements: the data read from the child's standard output and standard error, respectively. Returns nil if a block is provided.

Raises:

  • (ArgumentError) 


428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
 
# File 'lib/subprocess.rb', line 428

def communicate(input=nil, timeout_s=nil)
  raise ArgumentError if !input.nil? && @stdin.nil?

  stdout, stderr = "", ""

  # NB: Always force encoding to binary so we handle unicode or binary input
  # correctly across multiple write_nonblock calls, since we manually slice
  # the input depending on how many bytes were written
  input = input.dup.force_encoding('BINARY') unless input.nil?

  @stdin.close if (input.nil? || input.empty?) && !@stdin.nil?

  timeout_at = Time.now + timeout_s if timeout_s

  self.class.catching_sigchld(pid) do |global_read, self_read|
    wait_r = [@stdout, @stderr, self_read, global_read].compact
    wait_w = [input && @stdin].compact
    done = false
    while !done
      # If the process has exited, we want to drain any remaining output before returning
      if poll
        ready_r = wait_r - [self_read, global_read]
        ready_w = []
        done = true
      else
        ready_r, ready_w = select_until(wait_r, wait_w, [], timeout_at)
        raise CommunicateTimeout.new(@command, stdout, stderr) if ready_r.nil?
      end

      if ready_r.include?(@stdout)
        if drain_fd(@stdout, stdout)
          wait_r.delete(@stdout)
        end
      end

      if ready_r.include?(@stderr)
        if drain_fd(@stderr, stderr)
          wait_r.delete(@stderr)
        end
      end

      if ready_r.include?(global_read)
        if drain_fd(global_read)
          raise "Unexpected internal error -- someone closed the global self-pipe!"
        end
        self.class.wakeup_sigchld
      end

      if ready_r.include?(self_read)
        if drain_fd(self_read)
          raise "Unexpected internal error -- someone closed our self-pipe!"
        end
      end

      if ready_w.include?(@stdin)
        written = 0
        begin
          written = @stdin.write_nonblock(input)
        rescue EOFError # Maybe I shouldn't catch this...
        rescue Errno::EINTR
        rescue IO::WaitWritable
          # On OS X, a pipe can raise EAGAIN even after select indicates
          # that it is writable. Once the process consumes from the pipe,
          # the next write should succeed and we should make forward progress.
          # Until then, treat this as not writing any bytes and continue looping.
          # For details see: https://github.com/stripe/subprocess/pull/22
          nil
        rescue Errno::EPIPE
          # The other side of the pipe closed before we could
          # write all of our input. This can happen if the
          # process exits prematurely.
          @stdin.close
          wait_w.delete(@stdin)
        end
        input = input[written..input.length]
        if input.empty?
          @stdin.close
          wait_w.delete(@stdin)
        end
      end

      if block_given? && !(stderr.empty? && stdout.empty?)
        yield stdout, stderr
        stdout, stderr = "", ""
      end
    end
  end

  wait

  if block_given?
    nil
  else
    [stdout, stderr]
  end
end

#drain_fd(fd, buf = nil) ⇒ true, false

Do nonblocking reads from fd, appending all data read into buf.

Parameters:

  • fd (IO) —

    The file to read from.

  • buf (String) (defaults to: nil) —

    A buffer to append the read data to.

Returns:

  • (true, false) —

    Whether fd was closed due to an exceptional condition (EOFError or EPIPE).



398
399
400
401
402
403
404
405
406
407
408
409
 
# File 'lib/subprocess.rb', line 398

def drain_fd(fd, buf=nil)
  loop do
    tmp = fd.read_nonblock(4096).force_encoding(fd.external_encoding)
    buf << tmp unless buf.nil?
  end
rescue EOFError, Errno::EPIPE
  fd.close
  true
rescue Errno::EINTR
rescue Errno::EWOULDBLOCK, Errno::EAGAIN
  false
end

#poll ⇒ ::Process::Status?

Poll the child, setting (and returning) its status. If the child has not terminated, return nil and exit immediately

Returns:

  • (::Process::Status, nil) —

    The exit status of the process



379
380
381
 
# File 'lib/subprocess.rb', line 379

def poll
  @status ||= (::Process.waitpid2(@pid, ::Process::WNOHANG) || []).last
end

#send_signal(signal) ⇒ Integer

Does exactly what it says on the box.

Parameters:

  • signal (String, Symbol, Integer) —

    The signal to send to the child process. Accepts all the same arguments as Ruby's built-in Process::kill, for instance a string like "INT" or "SIGINT", or a signal number like 2.

Returns:

  • (Integer) —

    See Process.kill

See Also:

  • Process.kill


535
536
537
 
# File 'lib/subprocess.rb', line 535

def send_signal(signal)
  ::Process.kill(signal, pid)
end

#terminate ⇒ Integer

Sends SIGTERM to the process.

Returns:

See Also:



544
545
546
 
# File 'lib/subprocess.rb', line 544

def terminate
  send_signal("TERM")
end

#wait ⇒ ::Process::Status

Wait for the child to return, setting and returning the status of the child.

Returns:

  • (::Process::Status) —

    The exit status of the process



387
388
389
 
# File 'lib/subprocess.rb', line 387

def wait
  @status ||= ::Process.waitpid2(@pid).last
end