Class: Subprocess::Process

Inherits:
Object
  • Object
show all
Defined in:
lib/subprocess.rb

Overview

A child process. The preferred way of spawning a subprocess is through the functions on Subprocess (especially check_call and check_output).

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(cmd, opts = {}) {|process| ... } ⇒ Process

Create a new process.

Options Hash (opts):

  • :stdin (IO, Fixnum, String, Subprocess::PIPE, nil)

    The IO, file descriptor number, or file name to use for the process's standard input. If the magic value Subprocess::PIPE is passed, a new pipe will be opened.

  • :stdout (IO, Fixnum, String, Subprocess::PIPE, nil)

    The IO, file descriptor number, or file name to use for the process's standard output. If the magic value Subprocess::PIPE is passed, a pipe will be opened and attached to the process.

  • :stderr (IO, Fixnum, String, Subprocess::PIPE, Subprocess::STDOUT, nil)

    The IO, file descriptor number, or file name to use for the process's standard error. If the special value Subprocess::PIPE is passed, a pipe will be opened and attached to the process. If the special value STDOUT is passed, the process's stderr will be redirected to its stdout (much like bash's 2>&1).

  • :cwd (String)

    The directory to change to before executing the child process.

  • :env (Hash<String, String>)

    The environment to use in the child process.

  • :retain_fds (Array<Fixnum>)

    An array of file descriptor numbers that should not be closed before executing the child process. Note that, unlike Python (which has :close_fds defaulting to false), all file descriptors not specified here will be closed.

  • :preexec_fn (Proc)

    A function that will be called in the child process immediately before executing cmd. Note: we don't actually close file descriptors, but instead set them to auto-close on exec (using FD_CLOEXEC), so your application will probably continue to behave as expected.

Yields:

  • (process)

    Yields the just-spawned Subprocess::Process to the optional block. This occurs after all of Subprocess::Process's error handling has been completed, and is a great place to call #communicate, especially when used in conjunction with check_call.

Yield Parameters:

  • process (Process)

    The process that was just spawned.

Raises:

  • (ArgumentError)


236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
# File 'lib/subprocess.rb', line 236

def initialize(cmd, opts={}, &blk)
  raise ArgumentError, "cmd must be an Array" unless Array === cmd

  @command = cmd

  # Figure out what file descriptors we should pass on to the child (and
  # make externally visible ourselves)
  @child_stdin, @stdin = parse_fd(opts[:stdin], 'r')
  @child_stdout, @stdout = parse_fd(opts[:stdout], 'w')
  unless opts[:stderr] == STDOUT
    @child_stderr, @stderr = parse_fd(opts[:stderr], 'w')
  end

  retained_fds = Set.new(opts[:retain_fds] || [])

  # A control pipe for ferrying errors back from the child
  control_r, control_w = IO.pipe

  @pid = fork do
    begin
      require 'fcntl'

      FileUtils.cd(opts[:cwd]) if opts[:cwd]

      # The only way to mark an fd as CLOEXEC in ruby is to create an IO
      # object wrapping it. In 1.8, however, there's no way to create that
      # IO without it believing it owns the underlying fd, s.t. it will
      # close the fd if the IO is GC'd before the exec. Since we don't want
      # that, we stash a list of these IO objects to prevent them from
      # getting GC'd, since we are about to exec, which will clean
      # everything up anyways.
      fds = []

      # We have a whole ton of file descriptors that we don't want leaking
      # into the child. Set them all to close when we exec away.
      #
      # Ruby 1.9+ note: exec has a :close_others argument (and 2.0 closes
      # FDs by default). When we stop supporting Ruby 1.8, all of this can
      # go away.
      if File.directory?("/dev/fd")
        # On many modern UNIX-y systems, we can perform an optimization by
        # looking through /dev/fd, which is a sparse listing of all the
        # descriptors we have open. This allows us to avoid an expensive
        # linear scan.
        Dir.foreach("/dev/fd") do |file|
          fd = file.to_i
          if file.start_with?('.') || fd < 3 || retained_fds.include?(fd)
            next
          end
          begin
            fds << mark_fd_cloexec(fd)
          rescue Errno::EBADF
            # The fd might have been closed by now; that's peaceful.
          end
        end
      else
        # This is the big hammer. There's not really a good way of doing
        # this comprehensively across all platforms without just trying them
        # all. We only go up to the soft limit here. If you've been messing
        # with the soft limit, we might miss a few. Also, on OSX (perhaps
        # BSDs in general?), where the soft limit means something completely
        # different.
        special = [@child_stdin, @child_stdout, @child_stderr].compact
        special = Hash[special.map { |f| [f.fileno, f] }]
        3.upto(::Process.getrlimit(::Process::RLIMIT_NOFILE).first) do |fd|
          next if retained_fds.include?(fd)
          begin
            # I don't know why we need to do this, but OSX started freaking
            # out when trying to dup2 below if FD_CLOEXEC had been set on a
            # fresh IO instance referring to the same underlying file
            # descriptor as what we were trying to dup2 from.
            if special[fd]
              special[fd].fcntl(Fcntl::F_SETFD, Fcntl::FD_CLOEXEC)
            else
              fds << mark_fd_cloexec(fd)
            end
          rescue Errno::EBADF # Ignore FDs that don't exist
          end
        end
      end

      # dup2 the correct descriptors into place. Note that this clears the
      # FD_CLOEXEC flag on the new file descriptors (but not the old ones).
      ::STDIN.reopen(@child_stdin) if @child_stdin
      ::STDOUT.reopen(@child_stdout) if @child_stdout
      if opts[:stderr] == STDOUT
        ::STDERR.reopen(::STDOUT)
      else
        ::STDERR.reopen(@child_stderr) if @child_stderr
      end

      # Set up a new environment if we're requested to do so.
      if opts[:env]
        ENV.clear
        ENV.update(opts[:env])
      end

      # Call the user back, maybe?
      opts[:preexec_fn].call if opts[:preexec_fn]

      # Ruby 1.8's exec is really stupid--there's no way to specify that
      # you want to exec a single thing *without* performing shell
      # expansion. So this is the next best thing.
      args = cmd
      if cmd.length == 1
        args = ["'" + cmd[0].gsub("'", "\\'") + "'"]
      end

      # Ruby 1.9 changed the behavior of file descriptor retention in exec.
      # It also conveniently added the related spawn function, so
      # conditionalize on the presence of that in order to determine whether
      # or not we need to add the extra argument.
      if opts[:retain_fds] && Kernel.respond_to?(:spawn)
        redirects = {}
        retained_fds.each { |fd| redirects[fd] = fd }
        args << redirects
      end

      exec(*args)

    rescue Exception => e
      # Dump all errors up to the parent through the control socket
      Marshal.dump(e, control_w)
      control_w.flush
    end

    # Something has gone terribly, terribly wrong if we're hitting this :(
    exit!(1)
  end

  # Meanwhile, in the parent process...

  # First, let's close some things we shouldn't have access to
  @child_stdin.close if our_fd?(opts[:stdin])
  @child_stdout.close if our_fd?(opts[:stdout])
  @child_stderr.close if our_fd?(opts[:stderr])
  control_w.close

  # Any errors during the spawn process? We'll get past this point when the
  # child execs and the OS closes control_w because of the FD_CLOEXEC
  begin
    e = Marshal.load(control_r)
    e = "Unknown Failure" unless e.is_a?(Exception) || e.is_a?(String)
    # Because we're throwing an exception and not returning a
    # Process, we need to make sure the child gets reaped
    wait
    raise e
  rescue EOFError # Nothing to read? Great!
  ensure
    control_r.close
  end

  # Everything is okay. Good job, team!
  blk.call(self) if blk
end

Instance Attribute Details

#commandArray<String> (readonly)



193
194
195
# File 'lib/subprocess.rb', line 193

def command
  @command
end

#pidFixnum (readonly)



193
# File 'lib/subprocess.rb', line 193

attr_reader :command, :pid, :status

#statusObject (readonly)

Returns the value of attribute status



193
# File 'lib/subprocess.rb', line 193

attr_reader :command, :pid, :status

#stderrObject (readonly)

Returns the value of attribute stderr



184
# File 'lib/subprocess.rb', line 184

attr_reader :stdin, :stdout, :stderr

#stdinIO (readonly)



184
185
186
# File 'lib/subprocess.rb', line 184

def stdin
  @stdin
end

#stdoutIO (readonly)



184
# File 'lib/subprocess.rb', line 184

attr_reader :stdin, :stdout, :stderr

Instance Method Details

#communicate(input = nil) ⇒ Array<String>

Write the (optional) input to the process's stdin. Also, read (and buffer in memory) the contents of stdout and stderr. Do this all using IO::select, so we don't deadlock due to full pipe buffers.

This is only really useful if you set some of :stdin, :stdout, and :stderr to Subprocess::PIPE.

Raises:

  • (ArgumentError)


438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
# File 'lib/subprocess.rb', line 438

def communicate(input=nil)
  raise ArgumentError if !input.nil? && @stdin.nil?

  stdout, stderr = "", ""
  input = input.dup unless input.nil?

  @stdin.close if (input.nil? || input.empty?) && !@stdin.nil?

  self_read, self_write = IO.pipe
  self.class.catching_sigchld(pid, self_write) do
    wait_r = [@stdout, @stderr, self_read].compact
    wait_w = [input && @stdin].compact
    loop do
      ready_r, ready_w = select(wait_r, wait_w)

      # If the child exits, we still have to be sure to read any data left
      # in the pipes. So we poll the child, drain all the pipes, and *then*
      # check @status.
      #
      # It's very important that we do not call poll between draining the
      # pipes and checking @status. If we did, we open a race condition
      # where the child writes to stdout and exits in that brief window,
      # causing us to lose that data.
      poll

      if ready_r.include?(@stdout)
        if drain_fd(@stdout, stdout)
          wait_r.delete(@stdout)
        end
      end

      if ready_r.include?(@stderr)
        if drain_fd(@stderr, stderr)
          wait_r.delete(@stderr)
        end
      end

      if ready_r.include?(self_read)
        if drain_fd(self_read)
          raise "Unexpected internal error -- someone closed our self-pipe!"
        end
      end

      if ready_w.include?(@stdin)
        begin
          written = @stdin.write_nonblock(input)
        rescue EOFError # Maybe I shouldn't catch this...
        rescue Errno::EINTR
        end
        input[0...written] = ''
        if input.empty?
          @stdin.close
          wait_w.delete(@stdin)
        end
      end

      break if @status

      # If there's nothing left to wait for, we're done!
      break if wait_r.length == 0 && wait_w.length == 0
    end
  end

  wait

  [stdout, stderr]
end

#drain_fd(fd, buf = nil) ⇒ true, false

Do nonblocking reads from fd, appending all data read into buf.



415
416
417
418
419
420
421
422
423
424
425
426
# File 'lib/subprocess.rb', line 415

def drain_fd(fd, buf=nil)
  loop do
    tmp = fd.read_nonblock(4096)
    buf << tmp unless buf.nil?
  end
rescue EOFError, Errno::EPIPE
  fd.close
  true
rescue Errno::EINTR
rescue Errno::EWOULDBLOCK, Errno::EAGAIN
  false
end

#poll::Process::Status?

Poll the child, setting (and returning) its status. If the child has not terminated, return nil and exit immediately



396
397
398
# File 'lib/subprocess.rb', line 396

def poll
  @status ||= (::Process.waitpid2(@pid, ::Process::WNOHANG) || []).last
end

#send_signal(signal) ⇒ Object

Does exactly what it says on the box.



512
513
514
# File 'lib/subprocess.rb', line 512

def send_signal(signal)
  ::Process.kill(signal, pid)
end

#terminateObject

Sends SIGTERM to the process.



517
518
519
# File 'lib/subprocess.rb', line 517

def terminate
  send_signal("TERM")
end

#wait::Process::Status

Wait for the child to return, setting and returning the status of the child.



404
405
406
# File 'lib/subprocess.rb', line 404

def wait
  @status ||= ::Process.waitpid2(@pid).last
end