Python popen doesn't capture stderror

I need to be able read stdout and stderr as it occurs from a process that I spawn in Python, I am currently using:

task = Popen('sh job.sh', stdout=PIPE, bufsize=1)
with task.stdout:
    for line in iter(task.stdout.readline, b''):
        stream.append(line)
        fileHandle.write(line)

This is getting the stdout, but stderr is getting sent to the console:

./tmp_2edd9d49-4108-43e8-a09f-30f34488c531: line 1: @echo: command not found

I tried adding stderr=PIPE, but that made the errors vanish. Is there a way of doing this so I can read both (I really would like the error to occur at right place.

1 answer

  • answered 2017-11-12 20:20 Lorin

    You can't omit the stderr argument if you want to capture it!

    import subprocess as shell
    
    raw_cmd = 'sh job.sh'
    cmd_list = raw_cmd.split()
    
    task = shell.Popen.(cmd_lst, stdout=shell.PIPE, stderr=shell.PIPE)
    
    with task.stderr as stderr:
        for line in stderr:
            print line
    
    with task.stdout as stdout:
        for line in stdout:
            print line
    

    Basically the extern program writes into two files: stdout and stderr, we plug these "out-files" into our program. The way we are doing that in this example allows only to track the output of either stderr, or stdout in total, so right now there is no correlation.

    To track both files simultaneously, you would have to fall back to select, pool, or epoll. Depending on installed libraries and OS.

    e.g. on linux:

    ...    
    from select import select
    ...
    while 1:
        # `select` blocks until any file is ready !!!
        reads, writes, errors = select([task.stdout, task.stderr], [], [])
        for stdfile in reads:
           if stdfile == task.stdout:
              for line in stdfile: print "stdout:", line
           if stdfile == task.stderr:
              for line in stdfile: print "stdERR:", line
    
     ...
    

    Beware, the code above is untested, but would allow a tighter out/err correleation. This is also not an optimal solution, just a pointer to possible venues.

    You let select block until any of the specified files/PIPES are ready. Then you check which file is ready (e.g if stdfile == task.stderr) you print it and repeat the loop with select.

    If you don't want this loop to block, you could move them into a separate therad, or make select non-blocking and do multiple polls (see select).