Python POpen Stdout not returning back when running python

READ UPDATE!!!

I'm having an issue getting output from a Python POpen from the subproccess module.

It works fine on commands such as 'ls' , 'node' , and 'python3 -m pip install requests'.

However When I run a simple python file, such as 'python3 test.py' that contains:

print("Hello World!")

It does not output. Nor does it work with:

sys.stdout.write()

It runs the code, this can be proven by this code being put in test.py

f = open("text.txt","w")
f.write("Hello :P")
f.close()

And it Writing to the file just fine.

Here is the code extracted from the main file:

def executeCmd(cmd):
    filename = "cmdlog_{}.txt".format(cmd.replace(" ","-").replace(".","-").replace("/","-"))
    with io.open(filename, 'wb') as writer, io.open(filename, 'rb', 1) as reader:
        process = sb.Popen(cmd, stdout=writer,stderr=writer, cwd=oss.getcwd(),shell=True)
        while process.poll() is None:
            rd = reader.read()
            sys.stdout.write(rd.decode("UTF-8"))
            time.sleep(0.1)
        time.sleep(0.5)
        rd = reader.read().decode("UTF-8")
        sys.stdout.write(rd)
        time.sleep(0.5)
        oss.remove(filename)

As Extra Info, That function is called as a thread, from another thread that revives data from a socket. Hence the the dictionary 'data' bellow and the send function above.

threading.Thread(target=proccessListen,args=([data["cmd"],data["name"]]),daemon=True).start()

I'm at my wits end here, with this issue occurring on multiple os's and machines. With no fix yet found, and not related article a few hours worth of googling could not find, I request your help

Thank in advance, Sam.


UPDATE: After 1/2 hour of playing with code, and searching online to fix the new issue, I've hit a block again.

Here is my new code:

import io
import sys
import os
import subprocess
import time

def executeCmd(cmd):
    process = subprocess.Popen(cmd,stdout=subprocess.PIPE,stderr=subprocess.STDOUT,cwd=os.getcwd(),shell=True)
    print("Started")
    while process.poll() is None:
      line = process.stdout.readline()
      sys.stdout.write(line)
      #process.stdout.flush()
      time.sleep(0.5)
    print("Ended")

executeCmd("python3 test.py")

And in test.py

print("Hello")
count = 0
import time
while True:
  time.sleep(2)
  count = count + 1
  print("Output: {}".format(count))

The Problem is that process.stdout.readline() blocks the code, meaning it can never read the output, but why. This is taken from a forum page and modified. Neither Worked, Both with the same issue, I've searched and gone through many other stack overflow questions, and none of the solutions worked.I also tried something with an non-blocking flag (I'm not sure of what I was doing). It worked for others, but I got a OSERROR. Although that looks like a difficult route to take.

Any help would be appreciated.

Thanks Sam

2 answers

  • answered 2018-07-12 13:22 SamHDev

    Fixing only took an all nigher and some reading

    I would like to thank sarge. A great module!

    Here is my code:

    import sarge
    import time
    import sys
    
    cmd = sarge.run("python3 test.py",async_=True,stdout=sarge.Capture(buffer_size=1))
    print("Hello World")
    while True:
      time.sleep(0.5)
      sys.stdout.write("." + str(cmd.stdout.read()))
      sys.stdout.flush()
    

    The code can be modifed to get rid of the b'' and the .

    Sam


    Oh thanks for the question down-vote, you have made my already struggling rep go down back to 1. Thanks.

  • answered 2018-07-12 13:35 jasonharper

    process.poll() returns something other than None as soon as the subprocess finishes executing. That says absolutely nothing about whether or not you've read all of the subprocess's output! You need to keep calling process.stdout.readline() in a loop until it returns an empty string - in fact that's the only loop you need in this case, since process.poll() isn't giving you any information that you're actually using.