php read and parse streamed output, or alternately send blocks to file

The GOAL of this question is to find out how to parse streamed data from a process using PHP. For example, I can run this command:

kubectl get pods -n rias -w -o json 

from the CLI and it will continue to stream blocks of JSON. In other words, the process never ends.

How would you read and parse the JSON blocks that come out of this using PHP?

As an alternate approach, if there were some way (perhaps using Linux) I could write the blocks to files in a folder with a timestamp, I could process and delete them with a running PHP script. This would work but doesn't seem very efficient. Thanks.

1 answer

  • answered 2022-01-24 11:46 Oliver Williams

    I found an answer using a bash script. It's very rudimentary and there are things I want to do to make it more robust, but this does work:

    stream_kubectl_watcher() {
        # Note: `pods` and `rias` should be arguments, perhaps -o type
        # to make this function flexible
        kubectl get pods -n rias -w -o json
    }
    
    parse_kubectl_watcher() {
        i=0
        buffer=""
        file=""
        while read -r line; do
            ((i++))
            if [[ $line == '{' && $buffer == '}' ]]; then
                echo "starting new input at $i .."
                filename="file$i"
                # send to file; could also send to an API etc
                echo "writing $filename"
                echo $file > $filename
                file=$line
            fi
            file="$file$line"
            buffer=$line
        done < <(stream_kubectl_watcher 2>&1)
    }
    

    Big items to note:

    • this creates a file from each block but one could easily pipe the command to a PHP, Python or Go script as well
    • I've joined stdout and stderr and have not tested what will happen
    • there is an issue, the JSON lines are being trimmed on the left side and I think there is an $IFS variable I need to use to fix this. However, as long as this doesn't cause parse errors in the JSON, it's a feature not a bug, as the stream will be a bit smaller - perhaps even a lot in some cases

How many English words
do you know?
Test your English vocabulary size, and measure
how many words do you know
Online Test
Powered by Examplum