GNU Parallel: --line-buffer option doesn't work with --pipe

Im trying to parallelize some tasks that needs to be processed on real time, so i was using --line-buffer. I was processing very long strings, but then i noticed that sometimes it hits the line lenght limit, making a command line too long error, so i decided to pipe them

But when i use the --pipe option, --line-buffer stops working
I tested with simpler commands, and the issue still occurs

# Returns instantly, but pass the data as args
(echo 1; echo 2; sleep 100) | parallel -j1 --lb cat
# Pass the data to STDIN, but only after 100 seconds
(echo 1; echo 2; sleep 100) | parallel -j1 --lb --pipe cat

Im using parallel 20190422 on Arch Linux

1 answer

  • answered 2019-06-05 23:51 Ole Tange

    # Pass the data to STDIN, but only after 100 seconds
    (echo 1; echo 2; sleep 100) | parallel -j1 --lb --pipe cat
    

    This is due to GNU Parallel reads 1 MB by default. So GNU Parallel waits for more input. Only after 100 sec is the STDIN closed, and GNU Parallel gets an EOF.

    You can probably do something like this:

    (echo 1; echo 2; sleep 100) | parallel -j1 --block 1 -N1 --lb --pipe cat
    

    But if the lines are much longer, then increase --block.