Snakemake and bash script pipefails

I am attempting to create a snakemake pipeline that is robust to multiple error sources. I would like to call a bash script within my pipeline. One standard procedure for alerting a user of a problem is to use the following at the beginning of a bash script:

#!/bin/bash

set -e
set -u
set -o pipefail

The pipefail checks for errors when redirecting std out using pipes. My script has the following simple lines of code within it

# Read the first argument
input=$1

less $input | head -10

When I run my script using directly in the terminal using

bash script.sh input

it runs completely fine and no errors are detected.

I then wish to run this script within a snakemake pipeline in a rule as follows:

    shell:
        """
        bash script.sh {input}
        """

snakemake throws an error:

        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

If I then remove the pipefail condition

# set -o pipefail

and then re-run snakemake I have no problems whatsoever and the script executes beautifully.

Can anyone explain this interaction between a seemingly successful pipe and how snakemake treats it? (or at least give pointers to diagnose it).

1 answer

  • answered 2021-06-23 08:15 dariober

    Your script:

    #!/bin/bash
    
    set -e
    set -u
    set -o pipefail
    
    input=$1
    
    less $input | head -10
    

    Doesn't really exit clean if the input has more then 10 lines. You can check it with:

    bash script.sh longfile.txt
    echo $? # -> Prints 141
    

    echo $? returns the exit exit code of the previous process which is 141 here because head broke the pipe (google exit code 141 to know more about it).

    Snakemake detects the non-zero exit code and, accordingly, fails because snakemake executes shell scripts in strict mode (google snakemake strict shell mode). How you want to handle the case is up to you but this should be the explanation for the seemingly inconsistent behaviour you see.