run a bash for loop in parallel

I've got this script that does a credential lookup for each host, in an on-premise vault system, and then runs an ansible-playbook for it.

#!/bin/bash

for host in `cat ~/.ansible/hosts`
  do
    SECRET=`/opt/vault/bin/get-admin-credential --tag=$host`
    HOST=`echo $SECRET | cut -d ';' -f1`
    LOGIN=`echo $SECRET | cut -d ';' -f2`
    DOMAIN=`echo $SECRET | cut -d ';' -f3`
    PWD=`echo $SECRET | cut -d ';' -f4`

    if [ -z "$DOMAIN" ]; then
      ansible-playbook -i ~/.ansible/hosts ~/.ansible/windows.yml -e "ansible_host=$HOST ansible_user=$LOGIN ansible_password=$PWD" --limit $host
    else
      ansible-playbook -i ~/.ansible/hosts ~/.ansible/windows.yml -e "ansible_host=$HOST ansible_user=$LOGIN@$DOMAIN ansible_password=$PWD" --limit $host
    fi
  done

This loops over each host sequentially, I've tried stuff with GNU parallel but haven't been able to do what I want, running the for loop with 5 in parallel.

Anyone point me in the right direction?

2 answers

  • answered 2020-03-25 14:24 Mark Setchell

    I don't have any "ansibles" or "vaults", so this is completely untested but may get you close:

    doit(){
       host="$1"
    
       SECRET=$(/opt/vault/bin/get-admin-credential --tag=$host)
       HOST=$(echo $SECRET | cut -d ';' -f1)
       LOGIN=$(echo $SECRET | cut -d ';' -f2)
       DOMAIN=$(echo $SECRET | cut -d ';' -f3)
       PWD=$(echo $SECRET | cut -d ';' -f4)
    
       if [ -z "$DOMAIN" ]; then
          ansible-playbook -i ~/.ansible/hosts ~/.ansible/windows.yml -e "ansible_host=$HOST ansible_user=$LOGIN ansible_password=$PWD" --limit $host
       else
          ansible-playbook -i ~/.ansible/hosts ~/.ansible/windows.yml -e "ansible_host=$HOST ansible_user=$LOGIN@$DOMAIN ansible_password=$PWD" --limit $host
       fi
    }
    
    # Export doit function to subshells created by GNU Parallel
    export -f doit
    
    parallel -a ~/.ansible/hosts doit
    

    Stylistically, there are maybe a few improvements. Firstly, shell variables consisting of upper case letters are reserved, so you shouldn't maybe use HOST, DOMAIN etc. Also, you can probably simplify all that unsightly cutting and echoing to extract the variables from the SECRET by using an IFS=';' and a read like this:

    SECRET=$(/opt/vault/bin/get-admin-credential --tag=$host)
    IFS=';' read host login domain pwd <<< "$SECRET"
    

    So, my best and final answer is:

    doit(){
       host="$1"
    
       secret=$(/opt/vault/bin/get-admin-credential --tag=$host)
       IFS=';' read host login domain pwd <<< "$secret"
    
       if [ -z "$domain" ]; then
          ansible-playbook -i ~/.ansible/hosts ~/.ansible/windows.yml -e "ansible_host=$host ansible_user=$login ansible_password=$pwd" --limit $host
       else
          ansible-playbook -i ~/.ansible/hosts ~/.ansible/windows.yml -e "ansible_host=$host ansible_user=$login@$domain ansible_password=$pwd" --limit $host
       fi
    }
    
    # Export doit function to subshells created by GNU Parallel
    export -f doit
    
    parallel -a ~/.ansible/hosts doit
    

  • answered 2020-03-25 14:40 chepner

    You simply need to run ansible-playbook in the background using the & command terminator. Note, though, that the entire loop can be simplified and improved.

    run_playbook () {
      ansible-playbook -i ~/.ansible/hosts \
                       -e "ansible_host=$2 ansible_login=$3 ansible_password=$4" \
                       ~/.ansible/windows.yml --limit "$1"
    }
    
    while IFS= read -r host; do
        secret=$(/opt/vault/bin/get-admin-credential --tag="$host")
        IFS=";" read -r shost slogin sdomain spasswd _ <<< "$secret"
        if [[ -n $sdomain ]]; then
          login="$slogin@$sdomain"
        fi
        run_playbook "$host" "$shost" "$login" "$password" &
    done < ~/.ansible/hosts