How do you calculate sum of a column without creating temp files in shell script?

given an input file with data like this:

id,name
10,abc
20,xyz
30,def

I am trying to sum the values in the id column.

Note: the reason I am using awk -v header is because I have multiple files all of which have id as a common header name but at different positions.

The output I am expecting is the total sum of id (i.e., for the example above the output is 60).

The code below works and returns the expected output, but I have to create a temp file in the code and then calculate the sum.

I tried many variations unfortunately all my efforts failed.

I want to avoid writing the data to a temp file, intfile.txt, but I am stuck.

Any solutions/suggestions appreciated.

ps: I am relatively new to shell scripting and I know code is not written well, but I am working through it.

#!/bin/bash
awk -v header="id" '
BEGIN { FS=","; a=0 }
NR == 1 { for (i=1;i<=NF;i++) { if ($i==header) { a=i }} }
a=NR > 1 && a>0 { print $a }' testfile.txt>intfile.txt
awk '{s+=$1}END{print s}' intfile.txt

2 answers

  • answered 2018-11-08 00:51 Charles Duffy

    #!/usr/bin/env bash
    
    target="id"   # the field you want to sum
    target_idx=   # the column number of that field
    sum=0         # the sum that was found so far
    
    {
      # first, just read the header...
      IFS=, read -r -a header
      for idx in "${!header[@]}"; do  # and look for the target field in it
        [[ ${header[$idx]} = $target ]] && { target_idx=$idx; break; }
      done
      [[ $target_idx ]] || { echo "ERROR: No $target field found" >&2; exit 1; }
    
      # then, iterate over other lines
      while IFS=, read -r -a line; do
        sum=$(( sum + ${line[$target_idx]} ))
      done
    } <testfile.txt
    
    echo "$sum"
    

    See this running at https://ideone.com/MOnpFM


    Some references:

  • answered 2018-11-08 01:50 Sam

    Try this also:

    idline=$(grep id input.txt)
    IFS=',' read -ra elem <<< "$idline"
    idfld=0
    for i in "${elem[@]}"; do
        idfld=$(($idfld+1))
        if [[ "$i" = "id" ]]
        then
            break;
        fi
    done
    gt=0
    for num in `cat input.txt|grep "^[0-9]"|cut -d"," -f${idfld}`; do
        gt=$(($gt+$num ))
    done
    echo $gt