How to create a dynamic filter condition and use it to filter rows on a spark dataframe?

There are two variables that I read from a file and use it to filter to a dataframe. Count of variables will vary on a daily basis so I need to build the filter condition dynamically so code change is not required.


I read the above file row by row and split by "=" and load it to Map[String,String]. In order to get the value, I pass the key "clientid1" and split the value by ","

val clientid1 :Array[String] = parameterValues("clientid1").split(",")

I used these variables like below to filter the DF and the below is working fine.

val finalDF = customerDF
              .filter((col("clientid").isin(clientid1: _*) && col("clientref").isin(clientref1: _*))
                      ||(col("clientid").isin(clientid2: _*) && col("clientref").isin(clientref2: _*)))

Now I need to build the below part dynamically,

(col("clientid").isin(clientid1: _*) && col("clientref").isin(clientref1: _*))
                          ||(col("clientid").isin(clientid2: _*) && col("clientref").isin(clientref2: _*))

if clientid3 and clientref3 are present in the input file then the filter condition should be

(col("clientid").isin(clientid1: _*) && col("clientref").isin(clientref1: _*))
||(col("clientid").isin(clientid2: _*) && col("clientref").isin(clientref2: _*))
||(col("clientid").isin(clientid3: _*) && col("clientref").isin(clientref3: _*))

I used a function like below to generate the condition but it is returning a String and the filter condition is not accepting it,

def generateFilterCond(count: Int, parameterValues: Map[String,String]): String ={
     var filterCond = ""
     for ( i <- 1 to count){
     val clientid :Array[String] = parameterValues("clientid$i").split(",")
     val clientref :Array[String] = parameterValues("clientref$i").split(",")

        filterCond = filterCond + (col("clientid").isin(clientid: _*) && col("clientref").isin(clientref: _*)) + " || " 
        filterCond.substring(0, length-4).trim

Any suggestions on how to build this?