Terraform is throwing InvalidArgumentException Duplicate ProcessorParameter passed to ProcessingConfiguration on Kinesis Firehose dynamic partitioning

I'm trying to create a Kinesis Firehose using terraform with dynamic partitioning using two partition queries from the JSON I'm recieving, my processing configuration looks like this

processing_configuration {
  enabled = true
  processors {
    type = "RecordDeAggregation"
    parameters {
      parameter_name  = "SubRecordType"
      parameter_value = "JSON"
    }
  }
  processors {
    type = "MetadataExtraction"
    parameters {
      parameter_name  = "JsonParsingEngine"
      parameter_value = "JQ-1.6"
    }
    parameters {
      parameter_name  = "MetadataExtractionQuery"
      parameter_value = "{transaction_id:.transaction_id}"
    }
    parameters {
      parameter_name  = "MetadataExtractionQuery"
      parameter_value = "{stage:.stage}"
    }
  }
}

But when I execute this part of the code it returns a duplication error for the processing configuration. DuplicatedError

I also tried to create an appart processor for the new ExtractionQuery, it looks like this

processing_configuration {
  enabled = true
  processors {
    type = "RecordDeAggregation"
    parameters {
      parameter_name  = "SubRecordType"
      parameter_value = "JSON"
    }
  }
  processors {
    type = "MetadataExtraction"
    parameters {
      parameter_name  = "JsonParsingEngine"
      parameter_value = "JQ-1.6"
    }
    parameters {
      parameter_name  = "MetadataExtractionQuery"
      parameter_value = "{transaction_id:.transaction_id}"
    }
  }
  processors {
    type = "MetadataExtraction"
    parameters {
      parameter_name  = "JsonParsingEngine"
      parameter_value = "JQ-1.6"
    }
    parameters {
      parameter_name  = "MetadataExtractionQuery"
      parameter_value = "{stage:.stage}"
    }
  }
}

But it fails with an error that says only one MetadataExtraction processor is allowed. enter image description here

1 answer

  • answered 2022-05-06 16:56 Zorrero

    Solved by merging both queries in one using JQ format, that way firehose would separate them, tried it using this snippet and worked.

    processing_configuration {
      enabled = true
      processors {
        type = "RecordDeAggregation"
        parameters {
          parameter_name  = "SubRecordType"
          parameter_value = "JSON"
        }
      }
      processors {
        type = "MetadataExtraction"
        parameters {
          parameter_name  = "JsonParsingEngine"
          parameter_value = "JQ-1.6"
        }
        parameters {
          parameter_name  = "MetadataExtractionQuery"
          parameter_value = "{transaction_id:.transaction_id,stage:.stage}"
        }
      }
    }
    

How many English words
do you know?
Test your English vocabulary size, and measure
how many words do you know
Online Test
Powered by Examplum