Best way for a Lambda to start processing messages in a SQS Queue at a specific time of day

I have a SQS queue which fills up with messages throughout the day and I want to start processing all the messages at a specific time. The scenario would be:

  1. Between 9AM and 5PM the queue would receive messages
  2. At 6PM the messages should be processed by a lambda

I was thinking of:

  1. Enabler: Lambda A which will be executed using a CloudWatch Event Bridge ruleat 6PM. This lambda would create a SQS trigger for Lambda C
  2. Disabler: Lambda B which will be executed using a CloudWatch Event Bridge rule at 8PM . This lambda would remove the SQS trigger of Lambda C
  3. Executer: Lambda C which process the messages in the queue

Is this the best way to do this?

1 answer

  • answered 2022-01-25 14:36 MyStackRunnethOver

    I would aim for the process which requires the least complexity / smallest changes to your Lambda. You could use the AWS SDK to enable / disable your Lambda's subscription, rather than actually deleting and recreating it. See this question on how to do so and specifically the enabled parameter in the updateEventSourceMapping() method in the docs it links to:

    Enabled โ€” (Boolean)

    When true, the event source mapping is active. When false, Lambda pauses polling and invocation.

    Default: True

    The advantage is that the only thing you're changing is the enabled flag - everything else (the SQS-Lambda subscription, if you will) is unchanged.

    This approach still has the danger that if the enabler/disabler lambda(s) fail, your processing will not occur during your target hours. Particularly, I'm not personally super confident in the success rate of AWS's self-mutating commands - this might just be my bias, but it definitely leans toward "infra changes tend to fail more often than regular AWS logic".

    It's worth considering whether you really need this implementation, or whether the time-based aggregation can be done on the results (e.g., let this Lambda processing run on events as they come in and write the output to some holding pen, then at Xpm when you trust all events have come in, move the results for today from the holding pen into your main output).

    This approach may be safer, in that a failed trigger on the "moving" step could be easier / faster to recover from than a failed trigger on the above "process all my data now" step, and it would not depend on changing the Lambda definition.

How many English words
do you know?
Test your English vocabulary size, and measure
how many words do you know
Online Test
Powered by Examplum