Is there a way to find and remove multiple MongoDB documents in one query?

I'm trying to optimize a time-based event queue that is managed in MongoDB. I'm regularly querying for due events (by checking their due_time value), executing due events and then removing executed events.

A shortened version of my Python code:

def run(self):
    """
    Background process keeps pulling events from MongoDB and removes retrieved documents
    """
    while True:
        # Find all events that are due
        query = {'due_time': {'$lt': datetime.datetime.now()}}

        for event in self.mongo.db.events.find(query):
            # Do something
            pass
        
        self.mongo.db.events.remove(query)
        time.sleep(6)

I want to optimize my MongoDB calls. Is there a way I can reduce the two separate calls with the same query (find(query) and remove(query)) to one call, something like:

due_events = self.mongo.db.events.find_and_then_remove(query)

1 answer

  • answered 2021-06-10 18:10 D. SM

    Your code is going to break if

    • The system clock on the system which is processing the events is moved forward or backward
    • Clocks on systems that insert entries into the queue are moved forward or backward

    In either case you can remove entries that weren't processed.

    What you should do is remove the documents that you have actually processed, identified by their _ids.