Parts of multithreaded C++ program freezes at random times
I have been struggling for quite some time in a multithreaded C++ program.
It consists of two callback functions, one that grabs frames from a camera into the program, and one that sends data out from the program, as well as two concurrent threads that do various processing steps.
At most times it works as expected, and it can go on for hours without it failing. However, at seemingly random times all the concurrent threads freeze up, and only the callback function remains active, shown by printing lines from the various threads to the terminal during runtime.
I am using OpenCV to display various stages of the processing (all done in the main thread), which I can control in real time, and the problem seems to occur more frequently when I display one of the video feeds, and much more frequently when I display two. With one feed displaying the threads usually freeze within an hour (but it can be anywhere from 5 minutes to 60), with two they usually freeze within 15 minutes. Likewise, if I very frequently output lines from both callback functions and threads, all of them seem to freeze, and after only a few completed loops.
The image data is passed between the threads using pairwise arrays with corresponding mutex locks. The pairwise arrays are used to reduce potential bottle necks.
Attempting to cout frequently reveals very little – both threads and functions seem to be able to freeze at any given point. Also I have tried my best to see if there is a memory leak issue, but running for instance top does not show an increase of memory usage after the feeds have been activated, or before the freezing occurs (valgrind does not act friendly with much of OpenCV it seems). top does show that the CPU usage decrease drastically when a freeze occurs, from ~120 to ~30 %.
I am stumped on what the issue might be, so any thought or pointers would be very appreciated.
CallbackOne(cameraData) if lastCallbackOneFrame != 1 mutexCallbackOneMain1.lock() arrayCallbackOneMain1 = cameraData mutexCallbackOneMain1.unlock() lastCallbackOneFrame = 1; else mutexCallbackOneMain2.lock() arrayCallbackOneMain2 = cameraData mutexCallbackOneMain2.unlock() lastCallbackOneFrame = 2; main() initialize arrayCallbackOneMain1 initialize arrayCallbackOneMain2 initialize arrayMainThread1 initialize arrayMainThread2 initialize arrayThreadCallbackTwo1 initialize arrayThreadCallbackTwo2 initialize CallbackOne initialize CallbackTwo start Thread while(true) if lastCallbackOneFrame == 1 mutexCallbackOneMain1.lock() mainData = arrayCallbackOneMain1 mutexCallbackOneMain1.unlock() else mutexCallbackOneMain2.lock() mainData = arrayCallbackOneMain2 mutexCallbackOneMain2.unlock() //do some processing if lastMainFrame != 1 mutexMainThread1.lock() arrayMainThread1 = processedData mutexMainThread1.unlock() lastMainFrame = 1; else mutexMainThread2.lock() arrayMainThread2 = processedData mutexMainThread2.unlock() lastMainFrame = 2; if showFeed1 show feed1 //increases freeze risk if showFeed2 show feed2 //increases freeze risk Thread() while(true) if lastMainFrame == 1 mutexMainThread1.lock() threadData = arrayMainThreadOne1 mutexMainThread1.unlock() else mutexMainThread1.lock() threadData = arrayMainThreadOne1 mutexMainThread1.unlock() //some processing if lastThreadFrame != 1 mutexThreadCallbackTwo1.lock() arrayThreadCallbackTwo1 = processedData mutexThreadCallbackTwo1.unlock() lastThreadFrame = 1; else mutexThreadCallbackTwo2.lock() arrayThreadCallbackTwo2 = processedData mutexThreadCallbackTwo2.unlock() lastThreadFrame = 2; CallbackTwo() if lastThreadFrame == 1 mutexThreadCallbackTwo.lock() callbackTwoData = arrayThreadCallbackTwo1 mutexThreadCallbackTwo.unlock() else mutexThreadCallbackTwo.lock() callbackTwoData = arrayThreadCallbackTwo1 mutexThreadCallbackTwo.unlock() return callbackTwoData