YOLOv8 's' on APTEC-8 crashing after extended run

[jonK1984]

Hello,

My team is running a fully INT8 quantized YOLO v8 's' model on an Axis P3625 (APTEC-8 w/ 1GB of RAM). The model works great, very accurate object detections with a ~200ms inference time. Unfortunately, after an extended run (~8hrs to 20hrs depending on a few factors), the inference server crashes. Restarting the camera fixes things but hoping to get some advice on fixing the issue.

Some version info:
Axis Firmware 11.11.116
ACAP Runtime 1.3.1
Python v3.8.8

At first, we thought it was a memory leak in our python code, but after some testing, I'm not so sure that's the case. It does seem to be camera resource related (possibly memory or cpu or some process in the background). A few observations:

• Being more efficient with memory does seem to make the camera run longer. After making the code more memory efficient, we were able to run it for 68hrs before crashing (there is still some randomness in the exact time until crash).
• More network activity on the camera seems to accelerate the time until crash.
• There are some patterns to how the camera crashes. Below is a chart of the inference time each frame (y-axis) vs run time (x-axis). The pattern is the camera inference server starts to run in a degraded state (slow inference time) then crashes in a few hours. During this degraded state, the python post-processing code also slows down significantly.

Happy to share the model if that helps

[swarali-nandgaonkar]

Hi,

I just want to share that we are doing an extended run for the QR code detector app together with another app on the Axis Q1615 MK III camera. Both apps work well for certain days until the docker containers/docker daemon crashes later. Restarting the docker daemon resolves the problem.

Server logs show '2 DOCKERD RUNTIME ERROR'. We had set up some monitoring for the docker container stats and it shows that the CPU for the QR code detector container was 100% for a moment and the memory for inference server for axis keeps rising. This might be the reason for the crash our case.

We also observed that the camera restarted automatically after the docker daemon crash. Maybe this happened because of a 'watchdog' process in the camera.. I'm not so sure.