Slashing Perception Latency by 80%: Migrating Python Hot Paths to Multithreaded C++ on Jetson AGX

At Wastefull Insights, our autonomous waste sorting facility relied on 4 concurrent 1080p RTSP camera streams monitoring high-speed conveyor belts. The robots had less than $350\text{ ms}$ between when an item was first imaged by the overhead cameras and when the picking arm had to initiate its descent trajectory.

Our initial Python-based perception pipeline was struggling: total frame-to-decision latency was 195ms, and GPU utilization was pinned at 95% on expensive desktop-class RTX graphics cards. In this article, I break down how profiling the system revealed the real bottleneck, and how we cut latency to 42ms on low-power Jetson AGX hardware.

1. The Profiling Breakdown: The Latency Crisis

Before changing a single line of code, we profiled the pipeline using NVIDIA Nsight Systems and PyTorch CUDA profilers. The results were illuminating:

Pipeline Stage	Implementation	Time Taken	% of Budget
RTSP Ingestion & Decode	Python OpenCV VideoCapture	5 ms	2.5%
GPU Pre-Processing & Scaling	NumPy / Python CPU copies	50 ms	25.6%
Neural Model Inference	PyTorch FP32 YOLOv5	80 ms	41.0%
Bounding Box Tracking & NMS	Python loops / Shapely	60 ms	30.8%
Total Frame-to-Pick Latency		195 ms	100%

Notice that 110ms out of 195ms had nothing to do with neural model computation! It was completely consumed by Python interpreter overhead, host-to-device memory copies (`cudaMemcpy`), and inefficient CPU polygon operations.

2. Four Steps to 42ms Latency

Step 1: Model Quantization with TensorRT (80ms → 35ms)

We exported the trained YOLOv5 weights into ONNX and compiled an optimized TensorRT engine with FP16 precision and calibrated INT8 layers for backbone convolutions:

Layer fusion eliminated unnecessary memory round-trips to VRAM.

Kernel auto-tuning selected optimal CUDA kernel configurations for the Jetson AGX Xavier GPU architecture.

Step 2: Zero-Copy Hardware Decoding via DeepStream (50ms → 2ms)

We eliminated Python's CPU decode buffer. Using NVIDIA DeepStream SDK, video streams from all 4 cameras were decoded directly into NVMM (NVIDIA Memory Management) hardware memory. Frames stayed in GPU memory from initial H.264 decode through TensorRT inference—zero CPU-GPU transfers.

Step 3: Multithreaded C++ Hot Paths (60ms → 5ms)

We rewrote the post-processing tracking, non-maximum suppression (NMS), conveyor line-crossing triggers, and multi-robot task allocation in modern C++20:

Lock-free ring buffers (`std::atomic`) for frame metadata.

SIMD-accelerated polygon intersection checks for overlapping recyclable materials.

Step 4: Asynchronous Double-Buffering

Implemented asynchronous CUDA streams (`cudaStream_t`) so camera capture of frame $N+1$ overlapped seamlessly with TensorRT inference on frame $N$.

3. Quantified Impact

Before Optimization (Desktop RTX):
Latency: 195ms | GPU Load: 95% | Hardware Cost: $$$$

After Optimization (Jetson AGX Xavier):
Latency: 42ms  | GPU Load: 20% | Hardware Cost: $ (4x reduction)

The takeaway: In robotics, algorithm architecture and memory layout will always matter more than raw compute horsepower.