PEERNet: Benchmarking Networked Robotics on Wifi, 5G, and Beyond

Proceedings of the International Conference on Intelligent Robots and Systems (IROS)

TLDR: This post introduces PEERNet, a Python package for real-time benchmarking of networked robotic systems. It provides concise and modular methods for performance analysis of the entire system stack.

link to paper: PEERNet: Benchmarking Networked Robotics on Wifi, 5G, and Beyond

PEERNet is an end-to-end profiling tool for networked robotics. It excels in ease of deployment across various platforms and offers extensive benchmarking capabilities.

Why benchmark networked robotics?

Consider a mobile robot running a Machine Learning (ML) pipeline for perception, localization, and path planning. Here and in many other practical scenarios, offloading computationally expensive ML tasks to a cloud server over public networks is a viable optimization.

Introducing offloaded computation brings up several system-design questions which require hard statistics and real data to answer:

Which ML model maximizes decision accuracy while minimizing end-to-end latency?
How much better is the end-to-end latency when using a 5G network as opposed to a 4G LTE network?
How does data compression affect end-to-end latency in offloaded inference? Is the overhead of compression worth the faster networking latency? Does that answer change when using a 5G network as opposed to LTE or WiFi?

More generally, real-time benchmarking is required for:

1.Deployment in the real world: Before a robotic system can be deployed, decisions need to be made on hardware, ML models, networking protocols, etc.–these decisions must be informed by real-world statistics.

2.Debugging exising systems: Finding and removing bottlenecks in existing robotic systems requires hard numbers. ML model latency, network delay, or even hardware limitations could all be a system’s bottleneck.

3.Optimizing networked robotics: Many novel techniques for optimizing data sharing, offloading policies, and distributed ML are in active development. Assessing the performance of new algorithms on real hardware is essential before optimizations can be applied.

What makes benchmarking non-trivial?

Benchmarking networked robotics is fundamentally more challenging than simply timing ML models and testing sensors. The incorporation of networks into robotic systems significantly complicates the benchmarking problem by introducing:

1.Stochastic delay: Network delays are highly stochastic and system dependent. Delays are hard to predict and often change significantly from one setup to another.

2.Network Timing and Asymmetric Delay: Profiling networking operations by itself is difficult due to imperfect clock synchronization. Round-trip times, while commonly used, are of little use in networked robotic systems, where devices rarely upload and download similar amounts of data. For example, our mobile robot from earlier may upload gigabytes of images and lidar scans while downloading only bytes of labels. Hence, round-trip delay fails to provide any meaningful insight into this system.

Introducing PEERNet

We present PEERNet, the Profiler for End-to-End Real-time Networked Robotics. PEERNet is a highly modular and extensible package for profiling networked robotic systems, complete with one-way network delay estimation. PEERNet interfaces with industry standard hardware and software such as Nvidia embedded GPUs, Robot Operating System (ROS), and Zero MQ (ZMQ). PEERNet exposes a CLI for rapid benchmarking of offloaded inference systems, and we demonstrate PEERNet’s ease of use and versatility.

At a high level, PEERNet focuses on profiling networked robotics by looking at the life-cycle of data. When sensors sample data, serializable logging metadata is attached, and propagated through the system normally. This metadata includes all necessary information for tracking and profiling sampling time, ML inference latency, and one-way network delay.

Experiments

We illustrate and validate PEERNet by implementing and profiling three networked robotic systems. All of our experiments are conducted on physical hardware and live wireless networks, demonstrating that PEERNet is modular, robust, and capable of precisely profiling various systems.

Benchmarking offloaded inference

Our first experiment demonstrates how PEERNet can be used to profile and understand offloaded image classification, such as in our mobile robot example. By precisely quantifying latency costs across local and cloud hardware and for various model sizes, PEERNet allows for intelligent device/model selection.

PEERNet precisely quantifies inference costs of diffent ML classifiers at the edge and in the cloud.

Identifying non-intuitive behavior in Vison Language Models

In our second experiment, we use PEERNet to explore inference with Vision Language Models (VLMs) at the edge, an emerging task in robotic pipelines. We show that PEERNet is capable of identifying non-intuitive behaviors in VLM inference, such as a bimodal output token distribution.

PEERNet identifies that a Vision Language Model (VLM) returns a strongly bimodal distribution of caokens, speeds, and inference caimes.

Optimizing teleoperation of a robot arm

Our final experiment applies PEERNet to benchmark a full teleoperation pipeline. We explore all combinations of edge device, cloud compute, compression, and network on a teleoperated robot arm performing a pick-and-place task. We demonstrate that profiling with PEERNet solves the hardware selection problem, and is easily applicable to existing robotic pipelines.

Looking to try PEERNet out?

Our code is open-source! You can find it at our repo.