Swarm Lab

TensorCommitments: A Lightweight Verifiable Inference for Language Models

Sun, 04 Oct 2026 00:00:00 +0000

By Oguzhan Baser

A receipt for a cloud LLM run

Accepted to the 40th Conference on Neural Information Processing Systems (NeurIPS 2026).

Authors: Oguzhan Baser, Elahe Sadeghi, Eric Wang, Nico Vergauwen, Sam Kazemian, Hong Kang, Sandeep P. Chinchali, and Sriram Vishwanath.

TL;DR: TensorCommitments lets a lightweight client check selected hidden states from a remote LLM run using compact cryptographic commitments. On LLaMA 2-13B, the paper reports 98.6 ms of post-inference prover time and 12 ms of verifier time, with no verifier GPU.

Your browser does not support the video tag.

arXiv | Code

The question behind the answer

A cloud model returns text. What it does not return is much evidence of the computation that produced it. A client could rerun the model to compare results, but that assumes access to the weights and another GPU. The problem grows sharper when inference is part of a longer workflow: one altered prompt, weight block, or intermediate state can quietly change what comes next.

TensorCommitments asks a practical question: can the service leave a compact, checkable record of an inference while the client remains lightweight?

Keep the tensor structure

A commitment is a cryptographic tag for data: change the data, and the tag should no longer match. Many schemes first flatten model activations into long vectors. TensorCommitments instead commits to activation tensors through multivariate polynomials, preserving the axes that make the data a tensor in the first place. In the paper’s fixed-size interpolation experiment, moving from one to two dimensions reduces runtime from 4.1 seconds to 0.125 seconds. That observation motivates the construction; it is an interpolation benchmark, not an end-to-end inference speedup.

The commitments are organized in a Terkle Tree. A single root binds the evolving states of an interaction, while a verifier can request a path to a selected value and check it without replaying the model. The tree makes the record compact enough to query, and the multivariate openings retain the shape of the underlying data.

Figure 1. From activation tensors to a challenged proof checked by the verifier (paper Fig. 2).

Spend checks where they matter

Checking every layer of a large model would erase much of the efficiency gain. The paper therefore scores layers using spectral properties of their weights and allocates a fixed verification budget to influential blocks. This is a deliberate trade-off: the cryptographic opening verifies what was selected, while attack detection still depends on which layers are checked and on the verifier’s budget. TensorCommitments is not presented as a full zero-knowledge proof of every operation.

Figure 2. Attack manipulation coverage for layer-selection policies (paper Fig. 5).

What the experiments show

For LLaMA 2-13B on an A100, the paper reports 10.165 seconds for base inference. TensorCommitments adds 98.6 ms of post-inference prover work and 12 ms of verification: roughly 0.97% and 0.12% of that base time. The verifier’s reported GPU use is 0 GB. In the paper’s evaluated attack suite, detection accuracy is 96.02%, versus 82.59% for the TOPLOC baseline. The study also reports up to a 48% improvement under prompt-tampering attacks, and its layer-selection policy covers up to 75% more attacked layers than alternatives at the same budget.

Figure 3. LLaMA 2-13B system and detection comparison (paper Table 1).

Figure 4. Detection across tested attack scenarios (paper Fig. 7).

These numbers belong to the reported models, hardware, attacks, and selection budgets. They show a promising efficiency and detection profile; they do not imply that every possible tampering attempt will be caught.

Where this leads

Remote inference will be easier to trust when a client can ask for evidence, rather than accepting only a fluent answer. TensorCommitments offers one route toward that goal: proofs tied to model tensors, a tree that binds an interaction, and verification costs small enough for a client without a GPU.

The limits matter too. The design needs a trusted setup, the prover needs GPU access to the activations, and the paper does not claim full zero knowledge. Its robustness is bounded by the layers and checks a verifier can afford. Those constraints point directly to the next research questions: stronger coverage, more distributed trust, and privacy guarantees that remain practical at LLM scale.

AttentionViG: Cross-Attention-Based Dynamic Neighbor Aggregation in Vision GNNs

Thu, 01 Oct 2026 00:00:00 +0000

By Oguzhan Baser

AttentionViG: Cross-Attention-Based Dynamic Neighbor Aggregation in Vision GNNs

Accepted to the Fifth Learning on Graphs Conference (LoG 2026).

Authors: Hakan Emre Gedik, Andrew Martin, Mustafa Munir, Oguzhan Baser, Radu Marculescu, Sandeep P. Chinchali, and Alan C. Bovik.

TL;DR: AttentionViG teaches Vision GNNs which neighbors matter. Cross-attention dynamically weights neighboring nodes, helping sparse graphs suppress irrelevant features while reaching up to 83.9% ImageNet-1K top-1 accuracy.

Read the paper

PDF

Figure 1. Overview of the AttentionViG architecture across scales.

Motivation

Vision Graph Neural Networks (ViGs) represent image patches as nodes and exchange information through graph edges. Their performance depends on graph construction and on how each node aggregates its neighbors.

Dynamic k-nearest-neighbor graphs can capture semantic relationships, but neighbor search is expensive. Fixed sparse graphs such as SVGA are more efficient, yet may connect semantically unrelated nodes. Common aggregation operators also lack an explicit mechanism for learning how important each proposed neighbor should be.

We ask:

Q1 (Adaptive aggregation): Can a ViG learn which proposed neighbors are relevant instead of treating every connection equally?
Q2 (Efficiency and transfer): Can it retain sparse graph efficiency while improving classification and dense prediction?

Contributions

Cross-attention aggregation. Queries come from each node, while keys and values come from its neighbors. Cosine similarity is transformed through an exponential affinity function, giving neighbors independent relevance weights rather than forcing them to compete through softmax.
AttentionViG. We integrate this aggregation into a multi-scale hybrid CNN-GNN architecture. Inverted residual blocks perform local processing, while Grapher layers use efficient SVGA connectivity for non-local message passing.
Broad evaluation. We evaluate AttentionViG on ImageNet-1K, MS-COCO, and ADE20K.

Figure 2. Cross-attention aggregation: neighbor weighting and aggregation pipeline.

How AttentionViG Works

The key idea is to separate graph construction from neighbor importance. SVGA supplies candidate neighbors; cross-attention learns which should contribute strongly and which should be suppressed. This makes message passing adaptive to image content without expensive dynamic neighbor search. The parallelized Grapher implementation retains linear scaling with input resolution.

Results

On ImageNet-1K, AttentionViG-S/M/B achieve 81.3% / 83.1% / 83.9% top-1 accuracy at 1.6 / 3.2 / 4.8 GFLOPs. AttentionViG-B matches GreedyViG-B at 83.9% while using 4.8 rather than 5.2 GFLOPs. These results use the paper’s training setup with knowledge distillation from a RegNetY-16GF teacher.

ImageNet-1K performance summary. Values are reported in the paper.

AttentionViG-B also reaches 46.4 box AP and 42.3 mask AP on MS-COCO, and 47.8 mIoU on ADE20K, using Mask R-CNN and Semantic FPN, respectively.

COCO and ADE20K transfer performance summary. Values are reported in the paper.

In vanilla ViG, cross-attention reaches 74.3% top-1 at 1.6 GFLOPs, matching EdgeConv’s accuracy at about two-thirds of its FLOPs. In the affinity-function ablation, exponential affinity reaches 81.3% top-1 versus 80.8% with softmax.

Similarity visualizations show the model emphasizing semantically related regions while suppressing unrelated ones.

Figure 3. Query-key similarity visualizations highlighting semantically related regions.

Impact

AttentionViG shows that efficient graph construction does not require every proposed edge to be equally useful. Learning neighbor relevance allows Vision GNNs to remain sparse and efficient while becoming more selective about the information they propagate. Extending the aggregation idea to video, point clouds, and other graph-based domains is a promising direction for future work.

For the complete experiments, ablations, and implementation details, read the paper.

Constrained Posterior Sampling: Time Series Generation with Hard Constraints

Fri, 10 Oct 2025 00:00:00 +0000

By Sai Shankar Narasimhan

Forcing Diffusion Models to Respect the Rules: Constrained Posterior Sampling for Time Series

Proceedings of the 39th Conference on Neural Information Processing Systems (NeurIPS 2025)

TLDR: Constrained Posterior Sampling (CPS) is a diffusion-based sampling algorithm that aims to generate time series samples that can satisfy user-specified constraints.

arXiv

Motivation

Imagine you are a quantitative researcher who wants to stress-test trading strategies. You would want access to a tool that can precisely generate high-fidelity stock price time series for prompts like

Generate Tesla’s stock price for the next month with 5% volatility.

Additionally, for the stocks domain, there are some inherent rules that a generated sample should adhere to, such as the opening and closing prices for a day should be bounded by the highest and lowest prices for that day. This problem exists in almost all engineering domains, where certain domain-specific constraints may arise due to the laws of physics or the underlying process. We refer to this as the constrained time series generation problem.

Expectations Of A Desired Approach

Training-free: Training a generative model for specific constraints lacks scalability. For example, a model trained for mean constraints cannot easily adapt to argmax constraints.
Hyperparameter-free: The choice of guidance weights in guidance-based approaches with diffusion models significantly affects the sample quality. However, optimizing these weights becomes computationally intractable when dealing with hundreds of constraints.
Should be Independent of external models: Prior works often employ external models, like the discriminator in Generative Adversarial Networks (GANs), alongside the generative model to enforce realism, necessitating additional training and complex sampling procedures.

Figure 1 - High-Level Description of Constrained Posterior Sampling (CPS): Here, we show an example where CPS generates a daily stock price time series with natural constraints, such as the bounds on the opening and closing prices of the stock.

This post features Constrained Posterior Sampling, a training-free, plug-and-play approach that enforces hard constraints on the outputs from your diffusion generative models without compromising sample quality.

Constrained Posterior Sampling

CPS builds on diffusion models – a class of generative models that iteratively refine noise into data. In a diffusion model, you start with random noise and repeatedly “denoise” it to produce a sample. The key insight of CPS is: after each small denoising step, gently push the output back to satisfy the constraints before further denoising. In other words, at every step of the generation, we enforce the rules, at least a little. By the end, the output should follow all the rules by construction.

Under the Hood: How CPS Enforces Constraints in Diffusion Sampling

Given a specific constraint set from which we aim to generate a realistic time series sample, we perform the following steps:

Estimate the posterior mean. At each denoising step, we obtain an estimate of the clean sample.
Project the posterior mean estimate. We project the posterior mean estimate towards the constraint set. This is performed using off-the-shelf optimization routines.
Perform DDIM sampling. Using the projected posterior mean estimate, we obtain the noisy sample for the next denoising step using DDIM sampling. We refer the reader to our manuscript for further technical details.

This process continues for all the denoising steps. During projection, CPS performs an unconstrained optimization step that ensures minimal perturbation of the posterior mean estimate such that the constraint violation is reduced. This is concisely depicted in Figure 2.

Figure 2 - Constrained Posterior Sampling: We show the graphical model for one step of denoising in CPS: check Algorithm 1 in our manuscript.

To ensure minimal effects on the sample quality, we introduce penalty coefficients that minimize the perturbations of the posterior mean estimate during the initial denoising steps when the signal-to-noise ratio is very low. Towards the final denoising steps, the penalty coefficients are very large to ensure constraint satisfaction.

Note that CPS satisfies all the requirements listed above for a desired approach.

CPS is a training-free procedure and can be extended to any constraint class.
The proposed approach is hyperparameter-free. The penalty coefficients are fixed using the noise schedule of the diffusion model, and CPS leverages existing off-the-shelf optimization solvers to perform the projection step.
CPS does not require any external model to improve sample realism. In fact, we note that the adverse effects of projection are mitigated through the subsequent denoising steps.

Applications and Results Across Domains:

In our work, we showcase the efficiency of CPS in terms of sample quality and diversity on six diverse datasets, spanning environmental, traffic, and finance domains. Particularly, in terms of sample realism and utility metrics (check Figure 3), we show that CPS outperforms SOTA approaches by 10%.

Figure 3 - Main Qualitative Results: CPS provides high-fidelity synthetic time series samples that match real time series data. The real test samples from which the constraints are extracted are shown in blue. The samples generated using the extracted constraints are shown in red. Across all datasets, the baselines suffer from the adversarial effects of the projection step, whereas CPS generates high-quality samples.

Additionally, in terms of tracking real time series samples, we show that CPS outperforms SOTA methods by 42%. Specifically, CPS does not suffer from sample quality degradation for a large number of constraints, while other approaches break down in such settings (Figure 4). We refer the readers to Figures 3 and 4 to observe the sample quality and tracking abilities of CPS.

Figure 4: CPS tracks the real data samples as the number of constraints increases. Increasing the number of constraints reduces the size of the constraint set, and an ideal approach should effectively generate samples that resemble the real time series samples that belong to the constraint set. Here, we show a qualitative example from the Stocks dataset. Observe that CPS accurately tracks the real sample that concurs with the specified constraints, while other approaches suffer.

VIBE: Annotation-Free Video-to-Text Information Bottleneck Evaluation for TL;DR

Sun, 21 Sep 2025 00:00:00 +0000

By Po-han

VIBE: Annotation-Free Video-to-Text Information Bottleneck Evaluation for TL;DR

Proceedings of the 39th Conference on Neural Information Processing Systems (NeurIPS 2025)

TLDR: Annotation-free video summarization evaluation that boosts human decision-making accuracy by up to 61% and cuts response time by 75%!

Project Website | arXiv | Code | Dataset

Motivation

Many real-world tasks still require human oversight: a traffic officer sifting through dashcam footage, or a researcher screening long conference videos. Watching raw video is slow, and existing vision-language models (VLMs) often produce verbose, redundant captions that hinder efficiency. Current video-to-text evaluation methods depend on costly human annotations and ignore whether summaries actually help humans make decisions. We ask:

Q1 (Annotation-Free): Can we evaluate video summaries without relying on gold-standard human captions?
Q2 (Task-Aware): Can we measure how much a summary improves human performance on downstream tasks?

VIBE Framework Overview

Contributions

We introduce VIBE (Video-to-text Information Bottleneck Evaluation), a novel framework that evaluates and selects VLM summaries without annotations or retraining.

Grounding Score: Measures how faithfully a summary reflects the video using pointwise mutual information between video and text.
Utility Score: Measures how informative the summary is for a downstream task.
Annotation-Free Rejection Sampling: VIBE ranks multiple VLM-generated summaries and selects the one that maximizes grounding and/or utility, supporting efficient human decision-making.

VIBE Framework

VIBE: Mechanism Overview

VIBE adapts the information bottleneck principle to video summarization:

Grounding approximates I(V;T), testing how well the video supports reconstruction of a masked summary.
Utility approximates I(T;Y), testing how well the summary compensates for missing video information to predict a task label.

By maximizing both, VIBE selects concise, task-relevant summaries—without gold labels or retraining.

Results

VIBE: Results Overview

We validate VIBE through user studies with 243 participants across three datasets:

LearningPaper24 (research talk videos)
LongVideoBench (long instructional clips)
SUTD-TrafficQA (short traffic clips)

Correlation between Accuracy, Grounding, and Utility Scores

Key findings:

Accuracy Gains: VIBE-selected summaries boost task accuracy by up to 61.23%.
Speed Improvements: Response time drops by up to 75.77% compared to raw video.
Efficiency: VIBE achieves lower inverse efficiency scores (time/accuracy), showing superior speed–accuracy trade-offs.
Scalability: Grounding works in a fully self-supervised manner, while utility correlates strongly with higher task accuracy.

Impact

VIBE reframes video caption evaluation from a human decision support perspective. Unlike reference-based metrics, it scales to unseen data, works with black-box VLMs, and requires no human annotations. This makes VIBE a practical plug-in for improving video summarization in real-world settings: from scientific video search to public safety monitoring.

FairSynergy: Fair Resource Allocation for Fleet Intelligence

Wed, 10 Sep 2025 00:00:00 +0000

By Oguzhan B.

FairSynergy: Fair Resource Allocation for Fleet Intelligence

2025 IEEE Global Communications Conference (GLOBECOM 2025)

TLDR: Give or substitute each extra unit of compute or memory to the agent that benefits most to lift fleet accuracy by up to +25% in inference and +11% in training, with even bigger gains as fleets grow!

Your browser does not support the video tag.

arXiv | Code

Motivation

Intelligent fleets involving machine learning (ML) deployed drones, autonomous cars, or phones rarely look uniform. Some fleet agents have much better on-board compute or memory, while others face harder inputs, such as hard-to-parse images and complex prompts. While cloud assists fleets by allocating its limited resources, it is not really trivial how to allocate for optimal collective performance across the heterogeneous agents. If we split cloud resources evenly, we waste the budget on saturated parts of the accuracy-vs-resource curve while starving steep ones. We ask:

Q1 (Fairness): Where does the next unit of resource raise accuracy the most?
Q2 (Multivariate Utility): How can we substitute multiple types of ML resources available?

FairSynergy Overview

Contributions

We introduce FairSynergy, a novel framework to allocate cloud resources fairly across the intelligent heterogeneous agents.

Diminishing Marginal Returns: We empirically show that learning and inference curves across vision and language models consistently have the same concave pattern: accuracy improves with more resources (e.g., model size, data, compute) but with diminishing returns.
Fair Allocation in ML: With a concave objective, Network Utility Maximization (NUM) is the natural fit. It mathematically implies the condition turns into an intuitive policy: allocate so each agent’s next unit of resource yields the same marginal accuracy gain.
Cobb-Douglas: Like capital and labor in production, compute/model capacity and labeling/data curation both drive accuracy, each with diminishing returns and substitution. We form a multivariate utility capturing this to co-allocate the resources. (Figs. 5–6)

The Law of Diminishing Marginal Returns: Training

The Law of Diminishing Marginal Returns: Inference

FairSynergy Framework

Inference Setting (RTI) Univariate Case (Compute): At short intervals, the framework estimates each agent’s next-unit accuracy gain from extra cloud compute. Give the next unit to the highest gain, repeat until gains are roughly equalized—then reshuffle as conditions change. This hits the fairness/efficiency sweet spot without heavy tuning.
Learning Setting (DL) Bivariate Case (Compute + Labeling Effort): The framework uses the same “next-unit” idea with a quick two-step loop: hold labels fixed and split compute by the one-resource rule; then hold compute fixed and split labeling time by the same rule. A few rounds settle to a stable co-allocation, so compute-hungry agents get cycles and data-hungry agents get labels.
Handling Heterogeneity: Harder tasks show larger early gains, so the allocator leans into them first and naturally rebalances as gains even out. The result is proportional fairness and fleet-level accuracy that scales with more agents and changing workloads—no math knobs to tune.

**Extending Multivariate ML Utility**: Cobb-Douglas Production Function For a Given Capital and Labor

Results

We compare our method to common baselines and standard fair allocation methods:

Fair-Synergy (Ours) allocates compute (and labels) to equalize next-unit accuracy gains per agent using fitted concave utilities.
Random Allocation splits the available compute (and labels) at random among agents.
Uniform Allocation gives every agent the same share of compute (and labels), ignoring local differences.
Classical NUM optimizes a uniform log utility (identical response curves), so allocation follows equalized marginal gains without task-specific reweighting and agent heterogeneity.
Dominant Resource Fairness (DRF) equalizes each agent’s dominant resource share, targeting max–min fairness across resources.
Leximin prioritizes the worst-off first, maximizing the minimum utility, then the next, and so on. It is a stricted form of fairness compared to other methods.

How well does Fair-Synergy perform compared to the benchmarks?

How does Fair-Synergy scale with increasing number of agents?

Key findings:

Fairness-Accuracy Trade-off: We observe for stricter fairness conditions, while the worst case accuracy increases the aggregate accuracy decreases. Hovewer, we achieve the best trade-off by obtaining both the best performance and necessary fairness conditions.
Scalability: As fleets grow, our method converts heterogeneity into higher accuracy unlike (+6% RTI, +30% DL) and Uniform/DRF/Leximin (+13% RTI, +51% DL) at 100 agents.

Impact

Fair-Synergy treats fairness as physics, not philosophy. Fairness means no single agent experiences an increase in its accuracy while reducing the other’s accuracy more. As accuracy is concave, the right thing is to spend cloud resources where marginal gains are steepest and to do so optimize jointly over multiple substitutable resources. A fair allocation is the most efficient allocation because concavity makes “equalize marginal gains” optimal.

Exploiting Distribution Constraints for Scalable and Efficient Image Retrieval

Tue, 01 Apr 2025 00:00:00 +0000

By Mohammad Omama

Exploiting Distribution Constraints for Scalable and Efficient Image Retrieval

Proceedings of the International Conference on Learning Representations (ICLR)

TLDR: Image Retrieval with Foundation Models: Better, Faster, Distribution-Aware!

ArXiv

Project Website

Motivation

Image retrieval is pivotal in many real-world applications, from visual place recognition in robotics to personalized recommendations in e-commerce. However, current state-of-the-art (SOTA) image retrieval methods face two significant problems:

Scalability Issue: State-of-the-art (SOTA) image retrieval methods train large models separately for each dataset. This is not scalable.
Efficiency Issue: SOTA image retrieval methods use large embeddings, and since retrieval speed is directly proportional to embedding size, this is not efficient.

Our research specifically targets these challenges with two crucial questions:

Q1 (Scalability): Can we enhance the performance of universal off-the-shelf models in an entirely unsupervised way?
Q2 (Efficiency): Is it possible to design an effective unsupervised dimensionality reduction method that preserves the similarity structure and can adaptively perform well at varying embedding sizes?

Contributions

To tackle the scalability and efficiency challenges, our work introduces the follwoing novel ideas:

Autoencoders with Strong Variance Constraints (AE-SVC): Addressing scalability, AE-SVC significantly improves off-the-shelf foundation model embeddings through three rigorously enforced constraints: orthogonality, mean-centering, and unit variance in the latent space. We both empirically demonstrate and mathematically validate that these constraints adjust the distribution of cosine similarity, making embeddings more discriminative.
Single Shot Similarity Space Distillation ((SS)₂ D): To tackle efficiency, (SS)₂ D provides dimensionality reduction that preserves similarity structures and further allows embeddings to adaptively scale without retraining. This enables smaller segments of the embedding to retain high retrieval performance, significantly speeding up retrieval.

Methodology

Our proposed approach follows a two-step pipeline:

AE-SVC first trains an autoencoder with the constraints mentioned to enhance the embeddings from foundation models.
The improved embeddings from AE-SVC are then distilled using (SS)₂D, producing embeddings that are both efficient and adaptive at various sizes.

The training process ensures that the resulting embeddings, even at smaller sizes, preserve similarity relationships, making them highly effective for retrieval tasks.

Impact on Cosine Similarity Distribution

Our AE-SVC method profoundly impacts cosine similarity distributions, significantly reducing their variance. Lower variance in similarity distributions correlates with improved discriminative power as we mathematically prove in our paper. Our method shows remarkable benefits, particularly for general-purpose foundation models like DINO, compared to already optimized dataset-specific models such as Cosplace.

Results

Our experimental validation demonstrates:

AE-SVC consistently surpasses baseline PCA methods across multiple datasets, offering an average of 15.5% improvement in retrieval performance.
(SS)₂D, building upon AE-SVC, achieves up to a 10% further improvement at smaller embedding sizes, demonstrating superior performance compared to traditional dimensionality reduction methods like VAE and approaches the theoretical upper bound set by SSD.

This advancement represents a significant step towards more practical, scalable, and efficient image retrieval solutions, enhancing both speed and accuracy.

CSA: Bridging Modalities with Unimodal Power

Fri, 24 Jan 2025 00:00:00 +0000

By Po-han

CSA: Bridging Modalities with Unimodal Power

Proceedings of the International Conference on Learning Representations (ICLR)

TLDR: This post introduces CSA, a novel method for bridging modalities with unimodal power. CSA leverages existing unimodal encoders to create a shared multimodal feature space, achieving competitive results with minimal training data and computational resources.

arXiv link

Code

Introduction

In the world of artificial intelligence, multimodal models like CLIP have shown impressive capabilities in tasks ranging from image classification to cross-modal retrieval. However, these models often demand enormous amounts of training data, which can be a significant challenge. What if we could achieve similar results with far less data and computation? That is the question that led us to develop Canonical Similarity Analysis (CSA).

The Challenge of Multimodal Learning

Multimodal models aim to create a shared feature space where data from different modalities (like images and text) can be compared. The traditional approach involves training models on massive datasets of paired multimodal data. For example, the original CLIP model was trained on 400 million image-text pairs, requiring significant computational resources. This approach also faces challenges with data quality, as internet-sourced data can be noisy or mislabeled.

Our Approach: Leveraging Unimodal Encoders

We hypothesized that we could build powerful multimodal encoders by leveraging existing, well-developed unimodal encoders. Unimodal encoders, like DINO for images and GTR for text, are trained on single modalities and require significantly less data than their multimodal counterparts. Our idea is to map the features from these unimodal encoders into a shared multimodal feature space using limited data.

System Plot:

CSA: System Overview

The CSA system works by first using pre-trained unimodal encoders to extract features from different modalities, such as images and text. These encoders, which have already been trained on large datasets of single-modality data, create a representation of the input in a feature space specific to their modality. Then, Canonical Correlation Analysis (CCA) is applied to these unimodal features. CCA identifies the dimensions of each unimodal feature space that are most correlated with each other. It finds linear transformations that project the unimodal features into a shared, lower-dimensional space. Finally, CSA calculates a weighted cosine similarity between the transformed features of different modalities, using the correlation coefficients derived from CCA as weights. This similarity score is used for various downstream tasks, such as classification and retrieval. The hyperparameter s controls how many dimensions are considered when calculating the weighted cosine similarity, allowing for a trade-off between noise reduction and retention of information. The entire process is achieved without any training of neural networks, making it highly efficient.

Introducing Canonical Similarity Analysis (CSA)

To achieve this, we developed Canonical Similarity Analysis (CSA). CSA uses two unimodal encoders to encode data into unimodal features, and then projects these features into a joint multimodal feature space. The core of CSA is its method for mapping unimodal features to a multimodal space and a novel similarity function for replicating CLIP’s similarity score. We use Canonical Correlation Analysis (CCA) to find the bases in each unimodal feature space that maximize correlation. This process involves a matrix decomposition without the need for training neural networks, making CSA computationally efficient. We then calculate a weighted cosine similarity in this space. By discarding information from less correlated bases, CSA focuses on the most relevant multimodal information. How CSA Works

Unimodal Encoding: First, we use pre-trained unimodal encoders to extract features from each modality.
CCA Mapping: Next, we use CCA to find the optimal linear transformations that map these features into a common space. CCA identifies the dimensions of the unimodal feature spaces that are most correlated and obtain the correlation coefficients.
Weighted Cosine Similarity: We then calculate a weighted cosine similarity score to measure the similarity of two multimodal data points, using the correlation coefficients as weights.

Data Efficiency and Performance

One of the most remarkable features of CSA is its data efficiency. In our experiments, CSA matched or exceeded the performance of CLIP in image classification, mislabeled data detection, and misinformation detection while requiring $50,000$ times less paired multimodal data for fine-tuning.

On ImageNet classification, CSA only needs $35,000$ training samples to match CLIP’s performance.
In Leafy Spurge classification, CSA outperformed CLIP with only $800$ training images.
CSA also showed superior performance in detecting mislabeled ImageNet images compared to CLIP, ASIF, and LLaVA.
We tested CSA on detecting misinformative news captions from the COSMOS dataset, and it outperformed both CLIP and ASIF.
We also showed that CSA is robust to noisy data. Even with $50\%$ of training labels randomly shuffled, CSA maintained a high level of accuracy.
We’ve demonstrated CSA’s versatility with modalities beyond image and text, including audio and text, paving the way for new modality pairs, such as lidar and text.

Limitations and Future Directions

While CSA is effective, it currently focuses on bimodal data, unlike models like ImageBind, which can handle six modalities. In addition, the performance of CSA relies on the quality of the unimodal encoders used. Our future work will explore:

Extending CSA to more than two modalities.
Understanding the relationship between the training set size and performance.
Fine-tuning the unimodal encoders for improved feature spaces.
Applying CSA to intramodal data, such as multi-view images.

Conclusion

CSA offers a compelling solution to the challenges of multimodal learning. By leveraging the power of unimodal encoders and using a novel similarity analysis method, CSA achieves competitive results with minimal training data and computational resources. We believe that CSA opens up new possibilities for efficient and robust multimodal machine learning towards binding emerging modalities.

Fleet Supervisor Allocation: A Submodular Maximization Approach

Thu, 19 Sep 2024 00:00:00 +0000

By Oguzhan A.

Fleet Supervisor Allocation: A Submodular Maximization Approach

Link to paper

Introduction

Today, robotic fleets are deployed across diverse industries, performing tasks ranging from autonomous driving to healthcare and package delivery. These robots are often trained in simulated environments and often struggle to adapt to real-world scenarios, making continuous data collection crucial for high performance.

A common method for training robots is Imitation Learning (IL), where humans supervise robots to perform tasks. However, a significant challenge arises when the number of robots far exceeds the number of human supervisors, necessitating assigning limited humans to the most informative robots..

Additionally, human supervision is often provided through teleoperation over a network, especially when robots are geographically distributed. This introduces challenges related to network reliability which can cause interruptions in data collection.

To address these challenges, we introduce Adaptive Submodular Allocation (ASA), a novel approach that optimizes human supervision in multi-robots systems by dynamically allocating supervisors based on informativeness and network reliability of each robot.

Problem: The Supervisor Allocation Problem

Supervisor Allocation Problem

We consider a fleet of $N_{robot}$ robots, operating with shared policy $\pi_R^t$ at time $t$, where each robot has its own state representation $s_i^t \in S $. In each time step, human supervisors with policy $\pi_H$ can be allocated to the robots based on the allocation policy $\pi_A$. The allocation is subject to the network connectivity $c_i^t$, which can be unreliable.

Here our objective is to find an allocation policy $\pi_A$ that maximize the Return on Human Effort (RoHE) metric, which measures the performance of the robot fleet normalized by the human effort. This metric is proposed by Hoque et al. (2023) for Interactive Fleet Learning problem, and we extend it to account for uncertain network connectivity. Our objective can be formulated as:

\[\max_{\pi_A \in \Omega} \mathbb{E}_C \left[ \frac{N_{\text{human}}}{N_{\text{robot}}} \frac{\sum_{i \in I} \sum_{t=0}^T r(s_i^t, a^t_i)}{1 + \sum _{t=0}^T \| \pi_A(C^t, s^t, a^t, I) \|} \right]\]

Here $T$ is the time horizon, $r(s_i^t, a^t_i)$ is the reward function, and $\Omega$ is the set of all allocation policies. The RoHE metric evaluates the efficiency of human supervision by considering both the rewards and human effort.

Our Approach: Adaptive Submodular Allocation (ASA)

Our approach, Adaptive Submodular Allocation (ASA), leverages submodular maximization to optimize the allocation of human supervisors to robots.

Why Submodular Maximization?

Submodular functions are well-suited for optimization problems where diminishing returns occur. In our context, the more similar the states of selected robots,the less new information we gain by selecting another robot from a similar state. Submodularity helps us formalize this diminishing returns principle.

A Toy Example of Why Submodular Maximization

In the figure above, the blue contour represents uncertainty levels, with darker shades indicating higher uncertainty. Traditional score-based allocations (purple crosses) tend to select overlapping areas of high uncertainty resulting in redundant data collection. In constrast, submodular maximization (yellow plus signs) selects a more diverse set of robots, reducing redundancy while still capturing high uncertainty areas, leading to more informative data collection and better performance.

How ASA Works

We now explain how ASA works. We will first define the submodular objective function and then describe how ASA leverages this function to address the Fleet Supervisor Allocation problem.

Submodular Objective Function

Adaptive Submodular Allocation (ASA) leverages stochastic submodular maximization to address the Fleet Supervisor Allocation problem. The objective $f_{C^t}(X)$ captures the value of supervising a set of robots $X$ balancing diversity and informativeness:

\[f_{C^t}(X) = \sum_{i \in I} \max_{j \in X} c^t_j, M^t_{j,i}.\]

Here, $c^t_j$ represents the connection reliability of robot $j$ and $ M^t_{j,i} $ measures the value of supervision of robot $j$ for robot $i$. Here we define $ M_{j,i} $. However, the objective function $f_{C^t}(X)$ is stochastic due to the uncertain network connectivity. Instead, we optimize the expected value of the objective function $F(X) = \mathbb{E}_{C^t} [f_{C^t}(X)].$ Hence the problem becomes: $\max_{X \subseteq I} F(X),$

\[\text{subject to:} \; |X| \leq N_{\text{human}},\]

Adaptive Submodular Allocation (ASA) Policy

We now present our ASA policy which uses a greedy algorithm to iteratively select robots with the highest marginal gain based on the estimated expected value funtion, $\widehat{F}$. The algorithm is as follows:

Adaptive Submodular Allocation Algorithm

Starting with an empty set of selected robots, the algorithm iteratively selects the robot that provides the highest marginal gain in expected value function $\widehat{F}$. If the marginal gain falls below a preset threshold or the number of selected robots reaches the number of human supervisors, the algorithm terminates.

Here the key component is the expected value function $\widehat{F}$ which estimates the expected value of the submodular objective function $F$. Initially, $\widehat{F}$ is set to the connection probabilities and updated whenever a robot’s connection success is observed. This updating ensures ASA adapts to real-time network conditions.

The estimated value function $\widehat{F}$ is defined as:

\[\widehat{F}(X) = \sum_c f_c(X) \widehat{p}_\xi,\]

where $\widehat{p}_\xi$ represents the estimated connection probability of realization $ \xi $. The algorithm updates $\widehat{p}_{\xi} $ after each allocation if observations on the allocations are available, dynamically improving allocation decisions.

This policy is shown to approximate the optimal solution with a factor of $1 - 1/e$, providing robust and scalable performance across various network environments.

Experimental Results

We evaluated ASA in a simulated environment of 100 robots under various network conditions, including real-world 5G data. ASA outperformed benchmark methods, improving RoHE by up to 3.37× compared to traditional approaches, particularly in environments with unstable connectivity.

ASA Experimental Results

In the figure above, each column represents a different environment, while each row corresponds to a network configuration. We can see that ASA consistently outperforms other policies in terms of RoHE, demonstrating the effectiveness of our approach.

Conclusion

We introduce the ASA policy, offering a robust solution to the Fleet Supervisor Allocation problem. Our policy leverages stochastic submodular maximization to effectively balance human supervision and task performance across diverse environments. Through smarter allocation, ASA improves both fleet efficiency and data collection, laying the groundwork for more scalable autonomous fleets.

PEERNet: Benchmarking Networked Robotics on Wifi, 5G, and Beyond

Tue, 17 Sep 2024 00:00:00 +0000

By Aditya N.

PEERNet: Benchmarking Networked Robotics on Wifi, 5G, and Beyond

Proceedings of the International Conference on Intelligent Robots and Systems (IROS)

TLDR: This post introduces PEERNet, a Python package for real-time benchmarking of networked robotic systems. It provides concise and modular methods for performance analysis of the entire system stack.

link to paper: PEERNet: Benchmarking Networked Robotics on Wifi, 5G, and Beyond

PEERNet is an end-to-end profiling tool for networked robotics. It excels in ease of deployment across various platforms and offers extensive benchmarking capabilities.

Why benchmark networked robotics?

Consider a mobile robot running a Machine Learning (ML) pipeline for perception, localization, and path planning. Here and in many other practical scenarios, offloading computationally expensive ML tasks to a cloud server over public networks is a viable optimization.

Introducing offloaded computation brings up several system-design questions which require hard statistics and real data to answer:

Which ML model maximizes decision accuracy while minimizing end-to-end latency?
How much better is the end-to-end latency when using a 5G network as opposed to a 4G LTE network?
How does data compression affect end-to-end latency in offloaded inference? Is the overhead of compression worth the faster networking latency? Does that answer change when using a 5G network as opposed to LTE or WiFi?

More generally, real-time benchmarking is required for:

1.Deployment in the real world: Before a robotic system can be deployed, decisions need to be made on hardware, ML models, networking protocols, etc.–these decisions must be informed by real-world statistics.

2.Debugging exising systems: Finding and removing bottlenecks in existing robotic systems requires hard numbers. ML model latency, network delay, or even hardware limitations could all be a system’s bottleneck.

3.Optimizing networked robotics: Many novel techniques for optimizing data sharing, offloading policies, and distributed ML are in active development. Assessing the performance of new algorithms on real hardware is essential before optimizations can be applied.

What makes benchmarking non-trivial?

Benchmarking networked robotics is fundamentally more challenging than simply timing ML models and testing sensors. The incorporation of networks into robotic systems significantly complicates the benchmarking problem by introducing:

1.Stochastic delay: Network delays are highly stochastic and system dependent. Delays are hard to predict and often change significantly from one setup to another.

2.Network Timing and Asymmetric Delay: Profiling networking operations by itself is difficult due to imperfect clock synchronization. Round-trip times, while commonly used, are of little use in networked robotic systems, where devices rarely upload and download similar amounts of data. For example, our mobile robot from earlier may upload gigabytes of images and lidar scans while downloading only bytes of labels. Hence, round-trip delay fails to provide any meaningful insight into this system.

Introducing PEERNet

We present PEERNet, the Profiler for End-to-End Real-time Networked Robotics. PEERNet is a highly modular and extensible package for profiling networked robotic systems, complete with one-way network delay estimation. PEERNet interfaces with industry standard hardware and software such as Nvidia embedded GPUs, Robot Operating System (ROS), and Zero MQ (ZMQ). PEERNet exposes a CLI for rapid benchmarking of offloaded inference systems, and we demonstrate PEERNet’s ease of use and versatility.

At a high level, PEERNet focuses on profiling networked robotics by looking at the life-cycle of data. When sensors sample data, serializable logging metadata is attached, and propagated through the system normally. This metadata includes all necessary information for tracking and profiling sampling time, ML inference latency, and one-way network delay.

Experiments

We illustrate and validate PEERNet by implementing and profiling three networked robotic systems. All of our experiments are conducted on physical hardware and live wireless networks, demonstrating that PEERNet is modular, robust, and capable of precisely profiling various systems.

Benchmarking offloaded inference

Our first experiment demonstrates how PEERNet can be used to profile and understand offloaded image classification, such as in our mobile robot example. By precisely quantifying latency costs across local and cloud hardware and for various model sizes, PEERNet allows for intelligent device/model selection.

PEERNet precisely quantifies inference costs of diffent ML classifiers at the edge and in the cloud.

Identifying non-intuitive behavior in Vison Language Models

In our second experiment, we use PEERNet to explore inference with Vision Language Models (VLMs) at the edge, an emerging task in robotic pipelines. We show that PEERNet is capable of identifying non-intuitive behaviors in VLM inference, such as a bimodal output token distribution.

PEERNet identifies that a Vision Language Model (VLM) returns a strongly bimodal distribution of caokens, speeds, and inference caimes.

Optimizing teleoperation of a robot arm

Our final experiment applies PEERNet to benchmark a full teleoperation pipeline. We explore all combinations of edge device, cloud compute, compression, and network on a teleoperated robot arm performing a pick-and-place task. We demonstrate that profiling with PEERNet solves the hardware selection problem, and is easily applicable to existing robotic pipelines.

Looking to try PEERNet out?

Our code is open-source! You can find it at our repo.

ControlPay: An Adaptive Payment Controller for Blockchain Economies

Sun, 18 Aug 2024 00:00:00 +0000

By Oguzhan A.

ControlPay: An Adaptive Payment Controller for Blockchain Economies

Proceedings of the International Conference on Blockchain (ICB) 2024

Introduction

In recent years, decentralized platforms like Helium, BitTensor, and FileCoin have reshaped how services such as wireless networks, machine learning, and storage are provided. These platforms operate on a token-based economy, rewarding service providers with cryptocurrency tokens. However, managing these economies is a complex challenge, particularly when token supply strategies fail to account for market fluctuations. The collapse of Luna, for example, highlighted the devastating effects of inadequate token price stabilization, resulting in massive financial losses for users and raising concerns about the sustainability of blockchain economies. Our new paper, ControlPay: An Adaptive Payment Controller for Blockchain Economies, offers a solution by introducing a control-theoretic approach to stabilizing blockchain token economies.

The Problem

Decentralized platforms rely on native tokens to incentivize service providers. These tokens are essential for maintaining and expanding network infrastructure, but current approaches to managing token supply—like the “burn-and-mint” model—are often rigid and vulnerable to market volatility. For instance, maintaining a fixed exchange rate between tokens and USD can lead to the depletion of reserves, destabilizing the entire network. While some platforms have implemented adaptive mechanisms, these are often reactive heuristics lacking the predictive capabilities needed to maintain stability in the long term.

Additionally, the presence of strategic behavior by market participants further complicates the management of token economies. Token holders may act strategically to maximize their profits, leading to price fluctuations and network instability. Traditional control mechanisms are ill-equipped to handle these challenges, resulting in suboptimal token supply strategies and increased network costs.

Our Approach: ControlPay

ControlPay: Our Payment Controller for Blockchain Economies

To address these challenges, we introduce ControlPay, an adaptive payment controller for blockchain economies. ControlPay leverages advanced control theory and game-theoretic models to optimize token supply and stabilize token prices. Unlike traditional approaches, ControlPay is predictive, using forecasts of consumer demand and supplier growth to dynamically adjust token supply. By modeling the token economy as a dynamic system and incorporating strategic behavior, ControlPay provides a comprehensive solution for managing decentralized token economies.

Specifically, ControlPay combines the following key features:

Modeling the Token Economy as a Dynamic System: We develop a dynamic model of the token economy, integrating key variables such as token supply, price, reserves, income, and demand.
A Model Predictive Control Approach: We design a control mechanism based on model predictive control (MPC) to manage token supply and price, optimizing control actions to stabilize the token economy.
Strategic Behavior and Game Theory: We model the market as a Stackelberg game between token holders and the payment controller, anticipating strategic behavior and optimizing token supply to maximize overall welfare within the network.

In the following sections, we detail our approach to modeling the token economy, designing ControlPay, and incorporating strategic behavior through game theory.

Modeling the Token Economy as a Dynamic System

We begin by modeling the token economy as a dynamic system, integrating various factors such as token supply, price, reserves, income, and demand. Specifically, we consider the following components:

State Variables:

Circulating Supply $S_t$: The total number of tokens in circulation.
Reserves $R_t^{USD}$, $R_t^{Tok}$: The reserves in USD and tokens.
Price $p_t^{Tok}$: The price of the token.

Control Variables:

Token Payments $u^P_t$: The amount of tokens paid to service providers.
Token Buybacks $u^B_t$: The amount of tokens bought back from the market to stabilize the price.
Incentive Price $\Delta p_t$: The price difference between the token’s market price and the buyback price.

External Factors:

Income $Inc_t$: The income generated by the platform.
Demand $D_t$: The demand for tokens.

Using these variables, we define the dynamics of the token economies as a set of equations that describe how the state variables evolve over time. More specifically, we model the change in circulating supply, reserves, and price as:

\[S_{t+1} = S_t + u^P_t - \frac{u^B_t}{p_t^{Tok} +\Delta p_t},\] \[R_{t+1}^{USD} = R_t^{USD} - u^B_t + Inc_t,\] \[R_{t+1}^{Tok} = R_t^{Tok} - u^P_t + \frac{u^B_t}{p_t^{Tok} +\Delta p_t},\] \[p_{t+1}^{Tok} = \frac{D_{t+1}}{S_{t+1}}.\]

Here, circulating supply is updated based on token payments and buybacks, reserves are updated based on income and buybacks from the market, and the price is updated based on the demand and circulating supply of tokens. With these equations, we define the state $x_t$, control $u_t$, and forecasted external factors $s_t$ as follows:

\[x_t = [S_t, R_t^{USD}, R_t^{Tok}, p_t^{Tok}],\] \[u_t = [u^B_t, u^P_t, \Delta p_t],\] \[s_t = [\hat{D_t}, \hat{Inc_t}].\]

Finally, we can express the dynamics of the token economy $f$ as a function of the state, control, and external factors:

\[x_{t+1} = f(x_t, u_t, s_t) = \begin{bmatrix} x_t(0) + u_t(1) - \frac{u_t(0)}{x_t(3)+u_t(2)} \\ x_{t}(1) - u_{t}(0) + s_{t}(1) \\ x_{t}(2) + \frac{u_t(0)}{x_t(3)+u_t(2)} - u_{t}(1) \\ \frac{s_{t+1}(0)}{x_{t+1}(0)} \end{bmatrix}.\]

ControlPay: A Model Predictive Control Approach

With the token economy modeled as a dynamic system, we can now design a control mechanism to manage token supply and price. Our approach, ControlPay, is based on model predictive control (MPC), a control strategy that uses a model of the system to predict future behavior and optimize control actions. In the context of token economies, ControlPay uses forecasts of consumer demand and supplier growth to optimize token supply dynamically, providing a robust solution that can significantly reduce network costs while maintaining price stability. To achieve this we define the following optimization problem:

where $J$ is the cost function, $x_{t+1} = f(x_t, u_t, s_t)$ represents the dynamics of the token economy, and $\mathcal{X}$ and $\mathcal{U}$ are the state and control constraints, respectively. By solving this optimization problem, ControlPay can adaptively adjust token payments and buybacks to stabilize the token price and ensure the sustainability of the network.

An example cost function $J$ could be:

\[J(x_0, u_{0:H-1}) = \sum_{t=0}^{H-1} \begin{bmatrix} \beta_1 \\ \beta_2 \\ \beta_3 \end{bmatrix}^\top \begin{bmatrix} \left( x_{t+1}(3) - \bar{x}_{t+1}(3) \right)^2 \\ \left( u_t(0) - \bar{u}_t(0) \right)^2 \\ \left( u_{t}(1) - \bar{u}_t(1) \right)^2 \end{bmatrix},\]

where $\beta_1, \beta_2, \beta_3$ are weighting factors. This cost function penalizes deviations of the token price from the USD price, control actions, and changes in control actions, ensuring that the token price remains stable and control actions are smooth.

Unlike traditional approaches like PID controllers, which are reactive, ControlPay is predictive. It uses forecasts of consumer demand and supplier growth to dynamically optimize token supply, providing a robust solution that can significantly reduce network costs while maintaining price stability.

Strategic Behavior and Game Theory

A unique aspect of our approach is the incorporation of strategic behavior by market participants. We model the market as a Stackelberg game between token holders and the payment controller. Token holders, seeking to maximize the value of their tokens, may act strategically, affecting the stability of the token economy. ControlPay anticipates these actions and adjusts its strategy accordingly, ensuring that both token holders and the network benefit.

In this game, the payment controller acts as the leader, setting incentive price and buybacks to stabilize the token price. Token holders, acting as followers, adjust their behavior based on the controller’s actions. By modeling the token economy as a Stackelberg game, ControlPay can anticipate strategic behavior and optimize token supply to maximize overall welfare within the network. We also show that ControlPay achieves the Stackelberg equilibrium, where the payment controller maximizes the utility of token holders while minimizing control costs.

Experiments and Results

We validated ControlPay through extensive simulations using both synthetic and real-world data from the Helium network. We compared ControlPay with traditional methods like the “burn-and-mint” (No Control) model, which simply buys back and pays out the same amount of tokens, and the proportional integral derivative (PID) controller, which adjusts token payments and buybacks based on past data on token price and reserves. Here are some example simulation results for comparison:

Some example simulation results for comparison.

The two left columns show trajectories using real Helium data, demonstrating that ControlPay (green) closely follows the reference (purple), while the reactive PID (blue) is highly oscillatory. The right columns validate our approach’s adaptability using synthetic data, showing how ControlPay adjusts to a decreasing price trajectory. Unlike PID, ControlPay avoids overshooting, maintaining price stability even as demand shifts.

Below are the network cost and control effort comparisons for the three methods:

Network cost and control effort comparison.

ControlPay outperforms traditional methods by reducing network costs by up to 2.4× while using similar control effort. This demonstrates the effectiveness of our approach in stabilizing token prices and reducing network costs.

Additionally, we conducted experiments to evaluate ControlPay’s performance in environments with strategic behavior. Our results show that ControlPay effectively balances the interests of token holders and the payment controller, maximizing overall welfare within the token economy. The figure below illustrates how ControlPay finds a game strategy to sell only $\alpha^*_t$ tokens to maximize node utility and minimize control cost, compared to simply holding ($\alpha_t = 0$) or selling all tokens ($\alpha_t = 1$).

Network cost and utility of token owners.

Conclusion

As blockchain infrastructures continue to grow, the need for adaptive control mechanisms like ControlPay will become increasingly critical. By integrating advanced control theory and game-theoretic models, ControlPay provides a comprehensive solution for managing decentralized token economies. We are excited to present our findings at the International Conference on Blockchain (ICB) 2024 and look forward to further discussions on how ControlPay can help shape the future of blockchain economies.

Safe Networked Robotics with Probabilistic Verification

Sun, 07 Jul 2024 00:00:00 +0000

By Sai Shankar Narasimhan.

Link to paper

Motivation

Recently, an increasing number of robotic applications have adopted remote assistance through teleoperation, ranging from remote manipulation for surgery to emergency takeover of autonomous vehicles. Teleoperation is even used to control food delivery robots from command centers hundreds of miles away. What has contributed to this trend?

We identify the following two crucial reasons:

Mobile robots are constrained by their onboard computing power and cannot run powerful deep neural networks (DNNs) for perception and control tasks. Hence, it is preferred to offload sensor observations to a remote server via wireless networks, where these powerful DNNs can be run.
Autonomous robots cannot handle all possible edge cases. While encountering a new scenario (out of the training distribution), it is necessary to have a human-in-the-loop to take over control.

Though teleoperation has a lot of potential benefits, there are practical concerns that need to be addressed. For example, while using public communication networks, stochastic communication delays could potentially lead to the violation of key safety properties.

Therefore, in this work, we ask the following key question: How do we ensure safe networked control over wireless networks with stochastic communication delays? This blog will briefly explain our proposed solution and experimental results, including a real-world demonstration of safe networked control using F1/10th cars while transmitting sensor data through our university’s public Wi-Fi network.

To motivate the requirements of a safe networked control system, let us take a look at our proposed solution, shown in the image below.

Our proposed safe networked control approach

We implemented our proposed solution on a real-world leader-follower setup. The leader is uncontrolled and can exhibit stochastic motion. The follower is controlled remotely by a teleoperator or using a DNN. The sensor observations are transmitted to the remote server or cloud through a public Wi-Fi connection (step 1). The teleoperator or the DNN generates a control command, which is transmitted back to the follower (step 2). We propose a shield that restricts or modifies an unsafe control command based on its communication delay magnitude (step 3). Finally, the follower executes the “safe” control command (step 4).

What do we need to achieve safe networked control?

To develop a safe networked control system, as shown in the figure above, we require the following:

Ability to accurately model the interaction between the mobile robot and the environment in the presence of delay. Previous works make assumptions about the delay transitions, such as the delay is constant and non-decreasing. However, we end up with a very conservative model of the interaction with these assumptions.
Ability to trade off safety for efficiency according to the user's needs. This is important as we are dealing with a system with stochastic delays. Optimizing for absolute safety would translate to optimizing for the worst-case delay, which might occur very rarely.

Modeling the interaction between the robot and the environment in the presence of delay

Typically, Markov Decision Processes (MDPs) are used to model the interaction between a robot and its environment. To extend MDPs to the networked control setting, we developed Delayed Communication Markov Decision Processes (DC-MDP) without any conservative assumptions about the delay transitions.

Our key intuition is that by augmenting the states of the basic MDP (model of the interaction in the absence of delay) with fixed-length action buffers containing the number of actions corresponding to the actual delay, we can represent system states with arbitrary delays. Additionally, given the delay transition probability, we can accurately obtain the transition probability between the system states for two consecutive time steps with arbitrary delays. We refer the readers to Section 5.A of the manuscript for further details.

Ensuring Safety

We use Linear Temporal Logic (LTL) to formally represent our desired notion of safety for the system as safety specifications. We refer the readers to Section 3 of our manuscript for a detailed discussion on LTL and safety specifications. Given an MDP and a safety specification, the preferred approach to ensure safety is shielding. For any given system state, shielding allows those actions to be executed for which the maximum safety probability is 1.0. This ensures that the system is safe. Additionally, executing shielded actions ensures that the system always remains in a safe state. The literature on shielding is vast and will not be covered in this blog.

Achieving Safety-Efficiency Trade-Off

With the DC-MDP and the shield for the required safety specification, the naive solution is to apply shielding to the DC-MDP and ensure absolute safety for the networked control system. However, we note that this adversely affects the system’s task performance. This is attributed to the fact that ensuring absolute safety always considers the worst-case scenario, which is the maximum delay case. However, the maximum delay case is a very rare occurrence. Hence, we propose ε-shields that can trade off safety for task efficiency. We refer the reader to Algorithm 1 in our manuscript for the definition and synthesis of ε-shields. Intuitively, ε is a value between 0 and 1, and increasing ε increases safety. Our proposed ε-shields can guarantee any desired safety probability. This flexibility allows the user to trade off safety to achieve higher task efficiency.

Experimental Results

We test our approach on the following two simulation environments:

Leader-Follower Simulation Setup: The follower should get as close as possible to the leader while maintaining a safe distance. The state information contains the relative distance and the relative velocity.
Grid World Simulation Setup: The robot has to traverse an 8x8 grid from start to goal without colliding with an adversary. An episode is declared a win if the robot reaches the goal, a loss if the robot collides with the obstacle, and a tie if the robot does not reach the goal.

Now, we will describe the two main takeaways from our experiments with these two simulation environments.

The set of safe states expands for the DC-MDP. Since the DC-MDP accurately models the robot-environment interaction without any conservative assumptions on the delay transitions, we observe more safe states when compared to the previous models with conservative assumptions on delay transitions, such as constant maximum delays. This directly translates to aggressive robot behavior or higher task efficiency. In the figure shown below, note that for the same value of maximum allowable delay, the set of safe states for DC-MDP (Random Delay) is much larger than that obtained with the constant maximum delay assumption.
ε-shields trade off safety for efficiency. Our experiments show that we can decrease the value of ε to lower the achievable safety probability, thereby achieving higher robot task efficiency. This is shown by a reduced distance from the leader in the leader-follower environment and an increased number of wins in the grid-world environment.

The left figures show the expansion in the set of safe states when we use DC-MDP. The right figures show empirically how our proposed ε-shields gracefully trade off safety for efficiency.

We show similar results with the real-world demonstration (check the figure below). In the absence of shielding, we observe safety violations due to the communication delay while using our university’s public Wi-Fi network for sensor data transmission. Additionally, note that the distance between both cars is lower in the presence of ε-shields (Random Delay), highlighting better task efficiency.

The figure highlights the efficacy of our safe networked control approach in a real-world leader-follower setup. Observe the safety violations in the absence of shielding and higher task efficiency (lower distance between the cars) while using our proposed DC-MDP.

Conclusion

Our work, “Safe Networked Robotics with Probabilistic Verification”, provides a novel approach to ensure safety for networked control systems. Through multiple simulation and real-world experiments, we demonstrate how our approach can gracefully trade off safety for efficiency while dealing with stochastic communication delays.

Decentralized Data Collection for Robotic Fleet Learning: A Game-Theoretic Approach

Mon, 26 Feb 2024 00:00:00 +0000

By Oguzhan A.

Robotic Fleet Data Collection Through Game Theory

Link to paper

Motivation

In the rapidly evolving landscape of autonomous vehicles (AVs) and robotics, the efficient collection and utilization of data are crucial. Traditional data collection methods often lead to redundancy and inefficient use of bandwidth and storage. To deal with these challenges, we propose a strategic game between robots to sample data points to optimize data sampling, aiming to minimize the gap between current and target data distributions. This game-theoretic solution reaches a Nash equilibrium with only one message passing between AVs, offering an optimal strategy for data collection across robotic fleets.

The Challenge

AVs traditionally operate in isolation, prioritizing data collection based on individual, often greedy strategies. This isolation can lead to resource wastage and inefficiency. Coordination among AVs is essential to ensure the selection of the most valuable data points, avoiding redundancy and optimizing the collective data collection effort.

A motivating toy example with 2 robots and 2 classes

In the figure above, we show a toy example of why data collection needs to be coordinated. Here there are two robots observing data with only 2 classes. The axes represent the number of data points for each class. Our goal is to reach a target distribution (blue cross) where each class has 120 data points, represented by (120,120). The robots start at (0,0) with no data points in the cloud. The possible combinations that can be uploaded from robots 1 and 2 are shown as the shaded feasible action spaces (yellow and purple). This shaded region is determined by the robot’s local data distribution and vision model accuracy. Greedy (black) individually calculates the projection of the target distribution onto each robot’s feasible action space, but the sum of actions may not be optimal leading to a high error (red). However, Oracle accounts for the two robots’ action spaces and this minimizes the error between the target dataset and the sum of actions (grey).

A Game-Theoretic Solution

We frame the data collection problem as a potential game between networked AVs. Then, we show that this game reaches Nash equilibria based on the best-iterated response policy after only one message passings between the AVs. This equilibrium represents the collective optimization of data collection strategies, ensuring an optimal solution for the entire fleet.

How Interactive Policy Works

Game-Theoretic Data Collection

Each step in our cooperative algorithm is numbered in blue. First, each AV i observes a sequence of images in round r of data collection (step 1). Then, it classifies each image with a local vision model (step 2). Then it samples a limited set of images according to its action policy which governs what distribution of data points to upload. Crucially, the action to what data points to upload is chosen cooperatively with other AVs using a distributed optimization problem (step 3). Next, each AV transmits its local cache of data points to the cloud (step 4). The current cloud dataset is updated with the newly uploaded data points (step 5). The combined cloud dataset can be used to periodically retrain new model parameters (step 6), which are then downloaded by the AVs (step 7). All AVs share a goal of minimizing the distance between the collected cloud dataset and the target dataset.

Experimental Results

We show that our Interactive approach is able to achieve superior performance to the traditional distributed approach and achieves the same performance as an optimal Oracle policy. Our Interactive approach achieves performance improvements by up to 21% in classification accuracy compared to the standard benchmarks.

Experimental Results

In the figure above, each row represents a different dataset. Column 1: As expected by our theory, Interactive minimizes the L2-norm distance (optimization objective, y-axis) better than Greedy and matches the omniscient Oracle. Column 2: Clearly, Interactive achieves a much more balanced distribution of classes (target distribution is uniform) than benchmarks. Column 3: Since Interactive achieves a more balanced dataset, this experimentally translates to a higher DNN accuracy (statistically significant) on a held-out test dataset.

Conclusion

Our work “Decentralized Data Collection for Robotic Fleet Learning” introduces a novel, theoretically grounded, cooperative data sampling policy for networked robotic fleets, which converges to an Oracle policy upon termination. Additionally, it converges in a single iteration under mild practical assumption, which allows communication efficiency on real-world datasets. Our approach is a first step towards an increasingly timely problem as today’s AV fleets measure terabytes of heterogeneous data in diverse operating contexts.

Fleet Active Learning: A Submodular Maximization Approach

Sun, 25 Feb 2024 00:00:00 +0000

By Oguzhan A.

Fleet Active Learning with Submodular Maximization

Link to paper

Motivation

The recent success and popularity of machine learning (ML) models provide unique opportunities for developing ML models. Today, we can collect data from previously unseen diverse environments, which can be used to increase the robustness and overall performance of these ML models. This is especially true in the context of autonomous vehicles (AVs), where the collection and processing of diverse and informative data are pivotal for enhancing ML models. These models, integral to the vehicles’ perception, prediction, and planning capabilities, thrive on varied datasets that enable them to generalize across different environments. However, the current methods for data collection are not efficient enough to utilize the full potential of the data collection. Addressing this gap, our novel “Fleet Active Learning” framework emerges as a solution to three pivotal challenges:

How do we efficiently gather diverse and informative data in a fleet of robots?
How can we scale data collection methods to accommodate large fleets without overwhelming bandwidth and storage capacities?
How do we optimize the data collection process to enhance the performance of ML models?

The Challenge of Redundancy in Data Collection

When AVs operate independently to collect data, there is a significant risk of redundancy. Multiple vehicles may gather similar data points, which doesn’t add value to the collected dataset. This redundancy not only wastes resources but also fails to enrich the models’ learning with new, diverse information. Given constraints like bandwidth and computational resources, it is crucial for these vehicles to prioritize the collection of unique and informative data points that can substantially improve their ML models’ performance.

A motivating toy example with 3 robots

In the figure above, we show a toy example of why redundancy is a huge problem in data collection in a fleet setting. Here the robots observe 3 different data distributions that are mapped into a 2D space. The red crosses represent the data selected by each robot independently in a distributed manner, while the blue circles represent the data selected by robots interactively. Compared to the distributed approach, the interactive approach selects a more diverse set of data points, covering a broader range of scenarios.

An Interactive Solution through Submodular Maximization

Our work addresses these challenges by introducing a fleet active learning framework that leverages the submodular maximization approach. This approach is adept at handling situations where the value of adding data points diminishes over time - a concept known as diminishing returns. In essence, the method prioritizes data points that offer the most significant incremental value, ensuring that the selected dataset is not just large but also highly informative.

How Fleet Active Learning Works

Fleet Active Learning Framework: A collaborative approach to data gathering

Each robot in the fleet observes a stream of data points and processes them using its neural network to obtain predictions and embeddings. These embeddings and predictions are then used to select an action to sample data points, maximizing a submodular function while considering the previous robots’ actions. The aggregated action is passed to the next robot, repeating the process until all robots have taken an action. At the end of each round, the actions are shared with the cloud, which labels the newly acquired data points and updates the training dataset with the new data points. The model is retrained with the new dataset, and the updated model weights are shared with all robots, which update their model parameters accordingly.

Experimental Results

We show that our decentralized interactive approach is able to achieve superior performance compared to the traditional distributed approach and achieves the same performance as an upper-bound “Centralized” approach. Our FAL framework achieved up to a 25% increase in classification accuracy compared to a completely distributed baseline.

Experimental Results

In the figure above, each column represents a different dataset. Row 1 shows the submodular objective of the cloud dataset across the rounds. Both the Centralized and Interactive policies achieve similar objectives, while the Distributed fails to reach the same level of performance. Row 2 presents the accuracy of the retrained neural networks using the accumulated dataset in each round r. The accuracy plots exhibit a similar trend as the submodular objective, with the Interactive and Centralized policies consistently outperforming other benchmark policies.

Conclusion

Our work “Fleet Active Learning: A Submodular Maximization Approach” introduces a framework for enhancing the capabilities of AVs through smarter data collection methods. By adopting the FAL framework, fleets of autonomous vehicles can efficiently gather diverse data essential for training robust ML models, enabling the development of more robust and better-performing autonomous vehicles.

Online Foundation Model Selection in Robotics

Sat, 17 Feb 2024 00:00:00 +0000

By Po-han

How to Choose the Right Foundation Model for Robotics? Exploring Online Learning for Model Selection

Link to paper

Motivation

The integration of foundation models into robotics marks a new era of innovation, promising to transform the landscape with unprecedented efficiency and capabilities. However, this transition is not without its challenges, particularly in the selection between high-performance, expensive closed-source models and more accessible, albeit potentially less powerful, open-source alternatives. This paper presents a solution to this dilemma, introducing an online learning framework that adeptly navigates the intricate trade-offs between cost, efficiency, and performance of both closed-source and open-source models.

The Solution

The solution’s backbone is a user-centric online model selection pipeline (as shown below), utilizing a pre-trained open-source encoder to simplify complex data into low-dimensional embeddings for an online learning algorithm. This strategy not only avoids the steep costs associated with collecting data from closed-source models for supervised-learning methods but also leverages the adaptability of online learning to select models, considering factors such as performance, time efficiency, and financial limitations.

Online model selection pipeline

In the pipeline, a user sends their intentions in natural language and images to a model selected from a range of available options. To do so, an encoder first processes the language and visual inputs to extract features. These features help an online learning algorithm select the suitable model that maximizes accuracy and minimizes response latency and monetary costs. The algorithm should avoid selecting models that execute incorrectly, marked with red crosses. The above examples come from the ALFRED dataset.

Theory and Experiments

A thorough theoretical analysis underpins the superiority of contextual algorithms over non-contextual counterparts in model selection, demonstrating how the proposed method effectively balances these factors to make informed decisions. The experimental results further corroborate this, showcasing a significant improvement in task success rates across various language-based robotic applications, highlighting up to a 14% increase in efficiency compared to conventional methods.

Expanding on these findings, the paper delves deeper into the nuances of online learning algorithms, exploring their potential to revolutionize model selection processes. It examines the scalability of the proposed framework, considering its applicability across a broad spectrum of robotic tasks. The discussion extends to the implications of these advancements for the broader field of robotics, emphasizing the role of intelligent model selection in enhancing the autonomy and versatility of robotic systems.

In conclusion, this study not only provides a pragmatic solution to a pressing challenge in robotics but also lays the groundwork for future innovations. By bridging theoretical analysis with empirical evidence, it underscores the transformative potential of online learning algorithms in model selection, setting a new benchmark for efficiency, cost-effectiveness, and adaptability in robotic applications.

Task-aware Distributed Source Coding under Dynamic Bandwidth

Tue, 28 Nov 2023 00:00:00 +0000

By Po-han

Leveraging Neural Distributed Principal Component Analysis for Efficient Data Transmission

Link to paper

In today’s interconnected world, the constant exchange of data among various sensors and devices has never been more vital. The efficient transmission of the data presents several challenges:

How do we define important features from a data distribution?
How do we extend this to multiple data distributions or modalities settings?
How do we compress and prioritize the features so that we can transmit the important features first to save network bandwidth?

To tackle these challenges, our collaborative work with researchers from the Honda Research Institute has given birth to a transformative solution known as Neural Distributed Principal Component Analysis, or NDPCA. In this blog, we will briefly cover the intricacies of NDPCA and its pivotal role in simplifying complex data transmission processes.

Pipeline of NDPCA

The Setting

Here is our pipeline of NDPCA: each sensor independently compresses its data before transmitting it to a central node, where a decoder reconstructs all the data. This reconstructed data is then fed into a pre-trained machine-learning task model, which produces the desired output. It’s crucial to note that the quality of the final output hinges on compressing only the relevant features for the task at hand, and the overall performance is closely tied to available bandwidth.

For instance, the sensors are satellites compressing images of the Earth. The task model is a machine-learning model that can detect natural disasters from the images. The bandwidth is the communication channel between the satellites and the ground station.

The Challenge: Variable Bandwidth

In real-world scenarios, bandwidth availability can be unpredictable, with higher bandwidths typically yielding better task performance. To confront this challenge head-on, we design a novel framework capable of flexible data compression from multiple sources. This framework seamlessly adapts to varying bandwidth conditions while minimizing computing and storage overhead with a single model.

The Solution: NDPCA

Our solution, NDPCA, represents a breakthrough in this field. It comprises independent encoders and a joint decoder, facilitating efficient data compression and transmission. NDPCA’s standout feature is its adaptability to any available bandwidth using a single model, thereby simplifying computational and storage requirements significantly.

A Closer Look at NDPCA

Learning Low-Rank Task Representations

NDPCA harnesses the power of principal component analysis to learn low-rank task representations from the data. By focusing on the most essential features, the framework can efficiently compress the data without compromising task performance.

Distributed Compression

Each sensor in the network independently compresses its data using the learned low-rank task representations. This step ensures that only relevant information is transmitted, effectively reducing communication overhead between the sensors, which can be up to O(n^2).

Joint Decoding

At the central node, a joint decoder plays a pivotal role in reconstructing the compressed data from multiple sensors. This reconstructed data is then fed into the task model, ensuring that only the important task-relevant features are represented, ultimately leading to maintaining the task performance.

The Benefits and Significance

Adaptability to Varying Bandwidth

NDPCA can dynamically compress and transmit the most important features to match the available bandwidth, thereby maximizing the performance of the task model. This adaptability proves invaluable in real-world scenarios where bandwidth availability can fluctuate unpredictably.

Reduced Computing and Storage Overhead

By employing a single model and distributed compression, NDPCA significantly reduces the computational and storage requirements in comparison to traditional methods. This efficiency renders NDPCA viable even in resource-constrained environments.

Enhanced Task Performance

By compressing only the features that are pertinent to the task at hand, NDPCA ensures that the task model receives data of the highest quality. This, in turn, leads to improved overall performance and accuracy.

Conclusion

Our paper, titled “Task-aware Distributed Source Coding under Dynamic Bandwidth,” introduces NDPCA as a revolutionary solution to the challenges of data compression and communication in multi-sensor networks. By prioritizing the transmission of task-relevant features from multiple data sources and adapting to varying bandwidth conditions, NDPCA provides an efficient and flexible framework for optimizing data transmission. The significance of this work lies in its potential to enhance communication efficiency and improve task performance across a wide range of applications, from Internet of things devices to large-scale sensor networks.

Differentially Private Timeseries Forecasts for Networked Control: Protecting Privacy without Sacrificing Accuracy

Sun, 28 May 2023 00:00:00 +0000

By Po-han

Introduction

In the age of interconnected systems and data-driven decision-making, accurate forecasting plays a crucial role in optimizing control strategies. However, concerns regarding data privacy have led to the introduction of anonymization noise in forecasting models, which compromises their accuracy. Balancing the need for accurate forecasts and the privacy of input data is a challenging problem. In our recent paper titled Differentially Private Timeseries Forecasts for Networked Control, we propose a novel approach to address this trade-off by providing economic incentives to forecasting models while minimizing control costs. In this blog post, we will delve into the key insights and findings of this paper.

System graph: A controller incentivizes multiple forecasting models to reduce differentially private noise and combines the resulting forecasts to minimize its control cost.

The Challenge

Accuracy vs. Privacy Forecasting models rely on historical data to make predictions about future events. However, sharing this data openly poses privacy risks. To address this concern, anonymization noise is added to the data, which reduces the accuracy of the forecasts. The challenge lies in finding the optimal balance between accuracy and privacy, considering the cost implications of control decisions based on imperfect forecasts.

The Proposed Solution

We propose an approach that allocates economic incentives to forecasting models to encourage them to reduce anonymization noise while maintaining acceptable privacy levels. By reducing the noise, the accuracy of the forecasts improves, leading to better control decisions. We tackle this problem through a biconvex optimization framework, specifically focusing on linear quadratic regulators (LQRs), which are widely used in control systems.

Comparing Incentive Allocation Schemes

To evaluate the effectiveness of their proposed approach, we compare it to a uniform incentive allocation scheme. The results are promising, as the proposed solution demonstrates significant cost reductions. In the case of synthetic timeseries, control costs decrease by 2.5 times, while for the Uber demand forecast, costs decrease by 2.7 times. These findings highlight the potential of incentivizing forecasting models to enhance their accuracy without compromising privacy.

Results on the Uber demand: Ours (green) outperforms the uniform (blue) with lower control costs, while the more incentives the controller has, the lower the cost.

Implications and Future Directions

The research presented in this paper has important implications for networked control systems and the broader field of data privacy. By providing economic incentives to forecasting models, decision-makers can strike a balance between accurate forecasts and privacy protection. The findings of this study pave the way for future research on privacy-preserving forecasting techniques and incentive mechanisms in control systems.

Conclusion

Our paper “Differentially Private Timeseries Forecasts for Networked Control” introduces an innovative approach to address the accuracy-privacy trade-off in forecasting models. By allocating economic incentives to forecasting models, control costs can be significantly reduced while maintaining acceptable privacy levels. This research opens new avenues for improving decision-making in networked control systems while safeguarding sensitive data. As the field progresses, we can expect further advancements in privacy-preserving techniques and incentive allocation mechanisms, ultimately benefiting both accuracy and privacy in various domains.