UNIEGO uses proxies to unify egocentric vision

Egocentric video understanding has long suffered from the limits of a single viewpoint and modality. In a new paper, researchers propose UNIEGO, a unified egocentric encoder trained through a hierarchical multi-teacher distillation framework. The key twist is a dedicated layer of representation specific Proxy models that translate diverse teacher knowledge into a uniform egocentric space before any distillation happens. This mediator approach allows knowledge to flow from ego and exo viewpoints, RGB, depth, and skeleton streams, as well as from four foundation models, without the gradients colliding due to incompatible architectures or feature geometries.

The distillation pipeline unfolds in two stages. First, a layer of Proxy models acts as translators, reconciling heterogeneous teacher outputs into a shared, ego-friendly representation. Second, an adaptive stage called Selective Proxy Distillation (SPD) decides, on a per sample basis, which proxies are both correct and confident enough to contribute to the learning signal. By design, SPD suppresses unreliable guidance, preventing noisy supervision from derailing the student. The team further stabilizes learning by initializing UNIEGO as a learned convex combination of proxy parameters, placing the unified model in a well conditioned region of the loss landscape before the distillation begins.

The result is a model that achieves state of the art across three core egocentric video tasks, namely action recognition, video retrieval, and action segmentation, on three challenging ego-exo benchmarks. In experiments, UNIEGO outperforms naive multi-teacher distillation baselines and demonstrates that structured, proxy mediated knowledge transfer yields richer and more discriminative representations than distillation from any single teacher or from an indiscriminate ensemble.

For practitioners, the work offers several concrete implications. First, the proxy mediator concept provides a practical solution when aggregating signals from heterogeneous sources is desirable but the source architectures and feature spaces do not align. Second, SPD introduces a per example gating mechanism that reduces negative transfer by excluding unreliable supervision on the fly, a critical guardrail in real world deployments where data quality can vary widely. Third, the convex initialization trick highlights a broader engineering point: stabilizing a distillation-heavy training regime early can position the student model in a favorable region of the loss landscape, making heavy multi-source supervision more tractable. Finally, the framework acknowledges compute realities: distilling from nine teachers and four foundation models initially sounds expensive, but the payoff is a single compact encoder that inherits a broad, multi-modal, multi-view understanding without carrying that full training burden at inference time.

Industry watchers may see UNIEGO as part of a broader shift toward mediator style learning in AI systems, where messy, multi-faceted supervision is distilled through structured intermediaries rather than brute force ensembling. If the approach scales, it could reshape how teams fuse wearable, environment, and base-model signals into practical, deployable representations for action recognition, CLIPS of video search, or real-time segmentation in wearable robotics and augmented reality.

In sum, UNIEGO demonstrates that proxies can safely shepherd a chorus of pedagogues toward a unified, robust egocentric representation, delivering strong performance gains while offering clear knobs for engineers to tune training dynamics and data quality.