How Robot Dogs See the Unseeable: Improving Visual Interpretability via Peering for Exploratory Robots

If our eyes would be 1m in diameter, we would not need to do this.

Our CS team at JKU has been exploring optical synthetic aperture sensing for occlusion removal since 2017. To date, we have deployed aerial platforms (such as individual drones, drone swarms, and airplanes) for applications including search and rescue, wildfire detection, wildlife monitoring, and forest ecology. A core research challenge across these domains is optimally sampling the synthetic aperture. Adapting this technique to ground robots introduces distinct challenges: unlike aerial vehicles that can freely sample large areas, ground robots are constrained by physical obstacles. To achieve efficient sampling under these conditions, we developed a bio-inspired strategy based on insect peering behavior. Working with leading zoologists at the University of Sussex, we utilized experimental data on peering insects to model equivalent motion patterns for robotic platforms. In parallel, our collaborators in Bozen provided essential field robotics expertise, control strategies, and hardware. Ultimately, this interdisciplinary project brings together computer scientists, zoologists, and field robotics researchers.

Beyond the general difficulty of operating robots in complex environments, we encountered several practical engineering hurdles. For instance, executing the required swaying trajectories presented a unique challenge, as lateral motion is not supported out-of-the-box on conventional robotic platforms. The native control system initially restricted these non-standard movements, requiring low-level modifications to ensure stable and reliable execution.

The primary finding of this paper is that advanced vision reasoning fails in scene understanding when provided solely with standard, individual images (even when supplied with hundreds of frames captured during peering) due to persistent partial occlusion across all views. However, vision reasoning succeeds when these peering frames are computationally combined into a focal stack via synthetic aperture sensing. This enables the models to inspect the scene across distinct focal depths without occlusion. Consequently, peering alone is insufficient - the key lies in combining peering with synthetic aperture sensing, where peering provides the motion baseline required to generate the synthetic aperture.

Ongoing thread. More from Oliver Bimber to follow.