A white cane provides detailed information about the area immediately around a person, but many useful features of a scene lie farther away: an entrance, a specific object, or something approaching from outside the current path. Computer vision can detect much of this information. The challenge is deciding what is useful for the task and how best to communicate it.
What We Study
We develop systems that combine wearable sensing, computer vision, and accessible interfaces for navigation and spatial awareness. Our work covers camera placement and sensing, scene understanding and simplification, and guidance delivered through audio, haptics, augmented reality, or visual prostheses.
We evaluate these systems in immersive XR and real-world environments, including studies with blind and low-vision participants. A recurring question is how much information to provide: useful guidance needs to preserve the parts of a scene that matter without overwhelming the user. We also develop open-source tools for this work, including BionicVisionXR.
Current Directions
- Last-mile navigation and spatial awareness. Wearable sensing and guidance for the parts of a journey that conventional navigation tools do not address well.
- Scene simplification. Identifying and preserving the visual information needed for mobility when display bandwidth is limited.
- Task-aware guidance. Using language and task context to select which objects or parts of a scene should be emphasized.
- Out-of-view guidance. Audio, haptic, and visual cues that help users locate relevant objects outside the current field of view.
- Wearable systems. Camera placement, latency, power, and computation for real-time assistive devices.
