End-to-End Optimization of Bionic Vision

Our lack of understanding of multi-electrode interactions severely limits current stimulation protocols. For example, current Argus II protocols simply attempt to minimize electric field interactions by maximizing phase delays across electrodes using ‘time-multiplexing’. The assumption is that single-electrode percepts act as atomic ‘building blocks’ of patterned vision. However, these building blocks often fail to assemble into more complex percepts.

The goal of this project is therefore to develop new stimulation strategies that minimize perceptual distortions. One potential avenue is to view this as an end-to-end optimization problem, where a deep neural network (encoder) is trained to predict the electrical stimulus needed to produce a desired percept (target).

Importantly, this model would have to be trained with the phosphene model in the loop, such that the overall network would minimize a perceptual error between the predicted and target output. This is technically challenging, because a phosphene model must be:

  1. simple enough to be differentiable such that it can be included in the backward pass of a deep neural network,
  2. complex enough to be able to explain the spatiotemporal perceptual distortions observed in real prosthesis patients, and
  3. amenable to an efficient implementation such that the training of the network is feasible.

Project Team

Project Leads:

Project Affiliates:

Shivani Sista

Research Assistant

Sriya Aluru

ERSP Scholar

Principal Investigator:

Michael Beyeler

Assistant Professor

Project Funding

R00EY029329: Virtual prototyping for retinal prosthesis patients
PI: Michael Beyeler (UCSB)

September 2020 - August 2023
National Eye Institute (NEI)
National Institutes of Health (NIH)

Publications

What is the required stimulus to produce a desired percept? Here we frame this as an end-to-end optimization problem, where a deep neural network encoder is trained to invert a known, fixed forward model that approximates the underlying biological system.

We propose a perceptual stimulus encoder based on convolutional neural networks that is trained in an end-to-end fashion to predict the electrode activation patterns required to produce a desired visual percept.

Back to all research projects