Translation. Region: Russian Federal
Source: Novosibirsk State University –
An important disclaimer is at the bottom of this article.
The basic model of an autonomous intelligent assistant for visually impaired and blind patients was presented to Deputy Prime Minister of the Russian Federation — Head of the Government Staff Dmitry Grigorenko during his visit to Novosibirsk State University. The device is being developed using large language models and artificial intelligence technologies.
The idea of the development belongs to the Novosibirsk branch of the Federal State Autonomous Institution “National Medical Research Center “Microsurgery of the Eye” named after Academician S.N. Fedorov” of the Ministry of Health of the Russian Federation, which, together with scientists Research Center in the Field of Artificial Intelligence of NSU (NSU AI Center) creates a new device designed to make it easier for visually impaired and blind people to navigate in space using modern technologies.
As noted by the director of the Novosibirsk branch of the Scientific and Technical Complex "Microsurgery of the Eye", professor, doctor of medical sciences Valery Chernykh, today there are various approaches related to the possibility of giving a blind person a chance of functional orientation in the surrounding space, his adaptation and rehabilitation with the possibility of actively and independently living a full life.
First of all, these are social and rehabilitation-educational programs that require quite large financial investments. In addition, over the past 20 years, scientists from various countries have been attempting to implant expensive and high-tech chips either into the retina or directly into the occipital part of the cerebral cortex, which is responsible for the function of vision. These are very complex and expensive operations with a high risk of complications and, unfortunately, with a fairly low percentage of effectiveness, since the results obtained do not last long. Several such operations have also been carried out in our country, the results of which cannot yet be considered satisfactory.
— Considering the active development of technologies related to the capabilities of artificial intelligence, the idea was born to use the auditory analyzer of the brain for human orientation in space. When a person reads a book or listens to an audio recording, certain images of objects, etc. are formed in his brain. If a visually impaired or blind patient is given the opportunity to determine the distance to an object (doorway, window, vehicle), its size, etc. through the auditory analyzer, this will certainly help him in real life and adaptation in the social environment. Considering the previous successful experience of joint work, we decided to join forces with scientists from the Research Center for Artificial Intelligence of the Novosibirsk State University in order to create such a technology that should be autonomous and not use the Internet, — said Valery Chernykh.
At the first stage, the developers create a prototype of the device and train the model in accordance with the tasks set, which is actively carried out by the formed working group, which includes scientists from the NSU AI Center and the Scientific and Technical Complex "Microsurgery of the Eye".
— We use large language models to help this category of patients analyze the surrounding space — to create a guide for them that could give a concentrated verbal description of all objects in front of the user. The principle of this system is simple: a video camera will “see” instead of a person, the information from which will be sent to a portable computer, which will convert it into text. Then the text description is converted by a voice assistant and sent directly to the patient through headphones. This message provides the basic information that the user needs to know to navigate in space, — explained Alexey Okunev, head of the project at the NSU AI Center.
The device is developed using a multimodal language model, which allows it to already provide a fairly high-quality verbal description of surrounding rooms and objects. In the future, the user will be able to ask clarifying questions about the observed scene and receive detailed answers to them. This language model also perfectly recognizes texts, reads inscriptions, signs, obstacle warnings, etc.
Currently, a basic model has been created, a prototype of the device will be ready by the end of 2025. This is a compact wearable system consisting of a single-board 16-core computer placed in a shoulder bag, a video camera (for example, GoPro) and headphones. The device will be equipped with a battery, and all elements will be connected by wireless communication channels. The user will hold the video camera in his hand and point it in the desired direction. The total weight of the device with the battery will be about 1 kg.
According to the experts of the NSU Center for Information Technologies, it will take more than one year to develop the final product ready for replication, and now they are at the very start of the project. This concerns both the technical part and the training of artificial intelligence. After the prototype is created this year, long-term work will begin on setting up the device, setting modes, collecting a database and training AI.
The developers plan to teach the AI to prioritize information delivery, highlighting from the entire video sequence the details that are most important for the visually impaired person to orient themselves in space and assess the surrounding environment depending on the situation. It is assumed that the device will have several operating modes: a movement mode for moving in space, a reading mode for working with text documents, etc.
It will also be necessary to train a blind or visually impaired patient to work with the device in various modes and adapt it to new capabilities. As noted by the Scientific and Technical Complex "Microsurgery of the Eye", at this stage, the work of not only ophthalmologists, but also specialists in the field of neurophysiology will be required. And the third task that will need to be solved is the need to create a specialized ophthalmological rehabilitation center to prepare blind patients directly for working with the device.
“The creation of such a device is of interest in the ophthalmological community as an extremely relevant and necessary area, the implementation of which will enable blind patients to adapt to the environment, which is a socially significant project,” concluded Valery Chernykh.
Please note: This information is raw content obtained directly from the source of the information. It is an accurate report of what the source claims and does not necessarily reflect the position of MIL-OSI or its clients.
.