Novosibirsk State Technical University (NSTU) has developed software capable of automatically detecting road infrastructure facilities, generating a full text description of video scenes from a DVR and making a conclusion aboutAtraffic congestion. The project is based on computer vision technologies and neural networks.
The development by Maxim Semenov and Sergey Skorykh, fourth-year students of the Faculty of Automation and Computer Engineering, allows converting a video stream into a structured text description that includes not only a list of objects, but also an analysis of the traffic situation in dynamics. This is especially valuable for subsequent processing of large amounts of text data obtained from multiple video scenes, without having to watch hours of video.
According to Egor Antonyanets, the project's scientific supervisor, an assistant at the Department of Automated Control Systems at NSTU-NETI, the system analyzes the video stream in real time and generates a coherent text report.
"It is based on a convolutional neural network model that detects road infrastructure objects in the frame: markings, signs, traffic lights, pedestrian crossings, vehicles. The algorithm then analyzes their location and movement, after which it generates a full text description of the video scene. At the end, the program makes a final conclusion: "traffic is low," "average congestion," or "high traffic intensity," said Egor Antonyants.
Unlike simple detection systems that only display the names of objects; the new algorithm generates a coherent text description of the entire scene with a final assessment of the traffic situation. Technically, the work is built in several stages: first, the neural network selects objects on each frame, then the program tracks their dynamics and flow density, after which it uses language models to convert the data into readable text, culminating in a conclusion about traffic congestion.
According to the project developers, analogues of such solutions exist, but they either require large computing power, or are not adapted to Russian road conditions, or provide only a raw list of objects without analyzing the overall situation.
"Our development allows you not only to see which objects are in the frame, but also to get a full-fledged text description of the video scene with the final verdict on traffic. For example, "12 cars and 2 trucks were moving along the road for 30 seconds. The pedestrian crossing was not in use. The traffic light was working normally. The conclusion is that the traffic load is average." This format is convenient for automatic analysis of large amounts of data. The text is easy to index, search by keywords and process with algorithms," the developers noted.
The students also added that the tool can be useful for road services, analytical centers, and insurance companies. In the future, the system can be adapted to monitor traffic congestion in real time and integrate with existing video surveillance systems.