Translation. Region: Russian Federal
Source: Novosibirsk State University –
An important disclaimer is at the bottom of this article.
A student has created an intelligent robotic arm for the automatic harvesting of tomatoes in industrial greenhouses, which can distinguish ripe fruits from unripe ones. Faculty of Information Technology, Novosibirsk State University Anton Vlasenko. His robot is capable of analyzing the ripening time of different tomato varieties and harvesting only ripe ones. It leaves unripe ones on the bushes and returns to them as they ripen. The young researcher is currently testing his device at home, and plans are underway for industrial testing at the Tolmachevsky greenhouse complex, for which a preliminary agreement has already been reached.
We used computer vision algorithms to analyze the condition of the fruit and make decisions. The system also incorporates ultrasonic sensors. They help the robot estimate the distance to objects and avoid collisions. To prevent the robotic arm from accidentally crushing tomatoes when picking them from the branches, we equipped the device with sensors that regulate the force of compression. An interesting aspect relates to the "time to harvest" algorithm itself. We don't simply classify tomatoes as "green" or "red," but rather attempt to estimate how many days remain until the optimal harvest. To do this, we use color channel and saturation data. Using this data, the system predicts the harvest time. This will allow us not only to harvest the fruit "here and now," but also to plan when exactly to dispatch the robot to a specific plant. Our robotic arm doesn't simply determine the overall color of the tomato, but divides its image into a grid, like a chessboard. Each cell is analyzed individually based on the fruit variety, separating out areas that are red, green, or yellow. This way, the system understands whether the fruit is ripe, partially ripe, or still green, and then predicts the optimal time for harvesting, explained Anton Vlasenko.
To detect objects, the young researcher used the YOLOv8 (Ultralytics) core neural network in his development. It finds the bounding boxes of tomatoes in the frame. The robot's software is written in Python. The OpenCV (cv2) computer vision library handles several tasks: reading the video stream from the camera, image transformation (HSV, LAB), and creating color masks. Numerical calculations—channel averages, array operations, and pixel counting in masks—are performed using the NumPy library. An Orange Pi 5 controller powers the stepper motors and control drivers. This allows the robotic arm to receive tomato coordinates from YOLO, convert them into angles for the servo motors, and then pick the fruit.
The manipulator itself was manufactured using 3D printing. It consists of a gearbox, arm segments, brackets, and a gripper. A total of 115 parts were manufactured. After printing, each one underwent meticulous post-processing. A significant portion of this work was performed by the project's second participant, Yakov Gubarev, a student at the Siberian State University of Geosystems and Technology. Supports had to be removed from each part, contact surfaces had to be manually sanded, mounting holes for fasteners had to be drilled, and the accuracy of the mounting surfaces had to be verified.
"While working on printing the manipulator parts, we encountered a serious problem. It's a fairly large structure—if its "arm" is fully extended, it's about 1.5 meters long. Our existing printer couldn't handle this. We started looking for alternatives, and it turned out that printing ready-made 3D models would cost us more than a new printer with the capabilities we needed. So we had to buy a new 3D printer," said Anton Vlasenko.
The manipulator is currently assembled, and the young researchers will now fine-tune its motion and then assemble a mobile platform that will allow the robot to navigate between rows in greenhouses. After that, they will be able to move on to pilot testing in real-world conditions. Anton Vlasenko will defend his master's thesis, which will be the basis for his project. He also plans to submit it to a student startup competition.
The idea to create a robotic manipulator for this task came to me at a hackathon held by TRK. One of the tracks was to create a small robot that would use computer vision to pick certain types of fruit. The task wasn't difficult—we just needed to make sure the robot touched the fruit it had selected. Later, we decided that it would indeed be a good idea to create a robot that could pick tomatoes in industrial greenhouses. After speaking with Sergei Evgenievich Lozhnikov, the former director of the Tolmachevsky greenhouse complex, we learned that there was a real need for automated harvesters. Currently, this process is done manually, but there's a labor shortage, which is becoming a serious problem for greenhouse complexes. Our idea to create a robot that could perform this task found support, and we got to work. First, we studied existing robots, and then began considering which architecture to use to more effectively harvest tomatoes, as well as planning for future development. In the future, we plan to adapt our tomato picker to other vegetable crops, Anton Vlasenko shared his plans.
Material prepared by: Elena Panfilo, NSU press service
Please note: This information is raw content obtained directly from the source. It represents an accurate account of the source's assertions and does not necessarily reflect the position of MIL-OSI or its clients.