Robotic Gripper's Soft Touch Promises Gentler Blackberry Harvest Automation

An effort that was originally going to focus on grapes and wine has evolved into one that will help solve labor shortages and harvest issues for blackberries.

Yue Chen, a biomedical engineering roboticist at Georgia Tech, who was an assistant professor of mechanical engineering at the University of Arkansas when starting this work, was pursuing a Chancellor’s Innovation and Collaboration Fund grant in 2019. Chen wanted to develop a soft-touch robot for the agricultural space. So, he did a simple search through the University of Arkansas’ library for an agriculture expert, and found Renee Threlfall, a food science research scientist with the Arkansas Agricultural Experiment Station, the research arm of the University of Arkansas System Division of Agriculture.

Threlfall realized that grapes might not be the best starting point, so instead suggested blackberries.

“There are a lot more hand labor issues in harvesting fresh-market blackberries, so I thought that would better crop to focus on,” Threlfall recalled.
 

But before starting development on the robotic arm and gripper, the research team first had to determine just how much—or how little—force is needed and how many fingers are necessary to actually pluck a blackberry from the bush while doing as little damage to the fruit as possible.

This initial research started back in 2020 and outlined in “Determining Hand-harvest Parameters and Postharvest Marketability Impacts of Fresh-market Blackberries to Develop a Soft-robotic Gripper for Robotic Harvesting,” published last year in the journal HortScience. Andrea L. Myers, a food science graduate student at the University of Arkansas, was lead author, working in collaboration with Threlfall, Chen, and Anthony Gunderman, a doctoral student in robotics at Georgia Tech’s Institute for Robotics and Intelligent Machines, previously a doctoral student in mechanical engineering at the University of Arkansas. This spring, the paper was named an American Society for Horticultural Science’s Outstanding Fruit Publication Award winner for 2022.

After trying numerous types of gloves fitted with force sensors that didn’t provide the right kind of tactile response, the idea came to Threlfall to instead use silicone finger covers, much like the ones used to flip through books and documents. Gunderman carefully embedded resistive force sensors into holes that were cut into each sleeve, with the sensors extending just 0.5 mm past the sleeve itself.

Similar Reading: Robotic Gripper Helps Clean Up Hazardous Liquid Spills

“We had to have something on my hand or fingers with sensors so that I could actually touch and feel the berries when I harvest,” Myers added.

These sleeves were used on the thumb and three fingers: index, middle, and ring. Testing revealed that forces during harvest imparted by the thumb and middle finger were the greatest at 0.77 N and 0.37 N, respectively, while the index and ring fingers used lower forces at 0.16 N and 0.06 N, respectively, primarily to stabilize the blackberry, Threlfall explained.

In June and July 2020, the team used this setup to harvest 240 blackberries, which were then brought back to the University of Arkansas System Division of Agriculture Food Science Department to observe physical, composition, and marketability attributes at harvest, during storage, and after 21 days.
 

In 2021, Chen and Gunderman developed the first three-pronged, tendon-driven, soft robotic gripper with motion modeled after a human hand and equipped with a thick guitar string as a tendon. The team published a paper in 2022 describing the gripper, “Tendon-Driven Soft Robotic Gripper for Blackberry Harvesting.” One more paper followed in 2023, covering the second iteration, titled, "Tendon-Driven Soft Robotic Gripper with Integrated Ripeness Sensing for Blackberry Harvesting," published in the 2023 IEEE International Conference on Robotics and Automation.

“The gripper was engineered using forces that we determined on the first round of the actual harvesting using my fingers with the silicone finger sleeves and sensors,” Myers said. 

This effort showed that the gripper could use an amount of force as low as 0.5 N, with a mean error of 0.046 N, and could harvest berries with 16 percent red drupelet reversion (RDR) while maintaining a harvesting reliability of 95.24 percent at a rate of about 4.8 seconds per berry.

“When blackberries are ripening on the plant, they turn red to black. However, during or after harvest, drupelets on the berries can turn red, especially if the berry is damaged during harvest,” Threlfall explained. “The red color after harvest doesn’t have to do with ripeness but has more to do with damage to cellular structure. This is important because USDA has standards and grades for this red drupelet reversion so if a certain percentage of blackberries in a container have red drupelet reversion, they are rejected.” 

Become a Member: How to Join ASME

The first robotic gripper was designed from scratch. Gunderman’s initial thought was to develop the gripper to look like a tulip that would essentially envelop the berry and reduce any possible damage. In its most current form, the four sensors that were part of the initial build were lost, so the soft structure of the gripper simply envelops a berry and pulls using an enveloping type of grasp instead of a pinching grasp, Gunderman explained.

“Its structure was so well tuned to the berry retention force, that if the berry wasn't ripe it wouldn’t actually harvest it," he said. "So that's kind of part of our next step, doing the modeling of these soft nonlinear type structures. I'm kind of curious if you can drop the need for force feedback by inherently designing the gripper to prevent damage to the berry. If the berry is too unripe to harvest, it'll just stay on the plant without damage.” 

The latest gripper was also designed through CAD from scratch, but the team did draw on existing ideas in the literature, he added.
 

Artificial intelligence deep learning techniques will eventually help the soft touch robot and gripper to identify berry ripeness and location, allowing the robot to autonomously move and harvest, Chen explained. 

"Currently, there’s a human in the loop. But in the long term, we want to make sure everything is autonomous,” he added.

At the moment, researchers are working on camera feedback and integrating that with the gripper and a manipulator, something that the team foresees solving over the next year or two.

In March, Chen and Gunderman were among the authors on another paper, “Design, Modeling, and Redundancy Resolution of Soft Robot for Effective Harvesting,” which describes the robotic harvesting solution through the design, manufacturing, integration, and control of a pneumatically actuated, kinematically redundant soft arm with a tendon-driven soft robotic gripper. The study's lead author, Milad Azizkhani, is one of Chen’s students.

Research continues on the robot and gripper with a multi-institute team of Georgia Tech leading the engineering effort, the University of Arkansas focusing on post-harvesting analysis, and now Mississippi State University honing in on AI and camera integration. The team is exploring ripeness detection by using a state-of-the-art YOLOv7 model, and published a paper on this work, "Multi-ripeness level blackberry detection using YOLOv7 for soft robotic harvesting," in June.

You Might Also Enjoy: New 3D-printed Grippers for Short-term Manufacturing Runs

“We want to integrate the gripper and the soft robotic arm to the imaging system,” Chen said, with the goal of conducting in-field tests over the coming year or two.

“At the UA System, we're evaluating a rotating cross arm trellis system for blackberries, which has potential as a more harvestable trellis system for robotics because the system can be rotated flat on the ground during flowering, so that fruit is on one side of the trellis when the trellis is later in the upright position,” Threlfall added. Benefits would include keeping fruit close to the ground during frost events, but also making it so that plants would flower and bear fruit mostly on one side of the plant, making for easier automation with robots in the future.

“Just imagine a robot walking through a blackberry farm, but the robot has 10 arms,” Threlfall said.

Interest is strong in the innovation as well. While working with blackberry producers throughout the process, the top question the research team has been asked is where farmers can buy the robot and how much it will cost, Myers added. Application for other soft fruits, such as blueberries, could be on the horizon as well, and potentially even a return to the grape idea that started it all.

Louise Poirier is senior editor.