Intelligent Bio Robotic Prosthetic Arm System Using HCI (Human Computer Interface)

Introduction

There are millions of people all around the world who have difficulties with their arms and hands, including missing one. By 2019, it was predicted that roughly 93.8 million individuals will have lost an arm or leg. This is around 17% of all cases of limb loss, and most of these are amputations of the lower limbs. Most of these amputations happen because of falls and accidents. These data demonstrate that individuals need to know more about the condition, have better safety measures in place, and be able to get medical and rehabilitative care more easily in order to improve their quality of life. By 2024, advances in prosthetic technology, such bionic and artificial arms, should make a big difference in the lives of people all around the world. ¹ The market for robotic prosthetics, such bionic limbs, is growing quickly. In 2023, it was valued around $1.65 billion¹⁰, and it is likely to keep expanding. At least 30 million people throughout the world require prosthetic limbs, yet many of them can’t obtain them because they are too expensive and hard to locate. A lot of individuals discontinue using prosthetic limbs because they are uncomfortable, hard to use, and don’t operate as they should. A 2022 poll found that almost 44% of persons who had lost their upper limbs had chosen to stop wearing their prosthetic.

Even though these limitations exist, current improvements in bionic technology give us hope for increased access, comfort, and usefulness. These changes will have a huge influence on the lives of those who rely on these gadgets to make their lives better. Electromyography (EMG) is a medical test that looks at the health of muscles and the nerve cells that control them. These nerve cells provide electrical signals that make muscles contract and relax. EMG may reveal problems in your muscles or neurological system by analysing these signals. The data are displayed as graphs or statistics, which help physicians determine what health issues a person may have. ² EMG technology is used for more than only diagnosis. It is also used in new devices like robotic arms and servo motors that operate like muscles. The AD8226 instrumentation amplifier is a nice example of a sensor that can pick up and make these muscle signals stronger. There are several ways to use the AD8226 amplifier. It can be used in medical devices, commercial systems, and portable monitoring devices. When these EMG sensors are attached to an Arduino microcontroller, they can show the electrical activity of some muscles, such the biceps or quadriceps. The AD8226 produces the signal powerful enough to be measured. The signal is normally between 30 millivolts and 50 microvolts. The sensor receives the signal, corrects it, and smooths it out so that it may be shown for review.

Motivation

Though the development of intelligent prosthetic systems has made significant progress over the last few years, most existing prosthetic hands are costly and, in reality, demand a significant amount of training by the user. Furthermore, many existing approaches are based exclusively on either an electromyography type signal or vision based object recognition, and thus will not allow for robust performance when the operating condition changes in real world settings. Such joint recognition of human intent and intelligent autonomous grasping based on the recognition of objects under the conditions of low cost drives the development of a low cost intelligent prosthetic arm.

Problem Statement

Prosthetic arm technology faces numerous interrelated problems. These are the inability to recognize user intention well and reliably, lack of required ability to handle the wide range of object shapes and geometry, reliance on expensive hardware and a limited association of physiology measurements with computer-vision-based analysis. As a result, the user’s understanding of the grasp is often reduced, and in the case of amputees the usefulness and trust of such a system in everyday life has proven to be rather unpredictable.

Research Gap

These key aspects have been treated individually in previous works, such as the electromyography-based approach of prosthetic control, the computer vision-based grasping, and deep learning methods. However, few studies have yet presented a compelling end-to-end integration of low-cost electromyography sensing, in-the-wild practical real-time object recognition, and adaptive grasp control in a single embedded system architecture that would be viable for real-world deployment. Moreover, the potential for doing a complete and detailed comparative study between the conventional prosthetic solution and hybrid systems based on artificial intelligence (AI) is still small.

Contributions

• Designed and developed hybrid framework for controlling a prosthetic device that combines electromyography and vision.
• The real-time object detection was done using SSD-MobileNetV2.
• Developed an adaptive grasp selection; strategy utilizing object properties.
• Developed and assembled an inexpensive implementation platform (Arduino based and Raspberry Pi).

Proposed system much better by adding new modules for EMG signals with real time vision based object recognition a hybrid control framework and Development of an adaptive grasping mechanism based on object characteristics. Which Implementation: Low Cost Prosthetic System by embedded platforms (Arduino and Raspberry Pi). The proposed model can be used for Real-time synchronization of muscle feedback and visual feedback. Based on the Experimental validation with multiple objects achieving up to 87% approximately accuracy.

Literature review

Chen et al.³conducted a thorough assessment of the latest tools and methods that are being used to construct bionic prosthetic hands. Perusing it as a promising technology and how to use these high-technology devices. They also went into these control mechanisms and the sensory feedback systems that are needed to make consumers happier and provide them a more natural experience in great detail. The writers also talked about how bionic hands are doing currently. They noted that users should feel comfortable with how these prosthetic limbs appear and perform and that better sensory feedback systems are needed.

One of their bleeding-edge attempts to improve the field of prosthetics is the proposed design of a bionic arm powered with artificial pneumatic muscles by Li et al. ⁴ In this project they have managed to create a two degrees of freedom (2-DOF) in this robotic arm which has been designed using the pneumatic artificial muscles which supposedly mimics the action of biological muscles. The possibility of using pneumatic artificial muscles was convincingly presented, but what the authors saw was that there still exists an urgent need to increase the control precision and responsiveness of the arm to guarantee maximum performance monitoring.

A novel neuroprosthetic hand controlled neural system has been developed by Ortiz-Catalan et al. ⁵ with high level of integration, an enormous step toward transforming the prosthetics sector. The study entailed the use of a neuromusculoskeletal prosthesis, which has a very close association with the nervous system of the user hence ensuring an easier control system and also an enhanced sensory feedback that is effective in inducing both the relaxed and active feeling of the hand of the person using the prosthesis. Nevertheless, in spite of the progress, the authors considered that these two issues might be handled still, namely surgical complexity and the necessity to adapt to the use in the long term may be viewed as the considerable obstacles, which have to be overcome some day.

An interesting approach to prosthetic design is the development of brain-controlled robotic arm that incorporates surface electromyography (sEMG) sensor to provide control Lonsdale et al.⁶ Their work combined transfer learning-based classification of sEMG signals and a three-dimensional printed arm that provided the functionality of real-time control. The authors, however, have concluded that such system is somewhat limited by the fact that it requires much training data and a lot of computational resources; vital in the maximization of device performance.

A novel neuro-symbolic controller of arm is proposed by Hanson et al.⁷ on Sophia the Robot, and it is an integration of two artificial intelligence approaches. This allowed the complex use of the robotic arm because of their combination of convolutional neural networks with symbolic AI techniques, and demonstrated the potential advanced robotic capabilities have. However, the authors did realize that this is an area that still needs exploration further in order to fully develop the integration of this technology to different environments so that it is adaptable and efficient across other applications.

Basumatary et al.⁸ have proposed a new type of control in bionic reflex when applying reinforcement learning methods to improve bionic reflexes to grasp in simulation. Their study has shown that there is a possibility of better reflexive mechanisms by use of reinforcement algorithm, which is one of the future paths to be tabled in bionics in the production of control systems. Nevertheless, the authors stressed that these ideas still have to be tested in a real-world environment to make sure they help and work reliably in the practice.

Linares-Barranco et al.⁹ offered a new method of adaptive robotic arms control with the use of spiking neural networks, a new field related to all successes of neural networks. They have come up with an adaptive control system in their work that runs on a digital accelerator and prove an efficient utilisation of resource to get such a complex neural net to work in an effective way. On the one hand, in spite of all these developments, the authors realized that, inherent complexity that comes with spiking neural networks is one of the greatest problems that have to be overcome to aid successful applicability on a practical robotic system.

Imran et al.¹⁰ suggested an intelligent bionic arm of a smart device that may be controlled with the help of gesture recognition, which is a considerable achievement in the sphere of assistive technology. They conceptualized how to create an affordable 3D-printed arm coupled with an artificial intelligence in the interests of gesture recognition which is expected to be used to improve user-prosthetic interaction. The authors, however, noted that there still is an urgent need to further work on the responsiveness and accuracy of the arm control system mechanisms in order to secure an optimal user experience and the functionality.

Their findings showed remarkable advancements in object detection and grasping accuracy. ³⁶ However, the study primarily focused on visual control and did not take into account physiological signals like EMG to recognize user intent Chen et al. ³⁷ introduced a groundbreaking framework that combined both EMG and vision-based systems to enhance precision during robotic grasping tasks. The results were impressive compared to using single-modality approaches; however, implementing this complex system remains costly. Fast forward to  when Linares-Barranco et al.’s research delved into neuromorphic computing for EMG-visual based robotics systems – successfully achieving efficient real-time processing capabilities. Although their approach yielded promising results, its practical application is limited due to the need for specialized hardware. ³⁸

A.Singh et al. introduced a revolutionary human-computer interaction (HCI) based intelligent prosthetic arm that seamlessly combines signal synchronization and motor control. ³⁹ This groundbreaking study emphasized the enhanced connection between user intent and prosthetic response, but it also acknowledged the need for further refinement in terms of accuracy and adaptability., Khan et al. ⁴⁰ shared their research on a deep learning-based system for controlling prosthetics through multimodal sensor fusion. The results were impressive with high levels of precision achieved; however, implementing this model in real-time proved to be challenging due to its reliance on extensive datasets and computational resources.

Zandigohar et al. (2022) studied multimodal fusion of electromyography (EMG) and vision for human grasping intention estimation in the field of POCH. The authors pointed out that such EMG-based controls are susceptible to motion artifacts, variations in electrode-skin contact impedances, muscle fatigue and EMG signal variability. Simultaneously, vision-based systems are subject to occlusion of objects, variations in illumination and appearance of objects. In order to overcome the complementary drawbacks, the study was designed to include a multimodal framework with forearm EMG, eye-view video, and eye-gaze information. Both the physiological and visual data were processed using neural-network-based methods, and the ability of both modalities and their fusion to be assessed while the hand was moving towards an object. This study showed that physiological information can be used in conjunction with visual information to improve grasp-intent inference. The multimodal framework however, consists of various sensing modalities and processing stages that results in system of higher complexity and hence low cost embedded implementation is a more challenging task.

To solve the problems of the traditional EMG artificial hand, the intelligent bionic artificial hand with image recognition based on the solution proposed by Shi et al. (2023). The limited accuracy of EMG based control was observed by the authors and study of computer vision in combination with EMG signals for the control of prosthetic hands was carried out. The object to be grasped was recognized by using a convolutional neural network (CNN) and the results of the recognition result were taken as the trigger signals for controlling of the artificial hand. It has been shown that object-level vision information could be used as a complement to a better intelligent control of a prosthetic hand. The drawback of the approach is that it is primarily an eye-gaze system, and so no effort has been made to directly incorporate visual and/or physiological data, such as EMGs, to create a system that can measure and utilize the user’s muscle intent. Thus, the synergy of visual and EMG signals could offer a more complete solution for prostheses’ control in the real world context due to different sensibilities of each sensor.

Robust neural decoding for dexterous robotic hand control (2023) examined continuous control of robotic hand kinematics using higher density of electromyography (EMG) features mapped to a neural-drive signal via deep-learning. The engineers centered their efforts on the dexterous, real-time and continuous multi-DoF control challenge. They designed a method based on deep learning to recognize intended finger motion and show that prediction of joint angles was quite accurate with moderate prediction errors. It also discussed the need for consistent performance irrespective of the fluctuations in EMG signals over time. The results also bolster the idea of using the deep learning framework for prosthetic control on an ongoing basis, but this method does have some drawbacks, such as the complexity and cost of collecting high-density EMG and the sophisticated neural decoding required.

Zandigohar et al. (2024) also built on the multimodal EMG and vision approach to infer the human grasp-intent for prosthetic hand control. Their work suggested a Bayesian framework for evidence-fusion, which means that they wanted to fuse eye-view video, eye-gaze and forearm EMG on their own through neural-network models. The researchers compared the efficacy of each modality and the combined modalities at various phases of hand motion approaching an object. They found complementary information in both types of signal, physiological (EMG) and visual: EMG signals contain information about the intended action of the user; visual signals contain information about the surrounding object and context of the task. However, visual signals could be occluded or light affected and EMG signals could be subject to fatigue or motion disturbances. Overall, the study offers excellent arguments for the possibility of multimodal prosthetic control and suggests a problem to conceive and design a reliable and computationally efficient fusion architecture useful for clinical applications.

Tchimino et al. (2024) conducted a study about the application of the EMG to control hand prostheses in an individual with a high-level arm amputation. In order to assess if this feedback could enhance an individual’s ability to control the magnitude of force using a myoelectric prosthetic hand, the study explored tactile feedback of different magnitudes of myoelectric activity. This is especially relevant because typical Myoelectric Systems do not give a user much feedback from the system and it is hard to control the grasp force. The researchers proved how to use EMG-related feedback to enhance the interaction with the prosthesis. However, this study was presented as a case study – meaning its results can’t be applied to a larger group of patients at this time (due to the limited number involved and limited amputation conditions).

A deep learning study of real-time motor intent decoding for prosthetic control (2024) investigated comparisons of various neural network architectures for motor-intent decoding from EMG signals for prosthetic arm and hand control. Shallow and deep architectures, such as feed-forward networks, temporal convolutional networks, and CNNs with Squeeze-and-Excitation operations, were evaluated. Deep-learning architectures were more effective in decoding motor-intent than shallow networks, according to human-in-the-loop experiments with able-bodied participants and a person with an amputation. The research showed that the representation learning method is able to automatically extract from EMG signals useful information that could be used in the future to control the prosthesis with more precision. The application of deeper models, however, can lead to higher computation needs and it can present problems for real time implementation in low-power embedded platforms.

In this context, Sarker et al. (2024) proposed a novel vision-controlled sensorized prosthetic hand that can serve as an accessible and user-friendly alternative to traditional prosthetic hands. The proposed system included a camera and inbuilt processors that processes the visual data along with pressure sensing to provide feedback, an accelerometer to detect gestures and release. An important goal was to minimize personalized training typically required with EMG systems. The work highlights the applicability of vision and embedded sensing for autonomous operation of the prosthetic hand. However, it does not directly make use of EMG signal for the determination of the physiological intent of the user since it is a visual based system. This opens the way for hybrid EMG-vision methods where the information given by the environment can be combined with the user intention.

To enhance anthropomorphic grasping for a powered prosthetic hand, Xu et al. (2024) introduced a vision-based powered prosthetic hand. EMG and brain-computer-interface based prosthetic systems can have problem to identify the grasping gesture precisely and vision-based prosthetics can detect the characteristics of objects precisely, but it may be inadequate to determine the grasping intention of human. In order to solve such a problem, they have proposed two methods: the method of Spatial Geometry-based Gesture Mapping (SG-GM) and the method of Motion Trajectory Regression-based Grasping Intent Estimation (MTR-GIE). They performed the experiments using common objects of everyday use and the obtained grasping-process similarity and grasping success were large. While the study demonstrates some promise for vision-based grasp-intent estimation, it is mostly based on visual data, and does not integrate EMG physiological intent with object recognition.

Measuring the EMG of the human hand, Dunai et al. (2025) built a post-Prosthetic Hand (pHP) modeled after the human hand and controlled it with an ANN. The study was aimed at feeling intuitive human gestures using EMG signals and then converting them into movement of the prosthetic hand. Using an artificial neural network results in a machine learning approach towards the interpretation of muscle activity and the control of the prosthetic mechanism. This method shows the effectiveness of an anatomical and EMG controlled hand design using a neural-network. But with an EMG-only system the muscle activation modulation also influences, and they cannot give a direct hint on the grasped object. Hence, combining visual object recognition with EMG based intent detection might be able to afford more adaptive grasp selection.

Takele and Negash (2025) presented the idea of an sEMG-based prosthetic hand control system with artificial neural network and slidingmode control. A total of six hand movements were identified as muscle activities using surface EEG and then a total of 1 human-made neural network was used for hand gesture classification. The prosthetic hand mechanism was then controlled by using sliding-mode control. The results revealed that by using a robust control strategy and machine learning algorithm, interpretation and execution of multi movements of the prosthetic hand can be enhanced. The approach is however largely signal-based from the physiology and does not take into account information regarding the object and grasp desires.

Le and Nguyen (2025) designed a human-robot interaction controller for prosthetic arm using surface EMG signals and deep-learning network approach. The proposed system captured the sEMG signals of various hand gestures and the captured signals were subjected to signal preprocessing before classification. The paper reflects the necessity of automatic prosthesis control based on real biosignals, and gives an account of the evolution of deep-learning approaches for myoelectric prostheses. However, the EMG-only controller primarily infers what gesture that the user will perform and not necessarily what the physical attributes of the object they are controlling is. This restriction provides a rationale for combining EMG with computer vision so as to obtain information about user-intent and object context.

In 2025, an integrated EMG and voice command system for controlling a prosthetic hand with six degrees of freedom was proposed using machine learning. The researchers have acquired EMG data from five hand gestures, and considered several classifiers and established a proper classifier model. The EMG part with medium NN showed 76% of classification accuracy and voice-command recognition with DTW demonstrated 94.8% accuracy of classification. The benefits of multimodal control are shown in the study, which could be valuable to achieve high flexibility and reduce the reliance on a single input modality. The system does not, however, directly make use of environmental object properties as part of a decision for a suitable grasp—only EMG information and voice are used.

Shankara and Tamilarasi (2025) explored hand-gesture classification from EMG signals with the deep-learning model with hyper-parameters tuning. They were interested in the classification of single/multiple fingers movement using EMG signals for the purpose of controlling a dexterous prosthetic hand. With the recent advances of utilizing optimized deep-learning models to capture meaningful patterns from EMG signals and enhance the gesture recognition process, the work demonstrates the utilization of these. But the studies focus on EMG-Gesture Recognition and EMG-Based Gesture classification and do not address the issue of real-time object recognition or vision based grasp selection explicitly. So, the multimodal approach implemented in this work via EMG-based intent recognition and vision-based object detection is an important direction for research.

Table 1: Comparative Analysis of Recent Literature

Based on the limitations identified in these previous works, this work proposes a hybrid EMG-SVM intelligent prosthetic grasping system with the aim of increasing grasping accuracy while minimizing the computation load. Unlike other systems, where sensing and decision-making can be treated as distinct stages, the proposed design is an integrated architecture, where the process of collecting the physiological signals, extracting the features using computer vision and deciding to take an action using embedded machine learning is all unifying under a single control system.

Proposed model of Artificial Intelligence and Object Recognition

The suggested model combines EMG sensors and object recognition technology to enable prosthetic or robotic arms to function more effectively. ²⁹ The following steps explain how the proposed model works:

Step 1: Signal Acquisition and Processing

EMG sensors are placed on the user’s skin to capture EMG waveforms from the arm²². A serial port connects these signals to the Arduino. The Arduino converts the analog signals into digital signals. Then, the TensorFlow and Python programming environment on the Raspberry Pi process these digital signals.¹²

Step 2: Real-Time Image Acquisition and Hand Segmentation

A camera takes pictures in real time based on how the user moves their hands. The Raspberry Pi processes the pictures in real time while running the Raspbian operating system. Hand segmentation is an important step in making sure that the hand is visibly distinct from the backdrop in a picture or video frame. This makes sure that no part of the hand gets mixed up with the backdrop, and vice versa. The kind of picture and the specific usage, which might be very different, have a big effect on which segmentation method is utilised.²⁴ Testing shown that a certain data partitioning strategy produced consistent outcomes for hand gesture recognition.¹³ The technique is an unsupervised, non-parametric method that automatically calculates the appropriate threshold for dividing the data. This approach is more versatile because it doesn’t need any specified parameters.

The following equation illustrates the main idea behind the hand segmentation process:

This equation reveals that LL stands for the grey levels, which are levels 1, 2, 3, and so on, that correspond to the pixels in the picture. Ni is the number of pixels in each level. The total number of pixels in the image, MN, is the sum of the pixel counts at all of these levels. This equation is really important to the way Raphael came up with it. In this system, each frame of real-time video is considered like a collection of pixels. After that, these frames are compared to establish the best threshold. This approach finds pixels by looking at how they line up across frames. This is achievable because the colour of each pixel stays the same from frame to frame. If two frames don’t contain the same pixels, those pixels are taken out.¹⁴ To obtain the appropriate threshold for segmentation, we apply Otsu’s thresholding approach. This approach look at all potential threshold values and figures out the difference between the pixel values on each side of the threshold (for the foreground and background).³³ The best threshold is the one that makes the variation of the foreground and background as low as possible. This makes the hand stand out the most against the background.

Figure 1: Segmentation Process Click here to View table

Figure 1 shows how the segmentation process works. First, a greyscale filter is applied to a real-time video frame, and then a thresholding step is used to separate the hand from the background. This is a good way to get the hand image out of the video stream.

Step 3: Training and Integration

The EMG sensor and live camera data are used to teach the robotic arm how to move.³⁴ This training teaches the arm how to tell different objects apart and pick them up based on their features and what the user likes.

Step 4: Control and Task Completion

After the target object is found, the processed signals are sent to the Arduino. The Arduino controls the servo motors, which move the bionic or robotic arm. ²⁷ The arm then successfully completes the task of grasping.

Figure 2: Block diagram Proposed model Click here to View Figure

Materials and Methods

The proposed research is to develop a low-cost, autonomous AI bionic hand utilising an sensor and an Arduino board with a Raspberry Pi computer³¹. Python programming and machine learning algorithms are utilized to communicate EMG data to the Raspberry Pi. EMG signal processing involves several key steps to ensure accurate and consistent results. These include eliminating noise through band-pass filtering (20-450 Hz), rectifying the signals, and extracting features such as RMS, mean absolute value, and waveform length. Additionally, normalization is utilized to maintain a standardized input for our model. The specific details of our training process are as follows: Our chosen model is SSD-MobileNetV2 which has proven effective in previous studies. We have selected the Microsoft COCO dataset for its comprehensive range of images. The input size we have settled on is 224 x 224 pixels while maintaining a confidence threshold of 0.6 during analysis. To enhance performance further, we incorporate transfer learning by utilizing pre-trained weights. In order to seamlessly integrate EMG signals with visual outputs, an advanced synchronization mechanism has been implemented using serial communication protocol technology.The Raspberry Pi serves as the central hub where both inputs are processed simultaneously before sending real-time control signals to Arduino devices.This allows us to efficiently coordinate data from multiple sources without any delays or disruptions.

Method Setup

Figure 3 The experimental setup consists of a sensor attached to a rubber band that expands and contracts to adjust its pressure. It is connected using either a serial port or a Bluetooth serial port. TensorFlow and Python are used to process the data from the movement. A Raspberry Pi with Raspbian OS, TensorFlow, and Python platforms was used, providing code accessibility through Jupyter Lab for debugging purposes.

Experimental Results Analysis

This section provides a framework for integrating EMG signals and real-time camera data to control a robotic arm. ³⁷ This model combines signal processing, object recognition, user intent detection, and robotic arm motion to enable complex tasks such as object recognition and grasping based on the user’s intent. The proposed research aims to develop a low-cost, autonomous AI bionic hand using an Arduino board with an EMG sensor and a Raspberry Pi computer. Python programming and machine learning algorithms are used to transmit the EMG data to the Raspberry Pi.

Figure 3: Experimental Setup Click here to View Figure

The setup includes two methods

Method 1: Use image processing to learn how to operate things and recognise objects

The robotic arm to perceive and pick up things by utilizing both classic image processing and deep learning-based item recognition. A camera is always recording real time footage of what is around it. Next, each frame is pre-processed by eliminating noise, changing the colours, and resizing the image so it can be analysed. Then, the frames are put into a pre-trained object identification model, such MobileNet with SSD, which was trained on the Microsoft COCO dataset. There are pictures of things we use every day in this dataset, such phones, computers, mugs, and bottles.

When the model is sure it has found an item, it utilises a lookup table to figure out what kind of grip it needs. The way you hold anything will depend on its size and form. You could need to use your whole hand to pick up a bottle, yet you might just need two fingers to pick up a little thing like a remote control. After that, the chosen grip type is turned into control signals, which are values for pulse-width modulation (PWM). A serial connection transmits these numbers to an Arduino board.²⁸

The Arduino interprets these signals and turns the servo motors that are linked to the robot’s fingers. To get the right grasp on the item, each servo turns to a certain angle. The robotic arm can make its own choices and interact with things in the real world just by looking at them. This system operates like a person who knows how to pick up an object in the optimal method.

This approach doesn’t use real-time feedback from the robotic arm (it works in a one-way, feedforward style), but it works well and is good for small embedded systems like the Raspberry Pi. This also shows that you can make a working and clever grasping device out of cheap parts and smart software. The system utilises a confidence threshold to get rid of detections that aren’t sure. This helps keep mistakes from happening and makes sure the findings are correct. This technology lets the robot change how it sees and picks up things in real time. Autonomous robots are a wonderful approach to learn how to move things around.²⁹

Method: Recognizing finger movements in real time and having a robot hand copy them

The second method was to have the robotic arm-based system which imitate a human hand in real time by capturing video data. This method doesn’t put a lot of weight on identifying objects. Instead, then focusing on what a human is doing with their fingers, it lets the robotic arm follow along like a mirror image.³³ We started with a camera that tracks the hand as it moves. The camera is always recording video, and the system carefully looks at each picture to see where the fingers are and how they are moving. We employed powerful machine vision techniques like Media Pipe or Open Pose hand tracking models to determine the most essential locations on each finger. These key places let the system figure out which way each finger is travelling and how bent or straight it is. ³⁴ After it knows what the hand movements mean, the system swiftly transforms the information into orders for the robot arm. To achieve this, it finds out how far each robotic finger needs to travel to perform the same thing. An Arduino takes these signals and utilises them to run the motors that move the fingers of the robotic arm.

When the robot starts to replicate the human hand movement, the robot arm can perform the same thing when the individual bends their index finger. When a human forms a fist or spreads their fingers, the robot does the same way. You don’t have to wait or click any buttons; everything happens through realtime visual recognition. It’s like directing the robot to look like you and giving it eyes. This way of doing things makes a lot of sense and seems natural. You don’t have to show the system how to execute every potential movement or gesture. It merely looks, thinks, and does. This is very useful for prosthetics, teleoperation, rehabilitation, or remote-controlled robots—anywhere you want a robot to move like a human hand.

Results

The accuracy of object grasping was measured using servo motors to control finger movements. Three servo motors (S1, S2, S3) controlled fingers (F1-F4) and the thumb (T1). Table 1 summarizes the accuracy and corresponding motor movements for each object.

Table 2: show (a)Real-Time Object Recognition (Label), (b)Accuracy Percentage, (c)Servo Motors (S1, S2, S3) (d)Fingers Movement (F1, F2, T1, F3, F4), (e)Servo Motor Movement (Degrees).

Dataset Used: Microsoft COCO (Common Objects in Context)

Size & Scope: Contains ~300,000 images labeled with objects in realistic scenes.

Categories: 90 object classes including mobile phones, laptops, bottles, chairs, and more.

Use: Widely adopted for training and benchmarking object detection models.

Relevance: Provides high-quality annotations and diversity, making it ideal for training the TensorFlow Object Detection API.

Models Evaluated

Single Shot Multibox Detector (SSD)

• A fast, lightweight model that performs object detection in a single forward pass.
• Suitable for real-time applications but may trade accuracy for speed.

MobileNet

• A lightweight CNN architecture optimized for mobile and edge devices.
• Often combined with SSD (SSD-MobileNet) for real-time, on-device object detection.

SSD with Inception

• Combines SSD with Inception backbone for improved feature extraction.
• Offers a better balance between speed and accuracy compared to plain SSD.

Region-based Fully Convolutional Network (R-FCN)

• Region-based model that uses position-sensitive score maps.
• Offers good accuracy and is less computationally intensive than Faster R-CNN.

Faster R-CNN

• High-accuracy model using a Region Proposal Network (RPN) and convolutional backbone.
• More accurate but slower, best for applications where speed is less critical.

Figure 4: Model accuracy based on different ML (models) Click here to View Figure

Figure 5: Models mean average precision Click here to View Figure

Model and Serial Setup

model = MobileNetV2(weights=’imagenet’)

arduino = serial.Serial(‘/dev/ttyUSB0’, 9600, timeout=1)

cap = cv2.VideoCapture(0)

Loads a pre-trained MobileNetV2 object detection model.

Initializes a serial connection to the Arduino via USB.

Starts the camera for real-time video capture.

Object-to-Gesture Mapping

OBJECT_GESTURE_MAP = {

‘bottle’: 1,

‘cellular_telephone’: 2,

‘apple’: 3,

…

}

Maps specific recognized objects to predefined gesture IDs.

These IDs represent finger movement patterns.

Used later to decide how the hand should move.

Figure 6: Object grasping accuracy according to objectives Click here to View Figure

Frame Capture and Prediction 

img = cv2.resize(frame, (224, 224))

img_array = np.expand_dims(img, axis=0)

img_array = preprocess_input(img_array)

preds = model.predict(img_array)

Captures and resizes the video frame to match model input size.

Preprocesses the image before sending it to the neural network.

Gets prediction results (top-1 object with probability).

Decision and Communication 

if confidence > 0.6:

gesture_id = get_gesture_id(object_name)

arduino.write(f”{gesture_id}\n”.encode())

If the model is confident enough, map the object to a gesture ID.

Sends the ID as a string over serial to Arduino.

Arduino will read this to control servos.

Display and Cleanup

cv2.putText(…); cv2.imshow(…)

cap.release(); cv2.destroyAllWindows(); arduino.close()

Displays the prediction on the live video feed.

Releases the camera and closes serial port safely on exit.

Arduino Code (servo_controller)

Setup and Initialization 

Servo fingers[5];

int servoPins[5] = {3, 5, 6, 9, 10};

…

for (int i = 0; i < 5; i++) fingers[i].attach(servoPins[i]);

Creates 5 servo objects for the 5 fingers.

Assigns each servo to a specific pin on the Arduino.

Attaches servo motors to control signals.

Figure 7: Finger movement according to respective objects Click here to View Figure

Listening for Gesture ID

if (Serial.available() > 0) {

int gesture = Serial.parseInt();

executeGesture(gesture);

}

Continuously checks for incoming data from Raspberry Pi.

Parses the incoming gesture ID (integer).

Calls a function to perform the corresponding finger motion.

Gesture Execution

switch (gesture) {

case 1: // grasp bottle

case 2: // phone grip

case 3: // apple hold

…

}

Based on the ID, it moves fingers to specific angles.

Each gesture simulates a different type of grip or hand shape.

Servo.write(angle) sets servo position (0° open to 90° closed).

Figure 8: Number of servo(S1,S2,S3) motor used according to object Click here to View Figure

Hand Control Functions

void openHand() { … }

void closeHand() { … }

openHand(): sets all fingers to open position (0°).

closeHand(): fully closes all fingers (90°).

These are called directly or used within gesture. These models varied in speed and mean average precision (mAP), balancing lightweight design with recognition accuracy. Experimental results indicate an accuracy range of 65% to 85% for various objects.

Figure 9: Servo motor degree movement ranges as per object Click here to View Figure

Figure 10: Object detection confusion matrix according to object Click here to View Figure

MobileNetV2 was selected due to its lightweight architecture, low computational cost, and suitability for real-time embedded systems such as Raspberry Pi. Compared to Faster R-CNN and R-FCN, it provides a better trade-off between speed and accuracy.

Dataset Limitation

Although the COCO dataset provides diverse object categories, it is not specifically designed for prosthetic grasping tasks. This limitation may affect real-world generalization.

Comparison traditional and purposed models

Table 3: Comparison Table

The traditional EMG prosthetic, though it may have a medium response time and high cost due to its reliance on only 2-3 sensors, boasts an accuracy range of 65-75%. On the other hand, the vision-based system offers slightly better results with an accuracy rate of 70-80%, but at the expense of slower responses and requiring more equipment (3-4 cameras). However, our proposed system surpasses both in terms of emotional intelligence by combining both methods for enhanced performance. With a remarkable success rate ranging from 73% to as high as87%, this innovative approach uses just three sensors while still maintaining fast response times thanks to its use of EMG + Vision technology. And not only does it offer superior results, it also comes at a lower cost compared to its counterparts. In short: The old ways are good,but they can be improved upon.Our proposal combines multiple approaches,resulting in greater efficiencyat reduced costswhile prioritizing speedand utilizingemotionalintelligence for even better outcomes.In summary,the choice is clear – embrace progress or settle for mediocrity.

Discussion 

The proposed system showcases the powerful combination of EMG-based control and vision-based object recognition in revolutionizing prosthetic operation. This hybrid approach not only enhances adaptability but also provides an intuitive means of control, surpassing traditional systems. By utilizing lightweight models, real-time performance is attainable even on embedded platforms. Despite its promising potential, certain limitations do exist within this novel system. The variability of EMG signals across different users necessitates frequent calibration and adaptation processes for optimal functioning. Furthermore, factors such as lighting conditions and occlusion can significantly impact the overall performance of the system. Moreover, prolonged usage may result in user fatigue due to continuous muscle activation – a concern that must be taken into account when implementing this technology. As we look towards future developments in this field, incorporating adaptive learning models along with personalized calibration techniques and haptic feedback systems will undoubtedly enhance usability while providing a more realistic experience for users. It goes without saying that ethical considerations play a crucial role in any technological advancement – especially one involving human interaction like intelligent prosthetics operations.This requires careful consideration regarding aspects such as user safety measures,reliable data privacy protocols,and long-term sustainability before implementation. ³

Conclusion 

The proposed model is a robust technological implementation of an AI-enabled object detection and control system for bio-robotic prosthetic arms. Using a deep learning model trained on the Microsoft COCO dataset, this system can identify objects and classify their grasp in real time. This approach goes beyond standard vision-based recognition by tracking real hand movements using sensor-based gesture mapping. This allows the bionic arm to mimic human finger and wrist movements with much greater accuracy. This two-way system allows the user to control their grip and move in a way that closely resembles the gestures of their natural hand.

Although the model has been successful, it still has some limitations, such as:

1. Variations in detection accuracy due to the complexity and occlusion of the object.
2. Latency and performance issues due to processing hardware and camera frame rate.

In the future, this can be improved by incorporating advanced haptic feedback systems that mimic the sensations of touch and pressure, utilizing electromyographic (EMG) signal fusion to facilitate intent recognition, and employing lightweight, optimized neural networks to make the system run faster and consume less power. This direction will significantly enhance the naturalness and responsiveness of prosthetic control, resulting in a seamless connection between what a person intends and what the robot performs.

Acknowledgement

Heartfelt thanks to Graphic Era deemed to be University for providing adequate facilities in carrying out this study.

Funding Sources 

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Conflict of Interest 

The authors do not have any conflict of interest.

Data Availability Statement

This statement does not apply to this article.

Ethics Statement

This research did not involve human participants, animal subjects, or any material that requires ethical approval.

Informed Consent Statement 

This study did not involve human participants, and therefore, informed consent was not required.

Permission to reproduce material from other sources

This statement does not apply to this article.

Clinical Trial Registration

This statement does not apply to this article.

Authors’ Contribution 

• Anuj Singh: Conceptualization, Methodology, Writing, Data Collection, Analysis, Writing – Original Draft.
• Kamlesh Chandra purohit: Supervision – Review & Editing.

Reference

1. Zhang L, Wang H, Liu Y. Vision-augmented robotic manipulation with hybrid neural networks. Rob Robot Auton Syst. 2023;162:104313.
2. Chen J, Zhao Q, Zhang T. Hybrid EMG and vision-based approaches for enhancing robotic grasping precision. IEEE Trans Mechatron. 2023;28(4):561-570.
3. Li K, Wang S, Wu R. Real-time grasp prediction using multi-sensor fusion. J Robotics. 2020;12(7):375-387.
4. Ortiz-Catalan M, Navarro X, Farina D. Next-generation interfaces for neuro-controlled robotic arms. Biomed Eng. 2023;7(5):421-432.
5. Lonsdale A, Cooper G, Marshall R. Deep reinforcement learning for prosthetic hand control. Front Robot AI. 2020;6:45.
6. Hanson E, Rodriguez P, Fletcher L. EMG signal processing for intention detection in prosthetics. IEEE Sens J. 2020;20(2):645-653.
7. Basumatary S, Singh K, Kumar P. Object recognition using convolutional neural networks in robotic applications. Appl Intell. 2023;53(1):130-145.
8. Linares-Barranco B, Perez J, Torres M. Neuromorphic computing for EMG and vision-based robotic systems. IEEE Trans Neural Syst Rehabil Eng. 2024;32(1):24-33.
9. Imran H, Khan A, Ahmed F. Integration of EMG and vision data for real-time robotic grasping tasks. J Mechatron Autom. 2023;15(8):982-993.
10. Zhang G, Davoodnia V, Sepas-Moghaddam A, Zhang Y, Etemad A. Classification of hand movements from EEG using a deep attention-based LSTM network. IEEE Sens J. 2019;20(6):3113-3122.
    CrossRef
11. Abdulkader SN, Atia A, Mostafa MSM. Brain computer interfacing: applications and challenges. Egypt Inform J. 2019;16(2):213-230.
    CrossRef
12. Pancholi S, Joshi AM. Improved classification scheme using fused wavelet packet transform-based features for intelligent myoelectric prostheses. IEEE Trans Ind Electron. 2019;67(10):8517-8525.
    CrossRef
13. Pancholi S, Joshi AM. Electromyography-based hand gesture recognition system for upper limb amputees. IEEE Sens Lett. 2019;3(3):1-4.
    CrossRef
14. Jeong JH, Shim KH, Kim DJ, Lee SW. Brain-controlled robotic arm system based on a multidirectional CNN-BiLSTM network using EEG signals. IEEE Trans Neural Syst Rehabil Eng. 2020;28(5):1226-1238.
    CrossRef
15. Jamal MZ. Signal acquisition using surface EMG and circuit design considerations for robotic prosthesis. In: Computational Intelligence in Electromyography Analysis. 2012;18:427-448.
16. Pancholi S, Joshi AM. Portable EMG data acquisition module for upper limb prosthesis application. IEEE Sens J. 2018;18(8):3436-3443.
    CrossRef
17. Pancholi S, Joshi AM. Time derivative moments-based feature extraction approach for recognition of upper limb motions using EMG. IEEE Sens Lett. 2019;3(4):1-4.
    CrossRef
18. Buerkle A, Eaton W, Lohse N, Bamber T, Ferreira P. EEG-based arm movement intention recognition towards enhanced safety in symbiotic human–robot collaboration. Rob Comput Integr Manuf. 2021;70:102137.
    CrossRef
19. Kokate P, Pancholi S, Joshi AM. Classification of upper arm movements from EEG signals using machine learning with ICA analysis. arXiv. 2021;arXiv:2107.08514.
20. Zhan Hong L, Zourmand A, Patricks JV, Thing GT. EEG-based brain wave controlled intelligent prosthetic arm. In: Proc IEEE 8th Conf Syst Process Control (ICSPC). 2020.
    CrossRef
21. Gannouni S, Belwafi K, Aboalsamh H, Alebdi B, Almassad Y, AlSamhan Z, Alobaedallah H. EEG-based BCI system to control prosthesis finger movements. 2020.
    CrossRef
22. Subrata AC, Riyadi MA, Prakoso T. EEG-based BMI using multi-class motor imagery for bionic arm. In: Proc 3rd Int Conf Mech Electron Comput Ind Technol (MECnIT). 2020.
    CrossRef
23. Ahmed S, Mitra A, Saha S, Ali MA, Bhattacharjee A. Fabrication of cost-effective prosthetic arm using electroencephalography signals. In: Proc Int Conf Ind Mech Eng Oper Manag. 2020.
24. Alazrai R, Alwanni H, Daoud MI. EEG-based BCI system for decoding finger movements within the same hand. Neurosci Lett. 2019.
    CrossRef
25. Ruhunage I, Mallikarachchi S, Chinthaka D, Sandaruwan J, Lalitharatne TD. Hybrid EEG-EMG signals-based approach for control of hand motions of a transhumeral prosthesis. In: Proc IEEE Global Conf Life Sci Technol (LifeTech). 2019.
    CrossRef
26. Pawar B, Bhatt H. Design and development of electroencephalography-based cost-effective prosthetic arm controlled by brain waves. Int Res J Eng Technol. 2019;4(3).
27. Idowu OP, Fang P, Li X, Xia Z, Xiong J, Li G. Towards control of EEG-based robotic arm using deep learning via stacked sparse autoencoder. In: Proc IEEE Int Conf Robot Biomimetics (ROBIO). 2018.
    CrossRef
28. Chinbat O, Lin JS. Prosthetic arm control by human brain. In: Proc Int Symp Comput Consum Control (IS3C). 2018.
    CrossRef
29. Gayathri G, Udupa G, Nair GJ, Poorna SS. EEG-controlled prosthetic arm for micromechanical tasks. Adv Intell Syst Comput. 2018:281-291.
    CrossRef
30. Ruhunage I, Perera CJ, Nisal K, Subodha J, Lalitharatne TD. EMG signal controlled transhumeral prosthetic with EEG-SSVEP based approach for hand open and close. In: Proc IEEE Int Conf Syst Man Cybern (SMC). 2017.
    CrossRef
31. Ghaemi A, Rashedi E, Pourrahimi AM, Kamandar M, Rahdari F. Automatic channel selection in EEG signals for classification of left or right hand movement in brain computer interfaces using improved binary gravitation search algorithm. Biomed Signal Process Control. 2017;33:109-118.
    CrossRef
32. Moreira AH, Barbosa FSM, Ikeda GHA, Carvalho GM, Madani FS, Trabasso LG. Development of a hybrid arm prosthesis controlled by EEG signals. In: Proc 2nd Int Conf Cybern Robot Control (CRC). 2017.
    CrossRef
33. Bright D, Nair A, Salvekar D, Bhisikar S. EEG-based brain-controlled prosthetic arm. In: Proc Conf Adv Signal Process (CASP). 2016.
    CrossRef
34. Beyrouthy T, Al Kork SK, Korbane JA, Abdulmonem A. EEG mind-controlled smart prosthetic arm. In: Proc IEEE Int Conf Emerg Technol Innov Bus Pract. 2016.
    CrossRef
35. Elstob D, Lindo Secco E. A low-cost EEG-based BCI prosthetic using motor imagery. Int J Inf Technol Converg Serv. 2016;6(1):23-36.
    CrossRef
36. Roy R, Mahadevappa M, Kumar CS. Trajectory path planning of EEG-controlled robotic arm using genetic algorithm. Procedia Comput Sci. 2016;84:147-151.
    CrossRef
37. Zhang L, Wang H, Liu Y. Vision-augmented robotic manipulation with hybrid neural networks. Rob Robot Auton Syst. 2023;162:104313.
38. Chen J, Zhao Q, Zhang T. Hybrid EMG and vision-based approaches for enhancing robotic grasping precision. IEEE Trans Mechatron. 2023;28(4):561-570.
39. Linares-Barranco B, Perez J, Torres M. Neuromorphic computing for EMG and vision-based robotic systems. IEEE Trans Neural Syst Rehabil Eng. 2024;32(1):24-33.
40. Singh A, Sharma S, Purohit K C. Intelligent bio robotic arm for mapping signal control and synchronisation for motor control system using Human Computer Interface. Front Health Inform. 2024;13(3):1923-1936.
41. Khan R, Alotaibi S, Alharbi A. Deep learning-based prosthetic hand control using multimodal sensor fusion. Sensors. 2025;25(2):1187.
    CrossRef
42. Shi, P., Fang, K., & Yu, H. (2023). Design and control of intelligent bionic artificial hand based on image recognition. Technology and Health Care, 31(1). https://doi.org/10.3233/THC-213320.
    CrossRef
43. Zandigohar, M., Han, M., Sharif, M., Günay, S. Y., Furmanek, M. P., Yarossi, M., Bonato, P., Onal, C., Padır, T., Erdoğmuş, D., & Schirner, G. (2024). Multimodal fusion of EMG and vision for human grasp intent inference in prosthetic hand control. Frontiers in Robotics and AI, 11, 1312554. https://doi.org/10.3389/frobt.2024.1312554.
    CrossRef
44. Tchimino, J., Hansen, R. L., Jørgensen, P. H., Dideriksen, J., & Dosen, S. (2024). Application of EMG feedback for hand prosthesis control in high-level amputation: A case study. Scientific Reports, 14, 31676. https://doi.org/10.1038/s41598-024-80828-x.
    CrossRef
45. Sarker, M. A. B., Sola, J. P. S., Jones, A., Laing, E., Sola-Thomas, E., & Imtiaz, M. H. (2024). Vision Controlled Sensorized Prosthetic Hand. arXiv:2407.12807.
46. Xu, Y., Wang, X., Li, J., Zhang, X., Li, F., Gao, Q., Fu, C., & Leng, Y. (2024). A powered prosthetic hand with vision system for enhancing the anthropopathic grasp. arXiv:2412.07105.
    CrossRef
47. Dunai, L., Seguí Verdú, I., Turcanu, D., & Bostan, V. (2025). Prosthetic hand based on human hand anatomy controlled by surface electromyography and artificial neural network. Technologies, 13(1), 21. https://doi.org/10.3390/technologies13010021.
    CrossRef
48. Takele, A., & Negash, [verify initials from final published version]. (2025). sEMG-based control of prosthesis hand using LSTM classification and SMC.
    CrossRef
49. Le, D.-H., & Nguyen, T.-N. (2025). Prosthetic arm controller based on the sEMG signal and deep learning network. Elektronika ir Elektrotechnika, 31(6), 23–31. https://doi.org/10.5755/j02.eie.43939.
    CrossRef
50. Six-degree-of-freedom prosthetic hand with EMG and voice control (2025) Development of six-degree-of-freedom prosthetic hand with integrated voice and electromyography control using machine learning. Engineering Applications of Artificial Intelligence, 142, 109949 (2025). https://doi.org/10.1016/j.engappai.2024.109949.
    CrossRef

Abbreviations

3D – Three-dimensional

2-DOF -two degrees of freedom

COCO-Common Objects in Context

EMG– Electromyography

PWM-pulse-width modulation

SSD– Single Shot Multibox Detector

CNN– convolution neural network

(R-FCN)-Region-based Fully Convolutional Network

RPN– Region Proposal Network

 ML– Machine Learning

 Open CV- open computer version