Deep transfer learning in human–robot interaction for cognitive and physical rehabilitation purposes

Allbwn ymchwil: Cyfraniad at gyfnodolynErthygladolygiad gan gymheiriaid

Fersiynau electronig

Dangosydd eitem ddigidol (DOI)

  • Chaudhary Muhammad Aqdus Ilyas
    Aalborg University
  • Matthias Rehm
    Aalborg University
  • Kamal Nasrollahi
    Aalborg University
  • Yeganeh Madadi
    Aalborg University
  • Thomas B. Moeslund
    Aalborg University
  • Vahid Seydi
    Azad University
This paper presents the extraction of the emotional signals from traumatic brain-injured (TBI) patients through the analysis of facial features and implementation of the effective emotion-recognition model through the Pepper robot to assist in the rehabilitation process. The identification of emotional cues from TBI patients is very challenging due to unique and diverse psychological, physiological, and behavioral challenges such as non-cooperation, facial/body paralysis, upper or lower limb impairments, cognitive, motor, and hearing skills inhibition. It is essential to read subtle changes in the emotional cues of TBI patients for effective communication and the development of affect-based systems. To analyze the variations of the emotional signal in TBI patients, a new database is collected in a natural and unconstrained environment from eleven residents of a neurological center in three different modalities, RGB, thermal and depth in three specified scenarios performing physical, cognitive and social communication rehabilitation activities. Due to the lack of labeled data, a deep transfer learning method is applied to efficiently classify emotions. The emotion classification model is tested through closed-field study and installment of a Pepper robot equipped with the trained model. Our deep trained and fine-tuned emotional recognition model composed of CNN-LSTM has improved the performance by 1.47% on MMI, and 4.96% on FER2013 validation data set. In addition, use of temporal information and transfer learning techniques to overcome TBI-data limitations has increased the performance efficacy on challenging dataset of neurologically impaired people. Findings that emerged from the study illustrate the noticeable effectiveness of SoftBank Pepper robot equipped with deep trained emotion recognition model in developing rehabilitation strategies by monitoring the TBI patient’s emotions. This research article presents the technical solution for real therapeutic robot interaction to rehabilitate patients with standard monitoring, assessment, and feedback in the neuro centers.


Iaith wreiddiolSaesneg
Rhif yr erthygl653-677
Tudalennau (o-i)653-677
Nifer y tudalennau25
CyfnodolynPattern Analysis and Applications
Rhif y cyfnodolyn3
Dyddiad ar-lein cynnar22 Meh 2021
Dynodwyr Gwrthrych Digidol (DOIs)
StatwsCyhoeddwyd - Awst 2022
Cyhoeddwyd yn allanolIe
Gweld graff cysylltiadau