VR-Telecontrol of Multi-Arm Devices: Problems, Hypotheses, Problem Statement

Main Article Content

Abstract

The article discusses various solutions that exist in the field of remote control of robotic devices equipped with manipulators. New approaches are presented for organizing joint telecontrol of multiple manipulators using various user inputs. The following usage scenarios are considered: the architecture of a system with many manipulators and user control interfaces, including such promising areas as deep machine learning and neural interfaces.

Article Details

References

1. Pratt G., Manzo J. The darpa robotics challenge [competitions]. IEEE Robotics & Automation Magazine. 2013. V. 20(2). P. 10–12.
2. Artigas J. et al. Kontur-2: Force-feedback teleoperation from the international space station // Proc. IEEE Int. Conf. Robot. Automat. (ICRA). 2016. P. 1166–1173.
3. Bluethmann W. Robonaut: A robot designed to work with humans in space // Auton. Robots. 2003. Vol. 14, no. 2. P. 179–197.
4. Sheridan T. Space teleoperation through time delay: Review and prognosis // IEEE Trans. Robot. Automat. 1993. Vol. 9, no. 5. P. 592–606.
5. Negrello F. Humanoids at work: The walk-man robot in a postearthquake scenario // IEEE Robot. Automat. Mag. 2018. Vol. 25, no. 3. P. 8–22.
6. Hirche S., Stanczyk B., Buss M. Transparent exploration of remote environments by internet telepresence // Proc. Int. Workshop High-Fidelity Telepresence Teleaction jointly with Conf. HUMANOIDS. 2003. P. 1–20.
7. Khatib O. et al. Ocean one: A robotic avatar for oceanic discovery // IEEE Robot. Automat. Mag. 2016. Vol. 23, no. 4. P. 20–29.
8. Shukla A., Karki H. Application of robotics in onshore oil and gas industry-a review part I // Robot. Auton. Syst. 2016. Vol. 75. P. 490–507.
9. Kolyubin S.A. Dinamika robototekhnicheskih sistem / Uchebnoe posobie // SPb.: Universitet ITMO, 2017. 117 s.
10. Glynn S., Fekieta R., Henning R. Use of force-feedback joysticks to promote teamwork in virtual teleoperation // Proc. Hum. Factors Ergonom.Soc. Annu. Meeting. 2001. Vol. 45. P. 1911–1915,
11. Martinez-Palafox O., Lee D., Spong M.W., Lopez I., Abdallah C.T. Bilateral teleoperation of mobile robot over delayed communication network: Implementation // Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst., 2006. P. 4193–4198.
12. Selvaggio M., Giordano P.R., Ficuciello F., Siciliano B. Passive task- prioritized shared-control teleoperation with haptic guidance // Proc. Int. Conf. Robot. Automat. (ICRA), 2019. P. 430–436.
13. Materna Z. et al. Teleoperating assistive robots: A novel user interface relying on semi-autonomy and 3D environment mapping // J. Robot. Mechatronics, 2017. Vol. 29. P. 381–394.
14. Garate V.R., Gholami S., Ajoudani A. A scalable framework for multi-robot tele-impedance control // IEEE Trans. Robot., 2021. Vol. 37, no. 6. P. 2052–2066.
15. Clark J.P., Lentini G., Barontini F., Catalano M.G., Bianchi M., O’Malley M.K. On the role of wearable haptics for force feedback in teleimpedance control for dual-arm robotic teleoperation // in Proc. Int. Conf. Robot. Automat. (ICRA), 2019. P. 5187–5193.
16. Zhang T. et al. Deep imitation learning for complex manipulation tasks from virtual reality teleoperation // Proc. IEEE Int. Conf. Robot. Automat. (ICRA), 2018. P. 5628–5635.
17. Lipton J.I., Fay A.J., Rus D. Baxter’s homunculus: Virtualreality spaces for teleoperation in manufacturing // IEEE Robot. Automat. Lett., 2018. Vol. 3, no. 1. P. 179–186.
18. Koenemann J., Burget F., Bennewitz M. Real-time imitation of human whole-body motions by humanoids // Proc. IEEE Int. Conf. Robot. Automat. (ICRA), 2014. P. 2806–2812.
19. Rakita D., Mutlu B., Gleicher M. A motion retargeting method for effective mimicry-based teleoperation of robot arms // Proc. IEEE/ACM 12th Int. Conf. Hum.-Robot Interaction, 2017. P. 361–370.
20. Ajoudani A., Fang C., Tsagarakis N., Bicchi A. Reduced-complexity representation of the human arm active endpoint stiffness for supervisory control of remote manipulation // Int. J. Robot. Res., 2018. Vol. 37, no. 1. P. 155–167.
21. Ajoudani A., Tsagarakis N., Bicchi A. Tele-impedance: Teleoperation with impedance regulation using a body-machine interface // International Journal of Robotics Research. 2012. Vol. 31(13). P. 1642–1655.
22. Hocaoglu E., Patoglu V. Tele-impedance control of a variable stiffness prosthetic hand. // IEEE international conference on robotics and biomimetics (ROBIO). Piscataway, NJ: IEEE. 2012, P. 1576–1582.
23. Gribble P.L., Mullin L.I., Cothros N. et al. Role of cocontraction in arm movement accuracy // Journal of Neurophysiology. 2003. Vol. 89(5). P. 2396–2405.
24. Akazawa K., Milner T.E., Stein R.B. Modulation of reflex EMG and stiffness in response to stretch of human finger muscle // Journal of Neurophysiology. 1983. Vol. 49(1). P. 16–27.
25. Trumbower R.D., Krutky M.A., Yang B.S. et al. Use of self-selected postures to regulate multi-joint stiffness during unconstrained tasks // PLoS ONE. 2009. Vol. 4(5). Art. e5411.
26. Mussa-Ivaldi F.A., Hogan N., Bizzi E. Neural, mechanical, and geometric factors subserving arm posture in humans // Journal of Neuroscience. 1985. Vol.5(10). P. 2732–2743.
27. Perreault E.J., Kirsch R.F., Crago P.E. Voluntary control of static endpoint stiffness during force regulation tasks // Journal of Neurophysiology. 2002. Vol. 87(6). P. 2808–2816.
28. Franklin DW, Burdet E, Osu R, et al. Functional significance of stiffness in adaptation of multijoint arm movements to stable and unstable dynamics // Experimental Brain Research. 2003. Vol. 151(2). P. 145–157.
29. Edsinger A., Kemp C.C. Two arms are better than one: A. behavior based control system for assistive bimanual manipulation // Recent Progress in Robotics: Viable Robotic Service to Human. Berlin, Germany: Springer, 2007. P. 345–355.
30. Makris S., Tsarouchi P., Surdilovic D., Krüger J. Intuitive dual arm robot programming for assembly operations // CIRP Ann., 2014. Vol. 63, no. 1. P. 13–16.
31. Smithetal C. Dual arm manipulation–asurvey // Robot.Auton.Syst., 2012. Vol. 60, no. 10. P. 1340–1353.
32. Rakita D., Mutlu B., Gleicher M., Hiatt L.M. Shared control–based bimanual robot manipulation // Sci. Robot., 2019. Vol. 4, Art. no. eaaw0955.
33. Sun D., Liao Q., Loutfi A. Single master bimanual teleoperation system with efficient regulation // IEEE Trans. Robot., 2020. Vol. 36, no. 4. P. 1022–1037.
34. Lin T.-C., Unni Krishnan A., Li Z. Shared autonomous interface for reducing physical effort in robot teleoperation via human motion mapping // Proc. IEEE Int. Conf. Robot. Automat. (ICRA), 2020. P. 9157–9163.
35. Amanhoud W., Hernandez Sanchez J., Bouri M., Billard A. Contact-initiated shared control strategies for four-arm supernumerary manipulation with foot interfaces // Int. J. Robot. Res., 2021. Vol. 40, P. 1–29.
36. Laghi M. et al. Shared-autonomy control for intuitive bimanual telemanipulation // Proc. IEEE-RAS 18th Int. Conf. Humanoid Robots (Humanoids), 2018. P. 1–9.
37. Tyrva V.O. Sovmestnoe upravlenie ob"ektom v ergaticheskoj sisteme: modeli i realizacii // Vestnik Gosudarstvennogo universiteta morskogo i rechnogo flota imeni admirala S.O. Makarova. 2018. T. 10. No. 2. S. 430–443.
38. Tung A. et al. Learning multi-arm manipulation through collaborative teleoperation // Proc. IEEE Int. Conf. Robot. Automat. (ICRA), 2021. P. 9212–9219.
39. Kennel-Maushart F., Poranne R., Coros S. Manipulability optimization for multi-arm teleoperation // Proc. IEEE Int. Conf. Robot. Automat. (ICRA), 2021. P. 3956–3962.
40. Ozdamar I., Laghi M., Grioli G., Ajoudani A., Bicchi A. A Shared Autonomy Reconfigurable Control Framework for Telemanipulation of Multi-Arm Systems // IEEE ROBOTICS AND AUTOMATION LETTERS, 2022. Vol. 7, No. 4. P. 9937–9944.
41. Laghi M. et al. Shared-autonomy control for intuitive bimanual tele- manipulation // Proc. IEEE-RAS 18th Int. Conf. Humanoid Robots (Humanoids), 2018. P. 1–9.
42. Ashkinazi L.A. Mir Lema – slovar' i putevoditel'. 2004. 1068 s.
43. Ozdamar I., Laghi M., Grioli A., Ajoudani G., Catalano M.G., Bicchi A. A Shared Autonomy Reconfigurable Control Framework for Telemanipulation of Multi-Arm Systems // IEEE Robotics and Automation Letters, 2022. Vol.7(4). P. 9937–9944.
44. Florea B.C. Smartphone Controlled Autonomous Robotic Platform // 2018 15th International Conference on Electrical Engineering/Electronics, Computer, Telecommunications and Information Technology (ECTI-CON), Chiang Rai, Thailand, 2018. P. 620–623.
45. Mustafin M., Tsoy T., Martínez-García E.A., Meshcheryakov R., Magid E. Modelling mobile robot navigation in 3D environments: camera-based stairs recognition in Gazebo // 2022 Moscow Workshop on Electronic and Networking Technologies (MWENT), Moscow, Russian Federation, 2022. P. 1–6.
46. Bai Y. , Wang Y., Svinin M., Magid E., Sun R. Adaptive Multi-Agent Coverage Control With Obstacle Avoidance // IEEE Control Systems Letters, 2022. Vol. 6. P. 944–949.
47. Kugurakova V.V., Antonov I.O., Goncharenko B.V., Chajbar A.A. Cifrovoe predstavlenie v virtual'noj real'nosti mesta proisshestviya kak instrument ugolovnogo sudoproizvodstva // Programmnye sistemy: teoriya i prilozheniya, 2022. T. 13, vyp. 3. S. 193–223.
48. Kugurakova, V.V., Golovanova, I.I., Shaidullina, A.R. Digital Solutions in Educators' Training: Concept for Implementing a Virtual Reality Simulator // Eurasia Journal of Mathematics, Science and Technology Education. 2021. Vol. 17, Is. 9. P. 1–10.
49. Tianhao Zhang, Zoe McCarthy, Owen Jow Dennis, Lee Xi Chen, Ken Goldberg, Pieter Abbeel. Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation // eecs.berkeley.edu.URL: https://www2.eecs.berkeley.edu/Pubs/TechRpts/2020/EECS-2020-190.pdf (access date: 01.08.2022).
50. IVRE – An Immersive Virtual Robotics Environment // cirl.lcsr.jhu.edu. URL: https://cirl.lcsr.jhu.edu/research/human-machine-collaborativesystems/ivre/ (access date: 01.08.2022).
51. Reachy by Pollen Robotics, an open source programmable humanoid robot // pollen-robotics.com. URL: https://www.pollen-robotics.com/ (access date: 01.08.2022).
52. Kugurakova V.V., Hafizov M.R., Kadyrov S.A., Zykov E.Yu. Udalennoe upravlenie robotizirovannym ustrojstvom s ispol'zovaniem tekhnologij virtual'noj real'nosti // Programmnye produkty i sistemy. 2022. T. 35. No.3. S. 348–361.
53. Melero M., Hou A., Cheng E., Tayade A., Lee S.C., Unberath M., Navab N., Cutolo F. Upbeat: Augmented Reality-Guided Dancing for Prosthetic Rehabilitation of Upper Limb Amputees // Journal of Healthcare Engineering. 2019. Art. 2163705.
54. Nissler C., Nowak M., Connan M., Büttner S., Vogel J., Kossyk I., Márton Z.C., Castellini C. VITA – An everyday virtual reality setup for prosthetics and upper-limb rehabilitation // Journal of Neural Engineering. 2019. Vol. 16(2). Art. 026039.
55. Saraiji M.Y., Sasaki T., Matsumura R., Minamizawa K., Inami M. Fusion: full body surrogacy for collaborative communication // SIGGRAPH '18: ACM SIGGRAPH 2018 Emerging Technologies. 2018. Vol. 7. P. 1–2.
56. Takizawa R., Hivarna A., Verhulst A., Seaborn K., Fukuoka M., Kitazaki M., Inami M., Suqirnoto M. Parasitic body: Exploring perspective dependency in a shared body with a third arm // 26th IEEE Conference on Virtual Reality and 3D User Interfaces. VR 2019 – Proceedings. 2019. P. 1175–1176.
57. Parietti F., Asada H. Supernumerary Robotic Limbs for Human Body Support // IEEE Transactions on Robotics. 2016. Vol. 32(2). P. 301–311.
58. Sasaki T., Saraiji Y., Minamizawa K., Inami M. MetaArmS: Body remapping using feet-controlled artificial arms // UIST 2018 Adjun[ct – Adjunct Publication of the 31st Annual ACM Symposium on User Interface Software and Technology. 2018. P. 140–142.
59. Tong Y., Liu J. Review of Research and Development of Supernumerary Robotic Limbs // IEEE/CAA Journal of Automatica Sinica. 2021. Vol. 8. No. 5. P. 929–952.
60. Aly H.I., Youssef S., Fathy C. Hybrid Brain Computer Interface for Movement Control of Upper Limb Prostheses. 2018 International Conference on Biomedical Engineering Applications // ICBEA 2018 – Proceedings. 2018. P. 1–6.
61. Aljalal M., Ibrahim S., Djemal R., Ko W. Comprehensive review on brain-controlled mobile robots and robotic arms based on electroencephalography signals // Intelligent Service Robotics. 2020. Vol. 13 (4). P. 539–563.
62. Abdi E., Burdet E., Bouri M., Himidan S., Bleuler H., In a demanding task three-handed manipulation is preferred to two-handed manipulation // Scientific Reports. 2016. Vol. 6. P. 1–11.
63. Sasaki T., Saraiji Y., Minamizawa K., Inami M. MetaArmS: Body remapping using feet-controlled artificial arms // UIST 2018 Adjunct - Adjunct Publication of the 31st Annual ACM Symposium on User Interface Software and Technology. 2018. P. 140–142.
64. Drogemuller A., Verhulst A., Volmer B., Thomas B.H., Inami M., Sugimoto M. Remapping a Third Arm in Virtual Reality // 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), Osaka, Japan, 2019. P. 898–899.
65. Virtual'nyj trenazher upravleniya prizrachnymi konechnostyami cherez interfejs mozg-komp'yuter : Svidetel'stvo o gosudarstvennoj registracii programmy dlya EVM № 202211167 Rossijskaya Federaciya / V.V. Kugurakova, D.G. Ismailov, D.M. Timergalin; zayavitel' i pravoobladatel' Fed. gos. avtonom. obrazovat. uchrezhdenie vyssh. obraz. Kazanskij fed. un-t. №2022610311; zayavl. 13.01.2022; zaregistrirovano v reestre programm dlya EVM 20.01.2022. – [1] s.
66. Zhil'cov A.I., Zhukov V.S., Ryleev D.A. Upravlenie manipulyatorami s chislom stepenej svobody bolee shesti // Inzhenernyj zhurnal: nauka i innovacii, 2013, vyp. 10. 11 c.
67. Kugurakova V.V., Hafizov M.R., Kadyrov S.A., Zykov E.Yu. Udalennoe upravlenie robotizirovannym ustrojstvom s ispol'zovaniem tekhnologij virtual'noj real'nosti // Programmnye produkty i sistemy. 2022. T. 35. No. 3. S. 348–361.
68. Ushakova M.V., Ushakov Yu.A. Issledovanie energosberegayushchej besprovodnoj samoorganizuyushchejsya mnogoprotokol'noj seti peredachi dannyh internet ustrojstv // Sovremennye informacionnye tekhnologii i IT-obrazovanie. 2019. T. 15, No. 3. S. 733–745.
69. Miroshnikova N.E. Obzor sistem kognitivnogo radio // T-Comm – Telecommunications and Transport magazine. 2013. No. 9. S. 108–111.


Most read articles by the same author(s)

1 2 > >>