Real-time Hand Interaction and Machine Learning Agents in Immersive Learning Environments
The recent surge in the adoption of new technologies and innovations in connectivity, interaction technology, and artificial realities can fundamentally change the digital world. eXtended Reality (XR), with its potential to bridge the virtual and real environments, creates new possibilities to develop more engaging and productive learning experiences. Evidence is emerging that this sophisticated technology offers new ways to improve the learning process for better student interaction and engagement.
Recently, immersive technology has garnered much attention as an interactive technology that facilitates direct interaction with virtual objects in the real world. Furthermore, these virtual objects can be surrogates for real-world teaching resources, allowing for virtual labs. Thus XR could enable learning experiences that would not be possible in impoverished educational systems worldwide.
Interestingly, concepts such as virtual hand interaction and techniques such as machine learning are still not widely investigated in immersive learning. The need for touchless interaction technologies has exceptionally increased in the post-COVID world. Existing pedagogical approaches like kinesthetic learning have not been explored in detail in immersive learning. By implementing and evaluating real-time hand interaction technology for kinesthetic learning and machine learning agents for self-directed learning, this research has addressed these underutilized technologies to demonstrate the efficiency of immersive learning.
My PhD thesis has explored different hand-tracking APIs and devices to integrate real-time hand interaction techniques. These hand interaction techniques and integrated machine learning agents using reinforcement learning are evaluated with different display devices to test compatibility. The proposed approach aims to provide self-directed, more productive, and interactive learning experiences. Further, this research has investigated ethics, privacy, and security issues in XR and covered the future of immersive learning in the Metaverse.
Metaverse and Emerging opportunities in the Immersive Technologies
The recent surge in the adoption of new technologies and innovations in connectivity, interaction technology, and artificial realities has the potential to change the digital world fundamentally. The concept of the Metaverse is the most recent trend to encapsulate and define the potential new digital landscape. With the introduction of 5G with high speed and low latency, advancements in the hardware & software with the graphics power to display millions of polygons in 3D and blockchain technology, this concept is now no longer fiction.
This transition of today’s internet to one of a spatially embodied internet is at its core a transition from 2D interactions to 3D interactions, taking place in multiple virtual universes. The progress of augmented virtual reality in recent years has created possibilities in the private and professional spheres. The new virtual reality(VR) headsets and augmented reality(AR) glasses have the ability to provide immersion in the physical sense. To turn this concept into a complete reality with a full experience, technology must be able to provide a realistic experience for users.
Touchless/Contactless Interaction Technologies
Touchless Technology is facilitating the move to Zero User Interface(UI) propelled by the COVID-19 pandemic which has accelerated the use of this technology due to hygiene requirements. Zero UI can be defined as a controlled interface that enables user interaction with technology through voice, gestures, hand interaction, eye tracking, and biometrics such as facial recognition and contactless fingerprints. Smart devices, IoT sensors, smart appliances, smart TVs, smart assistants and consumer robotics are predominant examples of devices in which Zero UI is becoming increasingly adopted. These control interfaces include natural interaction modes such as voice or gestures.
Touchscreens and shared devices such as kiosks, self-service counters and interactive displays are present in our everyday lives. Each of these interactions however is a concern for consumers in a post-COVID-19 world where hygiene is of utmost importance.
The one-stop solution to hygienic interactions includes touchless technology such as voice control, remote mobile screen takeover, biometric, and gesture control as Zero User interfaces. With breakthroughs in image recognition and natural language processing, powered by advanced computer vision and machine learning, “Zero UI” is becoming a new normal.
Ethics, Privacy and security issues with Extended Reality
The adoption of Extended Reality (XR) has grown over the last few years. But this adoption has been accelerated with the impact of the pandemic, which has demonstrated the value of this technology. XR is becoming a popular solution to facilitate remote learning, remote conferences, and remote working. To mitigate the problems of remote learning, a trend is emerging among authorities to emphasize the potential for new technologies such as artificial intelligence and virtual, augmented, or mixed reality to create engagement, providing a kinesthetic aspect of learning and addressing students’ attention problems. Despite the fact that remote learning with XR offers several interesting educational advantages as compared to in-person classroom environments, it has its own downsides that have not been addressed in previous research and considerable research gaps remain in this area. When these devices are used in public places, they can infringe on other people’s rights as well. As for security concerns, the more we live our lives online and virtually, the more vulnerable we can become to hackers. These privacy and security risks include input and output data, user interaction data, and identification of both the user and device used. In the broader view, this research is focusing on a better understanding of the human value in terms of XR for learning purposes in the remote setting with more responsible design and usage.