The human brain contains about 86 billion neurons. They communicate via electrical impulses, which among other things initiate muscle movements. Wouldn’t it be elegant to be able to bypass the detour taken by traditional Human Machine Interfaces from the brain through the muscles to flipping a switch and instead directly control a device with the brain’s electrical impulses? Thanks to Brain Computer Interfaces or Brain Machine Interfaces (BMI), this is now possible.

Devices are becoming smaller and more affordable

In 1925, the German psychiatrist Hans Berger recorded the first human electroencephalogram (EEG). Since then, technology in the field of Brain Computer Interfaces and data processing has continually improved. For at least ten years, the trend in EEG hardware has been to make these devices smaller, wireless, portable and more affordable. Basic brain wave measurements can already be taken through relatively simple headsets, enabling more and more practical applications outside the laboratory. Ultimately, BCIs could be used not only for controlling neuroprosthetics but also for all computer-assisted devices like smartphones and tablets, or a smart home. A increasing number of non-medical use cases is also becoming more and more conceivable, ranging from the PC gaming industry to the simultaneous control of drone swarms. The U.S. Defense Advanced Research Projects Agency (DARPA) is even working on an advanced communication system based on BCIs: the idea is for soldiers and military personnel to be able to issue commands telepathically using the “Silent Talk” solution.

Medical applications still dominate

The medical sector is still the main market for BCIs. “Active and passive BCIs are already being used to improve movement control in Parkinson’s patience via deep brain stimulation, detect epileptic seizures and diagnose brain diseases. Digital and technological progress offers unprecedented new possibilities and has sparked broad scientific and economic interest,” explains Professor Florian Mormann from the German Society for Clinical Neurophysiology and Functional Imaging (DGKN).

BCIs can also translate brain activity into control signals for external devices such as prostheses, robots and exoskeletons. Bidirectional BCIs, moreover, allow for the brain to be electrically stimulated, for example to simulate a sense of touch when controlling a prosthesis.

“Medical technology has made huge progress during the last few years,” says Professor Alessandro Del Vecchio, head of the Neuromuscular Physiology and Neural Interfacing Laboratory (N-squared Lab) at the Friedrich-Alexander University Erlangen-Nuremberg (FAU). “However, much research and development still needs to be done in terms of fine motor skills, for example to enable movement of individual fingers of paralysed hands.” Together with the Institute for Factory Automation and Production Systems at the FAU, the N-squared Lab aims to develop a neuro-orthosis that restores hand function so that patients can perform more than 90 percent of everyday tasks independently. “Our aim is to move the fingers and the thumb of the hand independently of one another and with a high level of strength,” says Del Vecchio.

Increasing accuracy

BCIs can generally be divided into invasive systems that are implanted into the brain, semi-invasive and non-invasive systems. Non-invasive BCIs currently have the largest market share because they spare the patient from having to undergo a laborious and risky surgical brain procedure. “We have already developed a non-invasive BCI system that allows people with high spinal cord injuries to grip everyday objects by voluntarily changing their brain waves,” says Professor Surjo R. Soekadar, Einstein Professor for Clinical Neurotechnology at the Charité – Berlin University Medicine. “Despite considerable progress, it has not yet been possible to control complex hand movements with such a non-invasive system.” Yet this is precisely what Professor Soekadar’s team is striving to achieve: they are currently testing the use of ultra-precise sensors, so-called quantum sensors, which can measure brain activity with much higher accuracy on the surface of the head than EEG or other non-invasive methods. The basis of the high-tech sensors is gaseous atoms that act as magnetic field probes and respond to electrical brain signals.

Chip in the brain

Implanted BCIs can capture brain impulses even more precisely. Companies such as Synchron and Neuralink are already testing such implants on humans. Elon Musk, who co-founded Neuralink in 2016, has promised that the technology “will enable someone with paralysis to use a smartphone with their mind faster than someone using thumbs”. Ultimately, however, the implanted chip is intended to make the human brain more powerful – even to the extent of merging the brain with artificial intelligence. According to Professor Mormann, this is still pure science fiction based on current knowledge: “Neuro-enhancement means targeted and specific influencing of brain activity. The prerequisite for this is a detailed and mechanistic understanding of this activity. Our knowledge so far is still too incomplete and patchy.”

The post What are Brain Computer Interfaces? appeared first on Future Markets Magazine.

Machines have been indispensable partners to us humans in everyday life since time immemorial. It is not only in large production halls or complex industrial manufacturing processes where human and machine-supported processes closely interlock. The number of electronic devices used in the private sphere is also continually increasing – from washing machines to smartphones. Interfaces are the key to enabling humans to interact with these machines or systems.

HMIs use a broad spectrum of functional principles and technologies.

Variety of solutions

The design and functional principles of these Human Machine Interfaces can vary hugely – from a simple mechanical toggle switch to a touch display or a connected mobile device (such as a notebook or smartphone), to name a few.

The operation of machines has changed enormously over the centuries: the first human machines were operated predominantly via levers and cranks. This purely mechanical form of control was increasingly supplemented and eventually replaced by electrical buttons and switches when electrification started at the end of the 19th century. The next major change was brought about by automation and digitalisation: once computers started being used to control systems and machines, it became possible to visualise functions on screens and operate them via keyboards and input devices.

New possibilities through digitalisation

The invention of touchscreens has blurred the boundaries between the display and operation of functions: monitors can now recognise when a user touches a symbol or field and can subsequently translate this action into a digital command. Apple’s products, such as the iPod or iPhone, are iconic examples of how intuitive touchscreens have simplified the operation of devices. They are now also prevalent in industrial systems, in smart home applications and in cars.

Digitalisation has enabled devices and the operation of these to be increasingly adapted to the human user and their personal capabilities. Not only can HMIs now be customised to the individual operator, they can also be modified again and again to reflect new machine functions.

Augmented and virtual reality have also entered the world of Human Machine Interfaces: for example, maintenance and operating personnel can now use smart glasses that provide information from the machine directly “in front of the employees’ eyes”, meaning they no longer have to leaf through manuals to maintain the system.

AI opens up new possibilities for Human Machine Interfaces with the aim of more efficient interaction.

Smart interaction thanks to Artificial Intelligence

The invention of Siri and Alexa a few years ago marked the emergence of another technology on the market – one that has significantly changed the way humans interact with machines: voice control. In many areas today, it is almost a matter of course that a user can give commands directly to devices with their voice – whether in the car or when playing their favourite songs. According to Statista, around 4.2 billion voice assistants were used worldwide in 2020. By 2024, this number is expected to double to up to 8.4 billion. The technological advancements in the underlying artificial intelligence are what have enabled the use of effective voice recognition today. Even in industrial production, it is possible to operate machines by voice despite the interfering background noise.

AI also provides the basis for another form of operation: gesture control, whereby humans can interact with a machine through gestures and movements. Contactless gesture recognition is a complex system that uses several technologies in addition to AI, such as sensor fusion and image recognition using camera systems. So far, the use of gesture control has been largely limited to computer games. But now we’re seeing machines also being equipped with this innovative technology and it is already possible to control robots with gestures.

The latest HMI development is systems that allow the operator to control devices solely by thought. Some companies in the medical industry are already testing implanted chips that capture brain impulses and translate them into digital commands, for example for writing emails.

From pure operation to assistance

In the future, Human Machine Interfaces will provide various different assistance functions for commissioning, operating, servicing and repairing machines, and also support the process of training users on new systems. Research is already being conducted on solutions that can psychologically interpret human behaviour and register their attention and control capabilities. These so-called empathetic HMIs make it possible for machines such as cars and robots to recognise the intentions of humans in order to work with them flexibly, proactively and safely.

The post Diversity of Human Machine Interfaces appeared first on Future Markets Magazine.

James Cameron’s 1984 film “Terminator” is a classic that still shapes many people’s perception of Artificial Intelligence to this day. In the film, a computer system developed by the military was to start a devastating war against humanity in 1997 to protect itself from being shut down. Now in the 2020s, the horror scenarios predicted in “Terminator” and similar films have not come to pass. Nevertheless, AI has now become a part of professional and private everyday life, enabling a new era of human-machine collaboration: chatbots respond flawlessly to questions, smart home devices are controlled by voice and cobots work hand in hand with humans. AI systems are the basis for many innovative Human Machine Interfaces like voice and gesture controls.

Attempt at a definition

But what exactly is meant by Artificial Intelligence? There is, in fact, no universally accepted definition in science or practice. The draft of the European “AI Act” defines artificial intelligence as “software that is developed with one or more (…) techniques (…) for a given set of human-defined objectives and generates outputs such as content, predictions, recommendations, or decisions influencing the environments they interact with.” At its core, AI uses algorithms and complex mathematical models to analyse data and identify patterns. Like humans, AI is supposed to learn from experience, make judgments and solve problems independently – to be able to perform tasks increasingly better.

Several subareas

The term “artificial intelligence” covers several subareas, including Machine Learning and Deep Learning. In Machine Learning (ML), the system independently discovers connections based on example data. Thus, AI systems can learn from data and solve problems on their own without being explicitly programmed in the form of rules. Machine Learning is particularly suitable for recognising and generating so-called “patterns” from existing datasets – for example, the system can thus recognise which gesture a hand performs. Deep Learning goes a step further by automating further aspects of the learning and training process. Deep Learning algorithms can decipher unstructured datasets such as texts or images, so much less human intervention is required.

AI becomes creative

Thanks to ChatGPT, the latest technological developments in AI are currently a hot topic – the so-called generative AI and foundation models. They are capable of independently generating content such as software code, texts, images and music. This sets generative AI apart from “classic” discriminative AI, which is designed to differentiate and classify input but does not create new content. Compared to previous AI models, generative systems are particularly powerful as they are trained based on a very large amount of data. With this breadth and amount of information, foundation models can, for example, translate between languages and systematically work through tasks. In terms of HMIs, they offer advantages in voice control, among other things: the conversation with the machine is more natural, and responses can be given based on the context. Additionally, generative AI can process complex commands better. Instead of simple actions, users can give more detailed instructions: the AI can interpret these, ask follow-up questions if necessary and generate appropriate actions.

Legislative regulation

Despite all the advancements, artificial intelligence is still far less sophisticated than portrayed in Hollywood blockbusters. Nevertheless, regulations are needed so that AI can make the right decisions. An AI system is only as good as the database with which it was trained. There have already been several practical examples of AI systems having a certain “bias” because the database was not diverse enough. If, for example, a language model were trained only with a North German dialect, the later system would have problems understanding someone from southern Germany. To make AI safe and trustworthy, the European Union has introduced legislation to regulate its development and use: the AI Act. This regulation, which is expected to come into force in 2026, is intended to ensure that AI systems used in the EU are safe, transparent, understandable, non-discriminatory and environmentally friendly. AI systems should be monitored by humans and not by automation to prevent harmful results. Thus, the “Terminator” should remain pure science fiction in the future.

Boom in Edge AI processors

14.54 billion US dollars in 2022

54.38 billion US dollars in 2029

To realise AI functions in Human Machine Interfaces, so-called Edge AI processors are important as they can evaluate the data directly on-site and thus ensure a quick  response to commands. According to Maximize Market Research, the market will grow by an average  of 20.1 percent annually in the coming years.

The post Natural Interaction Thanks to AI appeared first on Future Markets Magazine.

Thumbs up, waving, the open hand as a stop sign – gestures are a natural form of communication for humans. Thanks to significant advancements in sensor technology and artificial intelligence in recent years, it is now possible to control machines and devices through gestures.

The breakthrough came with the introduction of Nintendo’s Wii console in 2007 and Microsoft’s Kinect motion control in 2010. Both solutions were developed for the gaming market – and entertainment electronics still dominate the gesture control market today. According to market analysts from Grand View Research, the segment had a revenue share of 59.4 percent in 2022.

However, other industries are also discovering the benefits of gesture control for operating devices and machinery: for example, both the automotive industry and healthcare sector have placed great emphasis on adopting gesture recognition. This technology makes it easy and intuitive for users to interact with computers and other devices. The COVID-19 pandemic has further focused attention on gesture control, as it enables contactless and thus hygienic operation.

Gesture recognition market in 2031: 88.3 billion US dollars

In 2021, the market had a volume of 13.9 billion US dollars. Accordingly, the expected average annual market growth is 20.6 percent.

Source: Allied Market Research

Control via Wearables

Various different technologies are used to detect user movements. One option is special wearables, such as bracelets or rings, equipped with motion sensors that capture the rotation rate or acceleration of the wrist. An intelligent algorithm recognises which gesture has been performed and issues the corresponding command.

Camera-based Solutions

Another approach is camera-based systems. In principle, 2D cameras can capture and interpret movements. However, the algorithms used have difficulty distinguishing movements in front of the screen correctly – the precise capture of distance as the third dimension is missing. For this reason, 3D cameras or image sensors are increasingly being used for gesture control. They have become more affordable in recent years and can be integrated into almost any device due to their small size. These systems complement 2D image data with depth information, mostly obtained through Time-of-Flight technology, which measures the travel time of a light pulse reflected by an object to determine the distance to the camera. Today’s image sensors can detect not only general hand movements but even the movements of each individual finger.

Detection via Thermal Imaging

However, camera-based systems require adequate lighting to reliably recognise gestures. This problem does not affect infrared sensors: they detect the infrared radiation emitted by the human body (passive sensors) or emit infrared radiation themselves as active sensors and capture the reflection. The corresponding algorithms then analyse the patterns and movements of this radiation. The sensors can also generate a depth image. Thus, various different gestures can be recognised depending on predefined movement patterns and algorithms. Nevertheless, systems based on infrared sensors tend to be more suitable for simple gestures. Since they are relatively cost-effective, they are used in many industrial, consumer and automotive applications.

Radar – Robust and Precise

Unaffected by lighting conditions, resistant to contaminants, and with high resolution, radar is increasingly conquering the field of gesture control. Even the smallest movements can be detected by a radar device, with the latest systems offering a resolution of just one millimetre. Radar sensors measure the speed, direction of movement, distance and angular position in real time to detect changes in the position of objects. This makes it possible to track and depict movements of persons or specific motion patterns. And for those who associate radar with the large rotating antennas on ships – the radar sensors needed for gesture recognition fit on a microchip.

AI and Edge Computing

No matter which technology is used for gesture control, one challenge remains: everyone performs gestures in a different manner. This means that the systems must be able to recognise numerous interpretations of a gesture. Artificial intelligence and machine learning processes are highly useful in this regard: through complex signal evaluations, gestures can be clearly identified and classified. To process sensor data in real time and achieve the fast response times necessary for device operation, machine learning algorithms are increasingly being executed locally on the chip, close to the sensor itself – typically referred to as the “edge.”

The post How does gesture control work? appeared first on Future Markets Magazine.

Tired? Angry? Inattentive? Or even sick? A person’s state of mind or wellbeing can have a significant impact on their ability to use and operate machinery and on safety. This is why HMIs are being equipped with technology for monitoring a user’s vital parameters in an increasing number of domains. These interfaces are not necessarily used to receive and execute control commands from the user. They are mainly utilised to monitor the user’s condition and trigger an action when certain changes in their vital parameters are registered.

The healthcare sector is the natural home of such HMIs: from the tiny pulse meter clamped on a finger to the highly advanced, sophisticated technology of artificial intelligence – everywhere, HMI technology is an essential part of the assessment, monitoring and treatment of patients. Typically, sensors from various devices are stuck to the patient’s skin to measure brain waves, impedance, motion, blood oxygen and temperature data. A local processor system can create individual warning messages for the patient based on the data obtained and automatically alert a caregiver if it detects unusual changes in the patient’s condition.

Recognising cardiac activity

Thanks to the success of smartphones and smartwatches, many of these parameters can now also be measured in high quality wherever the wearer is. This includes cardiac activity, where two methods have prevailed. The simplest is the single-channel ECG: here, two electrodes are integrated into a smartwatch, for example. The electrode on the back of the device is in contact with the wearer’s arm, and the second electrode on the top of the watch is activated by touching it with a finger on the other hand. In the automotive sector, so-called multi-touch ECGs are used. Here, the ECG sensor technology is integrated into various positions such as the steering wheel, gear lever or armrests. The system automatically detects which electrodes are in contact with the user. Thus, ECG measurements can take place unnoticed in the background, while the human has a high degree of freedom of movement.

Analysing vital parameters through light

Photoplethysmography (PPG), which measures a person’s heart rate optically using infrared light, has an entirely different operating principle. It detects how much light emitted by the system is reflected by the skin. This amount depends on how much blood flows through the superficial capillaries. Since the blood volume in the capillaries increases with each heartbeat, more light is absorbed and less reflected in that moment. The system converts the reflected amount of light into a pulse wave. The heart rate can then be determined through this pulse wave analysis. If RGB cameras are used to capture the light, the respiratory rate and oxygen saturation can also be determined contactlessly by analysing the red, green and blue components in the PPG signals. Recent studies have shown that the pulse wave signal can also be measured with a camera placed a few centimetres to metres away from the skin.

Radar-based sensors

Radar-based sensors can even capture heart and breathing values through clothing and over a distance of several metres. Electromagnetic waves with a frequency of, for example, 60 gigahertz are used, which are reflected by the body. Based on the reflected rays, the sensor detects the vibration of the skin caused by the pulse wave. Such systems are already used to monitor the driver’s condition in trucks, trains or aeroplanes.

Cameras read emotions

Camera systems also offer several possibilities for monitoring vital parameters. In addition to photoplethysmography, they can also recognise a person’s state of consciousness. Special CMOS cameras – usually with a resolution of one to two megapixels – take 30 or 60 frames per second in the infrared spectrum, depending on the model. A downstream system evaluates them and analyses, for example, the driver’s direction of gaze or the frequency of eyelid closure. From this, conclusions can be drawn about a distraction or increasing fatigue of the person, and if necessary, an alarm can be triggered. State-of-the-art solutions are able to recognise – in part thanks to AI – the smallest changes in behaviour, sleepiness, negative emotions and the possible influence of alcohol or drugs. Ultimately, such a system can form a complete picture of a person’s physical and emotional state.

The post Monitoring vital parameters with HMIs appeared first on Future Markets Magazine.

The interaction between humans and machines is usually not one-sided, but rather a kind of communication: the operator issues a command and receives feedback on whether the command has been received. For example, with toggle switches, it is the changed position, and with push-buttons, it is often a certain resistance that must be overcome, sometimes coupled with a control light. Important for operators of many machines is also direct feedback on the force or the angle of rotation triggered by the operating part. For example, an excavator operator can feel the strength of the force with which a shovel penetrates the ground through the resistance of the operating lever of a hydraulic system. The same applies to machine tools: through mechanical operating parts, force and vibrations are transmitted directly to the hand and body of the operator.

The market for haptic touchscreens is growing rapidly

17.32 billion US dollars in 2022

47 billion US dollars in 2030

Source: Verified Market Research

Operate with feeling

With the introduction of electronic controls and operating elements, however, this haptic feedback has been lost. But thanks to microelectronics, the feeling can be replicated in modern Human Machine Interfaces. In general, sensors detect the force that the human applies to the operating part. If necessary, additional sensors also measure the force or angle at the executing part – that is, the tool, the shovel of an excavator or the wheel of a car. Micro-actuators at the operating part can then, for example, cause a noticeable counter-movement based on these measured values. The technologies used for this are summarised under the term “force feedback”.

Gaming leads the way

Such systems have been well-known in the gaming sector for a long time, for example with driving simulators: a steering wheel with force feedback technology generates vibrations and simulates gravity. High-end systems dial into the physics and the audio engine of a game to enable an ultra-realistic experience. The player feels in real time the roar of the engine of their virtual race car, the tyre traction, the nature of the terrain of the track and the feedback of the steering wheel. So it feels as if you are sitting behind the wheel of a real car.

Feedback from the car

Force feedback solutions are found not only in virtual vehicles but increasingly also in real ones: especially in the context of highly automated driving, the classic steering wheel with mechanical steering column is being replaced by mechatronic actuators – this is called steer-by-wire. Thanks to force feedback, however, the mechatronic steering wheel conveys exactly the same feeling as a classic mechanical one. Additional functions can also be realised – for example a vibration of the steering wheel when the car detects that the driver is getting tired.

Accelerator and brake pedals are also equipped with force feedback. Such active accelerator pedals with integrated actuator and freely programmable haptic signals can, for example, help the driver to drive as fuel-efficiently as possible: depending on the selected driving program, a variable pressure point in the pedal travel can be generated, which signals the optimal accelerator pedal position to the driver.

Haptic touchscreens

Even touchscreens, which are becoming increasingly popular as HMIs, can be equipped with feedback functions. If touchscreens are supplemented by sensors that measure the pressure of the finger on the surface, it is not just touch that can trigger an action, but also the press of a button. The received command can then, for example, be acknowledged by the display with a vibration. This is caused, for example, by electrostatic actuators. This haptic technology is characterised by a stroke of up to 0.8 millimetres, which allows for button-like feedback. Alternatively, piezo actuators are integrated into the display. This technology delivers a stronger and more precise haptic event. In contrast to electrostatic actuators, the deflection with piezo actuators is significantly smaller and is in the range of up to 0.3 millimetres.

Addressing the sense of touch without contact

However, operating elements are increasingly being controlled completely without contact, for example through glances, hand gestures or voice control. Until now, users have received control-related feedback in the form of displays or acoustic signals. The Institute of Construction Technology and Technical Design at the University of Stuttgart is currently working on enabling haptic feedback here as well. This involves the use of ultrasound waves: they are projected onto the palm of the hand, for example as circles, triangles or moving points, and produce a tingling sensation. Once you have learned the language of these signals, this feeling helps to move the hand in the right direction and at the right speed.

Feedback technologies are an important part of Human Machine Interfaces. They improve the user experience and open up new possibilities for the optimisation of safe and efficient workflows.

The post Feedback technologies in HMIs appeared first on Future Markets Magazine.

It’s posited to be nothing less than the future of the Internet – the Metaverse. At its heart, it’s about a new, complex virtual world where one lives as a digital twin or avatar and may, in the future, also work and carry out daily activities. There are numerous connections with the real world: for example, virtual goods can be purchased with real money within the Metaverse. Behind avatars are real people, and behind virtual machines are real industrial complexes in the sense of a digital twin. “The Metaverse is one of the most thrilling visions of our time,” says Dr Bernhard Rohleder, CEO of the German industry association Bitkom.

The Metaverse is built upon technologies that are already well-established, such as augmented and virtual reality, blockchain, or artificial intelligence. The representation of virtual three-dimensional spaces, objects or individuals and the ability to interact with them are increasingly being employed not only in entertainment, marketing and eCommerce but also being tested and applied in industry, education and services. What has long been familiar to computer gamers is now being discovered as potential across various industries.

Realistic interaction

A core element of the Metaverse is the potential for realistic interaction in real time between different users or between users and machines. The immense advancements in hardware – both at the infrastructure level and in Human Machine Interfaces – have been prerequisites for the growing acceptance of this new virtual world. The major software companies are now entering the market with their hardware solutions: Meta purchased the VR headset developer Oculus in 2014 and now sells its own headsets under the name Quest 2. Apple has filed several patents over the last decade and aims to bring its VR headset to market in early 2024. Current systems also use controllers that enable targeted “touches” with virtual objects through finger, hand and arm movements. On the infrastructure side, faster fibre optic networks and the introduction of 5G are reducing the effects of latency times and increasing available bandwidth.

Collaboration in virtual space

Today, “serious” applications in industrial environments are possible. For example, Igus, a manufacturer of energy chains and plastic glide and linear bearings, has created its own virtual world where users from around the world can interact and collaborate on projects without the need for physical presence. Automation solutions can be planned, controlled and tested in virtual space using extended reality technologies like virtual reality and augmented reality. “In the future, we will see how collaborative work between humans in the Metaverse and, for instance, the control of robots from virtual worlds will change the way we work together,” says Marco Thull, Senior Marketing Activist at Igus.

Feeling the virtual world

In the future, the virtual world may even become literally “tangible.” Jürgen Steimle, a computer science professor at Saarland University, aims to achieve this through ultra-thin electronic films that can be applied to the body like transfer tattoos. Most applications of augmented reality have one thing in common: they only or primarily address the sense of sight. “The sense of touch is usually left out, although it’s a crucial factor in how we perceive our world,” explains Jürgen Steimle, who leads the research group on Human-Computer Interaction at Saarland University.

Steimle’s team has developed an ultra-thin, only 35 micrometres thick electronic film in the project “Tacttoo”, which can be applied to the skin and stimulate the sense of touch solely through electrical stimuli, without any moving parts. Because the film is so thin, objects can still be perceived and felt as before. Not only can haptic experiences for purely digital objects be created – as already possible with other solutions – but real objects can also be extended with other sensory impressions. For example, the technology could be used in product design: with the help of augmented reality and a physical prototype, the haptics of different materials could be tried out.

Standards for an open Metaverse

For the Metaverse to find broad application, a range of technologies for collaborative spatial data processing must be integrated, from interactive 3D graphics to physical simulations to the online economy. The Metaverse’s potential will only be unlocked if it is built on a foundation of open standards. To this end, the Metaverse Standards Forum was established in 2023, a non-profit consortium that doesn’t want to develop its own standards but brings together various Metaverse stakeholders to create consensus on interoperability requirements. “Interoperability is the key to the metaverse scaling to its full potential beyond siloed games, experiences and worlds, and the unprecedented level of participation in the Forum demonstrates strong industry interest in the broad cooperation necessary to bring that vision to life,” says Neil Trevett, president of the Khronos Group and initial president of the Metaverse Standards Forum.

The post Human Machine Interfaces in the metaverse appeared first on Future Markets Magazine.

Anyone who has ever been engrossed in a game knows what “immersive” means. It comes from the Latin “immersio,” meaning to dive in. In the context of computer games, the word describes the extent to which a player perceives themselves to become part of the game world. Besides the pure visuals, such as a first-person perspective, the interaction with the virtual world is decisive.

Familiarity with Digital Technology

The industry uses all the possibilities offered by modern Human Machine Interfaces, making it an exciting testing ground for new technologies. “Gaming is more than just entertainment. Through interactive gaming experiences, users are playfully familiarised with modern technology. As they dive into virtual worlds, they intuitively learn to handle digital interfaces, menus and control options,” says Dr Sebastian Klöß, head of Consumer Technology at Bitkom. Young people’s familiarity with complex Human Machine Interfaces is even utilised by the military, which uses HMIs resembling gaming controllers rather than military equipment – many systems can be used by young soldiers without intensive training.

Haptic feedback in controllers

To achieve the goal of an immersive gaming experience, manufacturers are constantly coming up with new ideas. This starts with the “simple” controller such as the PlayStation 5 from Sony: it makes the game action palpable through haptically fine-tunable vibrations. Adaptive triggers allow the adjustment of the analog shoulder buttons. In this way, certain activities in the game, such as shooting, accelerating and jumping, become more nuanced. Moreover, Sony has recently patented a new controller design that integrates an elastic area that can change its shape and temperature and respond to various physical inputs such as pressing or rubbing. This would offer new possibilities for immersion, such as the controller being able to enhance intense game moments with heat.

Measured pressure

Keyboards with force sensor technology are already a reality, although they may not be as visionary as some other technologies. They are used, for example, in special gaming laptops. Typically, the W, A, S and D keys are supplemented by a force function, making the keyboard much more intuitive and sensory for players to operate. “Imagine driving down your local highway and your accelerator is restricted to just 2 options, ‘off’ or ‘full speed’. Reality-based physics calls for acceleration dependent on how hard you press on the accelerator pedal, or how hard you turn the steering wheel. This is the difference Peratech intelligent tactile sensing makes,” explains Jim Thomas, CCO of Peratech. The company equips, for example, the Lenovo Legion 7i and 7 gaming laptops with corresponding force keys.

Capturing head movements

Lenovo’s top model can also capture the player’s head movements. The integrated software solution from Tobii Horizon uses the laptop’s built-in camera for this purpose. Players can thus control the field of view in the game with their head movements. When a player turns their head to the right, the camera follows his gaze to the right. This is particularly helpful in games where it is important to be aware of the surroundings, such as in first-person shooters or racing games.

Diving into Virtual Reality

Players can truly dive into a game through virtual reality and augmented reality technologies. Special VR glasses let the player sink into the artificially created, virtual world. These headsets can be mounts into which a smartphone is inserted, such as with Samsung Gear, or so-called “Head Mounted Displays”, which integrate the entire technology, such as Microsoft’s Hololens. In both approaches, the player is at the centre, and their virtual view follows their own movement. With its immersive and interactive nature, virtual reality has revolutionised the way games are experienced. But VR becomes truly “immersive” with input devices specially designed to make the virtual world seem more real.

Interacting with objects

Like the Oculus Touch: these handheld devices are designed to mimic hand movements in the virtual world. Equipped with sensors, buttons and triggers, the system allows the player to interact with objects and move precisely and effortlessly through virtual environments. The VR controller HTC Vive Wand also features haptic feedback – allowing the player to feel the virtual objects they interact with.

Playing with body movements

The gaming experience becomes completely realistic when the movements of the entire body can be used to guide the character through the virtual world: jumping, running, kneeling – all captured by full-body tracking systems like Omni One from Virtuix. It is an omnidirectional treadmill that allows players to walk or run in any direction through video games and other virtual environments. It is currently shipped with a Pico Neo 3 Pro headset with 6DoF technology (6 Degrees of Freedom) and additional hand controllers for hand and gesture tracking.

Empathise with the avatar

For players who still want more, in future they will be able to experience their characters’ suffering from the game. For instance, the game provider Ubisoft plans to offer a shirt with haptic feedback for its game Assassin’s Creed Mirage. With the Haptic Gaming  System from OWO, the player is supposed to experience sensations on the body that correspond with the actions in the game. “By leveraging OWO’s groundbreaking technologies and expertise, we are able to immerse players in the world, sounds and sensations of Assassin’s Creed Mirage in an innovative and enhanced way,” Fabian Salomon, lead producer at Ubisoft Bordeaux. The sensations that the shirt can convey range from wind or free fall to knife stabs, thus also exploring the limits of immersion.

Seamless interaction

But the future of VR controllers goes even beyond that. Companies like Neurable are working on Brain Computer Interfaces that allow players to control virtual environments with their thoughts. The company, a spin-off from the University of Michigan, is developing neurotechnological reference designs, APIs and capabilities that function outside of laboratory conditions and can be integrated into everyday technologies. Neurable licenses its technology for headphones, earbuds and AR devices. Dr Ramses Alcaide, CEO of Neurable, is sure that: “This is just the beginning for neurotechnology. We are at the cusp of being able to ethically create a seamless relationship between people and their technology.”

The post Technologies for an immersive gaming experience appeared first on Future Markets Magazine.

Lights controlled by an app, a smart thermostat that automatically turns off the heating when a window is open, a vacuum cleaner robot that starts with a voice command: smart home devices have long been established, and the market is a booming billion-dollar business. Analysts from Brainy Insights estimate that the global smart home market will grow from 90 billion US dollars in 2022 to 657.41 billion U.S. dollars in 2032. This market is a key driver for Human Machine Interfaces – as HMIs are the crucial component in the intelligent home to seamlessly integrate smart home devices into the daily lives of residents.

HMIs as a success factor

The further development of HMIs and the functionalities associated with them are closely linked to the growing demand for smart home devices. In a survey conducted by the global contract manufacturer Jabil together with SIS International Research among 200 decision-makers for smart home solutions and devices, 39 percent stated that intuitive Human Machine Interfaces are the most important factor for the success of their smart home solution.

Multi-touch and screens

In the Jabil survey, 84 percent of respondents stated they currently use multi-touch and display screens in their smart home solution – the second most common response. Moreover, 43 percent expect their customers to continue using displays to interact with smart home solutions in the future. One study participant highlighted a particular advantage: “An intuitive touchscreen display, simple controls that facilitate use, and a thoughtful design ensure that [the device] is easy to clean and maintain.”

Voice command operation

Voice assistants like Google Home and Amazon Alexa have revolutionised the operation of intelligent devices at home. “The smart home has become the main area of application for the voice control of devices,” said Bitkom President Achim Berg already in 2022.

Systems like Alexa or Siri work over the cloud – which repeatedly raises concerns about data protection. The data leaves the smart home and could be used by third parties. Indeed, more than three-quarters (76 percent) of manufacturers in the Jabil survey stated that they use HMI interactions, such as voice commands, to collect data on their smart home solutions. Systems that process speech locally offer more data protection security: for instance, Peaknx building control has offline voice control from ProKNX, which operates independently without internet access in the configured operation. The spoken commands are only processed locally and do not leave the building. No recording takes place either. Thus, following the motto “What’s said at home, stays at home,” the privacy of residents is always preserved.

Holography is not science fiction

The COVID-19 pandemic triggered a massive demand for touchless HMIs that enable interactivity like a touchscreen without the risk of contamination. One of the most effective ways to achieve this is holographic projections. Corresponding HMIs create a holographic image of a control element by directing the content of a display through a special holographic plate. In addition to a deflection of angle, a convergent bundling of rays occurs there, creating a virtual image floating freely in space. An infrared touch sensor is mounted below the virtual image. Its beam path is interrupted when the image is “touched”. The interruption is evaluated as a touch event. While manufacturers of public touchpoints are already using holographic HMIs for “germ-free” contactless operation, applications for the smart home are still in the development stage – for example for coffee machines or intelligent fitness equipment. Nevertheless, already four percent of the participants in the Jabil survey are using holograms in their HMI solutions.

Gesture control in low demand

As with holographic HMIs, COVID-19 has significantly increased the demand for solutions with gesture recognition. Just over a third of the participants in the Jabil survey currently use this technology for their HMIs. However, only three percent of them expect their customers to continue using gesture recognition to interact with their products in the future. However, facial recognition and eye-tracking solutions could gain market share in smart home HMIs. Such solutions could, among other things, improve the accessibility of smart home devices for people with mobility or speech disabilities – one could simply look at a control panel to change the temperature of a room or the volume of the television instead of using a remote control or giving a command.

Modern remote control

However, the most important HMI in the smart home is still the smartphone: according to Bitkom, 85 percent of people in Germany control their smart home applications and devices via a smartphone app, 20 percent via remote control. In the future, wearables will also be used to control smart home devices, which is also what the manufacturers surveyed in the Jabil study expect. Devices such as smartwatches or fitness trackers could, for example, send a signal to the smart home network as soon as the resident enters their home, in order to automatically activate the lights, heating or television.

The post How will we interact with smart home products? appeared first on Future Markets Magazine.

Whether in the rough daily routine of construction machinery, in food production or as a public terminal in the city – Human Machine Interfaces are often exposed to extreme environmental conditions. By using qualified materials and appropriately designed constructions, and choosing suitable operating technologies, HMIs can work reliably even in the most adverse conditions.

Replacing mechanical switches

Traditional mechanical buttons, which are operated by actual touch, are robust components, but germs or impurities can easily settle on mechanical and moving parts. Moreover, they cannot be installed flush in plants – an additional gateway for dust and the like. A resistant and hygienic alternative is provided by capacitive buttons.

They can be integrated flush into any system and have no mechanical elements where dirt particles or germs can settle. Thus, sealed, smooth and easy-to-clean user interfaces are possible, preventing the ingress of moisture or dust.

Sealed keyboards

Elastomer keyboards offer a high-quality, tactile feel with an underlying switch layer. They are characterised by high weather resistance in harsh environments, and moisture and chemical resistance. Elastomer keyboards are used when three-dimensional buttons are needed in environments with high moisture, such as in maritime or defence applications.

In the food, laboratory and medical sectors, the membrane keyboard plays an important role due to its closed surface, resistance to chemicals and its imperviousness to liquids and dust. The classic membrane keyboard consists of about eight layers, of which only the top layer, the “front foil,” is visible in the later application. The actual switching element, the snap disc, is placed between the different layers and is thus well protected against dust and moisture. Integrated acrylic inlays also prevent deformation of the snap disc.

Easy-to-clean touch panels

In harsh application areas, touch panels should be equipped with scratch-resistant and shock-insensitive surfaces and components. They must withstand high stresses from jet water, steam and dirt, and be sealed against harmful gases. The entire surface of the input system must be resistant to oil, solvents and aggressive chemicals. Touch panels should meet the IP67 protection class and have a continuous, closed front structure without dirt edges. This facilitates cleaning, which is of great importance in the food industry, for instance. The robustness against mechanical stresses can be ensured by choosing hardened glasses. In conjunction with a laminated composite glass pane on the back, for example, vandal-proof touch panels for public applications can be realised. Capacitive PCAP touch technology meets both industrial and medical requirements. It is resistant to water and saltwater, electromagnetic radiation and chemicals. Moreover, it can also be operated with gloves and can be equipped with antimicrobial surfaces.

Antimicrobial coatings

With antimicrobial coatings, HMIs are also equipped for particularly high hygiene requirements. Generally, a distinction is made between active and passive materials. Passive coatings prevent microbial colonisation through the surface structure alone. The so-called Lotus effect inhibits the adhesion of microorganisms to the material surface. In contrast, active antimicrobial materials contain biocidal components that attack microorganisms at the cell wall, in metabolism or in the genome. To meet high hygiene standards, it is recommended to provide switch handles and user interface for touch applications with coatings with Log reduction values of 5.25 (disinfection) to 6.05 (sterilisation).

The inside matters too

Vibrations, extreme temperatures and chemicals affect not only the surface of a Human Machine Interface. HMI assemblies generally consist of a multitude of layers or components. Adhesives and sealants ensure that thermal and electrical contacts remain consistently connected, possible damage from overheating is prevented and HMIs are protected from short circuits, contamination ingress or mechanical failure due to vibrations. For example, display layer bonds must have adequate peel adhesion, shear strength and impact resistance under harsh environmental conditions. Furthermore, vibration and shock absorbers and seals are indispensable components of HMI design. Thin and lightweight foams absorb vibrations and shocks very effectively and are flexible enough to fit into predefined housing dimensions and consider temperature change swelling and contractions.

All these materials must not lose their function in harsh conditions, for example by contracting, drying out or outgassing at extreme temperatures. Accordingly, careful attention should be paid to the materials for adhesives, seals and co. when designing an HMI for harsh environmental conditions.

The post Robust HMIs for harsh environments appeared first on Future Markets Magazine.