Active electronic skin: an interface towards ambient haptic feedback on physical surfaces

Abstract

In the era of ubiquitous computing with flourished visual displays in our surroundings, the application of haptic feedback technology still remains in its infancy. Bridging the gap between haptic technology and the real world to enable ambient haptic feedback on various physical surfaces is a grand challenge in the field of human-computer interaction. This paper presents the concept of an active electronic skin, characterized by three features: richness (multi-modal haptic stimuli), interactivity (bi-directional sensing and actuation capabilities), and invisibility (transparent, ultra-thin, flexible, and stretchable). By deploying this skin on physical surfaces, dynamic and versatile multi-modal haptic display, as well as tactile sensing, can be achieved. The potential applications of this skin include two categories: skin for the physical world (such as intelligent home, intelligent car, and intelligent museum), and skin for the digital world (such as haptic screen, wearable device, and bare-hand device). Furthermore, existing skin-based haptic display technologies including texture, thermal, and vibrotactile feedback are surveyed, as well as multidimensional tactile sensing techniques. By analyzing the gaps between current technologies and the goal of ambient haptics, future research topics are proposed, encompassing fundamental theoretical research on the physiological and psychological perception mechanisms of human skin, spatial-temporal registration among multimodal haptic stimuli, integration between sensing and actuation, and spatial-temporal registration between visual and haptic display. This concept of active electronic skin is promising for advancing the field of ambient haptics, enabling seamless integration of touch into our digital and physical surroundings.

In 1991, Mark Weiser presented the paradigm of ubiquitous computing in Scientific American, emphasizing the profound nature of technologies that seamlessly integrate into daily life until they are imperceptible¹. Ubiquitous computing envisions a world characterized by heightened intelligence and interconnectedness, where the technologies and tools enabling interaction and communication fade into the background, enabling individuals to concentrate on tasks and fostering natural, convenient human-computer interaction.

Over the past three decades, the paradigm of ubiquitous computing has been realized as “ambient visual display” to accommodate human vision channel (as shown in Fig. 1). Screens of varying sizes, including smartphones, laptops, LED walls and billboards, have ushered in an era of ambient visual information that brings us closer to Weiser’s blueprint. Esteemed international institutions like IBM, MIT, Microsoft, and BOE have dedicated significant effort to theoretical and technological research, pursuing the realization of this vision. Advancements in materials, technologies, and processes have been instrumental in achieving breakthroughs in visual display technologies.

Fig. 1

From ambient visual display to ambient haptic display.

While visual information provides typical object features (e.g. size, shape, and color), the absence of tactile feedback limits our perception of critical attributes such as texture, temperature, hardness, etc. A comprehensive understanding of natural human-computer interaction reveals that it necessitates not only visual input but also the involvement of the tactile channel. By synergistically combining visual and tactile perception, users can achieve efficient and convenient interaction with physical environments. In some scenarios, the effective addition of the tactile channel might be able to complement visual stimuli, mitigating information overload and reducing cognitive load.

Inspired by the success of “ambient visual display” and the similarity between human visual and haptic channels, one straightforward question is whether we can create a world with “ambient haptic display”. In 2009, MacLean introduced the concept of “Ambient Interfaces”², while she proposed the idea of leveraging tactile devices to address attention overload during human-computer interactions. However, haptic technologies remain in their nascent stages when compared to the remarkable capabilities of human touch, and remain significant challenges in faithfully replicating real and natural tactile information. Nowadays, vibration-based devices stand as the most mature haptic rendering technology, extensively employed in commercial devices like smartphones, gaming controllers, and massage chairs, offering information cues, enhanced tactile sensations, and emotional experiences. In research labs, lots of haptic devices, encompassing texture, softness and thermal feedback devices, as well as electronic skins with tactile sensing capabilities, have also been developed. Nonetheless, these devices primarily exist as research prototypes rather than industrial products, often failing to achieve sustained robustness, stability, and reliable tactile feedback. More importantly, these devices are hard to integrate with daily objects and thus fulfill the dream of Mark Weiser (“The most profound technologies are those that disappear. They weave themselves into the fabric of everyday life until they are indistinguishable from it”¹). Consequently, the realization of the goal of “ambient haptics” remains elusive.

In this paper, we propose a skin-like interface, i.e., active electronic skin (abbreviated as AE-Skin), to fulfill the goal of “ambient haptics”. To meet the demands for ambient haptic interactions, from a form factor perspective, the AE-Skin needs to be thin, transparent, and flexible, allowing convenient attachment to diverse object surfaces without obstructing its functionality. To enable bilateral haptic interactions between users and objects in the surroundings, the AE-Skin perceives user interaction behaviors and seamlessly delivers the required tactile information. In one aspect, it should possess the ability to faithfully reproduce diverse tactile information, encompassing textures, temperatures, vibration, and other physical attributes of various objects. In the other aspect, it should exhibit human-like perception capabilities, dynamically adjusting its output by discerning users’ interaction commands such as contact pressure, position, and temperature (as illustrated in Fig. 1). The AE-Skin aims to provide tactile feedback and sense human interaction actions anytime and anywhere, effectively reducing the interaction burden and fostering natural human-computer interaction.

The structure of this paper unfolds as follows: First, we introduce the concept of AE-Skin, and elaborate on its three fundamental characteristics: richness, interactivity, and invisibility. Second, attention is devoted to the two categories of potential applications of the AE-Skin, namely interaction with the physical world (e.g., intelligent home, intelligent car, and intelligent museum) and interaction with the digital world (e.g., haptic screen, wearable device, and bare-hand device). Third, the state-of-the-art key technologies of AE-Skin are encapsulated, encompassing texture display, temperature rendering, vibration rendering, multidimensional tactile sensing technology, materials and fabrication methods. Fourth, by identifying the gaps between present technologies and the goal of AE-Skin, four prospective research directions are delineated. Finally, we present concluding remarks.

Concept and fundamental features of AE-Skin

Concept of AE-Skin

As the primary sensory organ, the human skin covers the whole body and perceives diverse interaction information through mechanoreceptors, such as fabric texture, object temperature, and smartphone vibration. This information is transmitted to the brain, providing a judgment basis for the user’s interaction behaviors. During human-object interaction, the user can perceive multiple attributes of objects through the skin, but the object lacks the ability to sense human interaction information. To enable objects to perceive interaction information from humans or other objects in surrounding environments, researchers have pursued the development of electronic skin (e-skin) since the 1970s³. E-skin is an artificial skin with human-like perception capabilities, which can monitor the user’s interaction information by endowing interactive devices with remarkable sensing abilities⁴. In order to improve the mechanical properties of the e-skin to accommodate various curvature surfaces, researchers have developed flexible and stretchable e-skin using different sensing principles, enabling the sensing of multidimensional information including temperature, pressure, and strain.

Existing e-skin is limited to the ability of tactile sensing, unable to provide users with tactile feedback, making it impossible to achieve high-fidelity, natural, and bidirectional tactile interaction. The desired interaction mode immerses users in an intelligent interactive space where all objects are touchable. To establish such an intelligent interactive space, it is necessary to develop the next generation of intelligent tactile devices, known as AE-Skin. These devices can be attached to diverse object surfaces effortlessly, enabling the sensing and identification of the user’s multidimensional interaction information anytime, anywhere. Additionally, they can provide multi-modal tactile feedback, assisting users to perform various interaction tasks more conveniently and naturally. Moreover, the high transparency of AE-Skin is essential in specific interaction scenarios to prevent obstruction of objects, such as displays.

As shown in Fig. 2, the ultra-thin, transparent AE-Skin covers object surfaces. In the interaction process, the AE-Skin dynamically monitors and perceives the contact pressure, interaction position, sliding speed, finger temperature, etc., and the user’s interaction intention can be obtained according to these interaction behaviors. The microstructural features, vibration frequency, amplitude, and surface temperature of the AE-Skin can be modulated actively through a closed-loop control system, providing users with rich tactile feedback. The mechanoreceptors in human skin transmit the above tactile information to the brain’s central nervous system through ascending nerve channels. Through the processing and recognition of diverse neural cells in the brain, the user can perceive various tactile information from the AE-Skin.

Fig. 2

AE-Skin provides an interface between human skin and physical surfaces.

Fundamental features of AE-Skin

To meet diverse interaction requirements in ambient haptic applications, we present three fundamental features of AE-Skin: richness, interactivity, and invisibility (as shown in Fig. 2).

In terms of richness, AE-Skin should possess the sensing ability of multidimensional interaction to identify the user’s interaction position, speed, contact force, temperature, etc., and predict the user’s interaction intention. In order to accurately and reliably monitor the user’s interaction information, it is necessary to consider the sensing range, sensitivity, accuracy, repeatability, temperature excursion, etc. Furthermore, AE-Skin should provide multi-modal tactile feedback, including texture, vibration, and temperature. In practical applications, there are many factors that affect the efficiency and accuracy of users performing interactive tasks, such as the stability of force feedback, resolution and holding force in texture display, vibration frequency and amplitude of vibration feedback, and temperature range and latency of temperature feedback.

Regarding interactivity, to provide an intelligent, natural, and realistic tactile experience, AE-Skin requires the ability of closed-loop control, which automatically adjusts tactile feedback by identifying the user’s interaction information. For example, in an intelligent museum, large-scale AE-Skin can be used to simulate exquisite embroidery in the Tang Dynasty. When visitors touch different positions of the intelligent embroidery, the AE-Skin dynamically displays corresponding texture patterns by precisely controlling actuators in these areas, thus enabling realistic embroidery experiences. Additionally, to meet the spatial registration requirement of the actuators and sensors on the AE-Skin, it is necessary to ensure that the contact area between the fingertip and the AE-Skin covers at least one actuator and one sensor. The spatial layout of actuators and sensors allows the normal sensing and transmission of sensing signals, as well as reliable haptic display. In addition, the update rate of closed-loop control of the actuating and sensing signals should meet the real-time requirements of interactive applications.

For the feature of invisibility, AE-Skin should be as thin as possible, allowing it to hide in the interaction background and transform into an interaction boundary without diverting the user’s attention. When detecting the user’s interaction, AE-Skin should naturally and progressively perceive and display tactile information, avoiding sudden appearance or disappearance, thus minimizing the user’s cognitive load. To be integrated into the interaction environment without obstructing the appearance of objects, AE-Skin should exhibit a certain level of transparency. For example, when an AE-Skin is attached to the surface of the screen, it cannot affect the display of the visual information on the screen. In addition, it should also allow easy and rapid adhesion to various object surfaces, accommodate various curvature surfaces, and enable high freedom of movement for multi-joint/soft objects. To adapt to surfaces with different shapes and different physical properties, as well as to ensure high reliability during dynamic action, the range of bending angles and stretchability of the AE-Skin need to be considered.

Killer applications of the AE-Skin

Taxonomy of application

The application of AE-Skin can be divided into two main categories based on its ability to adhere to different objects. As shown in Fig. 3, the first category involves applying AE-Skin to the surface of physical objects, such as furniture, automobiles, and exhibits. By infusing these objects with “feelings”, AE-Skin brings them to life, enabling the digitization of the physical world. The second category focuses on applying AE-Skin to the surfaces of intelligent devices, including iPads, VR controllers, and haptic gloves. By giving these devices “physical embodiments”, AE-Skin facilitates the materialization of the digital world. Both scenarios allow users to interact more naturally and intuitively with everything around them for a more authentic and immersive interactive experience.

Fig. 3

The potential applications of AE-Skin.

Active skin for the physical world

In typical interactive scenarios such as intelligent home, intelligent car, and intelligent museum, AE-Skin can endow objects with tactile sensing and feedback capabilities to realize adaptive adjustments and personalized services possible, providing users with a more intuitive, affinitive, and comfortable interactive experience (as shown in Fig. 3).

Intelligent home: Smart wallpaper affixed with large-scale AE-Skin can alter its micro-surface morphology to present users with textured wall materials such as diatomaceous earth, stone, and ceramic, providing a richer tactile experience (as shown in Fig. 4). Smart tablecloth with medium-sized AE-Skin adjusts automatically its temperature based on seasons and weather conditions, as well as its surface texture patterns according to user preferences and needs. Small-sized AE-Skin can be applied to handheld objects such as doorknobs. When users return home from cold outdoors, the smart doorknob can sense their body temperature and activate the air conditioning in advance. Users can also be notified through vibrations if the door is not properly locked.

Fig. 4: AE-Skin for intelligent homes.

AE-Skin attached to the wall can dynamically regulate its micro-surface morphology. By touching AE-Skin, users can perceive different texture patterns and different materials, such as diatomaceous earth, stone, ceramic, etc.

Intelligent car: Steering wheel integrated with AE-Skin makes it possible to naturally monitor drivers’ physiological and behavioral parameters such as hand pressure and grip position. In cases of non-compliant or fatigued driving, the smart steering wheel activates vibration warnings. AE-Skin can also be affixed to the surface of car panels (e.g. instrument panels and car doors), transforming physical buttons into intelligent ones. When the driver touches the buttons while driving, the AE-Skin provides continuous tactile guidance, ensuring the accuracy of their actions, as well as notifying the completion of the operation (as shown in Fig. 5). Current tactile devices can only realize a single function (sensing or feedback), and cannot simultaneously record the driver’s interactive information and provide tactile feedback for drivers^5,6,7,8,9. For example, Luo et al.⁸ adopt the pressure signals obtained from piezoelectric sensors embedded in the carpet to tracking of the driver’s posture. Kim et al.⁶ have designed a soft vibrotactile interface to remind the driver of the vehicle’s navigation information. In terms of interior design, AE-Skin can simulate tactile sensations of different textures, providing designers with a more realistic and intuitive sensory experience when selecting materials. Additionally, designers can fine-tune simulated texture patterns (e.g., groove depth and spacing) using texture presentation devices based on application scenarios and customer needs. This design approach significantly shortens the design cycle, reduces product design and manufacturing costs, and improves product quality and user experience.

Fig. 5: AE-Skin for intelligent cars.

AE-Skin mounted on car panels can be used to control audio, wind speed, temperature, seat position and seat angle. In the initial state, the panel surface remains flat. When the driver touches the panel, the integrated sensor captures the pressure of the finger, and then the actuator generates a certain trajectory for interactive guidance. Under the guidance, the driver can be quickly and accurately moved to the specified position and press the designated position for auxiliary operation, while the actuator vibrates to remind the driver whether the operation is correct.

Intelligent museum: AE-Skin can enhance the vitality and authenticity of exhibits with dynamic patterns and temperature feedback, offering immersive and interactive experiences, such as smart books for exhibiting natural phenomena. For example, visitors can feel the temperature changes of heat exchanges, and the frictional sensation of objects sliding on surfaces with different morphological features by touching the AE-Skin. Compared to conventional video displays, tactile feedback of AE-Skin provides a more intuitive approach to presenting scientific knowledge, which could effectively stimulate the exploration interest and curiosity of younger generations. Furthermore, AE-Skin can be used to generate twin replicas of high-value and fragile artifacts. By digitizing the texture information of ancient inscriptions, engravings, or fabrics, and simulating their morphological features, visitors can touch and feel the artifacts in detail without damaging the original ones (as shown in Fig. 6). Exhibitions enhanced with tactile interactions encourage visitors to actively participate in the learning experience of related historical and cultural knowledge, improving their overall engagement and satisfaction levels.

Fig. 6: AE-Skin for intelligent museums.

AE-Skin can be conformally attached on the replica surface, dynamically simulating the texture information of ancient inscriptions, engravings or fabrics. With AE-Skin and VR glass, visitors can touch and see the artifacts in detail without damaging the original ones.

Active skin for the digital world

Unlike the physical world, digital devices possess limited tangible manipulation. Screens, as the most commonly-used digital interfaces, represent the 2D or 2.5D digital world, while VR devices extend the interaction space to 3D environments with different fixation options. In this section, haptic screens, wearable VR devices, and bare-hand VR devices are highlighted as examples, for describing the materialization of the digital world (as shown in Fig. 3). By applying AE-Skin to the above devices, the integration of sensing and driving capabilities enables closed-loop control of the different tasks and procedures, providing users with a more intelligent and efficient interactive experience.

Haptic Screen: Screens of various sizes have achieved a high level of ubiquity (e.g., smartphones, iPads, televisions, in-car displays, laptops, and advertising screens), presenting a wide range of information and greatly facilitating people’s daily lives and work (as shown in Fig. 7). AE-Skin provides the possibility to integrate the rich tactile experience into screens. For example, when attached to the surface of a phone-sized or pad-sized touchscreen, AE-Skin can render the tactile sensation of fabrics (i.e., texture and temperature), enabling users to have a tangible shopping experience with more comprehensive product information or simulate the outline of the clicked item on the screen, optimizing the accuracy and precision of users’ finger control (fat finger problem). Especially in in-car displays, tactile guidance allows the driver to perform operations with minimal or no visual input. Furthermore, benefiting from the integrated driving-sensing capabilities of the AE-Skin, product designers can achieve remote collaborative design activities in large-sized touchscreens, such as the manipulation of different materials and forms. However, the currently available screens are limited in their ability to provide multi-modal tactile feedback, as they are only capable of representing a single tactile modal^10,11,12. Based on the electrostatic adhesion or mechanical vibration, these screens can represent diverse textures to simulate the fabrics with different weaving processes and patterns. For instance, Nai et al.¹² study the use of periodic vibrotactile feedback to represent the patterned textures.

Fig. 7: AE-Skin for haptic screens.

The transparent AE-Skin can overlay the surface of the screen and display the same actual terrain features as the physical keyboard, enhancing the user’s interactive experience.

Wearable VR Devices: In interactive virtual environments, existing force feedback gloves attached with AE-Skin make it possible to simulate tactile sensations when a user touches various virtual surfaces such as sofas, tables, and walls, as well as thermal sensations of hot or cold drinks, greatly improving the realism of virtual experiences. Nevertheless, most force feedback gloves provide vibration feedback to the fingertips by integrating with different actuators, such as pneumatic actuators, motors, and piezo actuators. Currently, there is still a lack of force feedback gloves that can provide multi-modal and high spatial resolution tactile feedback^13,14,15. Smart vests with AE-Skin can dynamically adjust their thermal condition based on the user’s body temperature and the virtual environment, as well as provide users with embodied navigation and different physical contact. AE-Skin can even be integrated into shoe pads, simulating vibrotactile experiences of walking on grass, sand, mud, or other terrains.

Bare-Hand VR Devices: In recent years, researchers have developed a variety of handheld VR controllers with multiple capabilities that can render different tactile modals (e.g. shape, texture, size, and stiffness) when users interact with virtual objects^16,17,18. For example, Sun et al.¹⁷ propose a handheld device for bare-hand interactions in the virtual environment, which can render three object properties, namely, size, shape and stiffness. However, the lack of sensors makes it difficult for these VR controllers to detect and distinguish the interactive multi-finger synergistic gestures. AE-Skin-augmented VR controllers can offer enhanced virtual combat experiences by predicting users’ psychological status based on their tactile information such as grip pressure and palm temperature. Increasing grip pressure might indicate a higher combat engagement level. Moreover, by integrating AE-Skin with shape-rendering devices, users can experience virtual objects by touching the digital clay¹⁹ directly using bare hands. However, due to limited operational space, this approach is not suitable for extensive VR interactions. Therefore, AE-Skin can be attached to a mobile platform equipped with digital clay, allowing it to move with the user and render different shapes, temperatures, textures, and vibrations in real-time, and to achieve a more immersive closed-loop control between user perception and object display.

Key technology of the AE-Skin

To meet the application demands of ambient haptic interactions between users and the physical or digital world, AE-Skin needs to display multi-modal haptic stimuli and scene users’ interaction intention. In recent years, numerous researchers have focused on these objectives and have developed texture display devices, temperature rendering devices, vibration rendering devices, and multidimensional tactile sensing devices based on various materials and fabrication methods. In this section, we will provide a detailed overview of the research advancements in these areas.

To develop a high-fidelity AE-Skin with three fundamental characteristics, several key technologies must be taken into consideration, including multi-modal haptic display, multi-dimensional tactile sensing, materials, and fabrication methods (as shown in Fig. 8). During the design of AE-Skin, the required tactile modalities (e.g. texture, temperature, and vibration) and sensing signals (e.g. pressure, strain, and temperature) are selected based on the interactive scenarios. For a certain selected tactile stimuli or sensing signal, the appropriate materials (e.g. responsive material, and nanomaterial) and fabrication methods (e.g. molding process, soft lithography, and 3D printing) are adopted to prepare the corresponding actuator or sensor. With the use of the aforementioned technologies, the designed tactile device can provide users with an immersive interaction experience. Nonetheless, there are still some challenges that need to be overcome in the future, which will be described in the following section.

Fig. 8

The key technologies of the AE-Skin.

Texture display technology for AE-Skin

Texture perception mechanism

The texture is composed of different geometrical features (bumps and grooves) on the surface of an object. Depending on the scale of these features, textures can be categorized into millimeter-level features and micrometer-level features. The boundary value of the scale is reported to be about 1 mm²⁰. The combination of texture features at different scales forms specific texture patterns. Katz proposed the “Duplex Theory” of texture perception, perception of texture depends on two cues, i.e. spatial cues and temporal cues. Spatial cues are determined by the size, shape, and distribution of surface elements. Temporal cues are determined by the rate of vibration as skin is moved across finely textured surfaces. Research has shown that during the process of touching millimeter-level texture features, the SAI receptors within the skin are activated to encode the spatial patterns of the texture features²¹. In contrast, touching micrometer-level texture features induces high-frequency vibrations in the skin, activating the PA and PC receptors, which encode the vibration information of millimeter-level texture features into temporal peak patterns²¹. These encoded signals are transmitted through the ascending neural pathways to the perception area of the brain cortex, where they are further decoded and processed to identify different texture information.

Based on the different approaches to presenting texture information, existing texture display technologies can be divided into two categories. The first category is sensory substitution (for simulating micrometer-level texture features), which involves non-morphological changes to present texture information. Different methods, such as electrostatic adhesion, ultrasound, and mechanical vibration, are used to adjust the normal or tangential force between users’ fingers and the surface of the device to render different texture information. For detailed studies on non-morphological texture display methods, please refer to previous review papers²².

The second category is the faithful reproduction of object surface morphology (for simulating millimeter-level texture features), achieved by creating variable microstructures on the device surface using specific mechanical structures and actuation mechanisms. Users can perceive simulated texture patterns by touching the actuated microstructures. According to the different actuation principles, these methods mainly include dielectric-driven, electromagnetic-driven, shape memory-driven, pneumatic-driven, and optically-driven texture display methods. This paper focuses on exploring morphology-based texture display methods. Next, we will introduce the working principles and characteristics of each actuation method. The details of some representative texture display devices are presented in Fig. 9 and Table 1.

Fig. 9: Examples of the texture display device.

a Texture display method based on a single-layer electro-active polymer actuator. Reproduced with permission²⁹. Copyright 2015, Springer Nature. b Texture display method based on a multi-layer amplified electro-active polymer actuator. Reproduced with permission³⁴. Copyright 2018, IEEE. c Texture display method based on a hydraulically amplified self-healing electrostatic actuator. Reproduced with permission³⁰. Copyright 2020, John Wiley & Sons. d Texture display methods based on a rigid magnetic actuator. Reproduced with permission⁵². Copyright 2017, IEEE. e Texture display method based on a rigid magnetic actuator and flexible base. Reproduced with permission⁶⁸. Copyright 2016, Springer Nature. f Texture display method based on a soft magnetic actuator. Reproduced with permission⁵⁵. Copyright 2022, IEEE. g Texture display methods based on shape memory alloy actuator. Reproduced with permission⁷⁹. Copyright 2013, Elsevier. h Texture display method based on shape memory polymer actuator. Reproduced with permission⁸⁵. Copyright 2017, John Wiley and Sons. i Texture display method based on commercial valves. Reproduced with permission⁹³. Copyright 2012, IEEE. j Texture display method based on microvalves. Reproduced with permission¹⁰¹. Copyright 2014, Royal Society of Chemistry. k Texture display method based on chemical combustion. Reproduced with permission¹⁰². Copyright 2021, National Academy of Sciences. l Texture display method based on liquid crystal elastomer actuators. Reproduced with permission¹⁰⁴. Copyright 2014, Elsevier. m Texture display method based on artificial muscle actuators. Reproduced with permission¹⁰⁵. Copyright 2019, John Wiley & Sons.

Table 1 Summary of performance of representative texture display devices reported in the literature

Electro-active material based texture display method

Electro-active polymer (EAP) undergoes shape or size changes when subjected to external electrical stimulation (voltage or current)^23,24,25. Based on this working principle, EAP is sandwiched between upper and lower flexible electrodes to form a sandwich structure. Under the external electric field, the electrostatic force generated between the upper and lower flexible electrodes compresses EAP to produce deformation^26,27. To generate vertical deformation for texture display, a boundary constraint layer is typically prepared in the circumferential direction of the EAP actuator, which converts in-plane deformation into out-of-plane deformation, thus forming micro-scale surface morphology changes on the texture display device^28,29.

The EAP-based texture display method has a high response frequency, which can meet the real-time interaction requirements for haptic feedback. However, this method faces challenges of limited vertical deformation and low holding force^30,31,32. To address these issues, a multi-layer EAP actuator is stacked vertically to increase the vertical displacement and holding force of the texture display devices^33,34. In addition, EAP actuators require high driving voltage (several thousand volts), which poses a certain safety risk³⁵. To improve interaction safety, a flexible insulating film is typically applied to the surface of the EAP actuator to protect users from high-voltage shocks³⁴. In recent years, some researchers have developed dielectric elastomer materials with high dielectric constants, which are promising alternatives for reducing the driving voltage to hundreds of volts, improving the safety of the texture display device^36,37,38.

Different from the multi-layer superposition amplification method, the hydraulically amplified self-healing electrostatic (HASEL) actuator exploits the combination of hydraulic and electrostatic forces to realize the amplification of the holding force and vertical displacement^30,39. HASEL actuators employ liquid as the dielectric layer instead of elastomers, and the dielectric fluid is encapsulated within a sealed chamber formed by upper and lower electrodes. When a voltage is applied to the electrodes, the generated electrostatic force compresses the liquid to achieve high-speed electrohydraulic actuation, forming bumps on the surface of the tactile device. Compared to EAP actuators, HASEL actuators exhibit more stable characteristics and effectively avoid potential dielectric breakdown⁴⁰, but the challenges of fluid encapsulation and actuator miniaturization remain to be addressed.

Electromagnetic-actuated texture display method

Electromagnetic actuators have the advantages of fast response, simple structure and stable performance, and have been widely applied in soft robotics^41,42,43 and texture display devices^44,45,46,47. Under the exciting magnetic field generated by the electromagnetic coil or solenoid, the permanent magnet moves upward (named as the raised solution) and forms bumps on the device surface, and the bumps at different positions form a certain texture pattern^{48,49,50,51,52,53,54}. In the raised solution, the electromagnetic force is also used to provide a holding force to resist the finger’s pressure during the interaction. To provide sufficient holding force, it is necessary to increase the drive current or the size of the electromagnetic coil, which will reduce the spatial resolution of the texture display device. A promising alternative is to adopt a sunken solution, in which a groove rather bump is formulated to generate geometric profiles encoded in the target texture pattern^55,56. In the sunken solution, the holding force is provided by the base of the texture display device, and the electromagnetic force is solely used to create the grooves, which significantly reduces the demand for electromagnetic force and effectively improves the spatial resolution of the device.

In electromagnetic-actuated texture display devices, the magnetic interference between adjacent electromagnetic coils will reduce the stability of the device and seriously affect the user’s interactive experience. To address this challenge, a shielding layer is commonly employed to envelop each electromagnetic coil to change the magnetic flux distribution, thereby reducing the magnetic interference between electromagnetic coils^52,56,57,58. In addition, the electromagnetic-actuated texture display devices require numerous electromagnetic coils to keep continuously working to maintain a certain pattern. A long period of working will not only increase energy consumption, but also result in high temperatures and even burnout of the electromagnetic coil, affecting the user experience. The latch structure is employed to lock the moving position of the magnetic actuators, which can realize the preservation of the texture pattern without energy consumption^59,60. According to the latch principle, the existing latch mechanism can be categorized into mechanical structure-based latch method^61,62,63,64 and electromagnetic force-based latch method^56,65,66.

Most existing electromagnetic-actuated texture display devices consist of rigid electromagnetic coils, permanent magnets, and rigid bases, and the rigid components and structures cannot be attached to surfaces of objects with different curvatures, failing to meet the requirements of flexible and stretchable AE-Skin. A simple and effective solution is to embed rigid electromagnetic coils and permanent magnets into flexible substrate materials, which feature with small bending angle and lower stretchability^67,68. At a larger bending angle, the rigid components will be separated from the flexible substrate. Wang et al.,⁵⁵ developed a soft texture display device, which consists of soft magnetic units and a soft membrane bracket. Under the external magnetic field generated by the rigid electromagnetic coil, the magnetic units sink to form pits and represent a desired texture pattern. To achieve flexibility in electromagnetic coils, an effective solution is to encapsulate liquid alloy in a flexible silicone tube as a substitute for rigid electromagnetic coils^69,70,71. Compared to rigid electromagnetic coils, flexible electromagnetic coils have higher resistance, which requires a larger excitation current to generate the same electromagnetic field. Additionally, the size of the flexible electromagnetic coil based on the above fabrication process is relatively large, and the miniaturization technology of the flexible electromagnetic coil needs to be explored to meet the spatial resolution requirements of texture display devices.

Shape memory actuator-based texture display method

Compared with traditional actuators, shape memory actuators possess characteristics such as high strain, large power-to-weight ratio, and simple driving mechanism, making them extensively utilized in texture display devices^72,73,74. Most shape memory actuators belong to thermally driven materials. When the temperature reaches the phase transition temperature, the shape memory alloy (SMA) actuator contracts in length, which causes the pin to move upward, forming a bump of 2.6-mm height on the surface of the device⁷⁵. As the temperature decreases below the phase transition temperature, the SMA actuator gradually recovers to its original length, and the pin gradually descends. Thanks to the simple driving mechanism and structural layout, the spatial resolution of the device can reach 2 mm⁷⁶.

During the interaction, SMA actuators need to be energized for an extended period to maintain the raised state of the pin, which may lead to a rapid increase in the surface temperature of the texture display device, affecting the interaction experience. The surface temperature can be partially reduced by installing fans, and heat sinks⁷⁷, but the cooling effect is limited. To avoid prolonged energization, an effective solution is to install the magnetic components on the pin and the base respectively, and lock the position of the pin with electromagnetic force^78,79,80,81. When the pin reaches the top position under the action of the SMA actuator, the magnetic component on the pin is attracted to the magnetic component on the base. The resulting electromagnetic force enables the pin to maintain the protrusion state even after the SMA actuator is not continuously activated.

In recent years, shape memory polymer (SMP) actuators have garnered more attention as promising actuators for texture display devices because of their flexibility and excellent mechanical performance^82,83. Within a narrow temperature range, Young’s modulus of SMP can be regulated by several orders of magnitude⁸⁴. Benefiting from the controllable and reversible stiffness, some researchers developed a texture display device with the latch function. By patterning an array of miniature stretchable heaters on SMP actuators, one single pneumatic supply is adopted to rapidly and selectively reconfigure any or all elements^85,86,87. When the heater is powered on, Young’s modulus of the SMP actuators is reduced by >100-fold variation. Under positive/negative pressure, the SMP actuator deforms and produces bumps/grooves. The heater is turned off and the pressure is kept on to latch the morphological change in place as the SMP actuator cools, which takes ~2.5 s⁸⁵. Subsequently, the pressure is switched off, the device can achieve the preservation of the bumps/grooves without energy consumption.

Pneumatic-actuated texture display method

Pneumatic actuators have gained favor among researchers due to their wide range of power sources, low cost, and simple structure, and they are widely applied in texture display devices and wearable devices^88,89. Under positive pressure, the pin in the pneumatic chamber moves upward^90,91,92 or the soft membrane on the device surface deforms^88,93 to form the bump. The height of the bumps and the holding force depends on the value of the air pressure. By regulating the air pressure of different tactile units, the pneumatic-actuated texture display device can achieve a dynamic presentation of different texture patterns. To achieve independent control of the tactile units, each tactile unit requires a separate valve to control the on-off of the air pressure. The number of the values increases with the number of tactile units, and the increased cost and volume of the control system restrict the promotion and application of large-area pneumatic-actuated texture display devices.

In recent years, microfluidic systems based on soft lithography techniques have attracted wide attention due to their high integration, small size, precise control, and low fabrication cost^94,95,96. With the development and progress of this technology, microvalves with different functionalities (e.g. multiplexing, pressure regulation, and flow control) have been developed^97,98,99,100, providing a feasible solution for miniaturization and cost reduction of pneumatic-actuated texture display devices. By adjusting the open/closed states of microvalves, the surface morphological features of the texture display device can be switched to represent a certain texture pattern^89,101. In addition to the valves, the bulky pump also limits the miniaturization and portability of pneumatic-actuated texture display devices. Heisser et al.¹⁰² have proposed a promising solution that utilizes the exothermic reaction of methane combustion to achieve the deformation of the tactile unit. This method has the advantages of high response frequency (1 kHz), large deformation displacement (6 mm), and low power consumption (3 W), but it cannot provide a continuous and stable air source for the texture display devices.

Other methods for texture display

In addition to the abovementioned several actuation principles, liquid crystal elastomer (LCE) actuators and artificial muscle actuators can also be used on texture display devices. LCE materials can undergo reversible contraction and expansion in response to an applied light. Based on this characteristic, Camargo et al.¹⁰³ prepared the LCE actuator into a bulging shape. When the light source turns on, the raised LCE actuators contract, and the surface of the texture display device changes from a raised state to a smooth state. Based on the same driving principle, the LCE actuator is fixed at the bottom of the pin, and the pin is pushed upward to form a bump on the device surface through the contraction of the LCE actuators¹⁰⁴. LCE actuator-based texture display devices have the advantages of low power consumption and simple structure, but suffer from limitations such as low response frequency and low holding force.

Inspired by the cephalopods papillae, Lamuta et al.¹⁰⁵ developed a texture display device based on artificial muscle actuators. The artificial muscle actuator is composed of nylon wires and metal heating wires. At a 2-V voltage, the metal wire heats the artificial muscle actuator to produce the reversible deformation, forming a 0.45-mm bump on a device surface. The artificial muscle actuator-based texture display device has the advantages of low power consumption and simple structure, but it is unsuitable for high spatial resolution and dynamic texture display devices due to the limitation of the size and response frequency of the artificial muscle actuator.

Temperature rendering technology for AE-Skin

Temperature perception mechanisms

In 2021, the Nobel Prize in Physiology or Medicine was awarded to David Julius and Arden Patapoutian. They utilized capsaicin to identify receptors in the peripheral nerves of the skin that respond to heat, laying the groundwork for the discovery of other temperature receptors. Cold sensation and warmth sensation, collectively known as thermal sense or temperature sense, reveal the biophysical mechanisms of neural signal induction in response to different temperatures, advancing our understanding of the perception of cold and hot stimuli¹⁰⁶.

The perception of temperature in the human body is associated with the type of stimulus, whether it is cold or hot. Research has shown that the conduction velocity of afferent fibers in cold receptors (typically ranging from 10 to 20 m/s) is higher than that in heat receptors (typically ranging from 1 to 2 m/s)¹⁰⁷. Consequently, the body responds faster to cold stimuli compared to hot stimuli. Furthermore, the temperature perception in the human body is influenced by factors such as the baseline temperature of the skin, the temperature of the stimulus/environment, the location of the stimulated skin, and the rate of temperature change^{108,109,110,111,112}. In an environment ranging from 25 to 40 °C, a higher baseline skin temperature results in a lower threshold for warmth perception, whereas a lower baseline skin temperature leads to a lower threshold for cold perception. In an environment ranging from 30 to 36 °C, where both types of receptors are simultaneously firing, the human skin does not perceive the variation in temperatures¹¹³. Cold receptors are activated within the range of 15 to ~30 °C, while heat receptors are activated within the range of 36–45 °C, allowing individuals to distinctly perceive temperature changes^113,114. When the temperature exceeds 45 °C or falls below 15 °C, pain receptors are activated, causing a sensation of pain^114,115. Therefore, it is essential to control the output temperature of the AE-Skin within the range of 15–45 °C to avoid potential injure to the human body^113,116.

Temperature rendering methods

Temperature rendering devices achieve thermal exchange between the human skin and the device through physical contact, providing users with a tactile experience of different temperatures. They are widely applied in virtual reality^117,118, augmented reality^119,120, human-computer interaction^121,122,123, and other fields. Based on different working principles, existing temperature rendering devices can be classified into two main categories: Peltier elements^124,125,126 and thermo-resistive heaters^127,128,129. The details of some representative temperature rendering devices are presented in Fig. 10 and Table 2.

Fig. 10: Examples of the temperature display device.

a Temperature rendering device based on rigid Peltier patch. Reproduced with permission¹³³. Copyright 2019, Springer Nature. b Temperature rendering device based on flexible Peltier patch. Reproduced with permission¹¹⁶. Copyright 2020, John Wiley & Sons. c Temperature rendering device based on thermal resistance heating principle. Reproduced with permission¹³⁷. Copyright 2016, American Chemical Society. d Electronic skin based on thermal resistance heating and hydrogel cooling principle. Reproduced with permission¹⁴⁴. Copyright 2023, National Academy of Sciences.

Table 2 Summary of performance of representative temperature display devices reported in the literature

In the design of temperature rendering devices, the first consideration is flexibility and stretchability to ensure that the device fits well on the human skin surface^113,130, providing users with effective temperature feedback and reducing heat dissipation during the conduction and interaction. Secondly, temperature rendering devices need to have a fast response speed to meet the requirements of interaction scenarios with significant temperature differences¹³¹. For example, when simulating the feeling of touching hot coffee on a cold winter day, the temperature rendering device needs to first provide a cold sensation to simulate the winter chill and then quickly increase the temperature to simulate the warmth of hot coffee. Additionally, the temperature adjustment process needs to be dynamically controllable to minimize the temperature fluctuations and ensure that the temperature rendering range will not cause harm to humans (exceeding 45 °C or falling below 15 °C)^113,116,131.

Peltier devices consist of N/P-type semiconductors, electrodes, and substrates. Under an external excitation current, charged carriers move from one semiconductor to another semiconductor, resulting in the release or absorption of heat¹³². Based on this principle (thermoelectric effect), Peltier devices can directly transfer thermal energy using electrical energy to achieve cooling or heating and stimulate the user’s thermal receptors. By adjusting the magnitude and direction of the excitation current, precise control of heating/cooling temperature and heating/cooling direction can be achieved. To increase the temperature range and adjustment speed, a fan/cooling tablet is used to accelerate heat dissipation on the heating side to achieve better interactive effects¹³³. Commercial Peltier devices typically have N-type and P-type semiconductors that are usually fixed on rigid substrates^118,133,134. However, rigid Peltier devices cannot conform to the surface of the human skin, reducing the interaction area between the skin and the device and weakening the cooling/heating effect. To address this issue, an effective solution is to fix rigid P/N-type semiconductors on flexible substrates^135,136, utilizing the flexibility of the substrate to enable better attachment of the temperature-rendering device to the skin surface, thus enhancing the user’s interactive experience.

The working principle of thermo-resistive heaters is based on Joule’s law, where the generated heat is related to the resistance, current, and heating time. By adjusting the applied current/voltage, the surface temperature of the thermo-resistive heater can be precisely controlled. Moreover, this method can deliver a large amount of heat to the skin, achieving and maintaining a high output temperature¹³¹. With the continuous development and progress of materials and technologies, a large number of conductive materials have been developed, such as metal nanowires^137,138, carbon nanomaterials^139,140, and conductive polymers¹⁴¹. The flexibility of thermo-resistive heaters has been realized by embedding conductive materials into an elastic matrix. Additionally, some researchers have employed structures such as serpentine-mesh structures¹⁴², wrinkled structures¹⁴⁰, and weaving structures¹⁴³ to achieve the flexibility and stretchability of thermo-resistive heaters. Compared to Peltier devices, thermo-resistive heaters cannot provide cooling feedback to users during the interaction process. To address this issue, Rogers et al.¹⁴⁴ proposed a closed-loop temperature control device, which consists of a thermo-resistive heater based on a copper wire serpentine mesh structure, a passive cooling hydrogel, and an actively switched thermal barrier interface. In the cooling mode, the hydrogel efficiently couples heat with the thin thermal barrier interface, and the evaporation of moisture in the hydrogel lowers the surface temperature of the skin, providing a cold sensation to the user (27 °C). In the heating mode, the heat generated by the thermo-resistive heater causes the liquid inside the thermal barrier interface to undergo a phase change from liquid to vapor. This results in an expansion of the thermal barrier interface and isolates the hydrogel from the heater, effectively reducing heat dissipation from the heater and enabling the heater to reach 35 °C within 5 s.

Vibration rendering technology for AE-Skin

Vibration perception mechanism

The human skin is highly sensitive to vibration stimuli, and even weak mechanical vibrations can activate superficial tactile receptors, enabling individuals to perceive vibrotactile information. The perception process of vibrotactile information by receptors involves transduction and vibration encoding. When mechanoreceptors are stimulated by mechanical vibrations, they convert the vibration signals into action potentials through transduction. Simultaneously, the receptors encode the vibration information, such as vibration amplitude, frequency, and duration, into sequences of action potentials. The encoded action potentials are then transmitted through the ascending neural pathway to the cerebral cortex, enabling the perception of vibration signals¹⁴⁵.

The density of mechanoreceptors varies across different body parts¹⁴⁶. For example, in hairless skin areas such as the hands and feet, both Pacinian and Meissner corpuscles exhibit high densities, making it easier to perceive vibration stimuli^146,147. However, in areas with hairy skin, the spatial density of Pacinian corpuscles significantly decreases, and Meissner corpuscles are completely absent^148,149. Furthermore, each type of mechanoreceptor has a different perceptual range. Meissner corpuscles are sensitive to relatively high-frequency skin deformation^150,151 and have a frequency perception range of 5–50 Hz vibration^146,152. On the other hand, Pacinian corpuscles respond to a wider range of high-frequency vibrations and can perceive transient interactive information^153,154, with a frequency perception range of 40–400 Hz vibration^146,152. Additionally, when a vibration rendering device provides vibrotactile feedback to the user, it is essential to consider whether the presented vibration information can be perceived by the user. In psychophysical experiments, the absolute detection threshold and the relative discrimination threshold are often used to represent the perceptual ability of the human body in relation to external stimuli^155,156. The sensory detection threshold for a particular individual depends on various factors, including the vibration display device, testing procedure (e.g. vibration frequency, amplitude, and stimulus location), subject’s characteristics (e.g. age, gender, skin temperature, and stratum corneum thickness), and psychophysical testing protocols. Further comprehensive and in-depth reviews on this topic are available in the references^157,158.

Vibration presentation techniques

Mechanical vibration stimulation, as the most mature tactile feedback technology, is widely used in various aspects of our daily lives, such as rendering object properties, perceiving interactive positions, and conveying abstract information. During the interaction between humans and objects, mechanoreceptors perceive a significant amount of vibration information, and the surface properties of objects can be recognized by processing and analyzing these vibration signals. Based on this characteristic, vibrational stimuli applied to the contact surface can be used to modulate the perception of texture roughness¹⁵⁹ and friction sensation¹⁶⁰, thus rendering different surface properties of objects. Additionally, some researchers have employed the arrangement of vibration actuators on different parts of the body to simulate the perception of interactive positions. For example, in virtual military training, vibration stimulation is utilized to simulate the collision between a soldier’s body and virtual objects to enhance the soldier’s situational awareness of interactive scenarios¹⁶¹. In the realm of gaming and entertainment, the vibration actuators on various body parts are used to simulate different attack positions, greatly enhancing the interactivity and immersion of the game. In terms of conveying abstract information, vibration stimulation can be used for tactile communication for visually impaired people^162,163, navigation aids^164,165, emotional transmission¹⁶⁶, and vehicle warning^167,168. The details of some representative vibration rendering devices are presented in Fig. 11 and Table 3.

Fig. 11: Examples of vibration feedback devices.

a Tactile belt based on rigid vibration motor. Reproduced with permission¹⁷². Copyright 2011, IEEE. b Electronic skin based on flexible vibration motor. Reproduced with permission¹⁷⁶. Copyright 2019, Springer Nature. c Vibrotactile device based on the dielectric actuator. Reproduced with permission¹⁸⁶. Copyright 2021, John Wiley & Sons. d pneumatic actuator. Reproduced with permission¹⁷⁹. Copyright 2021, John Wiley & Sons.

Table 3 Summary of performance of representative vibration display devices reported in the literature

Mechanical vibration stimuli are applied directly to the human skin and are typically implemented through back-and-forth motion perpendicular to the skin surface^155,169. The most traditional way is to use electromagnetic-based motors to provide vibration feedback. Over the past few decades, many researchers have developed various tactile devices based on rigid motors to provide users with tactile feedback, such as vibration jackets^170,171, vibration belts^172,173, and vibration gloves^174,175. To ensure a satisfactory interaction experience, it is often necessary to use tapes, straps, or other tools to ensure close contact between the tactile device/motor and the skin, which significantly reduces the comfort and convenience of the interaction¹⁶⁹. To address this issue, some research teams have improved existing rigid motors by embedding key components (e.g. coils and permanent magnets) into elastic materials and using elastomeric materials as support structures to create flexible vibrotactile devices^176,177,178. Thanks to the intrinsic properties of the elastic material, the aforementioned tactile devices exhibit good flexibility and stretchability, enabling them to adapt to the curvature of different skin surfaces. Moreover, the elastic modulus of the silicone material is close to that of the human skin, which allows the vibrotactile device to fit closely to the skin surface and provide stable and reliable vibrotactile feedback to the user.

In addition to electromagnetic-based actuators, pneumatic actuators can also provide mechanical vibration stimulation^179,180,181. The frequency and amplitude of mechanical vibrations can be regulated by controlling the opening and closing period of the pneumatic valve and the amount of air pressure¹⁸². Compared to electromagnetic-based actuators, pneumatic actuators have a lower driving frequency (generally below 260Hz¹⁸¹), but can cover most of the perceptual range of the mechanoreceptors and basically meet the needs of vibrotactile interaction. At the same time, pneumatic actuators have the advantage of being simple, easy to prepare, soft, and wearable, which allows them to be easily attached to the skin surface to provide stable mechanical stimulation. However, pneumatic actuators require bulky auxiliary devices such as air pumps and valves, making it difficult to meet the demand for miniaturization and lightweight of the AE-Skin. In addition, some researchers have experimented with other types of actuators to present mechanical vibration stimuli. Among them, EAP actuators have received a lot of attention because of their combination of flexibility, large displacement, miniaturization, high response frequency, and low power consumption^183,184,185. Some researchers have developed vibrotactile devices based on EAP actuators that provide rich feedback to the user by controlling the frequency and amplitude of the excitation voltage^186,187. However, EAP actuators have problems such as high drive voltage, poor safety, and limited wearability, making them difficult to apply on a large scale.

Tactile sensors for AE-Skin

Transduction mechanisms of tactile sensors

Since the 1970s, researchers have been exploring sensors with tactile sensing capabilities and attempting to apply them in potential human-computer interaction scenarios such as prosthetics¹⁸⁸ and computer screens¹⁸⁹, laying a solid foundation for the further development and application of tactile sensors. However, most of these sensors were composed of rigid materials with low spatial resolution, limited to performing preliminary interactive operations. With the continuous progress of materials and fabrication processes, flexible and stretchable electronic devices have made significant breakthroughs and advancements in the following two decades, propelling the rapid development of flexible tactile sensing and its wide applications in fields such as robotics^190,191, healthcare^192,193, and human-computer interaction^194,195. Since the 21st century, various research institutions have successively developed large-area, low-cost, and high-performance flexible tactile sensors based on different conduction mechanisms. According to different conduction mechanisms, existing flexible tactile sensors can be mainly categorized into four types: resistive-based, capacitive-based, piezoelectric-based, and electromagnetic-based tactile sensors. This classification is helpful for researchers comprehending the fundamental working principles of tactile sensors. The details of some representative tactile sensors are presented in Fig. 12 and Table 4.

Fig. 12: Tactile sensors with different transduction mechanisms.

a Resistive Sensor. Reproduced with permission²⁰⁴. Copyright 2014, American Chemical Society. b Capacitive Sensor. Reproduced with permission¹⁹⁶. Copyright 2014, Springer Nature. c Piezoelectric Sensor. Reproduced with permission²²⁵. Copyright 2018, IOP Publishing. d Electromagnetic Sensor. Reproduced with permission²³⁷. Copyright 2021, AAAS. e Temperature Sensor. Reproduced with permission²⁵¹. Copyright 2015, Springer Nature. f Optoelectronic Sensor. Reproduced with permission²⁵³. Copyright 2019, AAAS. g Triboelectric Sensor. Reproduced with permission²⁵⁷. Copyright 2012, American Chemical Society.

Table 4 Summary of performance of representative tactile sensors reported in the literature

Piezoresistive sensors: Piezoresistive sensors measure interactive information based on the resistive effect. Under external stress, the resistance of the resistive material changes, and the applied force can be measured based on the resistance variation. Piezoresistive sensors have been extensively researched due to their advantages of simple structure, low power consumption, and wide detection range^196,197,198. However, they suffer from non-linear response, fatigue, and significant sensing hysteresis. Existing studies have shown that materials such as silicon, carbon nanotubes, carbon black, and graphene^{199,200,201,202,203} undergo changes in their energy band structures and carrier mobility under external stress, leading to changes in resistance. In addition to the energy band structure, the shape change of the sensing material can also alter the sensor’s resistance^204,205,206. For example, piezoresistive sensors composed of silicone rubber tubes and liquid alloys can monitor various deformations such as compression, bending, stretching, and torsion by changing the cross-sectional area of the silicone rubber tubes, exhibiting features of flexibility, stretchability, high sensitivity, high stability, and ultra-high resolution (5 mN)²⁰⁶. Furthermore, the resistance variation is also related to the contact area between components, enabling the detection of weak pressures^207,208,209. Bao et al.¹⁹⁶ developed a high-precision tactile sensor based on an elastic microstructure thin film with hollow spheres. This sensor can achieve ultra-high sensitivity pressure detection by monitoring the resistance variation between the structural film and the electrodes.

Capacitive sensors: Capacitive sensors consist of two parallel electrodes and a dielectric layer sandwiched between them. Under external mechanical stimuli, the storage capacitance changes, realizing the detection of normal force²¹⁰, shear force²¹¹, and strain²¹². Capacitive sensors offer advantages such as high sensitivity, high spatial resolution, large dynamic detection range, and low power consumption^3,213,214. However, capacitive sensors are susceptible to noise interference and have parasitic capacitance, so they are not suitable for operation in strong electromagnetic field environments²¹⁵. In 2012, Wong et al.²¹⁶ developed a capacitive sensor based on flexible electrodes. These electrodes are made by soft lithography, and the internal microfluidic channels of the electrodes are filled with liquid alloys. Compared with rigid electrodes, flexible electrodes have the advantages of anti-fatigue, large curvature deformation, and anti-bending. In addition to flexible electrodes, the introduction of dielectric materials has gradually become an important direction for the development of capacitive sensors^210,217. In 2020, Sharma et al.²¹⁸ used composite nanofiber materials as the dielectric layer to create a sensitive, low-cost, and reliable ultra-low-pressure flexible capacitive sensor. This sensor achieved linear measurement within a wide pressure range of 0–400 kPa, with a sensitivity of 0.51 kPa-1 and a minimum detectable pressure of 1.5 Pa. It can be used to monitor weak physiological signals such as pulse, respiration, muscle movement, and eye movement. The dielectric materials lay the technological foundation for the research and development of next-generation high-performance human-machine interaction devices.

Piezoelectric sensors: Piezoelectric sensors utilize the piezoelectric effect of materials to measure external stimuli. Under external mechanical stress, polarization occurs within the piezoelectric material, generating opposite charges on its upper and lower surfaces. The measurement of this potential difference enables the monitoring of external stress. Piezoelectric tactile sensors offer the advantages of high sensitivity, fast response, good dynamic performance, and low power consumption. They are widely used in the measurement of dynamic signals, such as vibration and slip^219,220,221. However, they face challenges related to sensitivity to temperature and humidity environments, and the inability to measure static signals. The piezoelectric constant is an important parameter for evaluating the energy conversion of piezoelectric materials. The higher the piezoelectric constant, the better the performance of the flexible tactile sensor. Inorganic materials have higher piezoelectric constants but limited flexibility, making them unsuitable for being deployed on curved surfaces. In recent years, researchers have developed piezoelectric polymer materials with high piezoelectric constants and constructed microstructures to fabricate high-performance piezoelectric sensors^222,223,224. For instance, Chen et al.²²⁵ proposed a piezoelectric-enhanced tactile sensor based on polyvinylidene fluoride-trifluoroethylene microcolumns. Vertically aligned microcolumns are fixed between four bottom electrodes and one top electrode, and elastic protrusions are fixed on the surface of the top electrode. The microcolumn structure significantly enhances the sensitivity of the piezoelectric sensor, allowing for precise measurement of various interactive forces such as shear force, normal force, and tensile force.

Electromagnetic sensors: In contrast to other types of tactile sensors, electromagnetic sensors present various advantages, including high sensitivity, low power consumption, high reliability, robustness, and the ability to achieve three-dimensional detection^226,227,228. They are widely applied in fields of robotics^229,230, texture discrimination^231,232,233, and healthcare^234,235. Electromagnetic sensors typically consist of a sensing element and an excitation magnetic field, and the detection of external stimuli is achieved by varying the intensity and direction of the excitation magnetic field²³⁶. Depending on the sensing element, magnetic sensors can be classified into various types, including Hall sensors^237,238,239, anisotropic magnetoresistive (AMR) sensors^240,241, giant magnetoresistive (GMR) sensors^242,243,244, and more. For example, Hall sensors are based on the Hall effect, where the carriers in a metal conductor or semiconductor produce a transverse potential difference under the influence of a magnetic field. The changes in the magnetic field can be calculated by measuring this potential difference. Hall sensors have a large magnetic field detection range and can be used for detecting strong magnetic fields. The excitation magnetic field is typically provided by rigid permanent magnets^245,246, flexible magnetic units^247,248, and electromagnetic coils^249,250. Compared to rigid permanent magnets, flexible magnetic units or electromagnetic coils can provide variable excitation magnetic fields by changing the proportion of magnetic particles or the magnitude of the current, thus further enlarging the measurement range of electromagnetic sensors. Additionally, flexible electromagnetic units have the advantages of miniaturization and ease of fabrication. They can be prepared in different shapes and sizes to meet various application requirements. Moreover, their inherent mechanical characteristics provide magnetic sensors with excellent flexibility.

Other transduction techniques: In addition to the aforementioned four methods, other transduction methods have been explored for the development of flexible tactile sensors. Temperature sensors based on thermoelectric materials consist of P-type and N-type semiconductors. Under the effect of a temperature gradient, thermoelectric sensors generate a thermoelectric potential on both sides, directly converting the thermal signal into an electrical signal for temperature measurement. They can be widely applied in wearable devices and the healthcare field^251,252. Optical sensors, on the other hand, consist of a light source, a transmission medium, and a detector. When subjected to external tactile stimuli, the intensity and wavelength of light in the transmission medium change, and the detector converts the collected light signal into an electrical signal, enabling the measurement of tactile information^{253,254,255,256}. In the operation of these sensors, an external power source is required, which may not only increase the size and mass of the sensor but also reduce its flexibility and stretchability. A promising solution is self-powered tactile sensors, which are based on the principles of triboelectricity or electrostatic induction^257,258,259. Since these sensors can scene interactive information without the external power source, self-powered tactile sensors that utilize triboelectrification or electrostatic induction have great potential to improve the integration and application feasibility of tactile devices.

Typical interaction signal of tactile sensors

During the interaction between humans and the external environment, various tactile information can be perceived through the skin, enabling the effortless accomplishment of complex tasks. To meet the diverse interaction requirements between users and the physical/digital worlds, AE-Skin needs to possess sensing capabilities akin to human skin, capable of detecting and recognizing multiple physical stimuli, including contact pressure, contact location, sliding velocity, tensile strain, contact temperature, etc. Based on the interaction signals, the tactile sensors can also be divided into pressure sensors, strain sensors, temperature sensors, and other types. This classification is beneficial for engineers to select the appropriate tactile sensor for their specific application scenarios.

Flexible Pressure Sensors: Compared to sensors based on other transduction mechanisms, piezoresistive and capacitive sensors not only enable the measurement of dynamic forces but also facilitate the measurement of static forces, and possess the advantages of simple structure and low fabrication cost²⁶⁰. An ideal pressure sensor should exhibit both high sensitivity and a large measurement range while maintaining the same sensitivity under different pressures. However, for highly sensitive sensors, their measurement range is usually limited, making it challenging to measure higher pressures. One feasible solution is to introduce microstructures (e.g. hemispheres, pyramids, and interlocking mechanisms^261,262,263) on the surface of the sensing layer. These microstructures can effectively enhance the sensitivity of the sensor while fulfilling a wide measurement range and high linearity^264,265. Additionally, due to the viscoelastic properties of the flexible sensing material, its deformation and recovery rates are different during the loading and unloading processes, resulting in measurement discrepancies (hysteresis) of the sensor under the same external load. Microstructures and porous sensing materials can effectively reduce the viscoelasticity of the material, thereby improving the sensitivity and reducing the hysteresis of the pressure sensor^210,266.

Flexible Strain Sensors: Similar to pressure sensors, strain sensors are typically fabricated based on resistive and capacitive sensing principles. Strain sensors are usually attached to the surface of the object. When the surface of the object is stretched or bent, the strain sensors will produce corresponding deformation. Traditional resistive strain sensors are composed of metal or semiconductor materials, and they are only suitable for small strain conditions (<1%) due to their mechanical properties²⁶⁷. To measure larger tensile and bending deformations, strain sensors need to possess flexible and stretchable characteristics, with their own deformation exceeding that of the object being measured. To achieve significant deformation, an effective solution is to embed conductive materials (e.g. graphene, carbon nanotubes, and silver nanowires^268,269,270) into elastic materials to enhance the stretchability of resistive strain sensors. However, due to the viscoelastic properties of the materials, there are significant issues of hysteresis and overshoot. Another effective solution is to inject conductive fluids (e.g. liquid alloys and ionic liquids^271,272) into silicone tubing or microchannels, utilizing the fluidic nature and high deformability of the conductive fluid. This allows the strain sensor to maintain excellent conductivity, low hysteresis, high durability, and high sensitivity under large strains.

Flexible Temperature Sensors: Existing temperature sensors can be categorized into three types based on the sensitive materials: resistive material-based temperature sensor, composite material-based temperature sensor, and thermoelectric material-based temperature sensor²⁷³. Among them, temperature sensors based on resistive materials are typically made of pure metals (e.g. platinum, gold, silver, and copper). The resistance of the metal material varies at different temperatures, and the external temperature can be determined by measuring the resistance variation. However, the inherent rigidity of metal materials makes it difficult to meet the requirements for flexibility and stretchability²⁶⁰. To address this challenge, researchers have adopted stretchable structures (e.g. buckling and rigid islands) to enhance the stretchability of thermal resistive sensors^274,275. Similar to resistive material-based temperature sensors, composite material-based temperature sensors also operate on the principle of thermal resistance. Such sensors are made of a composite material consisting of a polymer matrix and conductive fillers. Under external excitation, the substrate material undergoes expansion and separation, resulting in an increased distance between the conductive fillers and a change in the resistance variation of the sensor^276,277. Temperature sensors based on thermal expansion materials exhibit significant hysteresis and are not suitable for applications with high response requirements. In contrast, thermoelectric materials generate an electric current in the circuit connection between p-type and n-type semiconductors under different temperature conditions, directly converting thermal energy into electrical energy for temperature measurement^275,278. Compared to the other two types of temperature sensors, thermoelectric temperature sensors do not require an external power supply, facilitating miniaturization and lightweight design.

Other Types of Sensors: In addition to sensing the pressure and tension in the normal direction, it is also crucial to measure forces in other directions. For example, in object grasping control and friction force detection, it is essential to accurately perceive shear forces to ensure accurate and safe interaction^279,280. To sense both the magnitude and direction of contact forces, a viable solution is to combine multiple sensing units. Under the influence of forces in different directions, both the individual sensing units and the gaps between multiple sensing units undergo deformation, and multidimensional force sensing can be achieved by measuring electrical signals from multiple sensing units simultaneously^281,282,283. Different from the above multiple sensing units, Yan et al.²³⁷ designed a self-decoupled flexible tactile sensor based on a magnetic thin film with a sinusoidal magnetization direction. This sensor utilizes a single sensing unit to measure both normal and shear forces. Moreover, based on deep learning algorithms, this sensor can achieve super-high-resolution measurement of interactive information, and its sensing resolution is 60 times higher than that of the sensing unit’s resolution. Fine-textured surfaces exhibit spatial resolution and vertical height differences at the micrometer level, which makes it difficult for traditional sensors to accurately sense these topography features. Inspired by animal cilia, some researchers have developed cilia-inspired tactile sensors^284,285. By configuring parameters such as the height and diameter of the cilia, the cilia-inspired tactile sensor can achieve ultra-high measurement accuracy. For example, it can distinguish different roughnesses of sandpaper ranging from 9.2 um to 213 um, with a minimum detectable vertical height difference of 7 um²³². It can detect tiny contact forces as low as 1 mN, with a minimum sensing accuracy of 31 uN²⁴⁷.

Materials and fabrication methods for AE-Skin

In order to improve the performance of tactile devices, researchers have expanded the direction of exploration from conventional rigid materials to realize intelligent functional materials. This extension effectively addresses the limitations of conventional rigid materials, and increases the functionalities required for tactile devices such as richness, interactivity and invisibility that are not possible with conventional rigid materials.

Polymers are extensively utilized due to their numerous advantages, including softness, stretchability, non-toxicity, high stability, affordability, and ease of preparation. These properties have captured the interest of researchers and made polymers a preferred choice for flexible tactile device fabrication. By incorporating fillers, cross-linkers, and additives, the material properties of polymers can be tailored to meet specific requirements. Smart materials can respond to external stimuli, such as voltage, magnetic field, temperature, light, etc., and these materials have been widely used in the preparation of actuators and sensors for tactile devices. For example, electronic polymer provides functionalities through conversion between electrical and mechanical energy. Under an electric field, the large strain produced by the electronic polymer actuators can be used to provide texture patterns and vibration feedback for users. At the same time, electronic polymer can be used to prepare sensors for sensing user interaction information.

Similarly, magnetically responsive materials can convert between magnetic and electric fields according to Faraday’s laws of electromagnetic induction. According to this property, magnetic actuators and magnetic sensors can be prepared to provide tactile feedback (e.g. texture display and vibration) and sense interactive information. Another smart material category includes phase change materials, such as thermal responsive materials (e.g. SMA, SMP, and artificial muscles) and optical responsive materials. When the actuators are heated or exposed to light, the phase transitions of these materials occur, resulting in a large amount of elongation or contraction. Based on this deformation characteristic, researchers have developed different texture display devices to formulate patterns. Nanomaterials have good mechanical properties, electrical properties, and low cost advantages, and are widely used in flexible and stretchable tactile sensors. The common nanomaterials used for flexible tactile sensors include nanowire-based materials, CNT-based materials, graphene-based materials, conductive polymers, etc. Based on Joule’s law, nanomaterials can be also used to prepare soft thermo-resistive heaters for temperature-rendering devices. In addition, we also recommend several recent reviews that cover materials for tactile actuators and tactile sensors in greater detail^{3,286,287,288}.

Besides the selection of the abovementioned functional materials, the proper fabrication methods are also important in preparing the tactile actuators and sensors for AE-Skin^286,288. In the preparation of tactile actuators and tactile sensors, soft lithography and molding processes are widely used for creating large-area patterns with microscale resolution on flexible substrates. One advantage of these techniques is their versatility in creating patterns with different geometries and dimensions. They can be used to fabricate microscale features, such as lines, dots, or holes, as well as more complex structures, including channels, pillars, and patterns with varying heights or depths. This flexibility makes them suitable for a wide range of applications, including soft tactile displays, tactile sensors, and electronic circuits. Another advantage is the ability to create large-area patterns with high resolution. These techniques can be used to fabricate patterns over areas ranging from a few micrometers to several centimeters, depending on the size of the master mold or stamp. This enables the production of devices and systems that require both microscale resolution and large surface coverage. In addition, soft lithography and molding techniques offer advantages in terms of cost, simplicity, and scalability. The materials and equipment required for these techniques are relatively inexpensive and readily available. The process itself is relatively simple and can be performed in a standard laboratory environment without the need for specialized facilities.

In recent years, 3D printing technology has experienced significant advancements and has been widely applied in soft tactile devices. This technology, also known as additive manufacturing, allows for the creation of three-dimensional objects by layering materials based on digital designs. The application of 3D printing has revolutionized the way these devices are designed and manufactured. Traditional manufacturing methods often face challenges in producing complex geometries and intricate structures, especially when it comes to flexible and stretchable materials. However, 3D printing enables the production of customized and intricate designs with high precision and accuracy. This has opened doors for the development of innovative flexible tactile actuators, and soft sensors. In addition to the techniques described above, several other methods have been utilized to develop soft tactile devices, including microfabrication, laser processing, transfer printing, and so on. The combination of these fabrication technologies offers great potential for the development of the next generation of haptic devices in the field of human-computer interaction.

Research gaps and future topics

With the continuous development and progress of functional materials, sensing and actuating technologies, and fabrication processes, various skin-like prototypes for displaying textures, vibrations, and temperature, as well as skins for sensing pressure, strain, and temperature, have been developed. Although these advancements are exciting, several important challenges need to be overcome on the path to realizing the goal of ambient haptic interaction using AE-Skin.

Perception mechanisms of human skin

To develop multifunctional and high-performance AE-Skin, it is necessary to deepen our understanding on the physiological and perceptual mechanisms of human skin, including skin biomechanical models, perception thresholds, and haptic illusions. These studies are expected to establish a solid theoretical foundation for the design and performance evaluation of AE-Skin.

The process of bare-finger interaction involves contact, friction, and force transmission between the fingers and objects. Due to individual differences and different object characteristics, the biomechanical model of bare-finger interaction needs to consider geometric parameters (e.g. surface curvature and contact area), surface characteristics (e.g. roughness and friction coefficient), mechanical properties (e.g. anisotropy, Young’s modulus, and Poisson’s ratio), and the contact state (e.g. static contact or sliding contact). The mechanical model will be helpful in simulating the contact conditions during finger-object interactions and accurately predicting the real skin response. It might provide an important reference for the selection of actuation principles, the design of mechanical structures and performance evaluation. In addition, this mechanical model might provide guidelines for optimizing and characterizing the relationship between various parameters of the AE-Skin, such as the spatial resolution of tactile devices, holding force, deformation height, thickness, stretchability, power consumption, etc. Furthermore, in-depth research is needed to reveal the perceptual mechanisms of mechanoreceptors and the biochemical mechanisms of neural signal transmission. Although existing research has explored the functions and signal transduction of individual mechanoreceptors, it remains unclear how different types of mechanoreceptors work together and encode multiple mechanical stimuli at different levels to express complex mechanical behaviors. Additionally, how the brain decodes this multi-level information for accurate perception is another area of investigation. These studies will provide insights into the design of multimodal tactile sensors and signal processing algorithms.

Regarding perceptual mechanisms, the perceptual thresholds of single-modal tactile perception (absolute threshold and relative threshold) have been widely studied. However, it is still unclear how the perceptual thresholds change when multiple tactile stimuli simultaneously act on the human body. Some researchers have explored the coupling effect between temperature and vibration^289,290, or between temperature and texture²⁹¹, providing guidance for the development of temperature-vibration and temperature-texture feedback devices. However, the coupling effects of other modalities combined with texture, vibration, and temperature are still not well understood. In addition, due to the flexibility and stretchability of AE-Skin, the effect of its softness on texture, vibration, temperature and other perception thresholds needs to be further studied. This not only relates to the design of multidimensional sensor arrays but also to the integration strategies of multiple actuators.

Haptic illusions are another aspect of perceptual mechanisms that require research. Existing studies have shown that the temperature perception of the human body is related to the presentation mode of temperature stimuli^292,293. When harmless heat stimuli (36–42 °C) and cold stimuli (18–24 °C) alternate on the human hand, a burning sensation is perceived by the human skin. This temperature haptic illusion is highly stable and robust, with research indicating that 70 to 90% of subjects experience this haptic illusion phenomenon. Furthermore, haptic illusions based on texture, vibration, and multimodal stimulation have also been demonstrated^294,295. However, it is necessary to effectively utilize these tactile illusions to develop innovative interactions. On the other aspect, it is an intriguing topic to explore how to prevent misinterpretation of tactile information caused by tactile illusions.

Spatial-temporal registration among multimodal haptic stimuli

As mentioned above, many skin-like tactile devices have been developed, providing users with a single-modal interactive experience. To further enhance interaction realism, some researchers have integrated multiple actuators to achieve a synchronized presentation of multi-modal tactile information such as texture, temperature, and vibration^{67,296,297,298,299,300,301,302}.

Gallo et al.⁶⁷ integrated electromagnetic and piezoelectric actuators into a flexible silicone substrate to design a flexible tactile device, which can provide temperature and texture feedback. In contrast to the integration of multiple actuators, Kurogi et al.²⁹⁷ employed a single EAP actuator to achieve both temperature and vibration feedback. Under low-frequency voltage excitation, the EAP actuator generates vibrational stimuli, while under high-frequency voltage excitation (>20 kHz), the actuator generates heat based on the dielectric heating effect. The reuse of the actuator reduced the volume and structural complexity of the tactile device. However, only a single modality can be presented at any given moment, and multiple stimuli cannot be rendered simultaneously. As shown in Fig. 13, although a few studies have achieved multimodal tactile presentation, the number and the fidelity of multimodal presentation still differ from the ideal goal of AE-Skin, requiring further research in structural design, mode registration, and power supply.

Fig. 13

Development status of multimodal haptic display devices.

In terms of structural design, careful consideration should be given to the spatial arrangement of different actuators to ensure that they do not interfere with each other and ensure the simultaneous presentation of multimodal stimuli. For example, when Paltier and shape memory actuators are used for temperature and texture rendering, insulation measures are needed to prevent the heat generated by Paltier from affecting the performance of the shape memory actuators. In addition, the layout of different actuators also needs to meet the requirements of spatial registration, ensuring that users can perceive multiple tactile stimuli within a certain contact area (a small contact area on the fingertip). For example, when simulating the tactile experience of a marble surface, it is important to consider the spacing between the actuators that are responsible for rendering temperature and texture, ensuring that the user experiences the sensation of coldness while also perceiving the texture of the marble surface.

In addition to spatial registration, multimodal tactile stimuli also need to meet the requirements of temporal and even semantic registration to provide users with more natural and realistic tactile experiences. In the temporal dimension, different actuators have different response times, such as the slow response of Pertier and the faster response of electromagnetic actuators. The control system of AE-Skin needs to control different actuators to achieve temporal alignment of different tactile stimuli. Furthermore, different tactile stimuli represent different semantic information. Temperature represents the degree of hotness or coldness, and texture represents the roughness of an object’s surface. In the interaction process, different tactile stimuli need to be reasonably matched with the interaction context to enhance the realism and naturalness of the interaction, so as to ensure that the user’s interaction experience is close to the real world.

Furthermore, energy consumption needs to be considered during the design process of AE-Skin. First, in the selecting of the actuating methods, low-energy consumption actuators should be adopted while meeting the performance requirements. Second, in terms of power supply, reasonable power supply methods should be chosen based on different interaction contexts. For example, when AE-Skin is applied to wearable devices or mobile devices, ultra-thin batteries or wireless power supply technologies can be selected¹⁷⁶. When AE-Skin is attached to fixed surfaces such as touch screens in electronic cars, external power supply methods can be adopted. Thanks to high energy density, high power density, and long cycle life, Lithiumion batteries are adopted as one of the most conventional power sources, and are widely used for tactile devices^{177,178,303,304}. Despite their many advantages, the inflexible nature of Lithiumion batteries can restrict the flexibility and stretchability of tactile devices³⁰⁵. It is desirable to develop a promising power method for realizing thin and stretchable tactile devices. Near-field communication is a highly promising solution for power delivery, as it enables wireless power transfer over short distances. This eliminates the need for traditional wired power delivery methods, providing greater flexibility and portability for tactile devices¹⁷⁶. Another alternative solution is the stretchable batteries with excellent flexibility, mechanical stability and electrochemical performance. In the last decade, stretchable batteries have shown great potential for haptics and have integrated into various wearable electronics and tactile devices^{306,307,308,309}.

Integration between sensing and actuating units

To achieve the natural interaction between users and objects, AE-Skin needed to be able to simultaneously sense the user’s interactive behavior and provide multimodal tactile stimuli. To achieve this goal, the sensing and actuating units need to be integrated within the confined contact area covered by a fingertip. Different from the single-mode tactile sensor or tactile rendering device, the design and fabrication of the actuating-sensing integrated AE-Skin require careful consideration of the following issues: spatial layout, spatial registration, and functional compatibility to avoid mutual interference of sensing and actuating components at the spatial and signal levels. Fig. 14 illustrates the functions and spatial resolutions of some existing tactile devices.

Fig. 14: Functions and spatial resolutions of some existing haptic devices.

The values represent the spatial resolution of different tactile devices.

For actuating-sensing integrated AE-Skin, the first challenge lies in the spatial layout of the actuators and sensors to ensure that the actuating signals and sensing signals do not interfere with each other. A widely used layout is to place the sensors above the drivers^{182,310,311,312,313}, which allows the sensors to directly contact users’ fingertips, facilitating reliable tactile sensing. However, the overhead sensor might obstruct the movement of the actuators, thus impairing the performance of the tactile actuators. Conversely, when the actuators are positioned above the sensors, the actuators can provide the user with sufficient tactile stimuli³¹⁴. However, the presence of actuators may also impede the transmission of the user’s interactive information. When the force or temperature is transferred from the user’s fingertip to the tactile sensor, the interactive information may be attenuated, which can increase the measurement difficulty and reduce the measurement accuracy. To avoid mutual interference between the actuators and sensors, an effective solution is to alternately distribute the driver and the sensor on the same horizontal plane^315,316. Compared to the first two layouts, the alternating horizontal layout requires a larger area for arranging the actuators and sensors, which might reduce the spatial resolution of the tactile device, inducing a challenge for the spatial registration of the actuators and sensors. To achieve spatial registration, it is necessary to ensure that the contact area between the fingertip and the AE-Skin can cover at least one actuator and one sensor under any interaction positions. In this case, when the fingertip interacts with the AE-Skin, the user can not only perceive the tactile stimuli from the actuators but also trigger the sensing signals of the sensors. Therefore, the size of the actuators and sensor needs to be small enough to improve the spatial resolution of the tactile device to meet the requirements of spatial registration.

In addition to spatial registration, it is also necessary to consider the compatibility of different components’ performance to avoid mutual interference when they work simultaneously. For example, when the electromagnetic actuator is used for texture display, the use of magnetic sensors, piezoelectric sensors, and capacitive sensors should be avoided to ensure that the electromagnetic field generated by the electromagnetic actuators does not interfere with the sensing signal, thereby affecting the accuracy and sensitivity of the tactile sensors. Furthermore, to achieve a large-sized and high-spatial-resolution AE-Skin, efficient and real-time control of high-density large-scale arrays of actuators and sensors, ranging from thousands to millions or even more, needs to be addressed. To meet the large-scale control requirements, it is necessary to study the flexible and even stretchable circuit, signal line routing design, and high-performance control board.

Spatial-temporal registration between visual and haptic display

Unlike the physical world, when AE-Skin is applied to the digital world such as providing dynamic tactile feedback on a touch screen, it needs to be matched with the digital environment to provide consistent visual-tactile experiences to users, which imposes more requirements on the performance of AE-Skin. Therefore, it is necessary to consider the transparency, thickness, stretchability and virtual-real registration to ensure that the AE-Skin does not affect the visual information and touch-sensing functionality of the screens. Fig. 15 depicts the current state of existing tactile devices with respect to transparency, stretchability, and thickness.

Fig. 15: Current state of existing tactile devices with respect to transparency, stretchability, and thickness.

The height of the colored bar refers to the relative thickness of each tactile device, and the value within/or under the colored bar represents the absolute thickness of the device.

When AE-Skin is attached to screens such as smartphones, iPads, in-car displays, and large-sized LED screens, it needs to have high transparency to ensure that it does not obstruct or affect the visual information presented on the screens, thus avoiding any negative impact on the user’s interaction experience. Therefore, it is necessary to select high transparency of the substrate material, actuators and sensors in the design of AE-Skin. For example, transparent and stretchable polydimethylsiloxane (PDMS) can be chosen as the substrate material, pneumatic actuators can provide tactile feedback to users, and transparent sensors based on materials such as silver nanowires¹²⁷, graphene³¹⁷, and carbon nanotubes³¹⁸ can be used to capture user’s interaction behavior. Additionally, the AE-Skin should be as thin as possible to avoid compromising the touch-sensing functionality of the screen. Otherwise, thicker AE-Skin may prevent the formation of coupling capacitance between the finger and the touch screen, making it difficult to generate induced currents for detecting the finger’s contact position.

In virtual interactive scenes, visual-tactile fusion is one of the key technologies for achieving immersive experiences. Due to significant differences between virtual environments and real environments, there are still some challenges to be addressed in visual-tactile fusion. Firstly, there is a certain time delay in the generation and transmission of virtual visual and tactile signals, which may result in a mismatch between the perceived visual and tactile information in the temporal dimension, causing discomfort or even adverse reactions such as dizziness and nausea. Secondly, both the user’s and the virtual avatar’s positions are constantly changing during the interaction process. The sensors of the AE-Skin need to dynamically and accurately measure the user’s pose changes to ensure the precise matching of visual and tactile information regarding the positions and orientations of the virtual avatar and the real finger, ensuring the interaction experience consistency of the virtual and real space. Additionally, AE-Skin also needs to possess flexibility and stretchability to ensure its suitability for simulating objects with different shapes and curvatures, ensuring a good fit with the surface of the object without affecting its shape, size, and other attributes. For example, one important application scenario is to attach the AE-Skin on the surface of a shape display device to formulate “digital clay”¹⁹, which can provide multi-modal tactile stimuli for supporting bare-hand interaction with virtual objects.

Considering the technological maturity, potential application value, and societal impact, these future topics have been further refined into short-term and long-term goals. In the short term, significant advancements are required in the following aspects. First, user studies should be conducted to explore the effect of the multi-modal tactile stimuli on the perceptual thresholds and haptic illusion. Second, the minimization and performance improvement should be carried out by fully exploiting the potential of the existing actuators and sensors. Third, it is essential to take into account the spatial arrangement of various actuators, as well as the layout among actuators and sensors, to avoid mutual interference. Fourth, the temporal and semantic registration of multi-modal tactile stimuli should be considered to ensure that the representation of different tactile stimuli is consistent with the actual interaction scenarios. Based on these short-term studies, it is possible to develop a preliminary version of AE-Skin to provide users with a more immersive, realistic interactive experience.

In the long term, more time and effort need to be invested to make a breakthrough in the following aspects. In the perception mechanism, a biomechanics model needs to be constructed to simulate and analyze the force transmission and skin deformation when the finger is in contact with the object. Further, a comprehensive investigation is required to reveal the underlying perceptual mechanisms of mechanoreceptors and the biochemical mechanisms of neural signal transmission. In terms of advanced materials, it is necessary to explore other functional materials with high transparency, ultra-thin, and large stretchability. In addition, the development of ultra-thin, flexible, and high-capacity energy storage materials is a significant advancement toward addressing the challenge of energy consumption. In VR applications, it is crucial to conduct in-depth studies on visual-tactile registration to minimize mismatch between the visual and tactile information and prevent adverse reactions during prolonged usage. The solutions to these long-term challenges will propel the development of human-computer interaction, ushering human beings into an ambient haptic world.