5 projects to discover interactive projection mapping

Introduction - interactive art and projection mapping

Interactive art is a dynamic art form that reacts to its environment. The work produced thus creates a link between the environment, such as humans or nature, and the other elements that compose it. These elements can be a stage, a set, a sculpture, light, a computer system… In interactive installations, participants interact through sensors to produce a unique result generated live.

Example : Daniel Rozin – Mechanical Mirrors

Projection mapping, thanks to video projectors, makes it possible to adorn a space with light. It is therefore interesting for visual transcriptions of events occurring in the environment. These events can be for example the movement of a person, the action of a button, or a sound played by an instrument. In this article we offer a selection of interactive projection mapping projects in order to demonstrate the possibilities offered.

Two interactive projection mapping projects by Jesse James Allen

Jesse James Allen began his career as a sound engineer in the entertainment industry in the early 90s, at the dawn of the digital age. He gradually became a media designer for theme parks and learned to master many tools, all of which he sees as essential pieces of a puzzle to deliver emotional experiences through storytelling. He has participated in the design of 53 video games and this activity has truly shaped his career as a creative. The influence of video games can be felt throughout some of his projection mapping projects where he sometimes uses video game controllers to control the projection.

Jesse sees huge potential for HeavyM in interactive experiences and exhibitions to tell more compelling and sensational stories. In fact, he works with painters and illustrators as bringing the physical tactile world into the equation makes the effect even more vivid, more organic. According to Jesse, “The mix of traditional art and digital creates a special kind of magic. I’ve been doing research and development with HeavyM for about a year and I feel like I’m just scratching the surface, touching on a whole universe of possibilities.”

His collaboration with Charlie Jicha resulted in a painting made with conductive ink. This ink is connected to a Touch Board, which converts the information into MIDI for HeavyM. Since HeavyM’s sequencer is MIDI controllable, Jesse linked parts of the painting to specific sequences.

Based on this experience, Jesse James Allen envisioned and built the Seraph Angel, an 8-foot sculpture that was featured at Immerse in 2017, a huge arts festival hosted in Orlando by Creative City Project.

The Seraph combines projection mapping, motion capture and sound design for an immersive interactive experience. This is Jesse’s largest project to date with HeavyM, offering all viewers the opportunity to interact with the installation through their bodies and voices. Jesse harnessed the full power of HeavyM Live in this project.

The majority of the projected visuals are shaders, controllable by the visitor through motion recognition via Leap motion and touch interaction with the conductive ink. Finally, HeavyM’s sound analysis module was used to react to the sound of the visitor’s voice via a microphone.

Interactive projection mapping with motion tracking

Digital Essence produces interactive installations for corporate communication since 2014. The company relies on the collaboration between scenographers, computer graphics designers and engineers to design and implement interactive systems using video projection. For the Hacking de l’Hôtel de Ville de Paris 2020, an event dedicated to digital innovation, Digital Essence designed and produced an interactive projection mapping installation.

The main objective is to highlight a unique place, while proposing a digital animation enhancing the event. Thus, thanks to projection mapping, the steps of the main staircase react to the passage of the public by triggering video animations. The installation gives visitors an immersive and impactful experience they will remember.

Digital Essence used a ZED stereoscopic camera coupled with an image processing algorithm developed specifically for the installation to detect the presence and position of people on the stairs.

This data is sent to a shader which is displayed and deformed with HeavyM in order to turn on/off the stairs when the person walks by. Finally, a “glitter rain” video is played as soon as someone is present in the detection field.

Personalize an interactive projection mapping

Another Digital Essence project: an immersive box offering a personalized experience based on choices made on a tablet. Following a series of questions, the visitor is immersed in a box with a projection around him. The experience can be customized according to the choice of colors, music and destinations. The animations and music vary according to the answers given, to ensure an impactful experience.

For the technical part, the installation uses 2 Olga kits, 3 Full HD ultra-short-throw projectors, a computer and a tablet. Before entering the booth, visitors fill in a form on a web page displayed on the tablet. The answers are then entered into a database and processed by a program running on the computer broadcasting the show. Finally, this program controls the videos displayed by HeavyM thanks to OSC messages.

Interactive projection mapping on stage

To end this selection of interactive projection mapping projects, we have chosen to show its application in a scenic context. For this performance linking dance and video projection, Diomedi Mattia has implemented a real-time image generation system, based on the dancers’ movements.

This interactive ballet was produced by Laserentertainment using Blacktrax technology and TouchDesigner. Blacktrax is a motion tracking technology that relies on infrared LEDs. The dancers are equipped with a small electronic circuit consisting of a battery and LEDs whose light is invisible to the human eye, but not to specific cameras. The positions of the dancers can then be known thanks to an image processing software and are sent to the TouchDesigner software.

TouchDesigner allows you to program image generators through a system of nodes to be linked together.

Conclusion

This selection of interactive projection mapping projects gives a glimpse of the applications and possibilities. These are indeed very broad and will develop more and more in the years to come. Technology plays an essential role in interactive art because it is an integral part of the sensor systems that enable interaction. The accessibility of computer tools is therefore essential for the growth and multiplication of works in this discipline. HeavyM, with its ease of use and versatility, allows the realization of interactive projection mapping in a few clicks.