Create an Interactive Projection Mapping with the OSC
Impressive Hakanaï performance by Adrien M & Claire B
What is Projection Mapping?
Projection mapping, also known as video mapping or spatial augmented reality, is a groundbreaking technology that transforms ordinary surfaces into dynamic displays by projecting digital content onto them. This technique has revolutionized various fields, including marketing, entertainment, and education, by creating mesmerizing visual experiences that captivate audiences. Whether it’s conveying brand messaging, crafting unique stage backdrops, decorating walls and ceilings, or evoking emotions, projection mapping offers endless possibilities. By turning any surface into a canvas, it allows artists and creators to push the boundaries of visual storytelling and create immersive experiences that leave a lasting impression.
How Does Projection Mapping Work?
Projection mapping works by using specialized software to map digital content onto a physical surface, transforming it into a vibrant display. The process begins with planning the concept and selecting or creating the object for projection. Using virtual program software, the video content is meticulously mapped to the object’s surfaces. In 2D mapping workflows, images or videos are assigned to masks that correspond to the object’s surfaces. For 3D mapping, the scene is recreated in 3D specialized software, and the virtual version of the objects is projected onto their real-world counterparts. This involves setting precise coordinates for the object’s placement relative to the video projector, considering the projector’s XYZ orientation, position, and lens specifications. The result is a seamless integration of digital content with the physical world, creating an awe-inspiring visual experience.
What is Interactive Art ?
Interactive art is a dynamic art form that reacts to its audience or environment. The works are generally presented in the form of installations in which the audience can take part. They can also be experienced through performances during which one or more artists perform actions. Interactive displays are versatile tools used in various applications such as advertising, education, and events, enhancing user engagement and brand awareness.
Interactive art appeared in the late 1950s following the desire of artists to exhibit in places less exclusive than museums or galleries. Streets and stores became spaces of expression in order to propose more participative and inclusive experiences. The participants are no longer spectators, and their role can then resemble that of an actor.
The earliest examples of interactive art were sculptures that could be touched and played with. For example, Niki de Saint Phalle produced Golem, a monumental sculpture that has a slide for children to play with. Another example, in 1971, Gordon Matta-Clark cooked a pig under the Brooklyn Bridge and served 500 sandwiches as part of the performance. Interactive games are a key component of interactive art, engaging users of all ages through motion tracking and customizable projection options.
Some corporate events implement these processes, for example, in product launches, to engage their audiences and leave them with a memorable souvenir or offer a personalized experience.
From a technical point of view, interactive performances or installations are increasingly using computers and interactive projection. Sensors are used in combination with computers so that events in the real world can be interpreted by programs and then transcribed, for example, into sounds or images. Interactive systems integrate technology and art to create immersive installations that respond to audience movement, enhancing events with dynamic visual elements.
Open Sound Control (OSC) for interactive displays
Interactive art is based on the dialogue between several entities. Usually, these entities are not natively designed to dialogue with each other. For example, a music software and a video software installed on the same computer, a Mac and a PC on the same network, or a human being and a machine in the same room.
Standards exist to facilitate exchanges between different software and hardware. We talk about network protocols, such as UDP or TCP, to designate the rules that govern the interactions between the different entities on the same network. Among the standards used, there is also Open Sound Control, which is a transmission format based on UDP or TCP.
Protocols can be compared to rules in a discussion, such as not cutting each other off. Formats can be compared to the grammar of a language, for example, starting a sentence with the subject. If everyone follows the same standards, everyone understands each other!
Open Sound Control is a format inspired by MIDI (Musical Instrument Digital Interface) used in communications between musical instruments since the 1980s. It takes up this idea of connectivity between instruments and software through the exchange of messages, but with significant improvements. OSC can use a network connection such as the Internet or a local network, it can transport more information, and it offers an address-based system similar to file paths.
Originally designed for music applications, OSC has now become a standard used by hundreds of software applications. It is no longer limited to audio data and is capable of carrying images or information for motion tracking, for interactive projections as an example. OSC can also be used to synchronize sound effects with visual elements, enhancing the overall sensory experience of the interactive projection.
Projection Mapping Software and Tools for Interactive Art
There are numerous projection mapping software and tools available, each offering unique features and capabilities to cater to different project needs. Popular software options include Resolume, HeavyM and TouchDesigner, which provide a range of tools such as drag-and-drop features, warping tools, and edge blending to create complex and immersive projections. These platforms also offer advanced features like motion tracking, 3D modeling, and real-time rendering, allowing for the creation of highly interactive and dynamic displays. When selecting software, it’s crucial to consider the specific requirements of the project, including the type of projection surface, the size of the projection, and the desired level of interactivity. By choosing the right tools, creators can bring their vision to life and deliver stunning visual experiences.
Some examples of interactive art software compatible with Open Sound Control: PureData, Vézer, IanniX, Chataigne, Max/MSP, OpenFrameworks, Processing, HeavyM, TouchDesigner, Augmenta…
Depending on their use, these software applications allow you to control an interactive projection generated with HeavyM using external data, for example. Two programs that caught our attention are Vézer for the Mac and Chataigne for the PC. With these tools, you can control HeavyM with oscillators or variation curves that you draw beforehand.
Regarding the interactions between humans and computers, we speak of HMI for Human-Machine Interface. HMIs are the visible surfaces of software, and they allow users to manipulate the functionalities. In fact, you are currently using the HMI of your Internet browser, which allows you to read these lines. The hidden part of the browser allows you to exchange information with other computers and servers on the Internet.
Hexler develops TouchOSC, a tool for building HMIs to send OSC messages. With TouchOSC, you can create applications, usable on tablets and smartphones, to control OSC compatible software. Discover here the technical information to connect TouchOSC to HeavyM.
To make using TouchOSC with HeavyM easier for you, we offer pre-designed interfaces available for download here. These interfaces incorporate the main features of the software, and you simply need to install them on your tablet to remotely control HeavyM.
Finally, in addition to software, it is possible to use devices capable of driving software with OSC. This is the case, for example, with the Stream Deck created by Elgato. This device allows you to simply control the functionalities of interactive projection software by assigning a key. The main advantage of this system is that the keys are miniature screens, and you can therefore display an icon on them. It’s very easy to use!
Example: how Data Flaw created immersive experiences using Open Sound Control?
Romain Astouric, aka Data Flaw, is passionate about the digital revolution we are living in**. It affects all artistic fields, with the production of more and more works using technology in music, visual arts, and especially interactive art. Technology is democratized and is now just a click away. Artists now have access to affordable and easy-to-use tools for interactive projections. To achieve a seamless integration of video onto surfaces, it is crucial to create masks that match the shape of the objects being animated.
Data Flaw, as a creative technologist, teaches coding and the use of technology for creative purposes at ESME Sudria, an engineering school in Paris. He finds inspiration in the work of artists such as 1024 ArchitectureJoanie Lemercier, and the Scale collectiveIn his artistic work, Data Flaw regularly uses projection mapping and welcomes its opening to a wider audience thanks to tools like HeavyM.
For his project at the Makerlab of the Paris Interdisciplinary Research Center, Data Flaw integrated a notion of interactivity to include the viewer in the work. The desired approach is both aesthetic and playful. The public becomes an actor in the installation and becomes aware of the consequences of their actions on the result displayed as an interactive projection. Interactive walls can transform various surfaces into fully interactive displays, enhancing user engagement and creating immersive experiences.
The installation is a projection surface made with digital printing and cutting tools, on which visual animations generated by HeavyM software are projected. The artist used a popular accessory, a PlayStation 5 controller, connected to HeavyM via OSC, through a program created with Processing. Projection mapping projects can transform spaces by integrating movement and sound, allowing participants to interact with artistic installations.
The joystick can trigger various sets of effects and sequences. For example, it can control the shapes and their colors using the joysticks. Anyone can easily create beautiful live color combinations. The color palette and numerous gradients are inspired by a particular video game: Hyper Light Drifter. Video projectors are essential in creating immersive environments that respond to audience interactions, facilitating unique visual experiences.
Unlike a tangible object, computer code is never finished. There are always features to improve or new bugs to fix. This is what Data Flaw is particularly fond of. For his projects, he always needs to fine-tune, with a lot of back and forth, testing, and trials. HeavyM is an indispensable part of his creative process, both in the prototyping phase and in the final adjustments
With HeavyM, you can edit the interactive projection, modify or add shapes with just a few clicks without having to start over or go through a rendering stage. It is a flexible yet powerful tool, without unnecessary complexity.
The democratization of technology now allows a multitude of artists to express their creative talents. More and more interactive installations are being created, and the public is in demand for this type of experience, both aesthetic and playful. To get inspired, you can find examples of creations on Design Boom or Trend Hunter.
The Future of Interactive Projection Mapping
The future of interactive projection mapping is incredibly promising, with advancements in technology and software paving the way for more complex and immersive experiences. Interactivity is set to become the next frontier in projection mapping, with technologies that allow viewers to influence the mapping in real-time through sound, touch, and sight already making waves. Wearable tech is enhancing the interactive experience, and the integration of artificial intelligence and machine learning is expected to further elevate the technology. Despite challenges such as cost and production time, the potential for creating unique and memorable experiences is immense. As technology continues to evolve, interactive projection mapping will undoubtedly play a pivotal role in shaping the future of visual storytelling and immersive experiences.
To be continued
Learn more about projectors
We have created a detailed guide to choosing the right projector for your needs. Discover our tips and mistakes to avoid in this guide that explains the different features step by step.
Explore the HeavyM software
HeavyM is a video mapping software that is capable of generating visual effects and projecting them on volumes, decorations, or architectural elements. HeavyM is Mac and Windows compatible.
EN 
FR 
ES 
DE 
IT 
PT 
CN 
KR 
JA 
RU