Why don't we let what we hear merge with what we see? It all started with this question from Michaela Catranis. Together with data scientist Nikolay Jetchev, the concert pianist and composer of contemporary classical music "originally had the idea of translating live music and sound AI, supported in real time, into images in order to create a new kind of multisensory Gesamtkunstwerk," says Timo Hagenow, co-founder and CEO of the Berlin-based AI innovation lab LF1. Together with his colleagues and AI experts Duncan Blythe and Alexander Schlegel, he turned the idea into reality.

Clusters of stars in the black nothingness, colored lights that suddenly flash through the clouds and spread out - these are recordings from the "Hubble" space telescope that dance to the sounds of Michaela Catranis' piece "Synthingout," thus bringing the auditory to life visually as well. This translation of sound into images happens with the help of modern AI technologies using various neural networks. How this is done is described on the website of SHEEN AI - the name of the project: According to the sound, visual AI "paints" visuals on a dynamic canvas using its imagination. The imagination is based on visual concepts derived from visual reference inputs such as images and videos and brings them to life - even live on stage. Subsequently, audio AI perceives sound and breaks it down into auditory components, which in turn must be combined with visual features to create the desired look and feel of the image.

Immersion in water drops and acrylic paints

Synthingout is not the only piece of music in which this innovative technology has already been used. According to Hagenow, "diverse but little-known" sound works have already been visualized under SHEEN AI's brand. The AI does not always "paint" with images from outer space; water drops or an acrylic painting by artist Pawel Czerwinski - conducted by the technology - also allow all the senses to be immersed in the art experience.

SHEEN AI, which was launched in January 2022 as a project of LF1, is only at the beginning:

"We are currently working on a number of exciting collaborations, especially with electronic music artists," explains Hagenow, who says he is very enthusiastic about music and visual art. The entrepreneur is not yet allowed to reveal more details, but the vision of him and his small team is clear: "Our focus is on exploring the artistic potential of a symbiosis between artificial intelligence and traditional art forms," says the businessman, who is currently building up a network of freelancers and artists such as musicians, visual artists and VJs. "Our goal is to use AI to create a new art medium in which the auditory merges with the visual to create a multisensory, fully immersive environment - as a total work of art."