Painting Music uses Artificial Intelligence (AI) to create music from live painted drawings, in real time and unique for each performance. This research and development project began in 2018 in collaboration with AI developers Dr. Starkey and Jack Caven from Aberdeen University. The AI algorithm is based on the type of learning used by the human brain. It exploits areas of similarity within the two distinct artforms to respond to the live-painted elements and produce musical notes that reflect the development of the evolving artwork. Painting Music uses algorithms that are explainable, which is innovative even in the field of AI, and the combination of AI, painting and music represents to the best of our knowledge a world’s first.

Under the 2021 Connected Innovators‘ Award Painting Music underwent software and hardware optimisation in collaboration with composers, creative codes, and designers.

A portable and autonomous system designed and fabricated by Ray Interactive was engineered that enables live broadcasting of Painting Music through any physical and digital platform.

The AI music improvisation and sound generation is further developed using theory and practice of this field in consultation with the Edinburgh composers (e.g. Sam Healy @RayInteractive, Dr. Harry Whalley). In collaboration with composer Dr J Harry Whalley, they shortlisted several output software tools that allow them to programmatically generate musical outputs directly from the Painting Music software. The software selected gave them a wide range of control and provides much greater quality of sound than the previous output mechanism.

This venture, from initial concept, prototype development and testing, to a fully integrated product has paved the way to fully exploit cutting-edge technology and investigate questions on our relationship with these new technologies. Painting music offers an attractive package to galleries, institutes, residencies, commissions, physical and digital performances, and public engagement programmes.

Contact

Courtney Bates

Lead organisation

Kate Steenhauer

Hardware

Painting Music uses Artificial Intelligence (AI) to create music from live painted drawings, in real time and unique for each performance

Team

Kate Steenhauer, Founder/Visual Artist/Filmmaker, katesteenhauer@hotmail.com
Dr. Starkey, AI developer
Jack Caven, AI developer
Sam Healy, Consultant/Composer
Dr. Harry Whalley, Composer