How does an AI system generate music and audio in real time?

D'Accord Guitar · 2001

The problem was going from the chord chart to performance: choosing the positions on the guitar neck, modeling the guitarist's hand and animating the performance for learners. The system came out of the project "Agentes Inteligentes para Ambientes Complexos: estudo do acompanhamento musical em tempo real" (Intelligent Agents for Complex Environments: a study of real-time musical accompaniment) (1999–2001) and gave rise to Giordano Cabral's master's dissertation at CIn, D'Accord Guitar: um Sistema de Execução Violonística (2002, advisor Geber Ramalho, CNPq scholarship). Four papers describe the system, with Izabel Zanforlin, Hugo Santana, Rodrigo Lima and Geber Ramalho: Da Cifra para o Braço and Playing Along with D'Accord Guitar (VIII SBCM 2001, Fortaleza), Modelagem da Execução Musical em Violão e Guitarra (ENIA 2001) and D'Accord Guitar: an Innovative Guitar Performance System (Journées d'Informatique Musicale, Bourges, 2001; hal-02987455). The software D'Accord Violão Player (2001) and the company D'Accord Music Software came out of it. The SBCM PDFs are in the proceedings: research and discussion.

Pattern-based rhythmic accompaniment · 2004

Using Patterns to Generate Rhythmic Accompaniment for Guitar generates accompaniment from patterns extracted from real performances. The team included an ethnomusicologist, Carlos Sandroni, alongside Márcio Dahia, Hugo Santana, Ernesto Trajano and Geber Ramalho. Sound and Music Computing / JIM 2004, Paris; hal-03354323, zenodo.849240. The work precedes the doctoral thesis and builds on D'Accord Guitar.

Incremental parsing · 2005–2006

With Jean-Pierre Briot and François Pachet, Cabral adapted the IP algorithm, for real-time continuation of musical phrases, to accompaniment systems: Parsing incremental para acompanhamento em tempo real (10th SBCM, Belo Horizonte, 2005; hal-01416429, PDF in the proceedings) and Incremental Parsing for Real-Time Accompaniment Systems (FLAIRS 2006, AI track on music and art; hal-01338563, PDF).

Harmonisation automatique en temps réel, the thesis · 2008

Doctorate in generative artificial intelligence, with Jean-Pierre Briot (LIP6) and François Pachet (Sony Computer Science Laboratory, Paris), from 2003 to 2008, at LIP6, Université Pierre et Marie Curie (Paris 6), SMA team (Systèmes Multi-Agents), with a CAPES scholarship; from 2003 to 2007 the work was carried out together with the Sony CSL team. The thesis defines the système d'harmonisation automatique en temps réel: a system that automatically generates, in audio and in real time, the harmonization of a melody, possibly improvised, over a fixed rhythm, with partial information and mutual influence between system and user. The method extends music information retrieval techniques to work directly on the acoustic signal, in experiments per development cycle; the result is a framework of solutions evaluated against human musicians, and a simple solution outperformed all the others, including the more complex ones. Defense on 07/09/2008, très honorable mention; jury of Jean-Pierre Briot, François Pachet, Myriam Desainte-Catherine and Sergi Jordà (reviewers), Jean-François Perrot and Geber Ramalho. NNT registration 2008PA066236 at theses.fr; abstract and jury on the LIP6 page.

Ramin · 2008–2009

The thesis framework was given a name and software: Ramin: A Framework for Real-Time Improvised Accompaniment and Harmonization (IEEE International Symposium on Multimedia, Berkeley, 2008) and the program "Ramin, um Sistema de Harmonização em Tempo Real" (2008). The following year, Comparing Techniques for the Automatic Accompaniment of Audio Improvisation (IEEE ISM 2009, San Diego), with Briot, Pachet and Ramalho, compared the accompaniment techniques.

Capturing a Musician's Groove · 2015

Seven years after the thesis, the collaboration with Sony CSL resumed in an application demonstration at IJCAI 2015, in Buenos Aires: Capturing a Musician's Groove: Generation of Realistic Accompaniments from Single Song Recordings, with Mathieu Ramona. A concatenative synthesis engine takes a recording of a performer playing a song and generates the accompaniment of any other song with that content. The care lies in the timing deviations at the edges of the cut segments, which make up the groove: accidental ones are corrected and intentional ones are preserved. Abstract at IJCAI · PDF.

Brazyle · 2017

The Brazilian part of the Flow Machines project, by Sony CSL, on artificial intelligence for music creation: a Brazilian style base. Flow Machines produced the album Hello World (2018), one of the first albums created by AI in the world (album credits); the details are at press.

Chord Detection Challenge · 2021

An open chord detection challenge organized with Moises.ai, launched on March 16, 2021, with results in June (CIn portal, June 9, 2021). The paper evaluating the chord estimation and alignment tasks was published at SBCM 2021, with Valter Jorge da Silva (doi.org/10.5753/sbcm.2021.19425).

Cabral co-advised Uraquitan Sidney Cunha's doctoral research on automatic chord transcription (2010 and 2015 records in Lattes) and advised the undergraduate theses of Arthur Lubambo Peixoto Accioly (chord transcription with hybrid sources, 2017) and Bruno Tavares de Melo Gomes (chord following with dynamic time warping, 2018).

The related product line is iChords: a melody and harmony recognition engine (RHAE/CNPq, 2008–2010), iChords.Net (2008–2010), iChords Station (2011) and the app (2019), with the Banco do Nordeste and FACEPE/Tecnova projects of 2019, under the responsibility of Rodrigo Pereira. It is technological development at Daccord, of which Cabral is a founding partner.

Continues in: Music information retrieval and in the course Computação Musical (Music Computing).