JitterHeart

Listen to the Audio

About the Work

JitterHeart is an acousmatic work that ethically explores how AI can reimagine traditional musical notions of the score, performance, and expressivity. In this piece, I use the Max MSP programming environment in conjunction with Jitter (Jit.GL) to create a self-regulating, quasi-organic audiovisual system. The core material stems from a StyleGAN3 neural network I trained at the Yale Center for Research Computing (YCRC), using a dataset composed entirely of frames from my own digital music scores. The resulting AI-generated video is not merely representational because in a sense, it becomes a new score itself – the “heart” of the piece, the motor for both its sound and structure. The difference being that this score is read in a completely different way. Rather than interpreting the video in a traditional sense, I process it technically. Jit.GL extracts its pixel-level data into four dynamic matrices that act like the pulsating chambers of a heart, converting the video into long lists of numbers. These are mapped to musical parameters such as frequency, resonance, envelope shape, articulation density, and velocity. The video’s abstract jitteriness becomes a kind of LFO constellation – a network of oscillators – governing the sound in ways that feel organic and alive. The composition’s harmonic and timbral structure is shaped by a custom-built polyphonic subtractive synthesizer I programmed within Max MSP, using noise as the foundational signal – conceptually echoing the visual noise matrices typically used as generators in Jit.GL. Its behavior is refined by a “modalizer–microtonalizer” module I developed for spectral control. I quantize this noise-derived sound using scales, and further sculpt it with delay, reverb, and sequencing tools. The turbulent, layered and self–contained audio processing mirrors the structure of the video. The idea of the score is further expanded in this piece through a new Type of “reading” and extracting musical information: a data-rich object interpreted and generated by machines from its prior incarnation as a man–made symbolic system for human musicians. By giving the algorithmically generated video such a high degree of agency over sound, JitterHeart poses a speculative question: what if a score could not only instruct performance, but become it, pulsing with its own synthetic jittery heartbeat?

Performance History

Images