Body On Tune

Build a song with your body — one layer at a time, in a browser, with nothing but a webcam.

Coursework Working build Solo 2 days

Problem

Body-tracking music tools let you trigger sound, not make music. You wave, something beeps, you close the tab. Composing anything still means buying hardware and learning software first.

My role

Solo. I set the concept, audience, flow, interface and every design decision. Claude wrote all the code.

Key decision

An ordered-note system instead of free-movement mapping. Free mapping makes a better toy for dancers; placed notes let a beginner compose. I traded spontaneity for control.

Outcome

Tested unmoderated on two beginner musicians and one non-musician. Composing worked. My main assumption didn't: two of three played the finished track with their body, not the keyboard I gave them. So I rebuilt the performance stage around the body and fixed every finding.

One track, built and performed in a browser. Bass, mids, highs and beats recorded as separate layers, then launched together.

Context & constraints

Two days, a webcam, and no code of my own.

Self-directed coursework, two days — but the idea came first. One day, in a break between classes, I had a rhythm in my head, my laptop in front of me and no instrument near me. I wanted somewhere simple to put it. Then classmates showed a body-tracking project that released sounds, and I asked whether a body could order those sounds into a song.

Browser only, webcam only, nothing to install. The moment a user needs a depth camera or two apps talking to each other, it stops being for beginners. The real limit was me: I don't write code, so every decision had to be one I could specify precisely enough for a model to build.

Strategy

Everyone else built a trigger. I built a performer.

Audience: an amateur non-musician who wants to make a piece, not just a noise.

Every body-tracking audio project I looked at treats the body as a trigger. I treated it as a performer, and a performer has to know what they're playing before they move. That gave me the system: notes the user places within reach, organised by the key so a beginner cannot play a wrong one, four layers instead of one sound source, and a stage where layers launch in time with the bar.

What I rejected: free-movement mapping. Gesture-to-sound serves dancers, who already have something to express. Beginners need structure first. It looked like a feature decision; it was an audience decision.

The name. "MotionSynth" names the machinery. "Body On Tune" names the state. Person first, sensor second.

Process

Day one worked and was unusable.

Note zones were small buttons scattered across the frame, there was no metronome, and recording ran off the R key. Two things broke the moment I performed with it instead of testing it: hands drop below the frame and stop tracking, and reaching for a key ruins the take.

So the zones moved to the left and right edges and grew until I could hit them at speed — a smaller range I could reach beat a full range I'd miss. Recording moved to the eyebrows, the one part of the body that isn't playing. I found that by performing, not planning.

Recreated to show why the zones moved. Below this line the camera loses the hands and the note never fires.
Recording moved to the eyebrows because the hands were already playing.

AI in the workflow

It wrote every line. It decided nothing about the user.

Claude wrote all the code — MediaPipe tracking, Web Audio, the clip launcher. My method: an AI-run interview on my intentions produced a brief, then one chat per section so each part stayed coherent as the system grew. When I rebuilt the performance stage after testing, I used Higgsfield to generate references for a look I could describe but not draw — a dynamic visual system in the lineage of Studio Dumbar — then handed those images to a coding assistant with a written spec. The reference mattered more than the prompt.

Where I deliberately didn't use AI: every user-experience decision. Audience, flow, what each section is for, what gets cut, where a control lives. A model will happily optimise an interface for a user it has never met. No prompt produced the eyebrow control. Performing did.

Craft

Five pages, four layers, one colour each.

Four build pages and a performance stage. One colour per layer — blue bass, pink mids, yellow highs, red beats — carried from the main page through the note zones to the performance grid, so you always know which layer you're in.

The performance stage drops the camera image: a generative colour field driven by the music, with only the tracking skeleton over it. Watching yourself turns a performance into a mirror check.

Body On Tune main page showing four numbered layer sections: bass, mids, highs and beats, plus a performance page.
Four layers as four doors, not a menu. I numbered them; every tester ignored the numbers.
Zones sit at the frame edges, oversized on purpose. A smaller range I could hit beat a full range I'd miss.
Setting a key remaps off-scale notes instead of hiding them. The beginner keeps the whole grid and cannot play a wrong note.
Hands tracking mode: note zones at the left and right of the camera frame, triggered by the index fingertips.
Hands mode — seated, fingertips as triggers.
Full body tracking mode: note zones triggered by wrists and feet.
Full body mode — wrists and feet. Two postures, not two skill levels.
Performance stage: a four by four grid of recorded layers over a generative colour field, with the hand tracking skeleton drawn on top.
Only the tracking skeleton remains. The background is generated from the music — hue from the key, brightness from the balance of low and high frequencies — so what you see behind your hands is the track, not your room.

Outcome & learning

Composition held. My assumptions didn't.

A working build, performed live to my course. They watched it; nobody used it. So I ran an unmoderated test with three people: two beginner musicians who play for fun, and one non-musician. One instruction each — "build a track."

  • All three ignored my build order. I numbered the pages 1 to 4 starting with bass, because that’s how I usually build a song. Nobody followed the numbers — two opened beats, one opened highs.
  • Two of three wanted to play the final track with their body, not the keyboard. I had built the performance page around keys because that’s how the music software I know works. The testers went straight back to my original idea.
  • My best decision is also my worst. Eyebrow recording failed all three in three different ways — one asked "Is it recording?", one started it by accident and couldn’t stop it, one never found it. The insight was right; I never designed the signal that tells you it’s on.
  • Lighting broke the tracking again and again. I had only ever tested in my own room, which happens to be well lit.

Every finding has a change behind it. The performance stage is now played with the body: a 4×4 grid you touch, column launchers you fire the same way, keyboard demoted to a fallback. Beats gained a Free Drums mode. Recording gained a count-in and a REC indicator that stays on screen. Both note columns now carry the full range on pinch-drag octave carousels. The zones stayed placed and ordered — I moved the ordered system onto the body rather than switching to free mapping.

Still open: onboarding, which I'm leaving until I've watched more people fail at it, and lighting, for which I have no fix.

I spent two days building a system I could perform and never asked whether anyone else could. Two hours watching three strangers taught me more than the second day of building did.