Immortalizing movement

Every art form has quietly negotiated its own terms with time, and dance got the worst deal of any of them.

A painting is an object; it is the thing itself, and it survives by being looked after. A novel is a set of instructions for producing an experience in a head, and it survives by being copied. Music made the cleverest arrangement of the three: somebody worked out how to write down what the performer should do, so the performance can end and the piece does not, and five hundred years ago a printer in Venice worked out how to run off copies of it by the hundred¹. A quartet packs up, the room empties, and Schubert is still there in the morning.

Dance never got that. You can be in a room where something real happens (a body finds a phrase in the music that nobody planned, including the body) and an hour later there is no trace of it anywhere except in the memory of whoever was standing there. Most art leaves an object behind.

Dance leaves the dancer and the watcher, slightly changed, and little else.

1. Filming is not notation

The obvious answer is to film it, and filming is not nothing. I have hours of footage and I am glad of every minute of it. I have also watched almost none of it back. Footage piles up faster than I'd like to sit through and edit, the few best minutes from a long session are difficult to find among twenty-seven other good ones.

And even watched, a video keeps only how the dance looked from one fixed place in the room, under that light, at that angle, through whatever the camera decided to expose for. It does not keep the movement. You cannot ask a video how fast the wrist was travelling, or where the weight sat, or what the phrase looks like from the other side of the room. The information is in there somewhere, smeared and projected onto a flat pixelated plane, but it is not held in any form you can work with, and being able to work with it is the whole point of writing something down.

It is worth remembering the people who saw this problem coming and attacked it by hand. Rudolf Laban published a notation for movement in 1928²; the Benesh system arrived in the fifties and is still used to set ballet on new companies³. Both are real achievements and both are, in practice, a specialist skill nobody has: writing a dance down by hand takes longer than dancing it, and reading it back takes training. And the reading back was always the point of it: somebody else, in another room, years later, picking the page up, using the motion, dancing with it. The idea was right. The instrument was a pen.

2. What a skeleton keeps

Run the footage through a pose model and what comes out is thirty-three joint positions per frame, in metres, in three dimensions: the trajectory of how a body actually moved through space.

Not a picture of the dance. The dance's shape, as three dimensional coordinates.

That is notation that respects the true form of the dance, finally written at the speed of the dancing. And it behaves the way notation behaves rather than the way footage behaves: the file is small, it is durable, it can be replayed from any angle, measured, laid onto another body entirely⁴, or set beside the same phrase danced a year later to see what your body learned in between. It outlives the footage it came from (footage ages badly, in formats and codecs and the fashions of a lens), and it may even outlive the body that made it, which is a strange way to speculate about one's own arms but perhaps a true one.

I've built Immortal Kinematics from the ground up to explore solutions to the two queries above simultaneously. The kineclipper answers the pile: one can drop a dance video in, it finds the skeleton in your footage kinematically, and uses it to clip the most bangin' seconds. !D.A.N.C.E. answers the reading back: any dancer can publish a take, with skeleton computed, as a challenge, and anyone else can dance it, drop in their clip, and be scored against the challenger's skeleton on a leaderboard.

YTYCD?

3. The same file, two readers

Here is the part I find that most fascinating.

The thing that lets kineclipper find those sick thirty seconds is the same thing a machine could learn to move from.

On the human side, the skeleton is a motion signal, so the software can see where the dancing actually is: peaks of limb speed, sustained stretches of density, a hit that resolves into a shape. The kineclipper watches a thirty-minute dance session, finds all the seconds that cut the room up, and clips them for you. !D.A.N.C.E uses the same signal from the other side of the page: the skeleton on a challenge becomes the scoreboard, your clip is the performance, and what comes back is a score of how closely your skeleton kept to the other. That is Laban's idea expressed through computational pose estimation, no longer written by pen.

On the other side, clean 3D joint trajectories are exactly the substrate that embodied AI is short of. Explosive, off-balance, full-body movement that is constantly recovering its own balance is the slice the big motion archives underrepresent (the canonical benchmark for 3D human pose is eleven actors performing everyday scenarios in a lab⁵, the great unification of marker-based capture runs to a little over forty hours⁶, and the best-known dance set is about five⁷), for the unglamorous reason that it is expensive to capture and few of the computationally minded were pointing the cameras at dancers.

What is live today on the site is the kinematics: where the joints went, and how fast. Forces and contacts (how hard, how the floor pushed back) do not fall out of a video, and I am not going to pretend otherwise.

That said, we are planning for it.

🦾

4. Ten lamps in the dark

The measurement of the humanoid form has deep historical roots.

In 1973 the Swedish psychologist Gunnar Johansson fixed small flashlight bulbs to the main joints of an assistant in tight dark clothes and recorded the assistant moving with the room unlit. On the screen there was nothing but a handful of bright dots on black. When paused, the dots were never read as a body at all. The moment they moved, everyone who watched saw a person, and saw what the person was doing: ten or twelve points in the right motion were enough to call up walking, running, dancing⁸.

Four years later James Cutting and Lynn Kozlowski asked the stranger question: which person? They put lights on the joints of a group of friends, filmed them walking, and showed the dots back to the same group. The viewers could tell who was who, themselves included, with nothing else to go on, no feedback and very little practice⁹.

Movement is a form of handwriting it turns out. Everyone taught the same letters from the same copybook ends up with a hand their mother could pick out of a stack of envelopes, and everyone taught the same eight counts ends up with a way of landing them that is theirs. Nikolaus Troje and his colleagues tried to rub the hand out. They took the videos of point-light walkers and simulated various distortions: evened out their size, then the shape of their bodies, then the rhythm of their stride, then turned them ninety degrees to a view the watchers had never trained on, and the watchers still picked the right walker at nearly three times chance¹⁰. Scrub the ink off a signature and the pressure of the pen still sits in the paper.

The part I cannot stop thinking about is who reads the hand best. When Fani Loula and colleagues showed people point-light recordings of themselves, their friends and strangers, sensitivity was highest to their own movement¹¹. We almost never see ourselves move from the outside, and we know ourselves anyway, the way you know your own handwriting from the inside, from the writing of it.

Machines read it too. A team at Berkeley took recordings of more than fifty thousand people playing a rhythm game in virtual reality, slicing at blocks that arrive on the beat, with nothing but the path of a headset and two hand controllers, and could pick one player out of the whole crowd with 94 per cent accuracy from a hundred seconds of their motion¹². Three moving points and a song to a name.

The kineclipper's skeleton captures thirty-three points. So the file does not hold a loose mannequin walking through the steps. It holds a person, in their particular body, with their particular proportions, and their lazy way of arriving a hair late to the beat and catching up with their hips. It is why the !D.A.N.C.E board is worth reading: every attempt is the same sentence copied out in another body, purely for the fun of it, and the page of them together is a small anthology of people who share a shape and form. And it is the deeper sense of the word on our mantle: what a take keeps is you.

5. Gold, and why it is the anchor

Immortal Boutique is a house. Immortal Kinematics is the arm that keeps the movement; but the boutique's goal more broadly is to dress and address to serve the people dancing it up and making it. The through-line runs in both directions: a movement archive that does not decay, with artifacts built to last and grow.

Gold which holds the frame together, and as decoration.

For gold is the metal that refuses to tarnish. It is why pharaohs were buried in it: not because it was expensive, but because it was the one substance they knew of that time could not surmise¹³. Here we aim to make movement indestructible, to stand the test of time.

6. The invitation

You will dance better this year than you did last year and there will be no record of the difference, unless you make one. That is the ordinary, unremarkable version of the thing this is all about; the pharaohs and the robots are downstream of it.

Drop a dance video into the kineclipper and sweet seconds come back as clips. Publish one as a D.A.N.C.E challenge and watch the next dancer dance it out in their own hand. Both are at kinematics.immortal.boutique.

Dance, and keep it.

Sources

1. Ottaviano Petrucci, Harmonice Musices Odhecaton, Venice, 1501: the first book of polyphonic music printed from movable type, and the point at which a piece could be copied rather than remembered. Wikipedia.
2. Rudolf von Laban, Schrifttanz, 1928, the first published form of the system now called Labanotation or Kinetography Laban. Wikipedia.
3. Benesh Movement Notation, devised by Joan and Rudolf Benesh in the late 1940s and introduced publicly by Rudolf Benesh in 1955; the Benesh Institute merged into the Royal Academy of Dance in 1997 and the notation is still used to record and restage ballet. Wikipedia.
4. Michael Gleicher, "Retargetting Motion to New Characters", SIGGRAPH 1998: putting recorded motion on a body with different proportions. Author's page.
5. Catalin Ionescu, Dragos Papava, Vlad Olaru, Cristian Sminchisescu, "Human3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments", IEEE TPAMI 36(7), 2014. 3.6 million 3D poses, 11 professional actors, everyday scenarios in a lab. Project page.
6. Naureen Mahmood, Nima Ghorbani, Nikolaus F. Troje, Gerard Pons-Moll, Michael J. Black, "AMASS: Archive of Motion Capture as Surface Shapes", ICCV 2019. 15 marker-based datasets unified, more than 40 hours, over 300 subjects. Project page.
7. Ruilong Li, Shan Yang, David A. Ross, Angjoo Kanazawa, "AI Choreographer: Music Conditioned 3D Dance Generation with AIST++", ICCV 2021. 1,408 sequences, over 18,000 seconds of motion, 10 genres, captured multi-view in a studio. Project page.
8. Gunnar Johansson, "Visual perception of biological motion and a model for its analysis", Perception & Psychophysics 14(2), 201 to 211, 1973. Lamps on the main joints of an actor filmed in the dark; 10 to 12 moving points evoke walking, running, dancing, and a stopped display is never read as a body. Springer.
9. James E. Cutting and Lynn T. Kozlowski, "Recognizing friends by their walk: Gait perception without familiarity cues", Bulletin of the Psychonomic Society 9(5), 353 to 356, 1977. Viewers recognise themselves and others from lights on the joints alone, with no feedback and little practice. Springer.
10. Nikolaus F. Troje, Cord Westhoff, Mikhail Lavrov, "Person identification from biological motion: effects of structural and kinematic cues", Perception & Psychophysics 67(4), 667 to 675, 2005. Seven point-light walkers normalised for size, shape and walking frequency and rotated 90 degrees are still identified at nearly three times chance. PubMed.
11. Fani Loula, Sapna Prasad, Kent Harber, Maggie Shiffrar, "Recognizing people from their movement", Journal of Experimental Psychology: Human Perception and Performance 31(1), 210 to 220, 2005. In point-light displays of oneself, friends and strangers, sensitivity to one's own motion was highest. PubMed.
12. Vivek Nair, Wenbo Guo, Justus Mattern, Rui Wang, James F. O'Brien, Louis Rosenberg, Dawn Song, "Unique Identification of 50,000+ Virtual Reality Users from Head and Hand Motion Data", USENIX Security 2023. Replays of the VR rhythm game Beat Saber; 94.33% accuracy from 100 seconds of head and hand motion, out of a pool of 55,541 real users. Paper.
13. Gold is chemically unreactive and does not corrode; Tutankhamun's innermost coffin holds something like 112 kg of it. Royal Society of Chemistry, periodic table: gold. Why the Egyptians chose it is my reading, not the RSC's.