2018 2019

August 20, 2019

There are left-brain people, and there are right-brain people. It’s not often you meet someone who is equally creative and science-based—much less a whole team of creative, intelligent artists and scientists.

But you haven’t met the Machine Learning team from the Yale Blended Reality project at the Center for Collaborative Arts and Media (CCAM). 

Meet Chase Schimmin, a particle physicist who is an avid outdoorsman and photographer from California. There’s also Douglas Duhaime, a developer in the digital humanities lab whose job it is to bring digital transformation to the humanities. And Raymond Pinto is an actor and dancer on Broadway who’s always been interested in the intersection of data and dance and whose approach to dance is based on reason. Then there’s Mariel Pettee, a PhD candidate in particle physics who performs modern dance in her spare time. All four are deeply creative, passionate people who love the arts and the sciences equally and seek ways to merge the two in new, interesting ways that no one has thought of before.

Put them together in a digital media lab and give them virtually unlimited access to the latest virtual reality (VR) and artificial intelligence (AI) technology, and there’s no telling what they’d come up with. 

So that’s what we did, and the results are outstanding. The team has taught a computer how to dance. Seriously. Not only can it mimic moves, it can improvise and create new dance moves and routines that are both beautiful and intimidating at the same time. 

The idea is that dancers and choreographers can use this artificial intelligence to spark new ways to move the body, incorporate new movements in dance routines and teach other dancers the new moves.

So, How Do You Teach a Computer to Dance?

The first step is to capture real movements that dancers use in typical routines. Pettee and Pinto donned motion capture suits equipped with high-contrast sensors on the head, body and limbs and danced in a studio outfitted with 20 motion-capture cameras. 

<photo of dancers in suits>

Duhaime then took the raw data and fed it into a data processing model where it can be viewed and analyzed in different ways. 

<screenshot or short video of stick figure dancers>

Pettee and Schimmin, the two physicists, then created a machine learning algorithm that allowed the model to vary the movements slightly to create new moves. The model can also stitch together different moves to create entire routines.

<screenshot or short video of stick figure dancers with greater variation>

Machine Learning Dancing Applications

Eventually, the team will explore ways to then teach the computer’s routines to actual dancers—closing a 360-degree feedback loop that the team hopes will help people get out of creative ruts, inform more creative choreography and lead to better improvisations. 

The algorithm could also be used as a teaching tool for choreographers who aren’t physically able to demonstrate moves with proper technique due to age or physical ability. It can also be used in game design to create a troupe of dancers that aren’t in lockstep with each other—like in a night club or ballroom.

However it is used, the simple fact that this team taught a computer to dance is incredible. It really shows what is possible in the fields of machine learning and AI—giving hope to those right/left brained people who don’t know if they want to be an artist or scientist. They can be both.