November 15, 2019

Experiencing poems, testing phobias, visualizing surgery: these are just some of the projects the Yale Student Immersive Media Club (YSIM) has been working on this semester. Co-presidents of the club, Monique Baltzer and Noah Shapiro, shared with us some of the projects the group has been working on and some projects they’ve been commissioned to do by other departments here at Yale University. Monique and Noah along with another member of YSIM, Xavier, offer free classes on Unity, a gaming version that can be used for virtual and augmented reality, twice a week on Mondays and Saturdays. These classes soon led to four immersive projects that fall under the theme of experiencing media in new ways.

This stemmed from Xavier’s project to bring dresses designed in the Fashion Institute of Technology (FIT) into Tilt Brush, a gaming software that lets the user paint in 3D. Another project in the making involves the way the audience interacts with artifacts in a museum. For instance, as Noah explained, instead of reading where a particular artifact is from, the museum-goer picks up the relic, and the current setting fades away. In its place is the environment in which the piece would have been, like an ancient temple or altar. The third project involves living poetry.

Image from a YSIM virtual reality project

The poem that the team has chosen to work with first is Robert Frost’s Nothing Gold Can Stay. “We’re showing our interpretations of the poem and making it more accessible to others who don’t know how to interpret the poem. This poem mainly looking at a lot of nature imagery and lights floating around and coming together,”  Baltzer explained, “Flowers appearing around the user; we’re going to add interactions between the lines of the poem so the user guides the pace of the poem.” The team created all the models themselves in Tilt Brush. They already have more poems and even book passages lined up for their next pieces.

The fourth project allows the user to create music through body movements. Each movement triggers a particular sound and multiple, sequential movements can create one musical piece.

However, YSIM works on more than just their own projects. Yale Medical School and Yale Psychology have both asked YSIM to help them create projects to aid in their research. YSIM is helping the medical school create an animation to show patients in Tanzania how a kidney surgery, nephrostomy, will be done. So far, the user can only see one side of the surgery, but the group hopes to allow the patient to move around the animation by making a VR version of the process.

Kidney surgery visualization

For psychology, YSIM is helping to construct a VR environment that gives a mild fear stimulus to subjects to see how well people control themselves when they know something scary is coming. For this experiment, the fear stimulus involves watering a plant but spiders quickly start to appear from under the pot.

These are not the first projects this group has created. Last year, YSIM participated in YHack for game design. With a humorous spin, the group created a game to discourage trashing used toys. In this game, the player starts out doing spring cleaning, and in discarding unwanted objects, tosses a teddy bear in the trash. The player is then transported to an abandoned toy factory where they try to find their way out without knowing that the very teddy bear they trashed has now come back for revenge, wielding a blade in its furry, stuffed paw. In fact, this simulation was so engaging that a judge screamed when he turned and saw the stuffed animal!

From training fellow students to working with the medical school, YSIM is truly making a difference in multiple spheres. We’re excited to see the final products and all their future projects!