Integrating Digital Performances into Scholarly Work

Streaming audio and video files have become massively popular as internet providers now offer on-demand access to millions of digitized performances. Many people think immediately of the entertainment value of such services. However, many also quickly turn to audio or video for educational purposes, relying on podcasts or YouTube videos for instruction in the classroom as well as for DIY tips around the house. Digital humanities scholars have further investigated how computing technology can allow us to work with these digitized files for insight into the larger questions that the humanities ask. The following examples suggest some promising and innovative areas of research.

The Music Genome Project and Algorithms for Suggesting Similar Titles

Because the demand for online entertainment is high, companies providing streaming audio or video services have sought to improve their

Working with Performances 59 revenue by introducing audiences to additional streaming content that might interest them. This method keeps subscribers and/or generates more opportunities for viewership of revenue-generating advertisements. Though monetization is not the primary motivation for most digital humanities projects, the entertainment industry has relied on humanities scholars for some of its most innovative and profitable work. One of the most famous of these efforts, the Music Genome Project, was designed by Nolan Gasser, who used his experience and his PhD in Musicology to design a process by which popular music was categorized according to hundreds of characteristics so that software could suggest titles similar to each other.⁴

Pandora Radio relies on the Music Genome Project to generate its streaming stations, and their continued reliance on humanities scholars is evidenced on their corporate page explaining their work:

Each song in the Music Genome Project is analyzed using up to 400 distinct musical characteristics by a trained music analyst. These attributes capture not only the musical identity of a song, but also the many significant qualities that are relevant to understanding the musical preferences of listeners. The typical music analyst working on the Music Genome Project has a four-year degree in music theory, composition or performance, has passed through a selective screening process and has completed intensive training in the Music Genome's rigorous and precise methodology. To qualify for the work, analysts must have a firm grounding in music theory, including familiarity with a wide range of styles and sounds. All analysis is done on location.⁵

This insistence on analysts with humanities backgrounds corresponds to the project’s deep interest in the qualities of performance, those aspects that cannot yet be understood or captured by machine learning. When evaluating a song for the project, a scholar attaches attributes, such as genre or instrument, to a particular song. To date, the project’s scholars have analyzed tens of millions of songs, each labeled from options within the 400 possible characteristics. Such analysis means that when a listener “likes” a song on Pandora Radio by selecting a “thumbs up,” Pandora will then use the Music Genome’s software to suggest a song that sounds similar and has similar performance qualities. Often the combinations can be surprising as they group genres and artists that a user might not otherwise have thought to listen to.

The transparency of the Music Genome Project’s methods and its reliance on humanities scholars stands in contrast to other streaming video and audio services. Rather than deeply analyzing what an individual user might like, many services instead try to push certain content out to audiences. That push can be driven by advertising or licensing revenues or simply by what the content provider thinks may be popular at the time. For instance, sites such as Netflix highlight the titles that are most viewed in a country during a given week. Many sites also categorize content by genre, but their categories are often very general. Lack of transparency does not allow us to know whether any other sites approach the 400 characteristics attributable to performances that the Music Genome Project boasts.

Motion Capture Technology

Motion capture technology (also called mo-cap) has a strong connection to the entertainment industry as well. Though the technology has had a history of academic uses in the health sciences, many people first think of film and animation when they think of mo-cap. The study and production of film and animation is, of course, humanities work itself. Researchers exploring what mo-cap technology can reveal about performance take this work a step further to analyze those performances.

There are many applications for motion capture in the humanities. Mo-cap has been used to study dance for years, with a goal of capturing dance moves to instruct students.⁶ Some of this work is rooted in preservation of cultural tradition, such as the project to digitize Cypriot Folk Dances.⁷ Mo-cap can capture subtleties of performance that traditional videography might not. Other disciplines that focus on the importance of human movement to convey emotion, such as acting and communication studies, have also worked with motion capture technologies for analysis.⁸ Interdisciplinary possibilities abound. For instance, David Reed, Cheryl Briggs, and Rich Grula at the University of Central Florida asked acting students to wear mo-cap equipment to learn about how their bodies moved in space and created a collaborative opportunity for animation students to use the data captured to create new projects.⁹

The specialized equipment and software required for detailed data collection in mo-cap, however, has meant that such projects have historically required a great deal of funding or access to shared industry resources. As the technology becomes less expensive, such resources will become more accessible. Mo-cap’s potential applications in Virtual Reality and Augmented Reality make it likely that the technology will become more widely available in the near future.

Working with Performances 61

Analyzing Digital Dance: The Work of Harmony Bench

Harmony Bench is a scholar of dance who researches the impacts that digital technologies have had on the art form. In her book, Perpetual Motion: Dance, Digital Cultures, and the Common, she writes that

digital technologies, and especially internet technologies, have thoroughly saturated the practices, creation, distribution, and viewers’ experiences of dance. Why should this sea change in dance creation and reception matter? It is not only that digital media have radically reformatted dance for an era of information globalization, accelerating and expanding the ways that bodily motion proliferates as it is uploaded, downloaded, and shared as data—though these are important considerations. It is that, in thus remediating and reformatting dance, digital media throw open, magnify, and broadly disseminate dancing’s already powerful physical articulations of how we act in common.¹⁰

Her research underscores these twin influences of digital technologies on performance: they impact both creation and reception. The technology is ever evolving, and some platforms that highlight user-created performances, such as TikTok, can have an enormous impact on culture, as the Lil Nas X example that began this chapter reveals.

Video Games as Performance

Some scholars, including James Coltrain and Stephen Ramsay, have considered whether we can include video games in humanities scholarship.¹¹ Considering them within the scope of the humanities yields much opportunity for research, and games can employ many of the technologies already considered in this book, such as spatial navigation, richly informative visuals, and Virtual Reality. However, it is not yet clear whether humanities scholars in general have embraced the idea of video game as art form.

If we do accept that the study of video games is humanities work, is it possible to consider games as performance? When thinking about the concept of a game, there is no doubt that there are elements of performance involved. However, such performance follows a rule-oriented path of play; there is often a highly structured form of interaction. Whether the performance of playing a video game could be considered art depends very much on the game.

To date, much scholarly discussion of the role of video games in the humanities has focused on their potential use in the classroom as a method of or enhancement to instruction, with Second Life and MOOs (Multi-user domain, Object-Oriented) being two popular approaches. Nevertheless, many published games are deeply invested in storytelling, and some are remarkably artistic in their design. A notable example is the Stanley Parable, which the designers describe thus: “an exploration of story, games, and choice. Except the story doesn't matter, it might not even be a game, and if you ever actually do have a choice, well let me know how you did it.”¹² As we move forward, it is likely that the study of the artistry of storytelling and performance in video games will become an increasingly important subfield of the digital humanities.