III Access, control, and immersive media

Visuality as historical experience: immersive multi-directional narrative in the MIT Visualizing Cultures Project

Ellen Sebring

Visuality as historical experience

As we move through the digital age, from inception, through the spread of the Internet in the 1990s, and smartphone in 2007, visuality takes a larger role in the expression and reception of content. The text-based methodologies well established in the field of history are slowly expanding to include the visual historical record as a newly accessible resource and alternate form of testimony. Digitization aligns visual sources in unprecedented ways that blur the boundary between data and narrative. This chapter examines the creation of meaning in visual databases, and its communication through “Visual Narratives” that alter the relationship between author and reader.

An early investigation into image-driven scholarship, Visualizing Cultures (VC) was founded at the Massachusetts Institute of Technology (MIT) in 2002 by prominent Japan historian John W. Dower, and linguist Shigeru Miyagawa. With over 55 content units by 28 contributing historians, curators, and art historians to date, VC is a widely used resource for images and research on modern Japan, China, and the Philippines. The web-based platform delivers content freely worldwide, and has proved to be scalable and sustainable, receiving over a million site visits annually according to site host MIT OpenCourseWare. VC continues to be a testbed for historical representation with new content units in production some 17 years later. As Creative Director of Visualizing Cultures,

I collaborated with contributing scholars with the aim of pushing the project towards innovative digital paradigms that would expand the role of visual sources while maintaining credibility within the text-based corpus of traditional historical practice.

The turn towards the visual comes in different forms at different times. British historian and theorist Raphael Samuel, for example, observed that the media-induced turn towards visuality in the 1960s came with a pervasive embodiment: “the eclipse of radio by TV turned Britain, in the space of ten years, from a nation of viewers, to body politics, where picture windows, plate-glass doors and open-plan layouts gave office and shops a see-through look” (1994, p. 339).

My own research on the visual culture of the Boxer Uprising in China reveals a global response in an array of sophisticated media at the turn of the twentieth century. The colligated digitized sources—currently around 2,000 images—suggest a rich visual history that rivals the many published Boxer Uprising studies. But they are a resource without a publication mechanism. My essays for Visualizing Cultures, including several Boxer Uprising units co-authored with Yale University historian Peter C. Perdue (Perdue & Sebring 2016, 2017), present the visual sources, yet cannot replicate the immersive dynamism of the database. Somewhere between the raw data and the completed narrative lies a sensory experience of source that might, with new digital forms, be mediated as historical experience.

Visualizing Cultures makes a substantive case study in image-driven historiography and the digital future. What does the aggregation of source images and their organization into multiple content-driven units on VC tell us about visualizing history? What is next as platforms, media, and use patterns evolve?

Comparison: side-by-side

The first unit on VC, “Black Ships & Samurai” (Dower 2004/2010), created meaning within image sets through side-by-side comparison. Dower’s study of Commodore Perry’s 1853-1854 mission to force open long-secluded Japan juxtaposed the two sides of the encounter. The American view, captured by the official expedition artist and photographer, and the Japanese view, a spontaneous outpouring of sketches, studies, and scrolls, revealed by contrast how each visualized the “Other”. The digital format meant images could be reproduced in unlimited numbers and configurations, and that thematic chapters could be placed side-by-side, accessed in any order, with much of the content embedded in the visual layout. The cultural history described in the text was drawn from the visual record, reversing the secondary role images often play as illustrations for written analysis. (Fig. 7.1)

But always, in my case, the image was driving the structure. It was image- driven narrative. So the first thing we did, “Black Ships & Samurai”, which I still think was one of our great units, we were exploring everything under the sun in that unit. It’s completely image-driven. If I were sitting down to write a book on Commodore Perry and the opening of Japan, it would be an entirely different thing were it to be based on lots of texts and lots of data, and pictures wouldn’t be driving that book. At the same time, when I was writing it, I had to know what took place in the expedition. I had to be familiar with the written record. But it was the images that suggested to me how we can make this story unfold. And I don’t dislike the

Comparison of portraits of Commodore Matthew Perry as they appear in the MIT Visualizing Cultures project

FIGURE 7.1 Comparison of portraits of Commodore Matthew Perry as they appear in the MIT Visualizing Cultures project: left, Perry, ca. 1854 by unknown Japanese artist, Nagasaki Prefecture; right, detail of daguerreotype portrait of Perry by Mathew Brady, c. 1856, Library of Congress; John Dower, “Black Ships & Samurai: Commodore Perry and the Opening ofjapan (1853—1854)”, MIT Visualizing Cultures, 2004/2010.

word story, even though narrativity is a pejorative term these days. There are stories, and there are stories within stories and if you tell them, and let them unfold correctly, they can become very complex.

(John W. Dower, interviewed by Ellen Sebring, 1 October 2014)

Comparison: linear/non-linear

As the number of units grew, VC’s icon menu began to suggest multilinear relationships between historical events. The icon interface initially confused some users as it predated the icon menus that became ubiquitous after the introduction of the touch screen interface with Apple’s iPhone and iOS mobile operating system. VC thus developed a “table of contents” in both icons and text, making for comparative information systems. The text menu, a chronological list, aligned units in vertical, linear relationships. The icon menu of visual thumbnails, organized units geographically and chronologically (as much as possible) in a multilinear matrix. Users do not generally start at the top left with the earliest dated units, but move through the grid in all directions. Since the goal was to immediately engage the visual sense, the unit titles and dates appeared only on rollover.

Alternate menus for the content units on MIT Visualizing Cultures

FIGURE 7.2 Alternate menus for the content units on MIT Visualizing Cultures: left, list menu right, visual icon menu, designed 2004, captured 2019.

Unexpectedly, the sum total of the thumbnail icons—the grid itself—became a visual overview of the history presented therein. (Fig. 7.2)

Elevating visual media to the role of primary user interface created unusual readings within the icon menu. Visual data fields can convey provocative and understudied overviews of information translated from other mediums. The visual dictionary created by the MIT-IBM Watson AI Lab, for example, was an initial step in teaching computers to see: “a visualization of all the nouns in the English language arranged by semantic meaning. Each of the tiles in the mosaic is an arithmetic average of images relating to one of 53,464 nouns” (http:// groups.csail.mit.edu/vision/Tinylmages/; [Viewed 9 May].) The visual dictionary translated words to images by computationally averaging images of nouns, which, once compiled, created a unique visualization of the world’s people, places, and things.

In the VC menu, the icon grid reveals media types over time. The grid suggests that historical events have recognizable visual fingerprints tied to time and place. Even short time periods show marked changes in mass media, as with the emergence of battlefield photography during the 14 years between the First Opium War (1839-1842) and Second Opium War (1856-1860) in China. (Fig. 7.3)

In another example, a series of units by Dower on the Sino-Japanese and Russo-Japanese Wars revealed the peak and ebb of the woodblock print medium in Japan, and its demise with the emergence of consumer-based mass communication and picture postcards. The interplay of physical media types highlighted the range of intent behind their creation and mechanics of distribution, again exposing the relationships between historical events embedded in the interstices between units. (Fig. 7.4)

In “Black Ships & Samurai”, the Japanese view and the American view appear side-by-side in a horizontal rather than vertical relationship. This small deviation from linear exposition multiplies within the grid. Surrounding units— those to the left and right, above and below—become interrelated rather than isolated thematic episodes. The intrinsic pluralism of the medium supports the move towards transnational history that considers international developments over hyper-national specializations. The icon menu traverses time and place. “Black Ships & Samurai” leads to the “Yokohama Boomtown” unit depicting the first Western settlements in treaty-port Japan, and parallels developments in China. (Fig. 7.5)

Surprisingly, the “First Opium War” unit by Peter C. Perdue (2011) appears next to a unit (inserted several years later, as digital publication is not fixed but flexible) about the same war as it was depicted in a Japanese publication. The illustrated semi-historical account, Kaigai Shinwa (New Stories from Overseas) by samurai scholar Fuko Mineta would have been illegal when it was published in 1849. The unit, “The Opium War Through Japanese Eyes”, punctures the myth that information did not flow across Japan’s closed borders, and like the Perry unit, delves into views of the “Other”. (Fig. 7.6)

Two grids of Western graphics might be read as “fingerprints” reflecting the changing visual media between the First Opium War, 1839-1842

FIGURE 7.3 Two grids of Western graphics might be read as “fingerprints” reflecting the changing visual media between the First Opium War, 1839-1842 (on the left), and the Second Opium War, 1856—1860 (on the right), in China. The earlier war was rendered in lithographs, engineering drawings, and personal watercolours by a ship’s surgeon, Dr. Edward Cree. The Second Opium War saw the first battlefield photography, staged on site by Felice Beato, and French and British cartoons circulated, notably Daumier’s racist caricatures.

Source: Grids by Ellen Sebring using images Irom MIT Visualizing Cultures, 2018

A graphical sequence demonstrates the progression of mass media from woodblock print to postcard

FIGURE 7.4 A graphical sequence demonstrates the progression of mass media from woodblock print to postcard: top, “Illustration of Our Naval Forces in the Yellow Sea Firing at and Sinking Chinese Warships”, woodblock print by Kobayashi Kiyochika, October 1894; bottom, “Commemorating the Great Naval Battle of Japan Sea”, three-postcard series, 1906, Museum of Fine Arts Boston sources as they appear in the unit by John Dower, “Asia Rising: Japanese Postcards of the Russo-Japanese War (1904—05)”, MIT Visualizing Cultures, 2008.

FIGURE 7.5 A section of the Visualizing Cultures’ icon menu which began with two VC units on Commodore Perry’s 1853—1854 mission to Japan links both vertically to events in China and horizontally to later developments in Japan, MIT Visualizing Cultures, 2019.

VC can be read across the site as a single database of interrelated original sources, with each unit vivifying a particular story. Dower emphasized the importance of the complexity that would emerge in what he called “dense databases”.

What you’ve got to do with the images is see a vast number of images and, just as when you’re doing history, any of us trained as traditional historians went into the archives and read tens of thousands of pages of documents and diaries, memoirs, official documents, and so on. We read huge quantities of materials, and in sifting through that we could see the patterns, the contradictions, the tensions, the variations, the complexity of things. . . . with Visualizing Cultures we try to have dense databases, and we’re interested in giving mortar to complexity, the narratives out there, there’s a lot of sharp analytical points, but complexity, tension, contradiction is there all the time.

(John W. Dower, interviewed by Ellen Sebring 2014)

Convergence of database and story

While written narrative depends on a locked, forward-moving sequence of words, images appear in total, all at once. Supporting these contrasting modalities,

Kaigai Shinwa (New Stories from Overseas) by Mineta Fiiko

FIGURE 7.6 Kaigai Shinwa (New Stories from Overseas) by Mineta Fiiko, a rare 1849 Japanese graphical account of the Opium War in China, as it appears with details of embattled British troops in John Dower’s unit, “The Opium War in Japanese Eyes: An Illustrated 1849 ‘Story from Overseas”’. Book provided by Beinecke Rare Book and Manuscript Library, Yale University to MIT Visualizing Cultures, 2010.

especially the underserved ocular with its multilinear gaze, challenges digital design to move beyond mimicking pre-existing text-based formats. In interactive media, “readers” become “viewers” and “users”, who not only follow the author’s narrative but also explore on their own. VC’s narratives emerged from casting a wide net for visual sources, followed by close study of the resulting databases across a spectrum of image-text relationships.

VC decided to publish the image galleries used to develop the units to make the original sources available to readers free from the author’s essay. The project relied on simple galleries—thumbnail grids with large blowups—rather than diverting resources to creating the metadata needed for searchable databases. The galleries enabled authors and readers to view images in detail and in relation to surrounding images, facilitating observation and comparative looking, and were sufficient for small data sets. Unit creation evolved as a collaborative process, with the VC team adding visual streams that moved beyond illustration to become standalone narrative content arranged to create meaning through juxtapositions that could be understood by looking.

Colligation of diverse sources in digital data sets proves critical to image research, as isolating visual images as illustrations in written histories has led to misinterpretations. In the Boxer Uprising data set, for example, Chinese nianhua (New Year’s prints) depict military scenes inscribed with location, dates, and participants. However, in a more comprehensive search, identical woodblock prints appeared earlier, identified as different wars. Printmaking was a commercial enterprise, and the blocks with dramatic battle scenes were recycled for new wars. Side-by-side image comparison revealed the makers’ process, and undercut a reading of the images based on the text that appeared on the images. Many recycled sources can be found in popular media as the large databases of widely circulating postcards show, where any battle might be used with ad hoc labelling to suit the need. The provenance of an image, its multiple versions, and casual labelling can only be authenticated with large sets of sources. (Fig. 7.7)

Before digitization, original sources were only seen by the specialists able to visit multiple archives to view the media first hand. Physical media was not easily transported, and researchers relied on notes and photos to jog their memories.

The Boston Museum of Fine Arts very generously allowed me to go in and see everything in the Sharf Collection. . . . And I sat there for days looking, writing notes on my pad saying what this is and deciding, this I want copied. Then they would bring me another box and so there were loads of boxes being brought out. But it was like going to a museum exhibition. I would spend my five minutes or whatever on a print, I would write my notes, particularly if it looked like something I wanted I’d have to say what is it in here that makes me want it. If I didn’t want it I would still usually write something I had rejected, because it often happens that

In an example of recycled blocks, the texts on two Chinese woodblock prints label the scenes—identical with minor variations—as different wars

FIGURE 7.7 In an example of recycled blocks, the texts on two Chinese woodblock prints label the scenes—identical with minor variations—as different wars: above, the Sino-Japanese War of 1894, and bottom, the Boxer Uprising of 1900. Top: “Arrest and interrogation of Japanese soldiers”, Sino-Japanese War, 1894, nianhua (New Year’s Print), China; source: British Library. Bottom: “Drawing Depicting the Decapitation of Russian & Japanese Spies”, the Boxer Uprising, 1900.

Source: US National Archives

  • 200 prints later you begin to see, “Hey, that fits something here. I’ve got to go back and find that”. But usually, once I was done with the print, I gave it to them, it went into the shelves, I never saw it again .... I’ve found, when it came to sitting down with our digitized 400 images, I could linger and see things, and see things with a close up precision that was different than looking at the original. I’m not saying this means that you should go out and buy replicas and forget about trying to get originals. I’m not saying that at all, but for the purposes of the historian that wishes to use this as a window of opening up new perspectives, you could stay with it, and that is one of the fantastic things about the new digital technologies.
  • (John W. Dower, interviewed by Ellen Sebring 2014)

Digitization links images in multiple contexts, not only within the author’s text, but with surrounding sources within malleable, sortable database. VC developed Visual Narratives, that is, image sequences derived from a visual theme, to offer optical testimony as historical experience. The user can “see for themselves”, guided by the author’s expertise, and the organization of sources within the unit. In this way, data—which are fundamentally non-linear and non-narrative—and story—which is comprised of fixed, ordered sequences—become linked interdependencies. (Fig. 7.8)

2D to 3D: models for enhanced visuality and VR

Given the staggering numbers of visual sources available digitally, the next step is to break from traditional forms to create digital models that facilitate enhanced looking. Freed from the 2D “grid” of thumbnails on screens, images might convey deeper sensory experiences in 3D platforms. I began experiments with Visual Narrative in Virtual Reality (VR) seeking a model that supports a perceptual shift between reading (following) and looking (exploring). The linear process of reading contrasts with the iterative process of observation described in psychologist Eleanor J. Gibson’s (1969) book, The Principles of Perceptual Learning and Development. An interactive VR model could support the gestures of looking—looking close, looking far, comparing, turning away, and turning back to look again—that create meaning through the perception of objects in relation to setting. (Fig. 7.9)

Immersive media enhances ocular experience. Moving the database into VR might, as the quality of image resolution improves, simulate the physical experience of archives. There, a researcher can touch, move, examine, and sort physical objects. When visual sources appear in VR, similar interactions can occur as the digital 2D scan becomes, once again, a 3D object that moves in relationship to the viewer who, in VR, takes the first person rather than omniscient point of view. I tested the hypothesis with a VR Visual Narrative from my Boxer Uprising database. My students at Duke University built a 3D version of the ten-day march of the Allied troops from Tianjin to Beijing, a multinational expedition/incursion mounted to relieve the siege of the Foreign Legations, and capture the Forbidden City. This narrative represents only

“The Predictable Pose of the Hero” grid juxtaposes details from Japanese prints of the Sino-Japanese War to demonstrate a visual propaganda motif repeated in the central figure’s stance

FIGURE 7.8 “The Predictable Pose of the Hero” grid juxtaposes details from Japanese prints of the Sino-Japanese War to demonstrate a visual propaganda motif repeated in the central figure’s stance. John W. Dower, “Throwing Off Asia II: Woodblock Prints of the Sino-Japanese War (1894—95)”, images from the Museum of Fine Arts Boston (see the unit for complete list) as published on MIT Visualizing Cultures, 2008.

A spectrum of image—text relationships range from written text to visual images can be drawn on by authors in both linear and non-linear forms

FIGURE 7.9 A spectrum of image—text relationships range from written text to visual images can be drawn on by authors in both linear and non-linear forms. The reader’s perception likewise ranges from reading/following to looking/exploring. Chart by Ellen Sebring, presented at the Association for Asian Studies Annual Conference, Washington, DC, 2018.

one of many pathways within the data set, others including the photographic record taken within the siege, and pseudo-educational stereographic tours. “March on Beijing 1900” (Sebring et al. 2018) was further developed at the Harvard University Visualization Lab, and presented at the first Digital Workshop at the annual Association of Asian Studies (AAS) conference in Denver, March 2019. (Fig. 7.10)

Conclusion

Mv conclusion from the experiment was that indeed the VR space emulated archival immersion by isolating the researcher in relation to multiple sources— much like the constrained ambiance of closely controlled archives. Digitized sources regained physicality in relation to each other and the researcher who could move them, and move through them. The “March” prototype was intentionally simple, eliminating the computer graphic environments common to VR in order to focus on configuring the visual sources in spatially meaningful arrays. Source images that appeared in various media types around 1900 were placed as

Screenshots from Sebring’s Virtual Reality version of the invading Eight Nation Allied Army’s march on Beijing which reimagines archival experience as immersive

FIGURE 7.10 Screenshots from Sebring’s Virtual Reality version of the invading Eight Nation Allied Army’s march on Beijing which reimagines archival experience as immersive, non-linear, uniquely visual narrative; Ellen Sebring, “March on Beijing 1900”, for Oculus Rift, 2018.

objects on a period map that could be travelled in VR. Military photographs, Chinese nianhua, sketches, commemorative albums, illustrated news coverage, and commercial postcards appeared at the place and time they originated (photographs) or depicted (illustrations) along the route. This simple configuration restored the context of the visuals, and visualized the more granular logistics of the march as users followed the troops. While written accounts have brought the march to life in full detail, the visual sources carry a temporal stamp that forges a sensory link to the unfolding events. Viewers see the march, imperfectly and partially, as it was seen before the outcome was known, through the lens of the media generated at the time.

We’re saying look at these things. And then we’re saying to them, “You know, this is what somebody was looking at a hundred years ago in China”. That’s what they were seeing. That was shaping their view of the world.

(John W. Dower, interviewed by Ellen Sebring 2014)

The VR project achieved two outcomes: historical experience through the data itself and a link between data and narrative that reflected the complexity of converging viewpoints. The project indicated that source interactions could support multiple comparative data pathways, manipulated by authors, viewers, and, in time, AI image recognition programs, opening the possibility of machine-aided research and collaboration.

Just as two points are needed to make a line, narrativity and meaning emerge in the relationships between images. Narrative and database converge in the digital fabric, and each needs the other. Data without narrative remain obtuse, as N. Katherine Hayles wrote, “database can construct relational juxtapositions but is helpless to interpret or explain them, it needs narrative to make its results meaningful”, while narrative without data misses the computational power and flow of resources that “enhance its cultural authority and test the generality of its insights” (Hayles 2012, p. 176). While core narratives retain their value as communications emerging from expertise, the data field also “plays” a role as a place for looking and exploration. The immersion in archives and data sets that leads to historical understanding can merge with narrative as historical representation moves further towards digital-native paradigms.

References

Dower, J. W., (2004/2010). Black ships and samurai: Commodore Perry and the opening of Japan (1853-1854). MIT Visualizing Cultures [online]. Available from: http:// visualizingcultures.mit.edu

Gibson, E. J., (1969). The principles of perceptual learning and development. New York: Appleton-Century-Crofts.

Hayles, N. K., (2012). How we think: Digital media and contemporary technogenesis. Chicago: The University of Chicago Press.

Massachusetts Institute of Technolog)'., (2002-2020). MIT Visualizing Cultures [online].

Massachusetts Institute of Technology. Available from: http://visualizingcultures.mit.edu Perdue, P. C., (2011). The first opium war: The Anglo-Chinese war of 1839-1842. MIT Visualizing Cultures [online]. Available from: http://visualizingcultures.mit.edu Perdue, P. C. and Sebring, E., (2016). The Boxer Uprising-1: The gathering storm in north China (1860-1900). MIT Visualizing Cultures [online]. Available from: http://visualizing cultures.mit.edu

Perdue, P. C. and Sebring, E., (2017). The Boxer Uprising-2: War and aftermath (1900-1902).

MIT Visualizing Cultures [online]. Available from: http://visualizingcultures.mit.edu Samuel, R., (1994). Theatres of memory. London and New York: Verso.

Sebring, E., (2014). Civilization and barbarism: Cartoon commentary & the ‘White Man’s Burden’(1898-1902). MIT Visualizing Cultures [online|. Available from: http://visualizing cultures.mit.edu

Sebring, E., Zhao, Y. and Ao, Y., (2018). March on Beijing 1900. Virtual Reality prototype for Oculus Rift. Duke University, [unpublished item].

 
Source
< Prev   CONTENTS   Source   Next >