Intro: In this blogpost, inspired by Carlin Zia’s March 5th OHMA presentation on her book “Uncertain Journeys,” Lisa R. Cohen reflects on the fluid nature of shared memories in different spaces - traveling back and forth between spoken, written, physical and even, increasingly, the virtual world.

In her book, Uncertain Journeys, Carlin Zia creates an evocative, epic poem pieced together  from the memories of her grandfather; his life in China as a child, his journey to America… all the events that led to future generations - including Carlin herself. In the process, her grandfather’s memories became spoken words recorded by Carlin, which became written transcripts, then poetry, and ultimately the book of his life, Carlin’s Master’s thesis for Columbia’s Oral History MA program.

Carlin’s book also includes rare photos of that life, images of physical objects from the past, like this suitcase that made the voyage from China to the U.S. carrying Carlin’s grandmother’s belongings.

Among the book’s many unique features, Carlin physically created the book herself, stitching and binding hand-printed pages, into a finished piece of unique art readers can hold in their hands to experience the original memories for themselves.

This exploration of the fluid transformation of memory, one that shape-shifts from the physical to oral  to written and back again, runs parallel to a very different kind of work at another university,  this one in China, at the University of Nottingham’s Ningbo location.  The twist: research there has subjects access physical souvenirs in a virtual world, using virtual reality technology. Researchers then measure the memory responses that  these “virtual experiences” evoke.  

Here’s how it works: In Ningbo a Chinese student, a  young woman like Carlin, was also researching her roots as part of a study of digital tools to preserve cultural heritage. Unlike Carlin, she was able to  travel back to her grandparents’ village to document physical objects like the ones they had used.  On her field trip, the student  took pictures of the kitchen of an elderly couple, much like her grandparents. She documented cooking pots, utensils, the stove, for example -  that were reminiscent of her childhood visits to the village, taking scores of photos from many different angles for a full 360-degree view. It’s the first step in a process called “photogrammetry.” 

Back at the Ningbo campus, researchers virtually stitched these photos together to create a 360 degree view of the objects, adding a sense of depth and texture.  To illustrate this process, I used photogrammetry to capture  a (very) rough virtual version of Carlin’s book. You can get a sense of its depth and texture beyond what you see in the 2D photo at the top of this post. 

Here’s how it looks and behaves in a virtual space - click on the link to view the 3D model,  try moving it around, making it bigger and smaller,  and getting  a sense of its depth. 

My digital rendering of Carlin Zia’s book appears in a link. Clicking the link brings the 3D model of a white book with black hatch marks. It can be moved around using a computer mouse in desktop mode  or by moving around physically in mobile mode.

I also filmed a video clip showing the rendered book superimposed onto a real life scene, in this case a cobblestone street, for an effect called “Augmented Reality.” The video clip shows how I can walk around and interact with the digitized version. 

At the Ningbo campus,  researchers used the various kitchen images to construct  an entire virtual cooking scene, one that also could be experienced from all 360 degrees. Research subjects could move around in the virtual scene and see its every detail from every possible angle. Donning goggles that blot out the real world, they would enter a “virtual reality” version of the traditional Ningbo kitchen.

Using the goggles,  subjects  experienced the virtual images of the kitchen scene and they could move around the space, approach objects to examine them up close, even have the sense of touching and holding the object - grabbing utensils, opening drawers, etc.  

On the left is a digitally rendered version of the traditional Ningbo cookstove photographed in a traditional Ningbo kitchen. The details look less “real” than a photograph - you can tell it is a rendering, but it shows great detail. A small wooden footstool sits on the floor to the stove’s right. The right-hand image is also digitally rendered, of a wooden, round-backed kitchen chair next to a tall wooden kitchen cabinet with many rows of drawers. The bottom two rows of  drawers have traditional Chinese lattice work designs.

The Ningbo study examined research subjects’ response to their experience. Everyone in the small group of testers was a local resident, born and raised, familiar with local customs and history.  

In the study, subjects compared this experience to viewing the same scene with traditional video, also presented in goggles.  Both options triggered “memory recollections,” and evoked a sense of nostalgia. Although the traditional  video felt slightly more “real” to the participants, they remained impassive observers, while in the virtual space, subjects interacted with the objects.  Perhaps because of this, the virtual, reconstructed objects seemed to evoke more of a sense of familiarity:

 “In most cases, the participants compared the objects from the virtual environment with the old objects they remembered from the past, with responses such as ‘the wooden bench and the cupboard were the same as the ones in my grandma’s house’ (p13); ‘my grandma also used this brand of matchsticks’ (p11).  [1]

Subjects lingered longer in the virtual scene and made evocative comments suggesting a heightened sense of immersion. According to the study: 

 “I felt I went back to my grandparents’ house, watching them cook together, just in the same way I did when I was a child” (p2); “I could hear that they were talking in the local dialect” (p8); “I could feel that a real life is going on there” [1]

“Memories of relationships sweet and bitter embedded within objects distributed within such spaces diminish,” said PhD student Shenghan Cai, one of the project’s researchers.  “In time, the space’s home to these memories are lost, together with the living beings that used to dwell in them. We use VR technologies to accurately capture and reconstruct local heritage of the near past in our attempts to answer questions on whether memory recollections can occur in virtual spaces. Our aim is to store and share memory spaces across time and space for our future generations.” [2]

You can get a better sense of this as Cai demos a related experiment herself here. 

Or imagine this: In Carlin Zia’s’s book you’ll find this black and white photo of a garden filled with roses in the courtyards of her great-grandparents’ home in Changzhou. When her grandfather looks at that photo, it may remind him of certain childhood memories - the color of the roses or their fragrant perfume, the sensation of his grandmother or grandfather in their home.  The researchers at Ningbo posit that the memories evoked by such a photo are amplified, are richer,  when the roses - or his grandparents - can be approached, moved around, even touched; when a subject can have the sense of being in the scene themself.

The Ningbo research center’s director, Professor Eugene Ch’ng, believes preserving cultural heritage through technology holds great promise.  “Can we preserve our life and memory as gifts for our children, so that they can relive our present? Can the elderly relive their childhood by being brought back to their past?” [3]

In these times of global sickness and seclusion especially, when we are living so much of our current lives online, in our work, study and personal interactions, these questions take on a new urgency. 

To hear about other Oral History practices melding with innovative technology, check out the description of this recent OHMA Workshop Series event on “Oral History as Told by AI,” featuring a conversation with Stephanie Dinkins, a transmedia artist who explores artificial intelligence (AI) as it intersects with race, gender, aging, and our future histories. Visit her website https://www.stephaniedinkins.com/ for more on her work. 

Lisa R. Cohen is excited to be part of the OHMA cohort as a part time Masters Candidate after 30 plus years as a full time network news producer, author, documentary filmmaker, adjunct professor and university administrator. Currently, Lisa is the Director of Prizes administering the duPont-Columbia Awards for national and local audio and video reporting at the Columbia University Graduate School of Journalism. She taught reporting, video production, and long form narrative video classes there for over a decade, while also directing/producing documentaries; about a maximum security prison hospice staffed by the inmates for the OWN Doc Club, and about the inequities of cancer care in this country for HBO. She also authored a book about the historic disappearance of Etan Patz in 1979 that ushered in a profound change in child rearing in America. Previous to that she produced long form stories and documentaries at ABC and CBS News for over 20 years.