Do You Hear What I Hear in This Video Game?

Last month, National Public Radio’s All Things Considered aired a piece titled “Why Do We Hate The Sound of Fingernails On a Chalkboard?” Musicologist Michael Oehler reported that this sound produces a frequency that reaches the most sensitive spot of the human ear and creates an amplified “open ear gain.” He further explained that some of our reaction is also emotional. When Oehler played the sound for a group of volunteers, those who thought they heard music “perceived the sound to be less unpleasant.” Oehler’s report caused me to think about how sound affects game play and to revisit some of ICHEG’s collections.

Some attribute the first use of a self-contained LPC speech synthesizer (one of the most powerful and accurate artificial productions of speech) to Texas Instruments in the late 1970s. Larry Brantingham, Paul S. Breedlove, Richard H. Wiggins, and Gene A. Frant designed the TMC0280/TMS5100 for Texas Instruments to use in “Speak and Spell.” ICHEG has numerous examples of this toy and TI’s continued use of more advanced renditions of the chip for other products. In the early 1980s Atari began to use the TMS5220 for arcade games such as Star Wars and Indiana Jones and the Temple of Doom. ICHEG acquired both of these, and I can’t help but to chuckle when I hear the mechanical-sounding voice advise, “use the force Luke.”

Today, video game makers can do far more with audio, but they also face new production challenges. In the film industry, actors receive a script that clarifies a character’s motive and often provides a linear plot. Video games involve players in the fate of most characters. In Mass Effect 3, for example, a gamer first selects to play as either FemShep or BroShep. As she plays, she encounters an array of summaries, “I’m honored” or “This is unexpected.” Whichever summary she chooses determines how FemShep speaks and often how gameplay proceeds. Jennifer Hale, the voiceover actor for FemShep, told New Yorker reporter Tom Bissell that “it all has to be in my head. Environment, ambient noise, history with this person, what I need from this person—all these decisions have to be made on the fly.” She makes wise choices for the thousands of lines she reads. Kirk Hamilton, games editor at Paste Magazine, raves about FemShep, “Hale’s performance improves the experience….each pause and inflection accumulates over time until you can’t help but care for the character.” A successful voice actor evolves a character by creating a voice and verbal cues that emotionally impact the player.

Aside from dialogue, think about all of the sounds you encounter—your breath, ice clinking against a glass, nails on a chalkboard. Audio specialists need to capture noises to make game play realistic. In 1982, Williams’ sit-down arcade cabinet Sinistar created the illusion of directionality and audible perspective through the first use of stereo sound in a game.  Some might consider Sinistar’s audio melodramatic, but I still get a shiver when I head up to ICHEG’s lab to play it and I hear a voice boom out from the arcade, “Run coward…I hunger.” Like voiceover, sound effects have evolved. At the Audio Engineering Society conference last month, Chris Jahnkow, senior sound designer at Sony Computer Entertainment America, and Scott Selfon, senior audio specialist at Microsoft, discussed various aspects of video game audio. In film, audio specialists create sound for a sedentary audience, but current video games need to accommodate a player using motion for game control. A player that carries a sword in his right hand and slashes at the enemy sounds different than a player that bears a sword in his left hand and jabs at his victim. Some players rely solely on these sounds to defeat the game. College student Terry Garrett lost his eyesight at age 10. That same year his brother bought Odd World: Abe’s Oddysee. Garrett listened as his brother played the game. He explained to Wired Magazine’s Jason Schreier that he’s honed his hearing and memorized the footsteps, voices, and music of games. By listening to these sounds, Garrett played and beat Odd World.Like the sound of voices, environmental cues also impact the player emotionally. When Gamasutra’s Jeriaska Jeriaska asked sound designer Robin Arnott about his priorities for the upcoming first-person puzzle game Antichamber, he said bringing “an emotional context to the mental experience of the game.” Arnott said he had researched “how certain ambient soundscapes make people feel” and focused “on beautiful, naturalistic sound.” He and his team do not want a player to feel hurried. If she stops to think about the next move, Arnott said, she’ll hear rain splashing on leaves. The team has not announced a release date for the game, but mixing play with meditation sounds pleasant to me.

I have an auditory processing disorder and therefore my strengths do not include keeping rhythm, distinguishing between certain sounds, or winning an old-fashioned game of telephone. However, I appreciate the painstaking efforts that go into producing video game audio. I know my game play experience would include less of an emotional investment without the twang of a bug’s wings in Limbo or the alarm in Silent Hill.