THE PAST, PRESENT, AND PROMISE OF SONIFICATION

1.1. Past: The Rationale for the Use of Sonification

The rationale and motivation for displaying information using sound (rather than a visual presentation, etc.) have been discussed extensively in the literature for a long time (e.g., Bly et al., 1985Bly, Sara A.; Frysinger, Steven P.; Lunney, David; Mansur, Douglass L.; Mezrich, Joseph J. y Morrison, Robert C. (1985). Communicating with sound. In L. Borman & R. Smith (eds.) Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, San Francisco, California (CHI 85) (pp. 115-119). Association for Computing Machinery. https://doi.org/10.1145/317456.317477 ; Jeon, Walker, & Barrass, 2018Jeon, Myounghoon; Walker, Bruce N. y Barrass, Stephen (2018). Introduction to the Special Issue on Sonic Information Design: Theory, Methods, and Practice, Part 1. Ergonomics in Design, 26 (4), 3. https://doi.org/10.1177/1064804618797304 , 2019Jeon, Myounghoon; Walker, Bruce N. y Barrass, Stephen (2019). Introduction to the Special Issue on Sonic Information Design: Theory, Methods, and Practice, Part 2. Ergonomics in Design, 27 (1), 4. https://doi.org/10.1177/1064804618814232 ; Kramer, 1994Kramer, Gregory (1994). An introduction to auditory display. In G. Kramer (Ed.), Auditory Display: Sonification, Audification, and Auditory Interfaces (pp. 1-78). Reading, MA: Addison Wesley.; Nees & Walker, 2009Nees, Michael A. y Walker, Bruce N. (2009). Auditory Interfaces and Sonification. In Constantine Stephanidis (Ed.), The Universal Access Handbook (pp. 507-521). New York: CRC Press Taylor & Francis.; Peres et al., 2008Peres, S. Camille; Best, Virginia; Brock, Derek; Shinn-Cunningham, Barbara; Frauenberger, Christopher; Hermann, Thomas, et al. (2008). Auditory Interfaces. In P. Kortum (Ed.), HCI Beyond the GUI: Design for Haptic, Speech, Olfactory and Other Nontraditional Interfaces (pp. 147-196). Burlington, MA: Morgan Kaufmann.; Sanderson, 2006Sanderson, Penelope M. (2006). The multimodal world of medical monitoring displays. Applied Ergonomics, 37, 501-512.; Supper, 2014Supper, Alexandra (2014). Sublime frequencies: The construction of sublime listening experiences in the sonification of scientific data. Social Studies of Science, 44 (1), 34-58., 2015Supper, Alexandra (2015). Sound information: Sonification in the age of complex data and digital audio. Information & Culture, 50 (4), 441-464.; Walker & Nees, 2011Walker, Bruce N. y Nees, Michael A. (2011). Theory of Sonification. In T. Hermann, A. Hunt, & J. Neuhoff (Eds.). The Sonification Handbook (pp. 9-39). Berlin, Germany: Logos Publishing House. http://sonification.de/handbook/download/TheSonificationHandbook-chapter2.pdf ). Briefly, though, it has long been known that auditory displays exploit the superior ability of the human auditory system to recognize temporal changes and patterns (Bregman, 1990Bregman, Albert S. (1990). Auditory scene analysis: The perceptual organization of sound. Cambridge, MA: MIT Press.; Flowers, Buhman, & Turnage, 1997Flowers, John H.; Buhman, Dion C. y Turnage, Kimberly D. (1997). Cross-modal equivalence of visual and auditory scatterplots for exploring bivariate data samples. Human Factors, 39 (3), 341-351.; Flowers & Hauer, 1995Flowers, John H. y Hauer, Terry A. (1995). Musical versus visual graphs: Cross-modal equivalence in perception of time series data. Human Factors, 37 (3), 553-569.; Moore, 2013Moore, Brian C. J. (2013). An Introduction to the Psychology of Hearing (6th ed.). Leiden, NL: Brill.). In many instances, response times for auditory stimuli are faster than those for visual stimuli (Spence & Driver, 1997Spence, Charles y Driver, Jon (1997). Audiovisual links in attention: Implications for interface design. In D. Harris (Ed.), Engineering Psychology and Cognitive Ergonomics Vol. 2: Job Design and Product Design (pp. 185-192). Hampshire: Ashgate Publishing.). As a result, auditory displays may be the most appropriate modality when the information being displayed has complex patterns, changes in time, includes warnings, or calls for immediate action.

Additionally, it has long been known that in practical work environments the operator is often unable to look at, or unable to see, a visual display. The visual system might be busy with another task (Fitch & Kramer, 1994Fitch, W. Tecumseh y Kramer, Gregory (1994). Sonifying the body electric: Superiority of an auditory over a visual display in a complex, multivariate system. In G. Kramer (Ed.), Auditory display: Sonification, audification, and auditory interfaces (pp. 307-326). Reading, MA: Addison-Wesley.; Wickens & Liu, 1988Wickens, Christopher D. y Liu, Yili (1988). Codes and modalities in multiple resources: A success and a qualification. Human Factors, 30 (5), 599-616.), or the perceiver might be visually impaired, either physically or as a result of environmental factors such as smoke in a burning building or line-of-sight obstructions (Fitch & Kramer, 1994Fitch, W. Tecumseh y Kramer, Gregory (1994). Sonifying the body electric: Superiority of an auditory over a visual display in a complex, multivariate system. In G. Kramer (Ed.), Auditory display: Sonification, audification, and auditory interfaces (pp. 307-326). Reading, MA: Addison-Wesley.; Wickens, Gordon, & Liu, 1998Wickens, Christopher D.; Gordon, Sallie E. y Liu, Yili (1998). An introduction to human factors engineering. New York: Longman.); or the visual system may be overtaxed with information (see Brewster, 1997Brewster, Stephen (1997). Using non-speech sound to overcome information overload. Displays, 17, 179-189.; Brown, Newsome, & Glinert, 1989Brown, M. L., Newsome, Sandra L., & Glinert, Ephraim P. (1989). An experiment into the use of auditory cues to reduce visual workload. In K. Vice & C. Lewis (eds.) Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI 89) (pp. 339-346). Association for Computing Machinery. https://doi.org/10.1145/67449.67515 ).

In some cases, auditory and voice modalities have been shown to be most compatible when systems require the processing or input of verbal-categorical information (Wickens & Liu, 1988Wickens, Christopher D. y Liu, Yili (1988). Codes and modalities in multiple resources: A success and a qualification. Human Factors, 30 (5), 599-616.). Other features of auditory perception that suggest sound as an effective data representation technique include our ability to monitor and process multiple auditory data sets (parallel listening) (Fitch & Kramer, 1994Fitch, W. Tecumseh y Kramer, Gregory (1994). Sonifying the body electric: Superiority of an auditory over a visual display in a complex, multivariate system. In G. Kramer (Ed.), Auditory display: Sonification, audification, and auditory interfaces (pp. 307-326). Reading, MA: Addison-Wesley.).

Finally, advances in technology for the past several decades have simultaneously expanded visual information displays toward opposite extremes in physical size. Portable devices (e.g., the latest «smart» wristwatches) continue the trend toward smaller physical dimensions, thereby leaving appreciably less space (or perhaps even no space) for a visual display (see an early recognition of this, Brewster, 2002Brewster, Stephen (2002). Overcoming the lack of screen space on mobile computers. Personal and Ubiquitous Computing, 6 (3), 188-205.). Fixed work stations, on the other hand, have become characterized by multiple visual displays with increasingly large physical sizes, due in part to increases not only in the affordability of displays but also in the expanded computing power to support multiple concurrent displays. This extends to modern immersive virtual reality (VR) contexts with massive pixel counts. As a result, visually intensive workstations and other multitasking situations may overburden the visual modality (see Grudin, 2001Grudin, Jonathan (2001). Partitioning digital worlds: Focal and peripheral awareness in multiple monitor use. In J. Jacko & A. Sears (eds.) Proceedings of the 2001 SIGCHI conference on Human Factors in Computing Systems (CHI 01) (pp. 458-465). Seattle, WA. Association for Computing Machinery. https://doi.org/10.1145/365024.365312 , for another early recognition of this problem). Thus, the inclusion of nonspeech audio in interfaces can promote universal design principles such as flexibility in use and perceptible information (see Connell et al., 1997Connell, Bettye Rose; Jones, Mike; Mace, Ron; Mueller, Jim; Mullick, Abir; Ostroff, Elaine (1997). The Principles of Universal Design, Version 2.0. NC State University, Raleigh, NC: The Center for Universal Design.; McGuire, Scott, & Shaw, 2006McGuire, Joan M.; Scott, Sally S. y Shaw, Stan F. (2006). Universal design and its applications in educational environments. Remedial and Special Education, 3, 166-175.).

1.2. A Very Brief History of (the Field of) Sonification

Although investigations of audio as an intentional¹Intentional sounds are designed as an information display (see Walker & Kramer, 1996), as distinct from incidental sounds, which result organically from the normal operation of a system (e.g., a car engine running). Both can be informative information display modality date back over 75 years (see Frysinger, 2005Frysinger, Steven P. (2005, July 6-9). A brief history of auditory data representation to the 1980s. In Proceedings of the 11^thInternational Conference on Auditory Display (ICAD 2005) (pp. 410-413). Limerick, Ireland. http://hdl.handle.net/1853/50089 ; and see Worrall, 2018Worrall, David (2018). Sonification: A prehistory. In Proceedings of the 24^thInternational Conference on Auditory Display (ICAD 2018) (pp. 177-182). Michigan, USA. http://doi.org/10.21785/icad2018.019 for a «pre-history»), it was the advent of digital computing technology that really enabled sonification to gain the potential for ubiquity. Near the beginning of what might be considered the sonification era (about 1994 onwards), Edworthy (1998)Edworthy, Judy (1998). Does sound help us to work better with machines? A commentary on Rautenberg’s paper ‘About the importance of auditory alarms during the operation of a plant simulator’. Interacting with Computers, 10, 401-409. even argued that the advent of auditory displays and audio interfaces was practically inevitable given the ease and cost efficiency with which electronic devices can now produce sound. A quarter century later, we may finally be approaching that level; however, much remains to be done to truly unlock the power and potential of sonification and other auditory display technologies.

The formation of the International Community for Auditory Display (ICAD²ICAD Website: https://www.icad.org ), and the first of its now-annual conference the International Conference on Auditory Display (sharing the ICAD acronym) in 1992, was a seminal point in the rise of sonification as a systematic scientific tool and a flexible expressive medium (see Kramer, 1994Kramer, Gregory (1994). An introduction to auditory display. In G. Kramer (Ed.), Auditory Display: Sonification, Audification, and Auditory Interfaces (pp. 1-78). Reading, MA: Addison Wesley.). Members of the nascent ICAD community (many of whom are still active in the field today, more than thirty years later) produced the collaborative Sonification Report (Kramer et al., 1999Kramer, Gregory; Walker, Bruce N.; Bonebright, Terri; Cook, Perry; Flowers, John; Miner, Nadine (1999). The Sonification Report: Status of the Field and Research Agenda. Report prepared for the National Science Foundation by members of the International Community for Auditory Display. Santa Fe, NM: International Community for Auditory Display (ICAD). https://digitalcommons.unl.edu/psychfacpub/444 ) as a starting point for a more structured discussion of the theory of sonification by identifying four issues that should be addressed in a theoretical description of sonification. These included: (1) taxonomic descriptions of sonification techniques based on psychological principles or display applications; (2) descriptions of the types of data and user tasks amenable to sonification; (3) a treatment of the mapping of data to acoustic signals; and (4) a discussion of the factors limiting the use of sonification.

Since then, research into the where, when, and how of sonification has blossomed, encompassing researchers from such diverse fields as audio engineering, audiology, computer science, informatics, linguistics, mathematics, music, psychology, and telecommunications, to name but a few. Sonification began to be implemented in fields as disparate as STEM education (Bonebright et al., 2001Bonebright, Terri L.; Nees, Mike A.; Connerley, Tayla T. y McCain, Glenn R. (2001, July 29-August 1). Testing the effectiveness of sonified graphs for education: A programmatic research project. In Proceedings of the 7^thInternational Conference on Auditory Display (ICAD 2001) (pp. 62-66). Espoo, Finland. http://hdl.handle.net/1853/50654 ), astrophysics (Candey, Schertenleib, & Diaz Merced, 2006Candey, Robert M.; Schertenleib, Anton M. y Diaz Merced, W. L. (2006, June 20-23). Xsonify sonification tool for space physics. Proceedings of the 12th International Conference on Auditory Display (ICAD 2006) (pp. 289-290) London, UK. http://hdl.handle.net/1853/50697 ), and rowing (Schaffert et al., 2009Schaffert, Nina; Mattes, Klaus; Barrass, Stephen y Effenberg, Alfred O. (2009, December 3-4). Exploring function and aesthetics in sonifications for elite sports. Proceedings of the Second International Conference on Music Communication Science, Sydney, Australia. https://www.academia.edu/308233/Exploring_Function_and_Aesthetics_In_Sonifications_for_Elite_Sports ). An array of conferences, venues, and publications have further showcased sonification in science, education, and entertainment. Sonification tools have appeared regularly (see later discussion) and design guidelines (see later discussion) have continually evolved, to support all kinds of users and sonification use cases.

Progressively, an understanding of the theory underlying sonification has evolved. This started with an amalgam of important insights and generalizations drawn from the convergence of these many diverse fields (e.g., Barrass, 1997Barrass, Stephen (1997). Auditory information design. [Unpublished doctoral dissertation]. Australian National University.; Brazil, 2010Brazil, Eoin (2010). A review of methods and frameworks for sonic interaction design: Exploring existing approaches. Lecture Notes in Computer Science, 5954, 41-67. https://doi.org/10.1007/978-3-642-12439-6_3 ; de Campo, 2007de Campo, Alberto (2007, June 26-29). Toward a data sonification design space map. In Proceedings of the 13^thInternational Conference on Auditory Display (ICAD 2007) (pp. 342-347). Montreal, Canada. http://hdl.handle.net/1853/50042 ; Frauenberger & Stockman, 2009Frauenberger, Christopher y Stockman, Tony (2009). Auditory display design: An investigation of a design pattern approach. International Journal of Human-Computer Studies, 67, 907-922.; Hermann, 2008Hermann, Thomas (2008, June 24 - 27). Taxonomy and definitions for sonification and auditory display. In Proceedings of the 14th International Conference on Auditory Display (ICAD 2008) (pp. 1-8). Paris, France. http://hdl.handle.net/1853/49960 ; Nees & Walker, 2007Nees, Michael A. y Walker, Bruce N. (2007, June 26-29). Listener, task, and auditory graph: Toward a conceptual model of auditory graph comprehension. In Proceedings of the 13th International Conference on Auditory Display (ICAD 2007) (pp. 266-273). Montreal, Canada. http://hdl.handle.net/1853/50010 ; Neuhoff & Heller, 2005Neuhoff, John G. y Heller, Laurie M. (2005, July 6-9). One small step: Sound sources and events as the basis for auditory graphs. Proceedings of the 11^thInternational Conference on Auditory Display (ICAD 2005) (pp. 417-419). Limerick, Ireland. http://hdl.handle.net/1853/50174 ; Walker, 2002Walker, Bruce N. (2002). Magnitude estimation of conceptual data dimensions for use in sonification. Journal of Experimental Psychology: Applied, 8, 211-221. https://doi.org/10.1037/1076-898X.8.4.211 , 2007Walker, Bruce N. (2007). Consistency of magnitude estimations with conceptual data dimensions used for sonification. Applied Cognitive Psychology, 21, 579-599.), followed by more systematic statements of sonification principles) (see, e.g., Brazil & Fernstrom, 2009Brazil, Eoin y Fernström, Mikael (2009). Empirically based auditory display design. In F. Gouyon, A. Barbosa, & X. Serra (eds.). Proceedings of the 6th Sound and Computing Conference (SMC 2009) (pp. 7-12). Porto, Portugal. INESC.; de Campo, 2007de Campo, Alberto (2007, June 26-29). Toward a data sonification design space map. In Proceedings of the 13^thInternational Conference on Auditory Display (ICAD 2007) (pp. 342-347). Montreal, Canada. http://hdl.handle.net/1853/50042 ; Frauenberger, Stockman, & Bourguet, 2007Frauenberger, Chris; Stockman, Tony y Bourguet, Marie-Luce (2007, June 26-29). Pattern design in the context space: A methodological framework for auditory display design. In Proceedings of the 13th International Conference on Auditory Display (ICAD 2007) (pp. 513-518). Montreal, Canada. http://hdl.handle.net/1853/50020 ; Nees & Walker, 2007Nees, Michael A. y Walker, Bruce N. (2007, June 26-29). Listener, task, and auditory graph: Toward a conceptual model of auditory graph comprehension. In Proceedings of the 13th International Conference on Auditory Display (ICAD 2007) (pp. 266-273). Montreal, Canada. http://hdl.handle.net/1853/50010 ). The field marked milestones with encyclopedia entries (e.g., Walker & Kramer, 2006Walker, Bruce N. y Kramer, Gregory (2006). Sonification. In W. Karwowski (Ed.), International Encyclopedia of Ergonomics and Human Factors (2nd ed.) (pp. 1254-1256). New York: CRC Press. https://www.taylorfrancis.com/chapters/edit/10.1201/9780849375477-272/sonification ; Nees & Walker, 2009Nees, Michael A. y Walker, Bruce N. (2009). Auditory Interfaces and Sonification. In Constantine Stephanidis (Ed.), The Universal Access Handbook (pp. 507-521). New York: CRC Press Taylor & Francis.), and with the publication of The Sonification Handbook (Hermann, Hunt, & Neuhoff, 2011Hermann, Thomas; Hunt, Andy y Neuhoff, John (2011). The Sonification Handbook. Berlin: Logos Publishing House. ). Interest in, and need for, periodic summaries of the theory, methods, and practices for sonification has continued to the present (see, e.g., Jeon, Walker, & Barrass, 2018Jeon, Myounghoon; Walker, Bruce N. y Barrass, Stephen (2018). Introduction to the Special Issue on Sonic Information Design: Theory, Methods, and Practice, Part 1. Ergonomics in Design, 26 (4), 3. https://doi.org/10.1177/1064804618797304 , 2019Jeon, Myounghoon; Walker, Bruce N. y Barrass, Stephen (2019). Introduction to the Special Issue on Sonic Information Design: Theory, Methods, and Practice, Part 2. Ergonomics in Design, 27 (1), 4. https://doi.org/10.1177/1064804618814232 ; Walker, 2021Walker, Bruce N. (2021). Sonification. In Charles LaPierre (Ed.), DIAGRAM Reports 2017-2020 (pp. 554-582). Palo Alto, CA: Benetech. https://www.bookshare.org/browse/book/3800695/DIAGRAM%20Reports%202017%20-%202020 ; and Walker & Nees, 2011Walker, Bruce N. y Nees, Michael A. (2011). Theory of Sonification. In T. Hermann, A. Hunt, & J. Neuhoff (Eds.). The Sonification Handbook (pp. 9-39). Berlin, Germany: Logos Publishing House. http://sonification.de/handbook/download/TheSonificationHandbook-chapter2.pdf , as just some examples). Nevertheless, there remains concern about whether there is such a thing as a theory of sonification. Nees (2019)Nees, Michael A. (2019, June 23-27). Eight components of a design theory of sonification. Proceedings of the 25^thInternational Conference on Auditory Display (ICAD 2019) (pp. 176-183). Newcastle, UK. https://doi.org/10.21785/icad2019.048 recently summarizes the weaknesses of the field of sonification, as it relates to «theory», and discusses, instead, the potential for a «design theory» of sonification. Regardless of the state of theorizing, it is still possible, and useful, to describe the state of play in sonification, as a point of departure for discussing where the field may be (or may need to be) heading. This is congruent with the recent comprehensive overview by Andreopoulou and Goudarzi (2021)Andreopoulou, Areti y Goudarzi, Visda (2021, June 25-28). Sonification First: The Role of ICAD in the Advancement of Sonification-Related Research. In Proceedings of the 26^thInternational Conference on Auditory Display (ICAD 2021) (pp. 65-73). Virtual conference. https://doi.org/10.21785/icad2021.031 of 30 years of ICAD conference papers, showing a growth in the focus on «sonification», the popularity of the concept of «design», and an increase in interest for / use of more rigorous evaluation methods in relation to sonification.

2.1. Functions of Auditory Displays

Alerts and notifications are sounds that indicate that something has occurred, or is about to (see seminal work by Buxton, 1989Buxton, William (1989). Introduction to this special issue on nonspeech audio. Human-computer Interaction, 4, 1-9.; Sanders & McCormick, 1993Sanders, Mark S. y McCormick, Ernest J. (1993). Human Factors in Engineering and Design (7th ed.). New York: McGraw-Hill.; Sorkin, 1987Sorkin, Robert D. (1987). Design of auditory and tactile displays. In G. Salvendy (Ed.), Handbook of human factors (pp. 549-576). New York: Wiley & Sons.). Alerts and notifications tend to be simple and particularly overt. The message conveyed is information-poor. For example, a beep is often used to indicate that the cooking time on a microwave oven has expired. There is generally little information as to the details of the event-the microwave beep merely indicates that the time has expired, not necessarily that the food is fully cooked.

Alarms and warnings are alert or notification sounds that convey the occurrence of a constrained class of events, usually adverse, that carry particular urgency in that they require immediate response or attention (Haas & Edworthy, 2006Haas, Ellen y Edworthy, Judy (2006). An introduction to auditory warnings and alarms. In M. S. Wogalter (Ed.), Handbook of Warnings (pp. 189-198). Mahwah, NJ: Lawrence Erlbaum.). However, the specificity of the information about the event is generally limited. Fire alarms identify an adverse event (a fire) that requires immediate action (evacuation), but the alarm does not indicate the location of the fire. More complex (and modern) kinds of alarms encode more information into the auditory signal, such as medical information (e.g., Anderson & Sanderson, 2004Anderson, Janet y Sanderson, Penelope (2004). Designing Sonification for Effective Attentional Control in Complex Work Domains. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 48 (16), 1818-1822. ; Sanderson, Liu, & Jenkins, 2009Sanderson, Penelope M.; Liu, David y Jenkins, Simon A. (2009). Auditory displays in anesthesiology. Current Opinion in Anesthesiology, 22, 788-795.; and see the development of the international standard for hospital alarms, Edworthy et al., 2018Edworthy, Judy Reed; McNeer, Richard R.; Bennett, Christopher L.; Dudaryk, Roman; McDougall, Siné J. P.; Schlesinger, Joseph J.; Bolton, Matthew L.; Edworthy, Jonathan D. Reed; Özcan, Elif; Boyd, Andrew D.; Reid, Scott K. J.; Rayo, Michael F.; Wright, Melanie C. y Osborn, David (2018). Getting Better Hospital Alarm Sounds Into a Global Standard. Ergonomics in Design, 26 (4), 4-13.).

Object, item, and status indicators . Sounds such as earcons (e.g., Blattner et al., 1989Blattner, Meera M.; Sumikawa, Denise A. y Greenberg, Robert M. (1989). Earcons and icons: Their structure and common design principles. Human-Computer Interaction, 4, 11-44. ; Bonebright & Nees, 2007Bonebright, Terry L. y Nees, Michael A. (2007, June 26-29). Memory for auditory icons and earcons with localization cues. In Proceedings of the 13^thInternational Conference on Auditory Display (ICAD 2007) (pp. 419-422). Montreal, Canada. http://hdl.handle.net/1853/50014 ; Brewster, Wright, & Edwards, 1993Brewster, Stephen; Wright, Peter C. y Edwards, Alistair D. N. (1993, April 24-29). An evaluation of earcons for use in auditory human-computer interfaces. In B. Arnold, G. van der Veer, & T. White (eds.) Proceedings of the INTERACT’93 and CHI’93 Conference on Human Factors in Computing Systems (CHI 93) (pp. 222-227). Amsterdam. Association for Computing Machinery. https://doi.org/10.1145/169059.169179 ; McGookin & Brewster, 2004McGookin, David K. y Brewster, Stephen A. (2004). Understanding concurrent earcons: Applying auditory scene analysis principles to concurrent earcon recognition. ACM Transactions on Applied Perception, 1, 130-150.), auditory icons (e.g., Bonebright & Nees, 2007Bonebright, Terry L. y Nees, Michael A. (2007, June 26-29). Memory for auditory icons and earcons with localization cues. In Proceedings of the 13^thInternational Conference on Auditory Display (ICAD 2007) (pp. 419-422). Montreal, Canada. http://hdl.handle.net/1853/50014 ; Gaver, 1989Gaver, William W. (1989). The SonicFinder: An interface that uses auditory icons. Human-Computer Interaction, 4 (1), 67-94.; Keller & Stevens, 2004Keller, Peter y Stevens, Catherine (2004). Meaning from environmental sounds: Types of signal-referent relations and their effect on recognizing auditory icons. Journal of Experimental Psychology: Applied, 10 (1), 3-12.), and spearcons (Palladino & Walker, 2007Walker, Bruce N.; Kim, Jonathan y Pendse, Anandi (2007). Musical soundscapes for an accessible aquarium: Bringing dynamic exhibits to the visually impaired. In Proceedings of the International Computer Music Conference (ICMC 2007) (pp. 268-275). Copenhagen, Denmark. http://hdl.handle.net/2027/spo.bbp2372.2007.164 ; Walker, Nance, & Lindsay, 2006Walker, Bruce N.; Nance, Amanda y Lindsay, Jeffrey (2006, June 20-23). Spearcons: Speech-based earcons improve navigation performance in auditory menus. In Proceedings of the 12^thInternational Conference on Auditory Display (ICAD 2006) (pp. 63-68). London, UK. http://hdl.handle.net/1853/50642 ) provide information about the nature of the underlying action or event. These sounds are often used to facilitate user interface tasks (e.g., Brewster, Wright, & Edward, 1994Brewster, Stephen; Wright, Peter C. y Edward, Alistair D. N. (1994, April 24-28). The design and evaluation of an auditory enhanced scrollbar. In B. Adelson, S. Dumais, & J. Olson (eds.) Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI 94) (pp. 173-179). Boston, MA, Association for Computing Machinery. https://doi.org/10.1145/191666.191733 ; Winberg & Hellstrom, 2003Winberg, Fredrik y Hellstrom, Sten Olof (2003, November 10-11). Designing accessible auditory drag and drop. Proceedings of the 2003 Conference on Universal Usability (CUU 2003). Vancouver, Canada. https://cid.nada.kth.se/pdf/CID-242.pdf ). Earcons are abstract, artificial sounds that bear no ecological relationship to the represented process or event (e.g., beeps, chimes, abstract sound motives, etc., see Blattner et al., 1989Blattner, Meera M.; Sumikawa, Denise A. y Greenberg, Robert M. (1989). Earcons and icons: Their structure and common design principles. Human-Computer Interaction, 4, 11-44. ). They can, however, be designed with a hierarchical structure or grammar, thereby enhancing their communicative power (e.g., McGookin & Brewster, 2011McGookin, David K. y Brewster, Stephen A. (2011). Earcons. In T. Hermann, A. Hunt, & J. Neuhoff, The Sonification Handbook. Berlin: Logos Publishing. https://sonification.de/handbook/ ). Auditory icons are more natural sounds that have some real world relationship with their referent process or event. One simple example is the sound of a camera shutter being used in a (shutter-less) digital camera to indicate when a picture has been taken (see, e.g., Gaver, 1989Gaver, William W. (1989). The SonicFinder: An interface that uses auditory icons. Human-Computer Interaction, 4 (1), 67-94.). As an alternative to earcons and auditory icons, spearcons (and their «cousins», nearcons⁴Spearcons are created by speeding up an audio recording of a spoken word or phrase using a simultaneous overlap and add (SOLA) algorithm that preserves pitch contours, consonant/vowel ratios, etc. In contrast, nearcons are created by speeding up the speech rate of a text-to-speech engine, which leads to fast-talk that often truncates vowels more than consonants.) use temporally compressed speech to represent objects, items, or processes with sound (Palladino & Walker, 2007Palladino, Dianne y Walker, Bruce N. (2007, June 26-29). Learning rates for auditory menus enhanced with spearcons versus earcons. In Proceedings of the 13th International Conference on Auditory Display (ICAD 2007) (pp. 274-279). Montreal, Canada. http://hdl.handle.net/1853/50011 ; Walker et al., 2006Walker, Bruce N.; Godfrey, Mark T.; Orlosky, Jason E.; Bruce, Carrie y Sanford, Jon (2006, June 20-23). Aquarium sonification: Soundscapes for accessible dynamic informal learning environments. Proceedings of the 12^thInternational Conference on Auditory Display (ICAD 2006) (pp. 238-241). London, UK. http://hdl.handle.net/1853/50426 ). Spearcons have been shown to outperform both earcons and auditory icons (Walker et al., 2006Walker, Bruce N.; Godfrey, Mark T.; Orlosky, Jason E.; Bruce, Carrie y Sanford, Jon (2006, June 20-23). Aquarium sonification: Soundscapes for accessible dynamic informal learning environments. Proceedings of the 12^thInternational Conference on Auditory Display (ICAD 2006) (pp. 238-241). London, UK. http://hdl.handle.net/1853/50426 ) and may be especially useful in the design of flexible auditory menus (see Palladino & Walker, 2007Palladino, Dianne y Walker, Bruce N. (2007, June 26-29). Learning rates for auditory menus enhanced with spearcons versus earcons. In Proceedings of the 13th International Conference on Auditory Display (ICAD 2007) (pp. 274-279). Montreal, Canada. http://hdl.handle.net/1853/50011 ) or for representing a large number of items.

Auditory menus are speech-based hierarchical lists (aka, menus) that present a set of options. Such menus can be simple (i.e., just presenting text-to-speech versions of the menu and sub-menu items), or may be more sophisticated constructions involving multiple voices, louder and softer speech, whispers, and additional elements representing scrollbars and spoken indexes. Menus may be navigated actively by the user («pull» menus) or more passively as the options are presented serially to the user («push» menus). For an overview of auditory menus see Yalla and Walker (2007)Yalla, Pavani y Walker, Bruce N. (2007). Advanced Auditory Menus. Georgia Institute of Technology GVU Center Technical Report # GIT-GVU-07-12. October. http://hdl.handle.net/1853/26170 , and Jeon et al. (2015)Jeon, Myounghoon; Gable, Thomas M.; Davison, Benjamin K.; Nees, Michael A.; Wilson, Jeff y Walker, Bruce N. (2015). Menu navigation with in-vehicle technologies: Auditory menu cues improve dual task performance, preference, and workload. International Journal of Human Computer Interaction, 31, 1-16. http://dx.doi.org/10.1080/10447318.2014.925774 .

Status and progress indicators convey the state of an ongoing process, such as downloading a file. In these instances, sound takes advantage of «the listener’s ability to detect small changes in auditory events or the user’s need to have their eyes free for other tasks» (Kramer et al., 1999, p. 3Kramer, Gregory; Walker, Bruce N.; Bonebright, Terri; Cook, Perry; Flowers, John; Miner, Nadine (1999). The Sonification Report: Status of the Field and Research Agenda. Report prepared for the National Science Foundation by members of the International Community for Auditory Display. Santa Fe, NM: International Community for Auditory Display (ICAD). https://digitalcommons.unl.edu/psychfacpub/444 ). Soundscapes have been designed to mimic natural sounds (e.g., a thunderstorm with rain), and parameters of the soundscape are mapped to variables in a multidimensional data set (e.g., Mauney & Walker, 2004Mauney, Bradley S. y Walker, Bruce N. (2004, July 6-9). Creating functional and livable soundscapes for peripheral monitoring of dynamic data. Proceedings of the 10th International Conference on Auditory Display (ICAD 2004). Sydney, Australia. http://hdl.handle.net/1853/50847 ). While the listener may not necessarily act upon every change in the soundscape, the display allows for on-going monitoring and awareness of a changing situation.

Art, entertainment, sports, and leisure-based auditory displays have long been provided for simple, traditional games like the Towers of Hanoi (Winberg & Hellstrom, 2001Winberg, Fredrik y Hellstrom, Sten Olof (2001, July 29-August 1). Qualitative aspects of auditory direct manipulation: A case study of the Towers of Hanoi. Proceedings of the 7^thInternational Conference on Auditory Display (ICAD 2001) (pp. 16-20). Espoo, Finland. http://hdl.handle.net/1853/50633 ) and Tic-Tac-Toe (Targett & Fernstrom, 2003Targett, Sue y Fernstrom, Mikael (2003, July 6-9). Audio games: Fun for all? All for fun? In Proceedings of the 2003 International Conference on Auditory Display (ICAD 2003) (pp. 216-219). Boston, MA. http://hdl.handle.net/1853/50444 ), and more complex game genres such as arcade games (e.g., space invaders, see McCrindle & Symons, 2000McCrindle, Rachel J. y Symons, David (2000). Audio space invaders. In P. Sharkey, A. Cesarani, L. Pugnetti & A. Rizzo (Eds.), Proceedings of the 3rd International Conference on Disability, Virtual Reality, & Associated Technologies (pp. 59-65). Alghero, Italy. The University of Reading. https://www.icdvrat.com/2000/P/2000_09.pdf ) and role-playing games (Liljedahl, Papworth, & Lindberg, 2007Liljedahl, Mats; Papworth, Nigel y Lindberg, Stefan (2007, June 26-29). Beowulf: A game experience built on sound effects. Proceedings of the 13th International Conference on Auditory Display (ICAD 2007) (pp. 102-106). Montreal, Canada. http://hdl.handle.net/1853/49983 ). There are many audio-only games, too, of course (see, e.g., the Survive the Wild audio game⁵ http://www.samtupy.com/games/stw/ ; and the Audio Games website⁶ https://www.audiogames.net currently lists over 850 titles). Auditory displays also have been used to facilitate sports «watching» (e.g., Savery et al., 2019Savery, Richard; Ayyagari, Madhukesh; May, Keenan R. y Walker, Bruce N. (2019, June 23-27). Soccer Sonification: Enhancing Viewer Experience. In Proceedings of the 25^thInternational Conference on Auditory Display (ICAD 2019) (pp. 207-213). Newcastle-upon-Tyne, UK. https://doi.org/10.21785/icad2019.037 ) and sports playing (e.g., soccer, Stockman et al., 2007Stockman, Tony; Rajgor, Neil; Metatla, Oussama y Harrar, Lila (2007, June 26-29). The design of interactive audio soccer. In Proceedings of the 13th International Conference on Auditory Display (ICAD 2007) (pp. 526-529). Montreal, Canada. http://hdl.handle.net/1853/50045 ; rowing, Schaffert et al., 2009Schaffert, Nina; Mattes, Klaus; Barrass, Stephen y Effenberg, Alfred O. (2009, December 3-4). Exploring function and aesthetics in sonifications for elite sports. Proceedings of the Second International Conference on Music Communication Science, Sydney, Australia. https://www.academia.edu/308233/Exploring_Function_and_Aesthetics_In_Sonifications_for_Elite_Sports ; speed skating, Godbout & Boyd, 2010Godbout, Andrew y Boyd, Jeffrey E. (2010, June 9-15). Corrective sonic feedback for speed skating: A case study. In Proceedings of the 16th International Conference on Auditory Display (ICAD 2010) (pp. 23-30). Washington, DC. http://hdl.handle.net/1853/49865 ). Auditory displays have also been used as a means to bring some of the experience and excitement of dynamic exhibits to the visually impaired (e.g., sonified soundscapes to convey dynamic movement of fish in an «accessible aquarium», Walker et al., 2006Walker, Bruce N.; Nance, Amanda y Lindsay, Jeffrey (2006, June 20-23). Spearcons: Speech-based earcons improve navigation performance in auditory menus. In Proceedings of the 12^thInternational Conference on Auditory Display (ICAD 2006) (pp. 63-68). London, UK. http://hdl.handle.net/1853/50642 ; Walker, Kim, & Pendse, 2007Walker, Bruce N.; Kim, Jonathan y Pendse, Anandi (2007). Musical soundscapes for an accessible aquarium: Bringing dynamic exhibits to the visually impaired. In Proceedings of the International Computer Music Conference (ICMC 2007) (pp. 268-275). Copenhagen, Denmark. http://hdl.handle.net/2027/spo.bbp2372.2007.164 ; and more recently to sonified planetarium exhibits, e.g., Quinton, McGregor, & Benyon, 2016Quinton, Michael; McGregor, Iain y Benyon, David (2016, July 2-8). Sonifying the Solar System. In Proceedings of the 22^ndInternational Conference on Auditory Display (ICAD 2016). Canberra, Australia. https://doi.org/10.21785/icad2016.003 ; Tomlinson et al., 2017Tomlinson, Brianna J.; Winters, R. Michael; Latina, Christopher; Bhat, Smruthi; Rane, Milap y Walker, Bruce N. (2017, June 20-23). Solar System Sonification: Exploring Earth and its Neighbors Through Sound. In Proceedings of the International Conference on Auditory Display (ICAD 2017) (pp. 128-134). State College, PA, USA. https://doi.org/10.21785/icad2017.027 ).

Wayfinding and navigation can also be supported by auditory displays and sonification, often leveraging virtual spatial audio. Historical examples include the System for Wearable Audio Navigation (Wilson et al., 2007Wilson, Jeff; Walker, Bruce N.; Lindsay, Jeffrey; Cambias, Craig y Dellaert, Frank (2007). SWAN: System for wearable audio navigation. Proceedings of the 11th International Symposium on Wearable Computers (ISWC 2007) (pp. 91-98). Boston, MA. https://doi.org/10.1109/ISWC.2007.4373786 ; updated as SWAN2.0, see Walker & Wilson, 2021Walker, Bruce N. y Wilson, Jeff (2021, March 23-24). SWAN 2.0: Research and Development on a New System for Wearable Audio Navigation. In Proceedings of the WirelessRERC State of the Technology Forum 2021. Virtual conference. http://www.wirelessrerc.gatech.edu/sites/default/files/publications/walker_swan2-wirelessrerc-sot2021-proceedings-v16-withalttext.pdf ), the Personal Guidance System (PGS, Golledge et al., 1991Golledge, Reginald G.; Loomis, Jack M.; Klatzky, Roberta L.; Flury, Andreas y Yang, Xiao Li (1991). Designing a personal guidance system to aid navigation without sight: Progress on th eGIS component. International Journal of Geographical Information Systems, 5, 373-395.; Loomis, Golledge & Klatzky, 1993Loomis, Jack M.; Golledge, Reginald G. y Klatzky, Roberta L. (1993). Personal guidance system for the visually impaired using GPS, GIS, and VR technologies. In Proceedings of the First Annual International Conference, Virtual Reality and Persons with Disabilities (pp. 71-74). Center for Disabilities, California State University, Northridge, CA.; Loomis et al., 2005Loomis, Jack M.; Marston, James R.; Golledge, Reginald G. y Klatzky, Roberta L. (2005). Personal guidance system for people with visual impairment: A comparison of spatial displays for route guidance. Journal of Visual Impairment & Blindness, 99, 219-232.), and computer-vision and navigation systems by Revuelta Sanz and colleagues (Revuelta Sanz et al., 2014aRevuelta Sanz, Pablo; Ruiz Mezcua, Belén; Sánchez Pena, José Manuel y Walker, Bruce N. (2014a). Scenes into sounds: A taxonomy of image sonification methods for mobility applications. Journal of the Audio Engineering Society, 62 (3), 161-171., 2014bRevuelta Sanz, Pablo; Ruiz Mezcua, Belén; Sánchez Pena, José Manuel y Walker, Bruce N. (2014b). Evaluation of the Sonification Protocol of an Artificial Vision System for the Visually Impaired. International Journal of Computer and Information Technology, 3 (3), 469-481., 2014cRevuelta Sanz, Pablo; Ruiz Mezcua, Belén; Sánchez Pena, José Manuel y Walker, Bruce N. (2014c). Evaluation of a New Mobility Assistive Product for the Visually Impaired. International Journal of Computers and Technology 13 (2), 4191-4205.). More recently, Microsoft has developed Soundscape⁷ https://www.microsoft.com/en-us/research/product/soundscape/ and XRNavigation has developed AUDIOM⁸ https://xrnavigation.io ; both are available for audio-based navigation and wayfinding.

Data exploration interfaces are what is generally meant by the term «sonification», and are usually intended to encode and convey information about an entire data set or relevant aspects of the data set. Sonifications designed for data exploration differ from status or process indicators in that they use sound to offer a more holistic portrait of the data in the system rather than condensing information to capture a momentary state such as with alerts and process indicators, though some auditory displays, like soundscapes, blend status indicator and data exploration functions. Auditory graphs are a common approach to basic data exploration sonifications, and most commonly use changes in auditory frequency to correspond to changes in data values along the visual Y axis, while time corresponds to the visual X axis. Nees and Walker (2007)Walker, Bruce N.; Kim, Jonathan y Pendse, Anandi (2007). Musical soundscapes for an accessible aquarium: Bringing dynamic exhibits to the visually impaired. In Proceedings of the International Computer Music Conference (ICMC 2007) (pp. 268-275). Copenhagen, Denmark. http://hdl.handle.net/2027/spo.bbp2372.2007.164 proposed a conceptual psychological model of auditory graph comprehension. There have been auditory versions of numerous traditional display formats, including auditory scatterplots (e.g., Bonebright et al., 2001Bonebright, Terri L.; Nees, Mike A.; Connerley, Tayla T. y McCain, Glenn R. (2001, July 29-August 1). Testing the effectiveness of sonified graphs for education: A programmatic research project. In Proceedings of the 7^thInternational Conference on Auditory Display (ICAD 2001) (pp. 62-66). Espoo, Finland. http://hdl.handle.net/1853/50654 ; Flowers, Buhman, & Turnage, 1997Flowers, John H.; Buhman, Dion C. y Turnage, Kimberly D. (1997). Cross-modal equivalence of visual and auditory scatterplots for exploring bivariate data samples. Human Factors, 39 (3), 341-351.), box-whisker plots (Flowers & Hauer, 1992Flowers, John H. y Hauer, Terry A. (1992). The ear’s versus the eye’s potential to assess characteristics of numeric data: Are we too visuocentric? Behavior Research Methods, Instruments & Computers, 24(2), 258-264.; Peres & Lane, 2003Peres, S. Camille y Lane, David M. (2003, July 6-9). Sonification of statistical graphs. Proceedings of the International Conference on Auditory Display (ICAD 2003) (pp. 157-160). Boston, MA. http://hdl.handle.net/1853/50486 , 2005Peres, S. Camille y Lane, David M. (2005, July 6-9). Auditory graphs: The effects of redundant dimensions and divided attention. Proceedings of the 11^thInternational Conference on Auditory Display (ICAD 2005) (pp. 169-174). Limerick, Ireland. http://hdl.handle.net/1853/50093 ), histograms (Flowers & Hauer, 1993Flowers, John H. y Hauer, Terry A. (1993). «Sound» alternatives to visual graphics for exploratory data analysis. Behavior Research Methods, Instruments & Computers, 25 (2), 242-249.), multidimensional data sets (see Hermann & Hunt, 2005Hermann, Thomas y Hunt, Andy (2005). An introduction to interactive sonification. IEEE Multimedia, 12(2), 20-24.), and tabular data (Stockman, Hind, & Frauenberger, 2005Stockman, Tony; Hind, Greg y Frauenberger, Christopher (2005, July 6-9). Interactive sonification of spreadsheets. In Proceedings of the 11^thInternational Conference on Auditory Display (ICAD 2005) (pp. 134-139). Limerick, Ireland. http://hdl.handle.net/1853/50166 ).

As a bit of an aside, some organizations, such as NASA, are producing sonifications of data using many of the methods that a scientist might use to explore their data; and then releasing the sonifications to the public as a form of outreach. Recent such outreach examples come from the Chandra X-ray Observatory⁹ https://chandra.si.edu/sound/ and the James Webb Space Telescope¹⁰ https://www.nasa.gov/feature/goddard/2022/nasa-webb-s-first-full-color-images-data-are-set-to-sound . The point is that the sonifications are not, in those cases, really intended to be used for scientific discovery. Rather, many listeners simply enjoy the novel sounds as a sort of “astronomical artwork” and (hopefully) become excited about the activities of the scientists. This highlights that while one might list out a taxonomy of sonification types, the categories are really much less distinct, the boundaries less clear…and it probably matters relatively little what, exactly, a sound is called, compared to the ultimate utility it has in conveying information or achieving some other purpose.

2.2. Representational Approaches for Sonifications

Another way to organize and define sonifications is to describe them according to the sonification technique or approach. De Campo (2007)de Campo, Alberto (2007, June 26-29). Toward a data sonification design space map. In Proceedings of the 13^thInternational Conference on Auditory Display (ICAD 2007) (pp. 342-347). Montreal, Canada. http://hdl.handle.net/1853/50042 offered a sonification design map that featured three broad categorizations of sonification approaches: (1) event-based; (2) model-based; and (3) continuous.

Parameter mapping sonification represents changes in some data dimension with changes in an acoustic dimension to produce a sonification. Auditory graphs and many sonifications fall into this category. Sound has a multitude of changeable dimensions (see Kramer, 1994Kramer, Gregory (1994). An introduction to auditory display. In G. Kramer (Ed.), Auditory Display: Sonification, Audification, and Auditory Interfaces (pp. 1-78). Reading, MA: Addison Wesley.; Levitin, 1999Levitin, Daniel J. (1999). Memory for musical attributes. In P. Cook (Ed.), Music, Cognition, and Computerized Sound: An Introduction to Psychoacoustics (pp. 209-227). Cambridge, MA: MIT Press.) that allow for a large design space when mapping data to audio. These approaches to sonification have typically employed a somewhat passive mode of interaction, in that the sonification is «played» and the listener attempts to understand what is happening in the data set.

Model-based sonification (e.g., Hermann, 2002Hermann, Thomas (2002). Sonification for exploratory data analysis. [Unpublished PhD. thesis]. Bielefeld University.; Hermann & Ritter, 1999Hermann, Thomas y Ritter, Helge (1999). Listen to your data: Model-based sonification for data analysis. In G. D. Lasker (Ed.), Advances in Intelligent Computing and Multimedia Systems (pp. 189-194). Baden-Baden: IIASSRC.) involve a virtual model whose sonic responses to user input are derived from data. A model, then, is a virtual object or instrument with which the user can interact, and the user’s input drives the sonification such that «the sonification is the reaction of the data-driven model to the actions of the user» (Hermann, 2002, p. 40Hermann, Thomas (2002). Sonification for exploratory data analysis. [Unpublished PhD. thesis]. Bielefeld University.). The user comes to understand the structure of the data based on the acoustic responses of the model during interactive probing of the virtual object. These types of sonifications tend to involve high data dimensionality and large numbers of data points.

Audification is the (nearly) direct conversion of data into sound: waveforms of periodic data are translated into sound (Kramer, 1994Kramer, Gregory (1994). An introduction to auditory display. In G. Kramer (Ed.), Auditory Display: Sonification, Audification, and Auditory Interfaces (pp. 1-78). Reading, MA: Addison Wesley.). For example, seismic data have been audified in order to facilitate the categorization of seismic events with accuracies of over 90% (see Dombois, 2002Dombois, Florian (2002, July 2-5). Auditory seismology - On free oscillations, focal mechanisms, explosions, and synthetic seismograms. In Proceedings of the 8th International Conference on Auditory Display (ICAD 2002) (pp. 27-30). Kyoto, Japan. http://hdl.handle.net/1853/51334 ; Speeth, 1961Speeth, Sheridan Dauster (1961). Seismometer sounds. Journal of the Acoustical Society of America, 33, 909-916.). This approach may require that the waveforms be frequency- or time-shifted into the range of audible waveforms for humans.

3.1. Detection and Discrimination

An auditory display is useless if the listener cannot hear the sounds in the system’s environment of operation. To ensure detection, a consideration of the acoustic spectra of both the sonification (the «signal») and the environmental sounds (the «noise») is critical. Considerations of detection thresholds (e.g., Hartmann, 1997Hartmann, William M. (1997). Sounds, signals, and sensation: Modern acoustics and signal processing. New York: Springer Verlag.) and masking theories may help (for a discussion, see Watson & Kidd, 1994Watson, Charles S. y Kidd, Gary R. (1994, November 7-9). Factors in the design of effective auditory displays. In Proceedings of the 2nd International Conference on Auditory Display (ICAD 1994) (pp. 293-303). Sante Fe, NM. http://hdl.handle.net/1853/50864 ). And ecologically valid evaluation is important (Brewster, 2002Brewster, Stephen (2002). Overcoming the lack of screen space on mobile computers. Personal and Ubiquitous Computing, 6 (3), 188-205.; also see Walker & Kramer, 2004Walker, Bruce N. y Kramer, Gregory (2004). Ecological psychoacoustics and auditory displays: Hearing, grouping, and meaning making. In J. Neuhoff (Ed.), Ecological psychoacoustics (pp. 150-175). New York: Academic Press.). A second consideration is the discriminability of sounds with distinct meanings, with a long-standing literature of perception research available for guidance on the psychology of hearing (e.g., Moore, 2013Moore, Brian C. J. (2013). An Introduction to the Psychology of Hearing (6th ed.). Leiden, NL: Brill.), pitch (e.g., Stevens, Volkmann, & Newman, 1937Stevens, Stanley S.; Volkmann, John, y Newman, Edwin B. (1937). A Scale for the Measurement of the Psychological Magnitude Pitch. Journal of the Acoustical Society of America, 8, 185.; Turnbull, 1944Turnbull, William W. (1944). Pitch discrimination as a function of tonal duration. Journal of Experimental Psychology, 34, 302-316.), loudness (e.g., Stevens, 1936Stevens, Stanley S. (1936). A scale for the measurement of a psychological magnitude: Loudness. Psychological Review, 43, 405-416.), tempo (e.g., Boltz, 1998Boltz, Marilyn G. (1998). Tempo discrimination of musical patterns: Effects due to pitch and rhythmic structure. Perception & Psychophysics, 60 (8), 1357-1373.), and duration (e.g., Jeon & Fricke, 1997Jeon, Jin Y. y Fricke, Fergus R. (1997). Duration of perceived and performed sounds. Psychology of Music, 25 (1), 70-83.), to name but a few.

3.2. Annoyance / Attention

Sounds that annoy the user may be ignored or turned off, even when the sounds are beneficial. Aesthetic considerations intersect with performance concerns. Some recommend musical sounds (Brown et al., 2003Brown, Lorna M.; Brewster, Stephen A.; Ramloll, Ramesh; Burton, Mike y Riedel, Beate (2003, July 6-9). Design guidelines for audio presentation of graphs and tables. In E. Brazil & B. Shinn-Cunningham (eds.) Proceedings of the 9th International Conference on Auditory Display (ICAD 2003) (pp. 284-287). Boston, MA. http://hdl.handle.net/1853/50454 ; Childs, 2005Childs, Edward (2005, July 6-9). Auditory graphs of real-time data. In Proceedings of the 11^thInternational Conference on Auditory Display (ICAD 2005) (pp. 402-405). Limerick, Ireland. http://hdl.handle.net/1853/50086 ; Ramloll et al., 2001Ramloll, Rameshsharma; Brewster, Stephen; Yu, Wai y Riedel, Beate (2001). Using non-speech audio sounds to improve access to 2D tabular numerical information for visually impaired uses. Proceedings of the IHM-HCI 2001 (pp. 515-530). Lille, France.), though that, in itself, will not guarantee a pleasant experience for all users, tasks, and environments. Clearly, developing an auditory interface is, in all regards, a design task, with all the inherent difficulties associated with design (and, as noted above, see Nees, 2019Nees, Michael A. (2019, June 23-27). Eight components of a design theory of sonification. Proceedings of the 25^thInternational Conference on Auditory Display (ICAD 2019) (pp. 176-183). Newcastle, UK. https://doi.org/10.21785/icad2019.048 ).

3.3. Mapping and Choice of Display Dimension

Data-to-display mapping refers to the attribute of sound that is used to represent changes in data. Walker has studied the appropriate acoustic dimension for a given type of data by examining mappings between numerous conceptual data dimensions (e.g., temperature, pressure, danger) and three acoustic dimensions (pitch, tempo, and spectral brightness; Walker, 2002Walker, Bruce N. (2002). Magnitude estimation of conceptual data dimensions for use in sonification. Journal of Experimental Psychology: Applied, 8, 211-221. https://doi.org/10.1037/1076-898X.8.4.211 , 2007Walker, Bruce N. (2007). Consistency of magnitude estimations with conceptual data dimensions used for sonification. Applied Cognitive Psychology, 21, 579-599.). This is complicated by the fact that many acoustic dimensions (e.g., pitch and loudness) interact with one another (see, e.g., Moore, 2013Moore, Brian C. J. (2013). An Introduction to the Psychology of Hearing (6th ed.). Leiden, NL: Brill.). Nees and Walker (2007)Nees, Michael A. y Walker, Bruce N. (2007, June 26-29). Listener, task, and auditory graph: Toward a conceptual model of auditory graph comprehension. In Proceedings of the 13th International Conference on Auditory Display (ICAD 2007) (pp. 266-273). Montreal, Canada. http://hdl.handle.net/1853/50010 discuss the convention of mapping data values onto changes in pitch in auditory graphs. Sonification designers should note that not all acoustic mappings are equally effective, and best designs will arise from an awareness of both the historical literature and pilot testing of any displays.

3.4. Mapping Polarities

The polarity of the data-to-display relationship refers to whether increases in a given acoustic dimension (e.g., pitch, tempo, etc.) represent increases in the data represented (a positive mapping polarity, Walker, 2002Walker, Bruce N. (2002). Magnitude estimation of conceptual data dimensions for use in sonification. Journal of Experimental Psychology: Applied, 8, 211-221. https://doi.org/10.1037/1076-898X.8.4.211 , 2007Walker, Bruce N. (2007). Consistency of magnitude estimations with conceptual data dimensions used for sonification. Applied Cognitive Psychology, 21, 579-599.), or decreases in the data (a negative polarity). Listeners might agree that increasing pitch suggests increasing temperature, yet the same group of listeners may feel that increasing pitch offers a more intuitive representation of decreasing size. Walker and Lane (2001Walker, Bruce N. y Lane, David M. (2001, July 29-August 1). Psychophysical scaling of sonification mappings: A comparison of visually impaired and sighted listeners. In Proceedings of the 7th International Conference on Auditory Display (ICAD 2001) (pp. 90-94). Espoo, Finland. http://hdl.handle.net/1853/50630 ; see, also, Mauney & Walker, 2010Mauney, Lisa M. y Walker, Bruce N. (2010). Universal design of auditory graphs: A comparison of sonification mappings for visually impaired and sighted listeners. ACM Transactions on Accessible Computing, 2 (3), Article 12.) showed early on that some polarity mappings were reversed for visually impaired as compared to sighted listeners.

3.5. Scaling

The scaling refers to the amount of change in an acoustic dimension that will be used to represent a unit of change in the data. Magnitude estimation has been employed to describe the intuitive slopes for scaling frequency to a number of conceptual data dimensions (Walker, 2002Walker, Bruce N. (2002). Magnitude estimation of conceptual data dimensions for use in sonification. Journal of Experimental Psychology: Applied, 8, 211-221. https://doi.org/10.1037/1076-898X.8.4.211 , 2007Walker, Bruce N. (2007). Consistency of magnitude estimations with conceptual data dimensions used for sonification. Applied Cognitive Psychology, 21, 579-599.), and the conceptual data dimension being represented impacts the choice of scaling factor in the display. A match between the listener’s preferred or intuitive internal scaling function and the display’s scaling function may improve comprehension, though usability testing will help determine the best scaling for a given situation.

3.6. Concurrent presentation of multiple data streams/series

Some data analysis tasks require the comparison of values from different data streams, whereas in other cases it is preferable to fuse streams into a perceptual whole. Bregman (1990)Bregman, Albert S. (1990). Auditory scene analysis: The perceptual organization of sound. Cambridge, MA: MIT Press. discusses what acoustic properties support or inhibit stream segregation, with key dimensions being timbre, spatial location (or stereo panning), pitch/frequency, and onset/offset of sounds.

3.7. Context

Context refers to the purposeful addition of non-signal information to a display (Smith & Walker, 2005Smith, Daniel R. y Walker, Bruce N. (2005). Effects of auditory context cues and training on performance of a point estimation sonification task. Applied Cognitive Psychology, 19 (8), 1065-1087.; Walker & Nees, 2005Walker, Bruce N. y Nees, Michael A. (2005, July 6-9). An agenda for research and development of multimodal graphs. In Proceedings of the 11^thInternational Conference on Auditory Display (ICAD 2005) (pp. 428-432). Limerick, Ireland. http://hdl.handle.net/1853/50098 ). Sonifications need to include contextual cues equivalent to axes, tick marks and labels, so the listener can perform the interpretation tasks. For example, adding a series of clicks to the display can help the listener keep track of the time better, which aids in their interpretation of the graph values (see, e.g., Smith & Walker, 2005Smith, Daniel R. y Walker, Bruce N. (2005). Effects of auditory context cues and training on performance of a point estimation sonification task. Applied Cognitive Psychology, 19 (8), 1065-1087.).

3.8. Individual Differences

The perceptual and cognitive capabilities, limitations, and experiences of listeners, as well as transient states (like mood and level of fatigue) will all impact performance outcomes with auditory displays. By understanding ranges in individual difference variables, a designer can build a display that accommodates most users in a given context (e.g., universal design, see Iwarsson & Stahl, 2003Iwarsson, Susanne y Stahl, Agneta (2003). Accessibility, usability, and universal design--positioning and definition of concepts describing person-environment relationships. Disability and Rehabilitation, 25 (2), 57-66.). It is interesting to note that for many years researchers predicted and anticipated that musicians would outperform non-musicians on tasks involving sonifications. However, research has very rarely found any correlations between musical experience and performance (e.g., Lacherez, Seah, & Sanderson, 2007Lacherez, Philippe; Seah, Eunice L. y Sanderson, Penelope M. (2007). Overlapping melodic alarms are almost indiscriminable. Human Factors, 49 (4), 637-645.; Neuhoff & Wayand, 2002Neuhoff, John G.; Knight, Rebecca y Wayand, Joseph (2002, July 2-5). Pitch change, sonification, and musical expertise: Which way is up? Proceedings of the 8^thInternational Conference on Auditory Display (ICAD 2002) (pp. 351-356). Kyoto, Japan. http://hdl.handle.net/1853/51370 ; Sandor & Lane, 2003Sandor, Aniko y Lane, David M. (2003, July 6-9). Sonification of absolute values with single and multiple dimensions. In Proceedings of the 2003 International Conference on Auditory Display (ICAD 2003) (pp. 243-246). Boston, MA. http://hdl.handle.net/1853/50487 ). One explanation for the lack of relationship is the crude nature of oft-used self-report metrics of musical experience. Indeed, in a systematic investigation Schuett (2019)Schuett, Jonathan Henry (2019). Measuring the effects of display design and individual differences on the utilization of multi-stream sonifications. [Unpublished PhD Dissertation]. Georgia Institute of Technology, Atlanta, USA. http://hdl.handle.net/1853/61808 determined that a more sophisticated measure of musical sophistication, leaning largely on engagement with music, can be predictive of performance. Visual impairment also has been shown to have a potentially profound impact on the perception of sonifications. As mentioned, it has been shown (Mauney & Walker, 2010Mauney, Lisa M. y Walker, Bruce N. (2010). Universal design of auditory graphs: A comparison of sonification mappings for visually impaired and sighted listeners. ACM Transactions on Accessible Computing, 2 (3), Article 12.; Walker & Lane, 2001Walker, Bruce N. y Lane, David M. (2001, July 29-August 1). Psychophysical scaling of sonification mappings: A comparison of visually impaired and sighted listeners. In Proceedings of the 7th International Conference on Auditory Display (ICAD 2001) (pp. 90-94). Espoo, Finland. http://hdl.handle.net/1853/50630 ) that blind and sighted listeners can have opposing intuitions about the polarity of the pairing of some acoustic dimensions with conceptual data dimensions. Individual differences between visually-impaired and sighted listeners require more research and a careful testing of auditory displays with the intended user population.

3.9. Authoring

As Nees (2019)Nees, Michael A. (2019, June 23-27). Eight components of a design theory of sonification. Proceedings of the 25^thInternational Conference on Auditory Display (ICAD 2019) (pp. 176-183). Newcastle, UK. https://doi.org/10.21785/icad2019.048 discusses, there have been a (long) series of largely one-off software tools to create sonifications (too many to list exhaustively, here). They have varied greatly in terms of the platform and programming language, the approach to creating sounds, the process for defining mappings, context, and other attributes. Examples range from sonification toolkits focusing on a specific domain (e.g., xSonify: Candey, Schertenleib, & Díaz Merced, 2006Candey, Robert M.; Schertenleib, Anton M. y Diaz Merced, W. L. (2006, June 20-23). Xsonify sonification tool for space physics. Proceedings of the 12th International Conference on Auditory Display (ICAD 2006) (pp. 289-290) London, UK. http://hdl.handle.net/1853/50697 ) to more general frameworks (e.g., SoniPy: Worrall et al., 2007Worrall, David; Bylstra, Michael; Barrass, Stephen y Dean, Roger (2007, June 26-29). SoniPy: The design of an extendable software framework for sonification research and auditory display. In Proceedings of the 13^thInternational Conference on Auditory Display (ICAD 2007). Montreal, Canada. ). The Sonification Sandbox¹¹Georgia Tech Sonification Sandbox: http://sonify.psych.gatech.edu/research/sonification_sandbox/index.html was, for many years, a toolkit that was intended to serve the needs of diverse STEM fields (Davison & Walker, 2007Davison, Benjamin K. y Walker, Bruce N. (2007, June 26-29). Sonification Sandbox reconstruction: Software standard for auditory graphs. In Proceedings of the 13th International Conference on Auditory Display (ICAD 2007) (pp. 509-512). Montreal, Canada. http://hdl.handle.net/1853/50030 ), though it still had the limitation of being optimized for basic auditory graphs, and not a broader range of sonification methods. The recently released Highcharts Sonification Studio (HSS)¹²Highcharts Sonification Studio: https://sonification.highcharts.com is a re-implementation of the Sonification Sandbox as a web application, supported by the power of the Highcharts visualization engine that incorporates extensive sonification capabilities. Backed formally by a mainstream data visualization company (HighSoft), the HSS seems to represent the first corporate mainstreaming of sonification tools (Cantrell, Walker, & Moseng., 2021Cantrell, Stanley; Walker, Bruce N. y Moseng, Øystein (2021, June 25-28). Highcharts Sonification Studio: An online, open-source, extensible, and accessible data sonification tool. In Proceedings of the 26^thInternational Conference on Auditory Display (ICAD 2021) (pp. 210-216). Virtual Conference. http://hdl.handle.net/1853/66348 ), and is notably built to be accessible to screen reader users.

3.10. Audio Delivery Hardware

Historically, the «last mile», or the actual output of sound was often a challenge. Systems might not be able to produce sound, or if so, might need speakers or headphones as an additional piece of equipment. A sonification designer could never know what the actual listening equipment would be. Now, however, nearly all modern digital devices, from phones to tablets to laptops to smartwatches to smart speakers, are capable of producing high-fidelity sound, with most now including speakers, even if small. An output jack (e.g., audio only or HDMI) or Bluetooth capability is largely standard. As such, it is generally straightforward to play a sonification.

5.1. Technology Acceptance Model: Utility and Usability

The Technology Acceptance Model (TAM; Davis, 1989Davis, Fred D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, 13 (3), 319-340. https://doi.org/10.2307/249008 ) posits that the adoption of a technology depends largely on two main factors: perceived usefulness or perceived utility (PU) and perceived ease-of-use (PEU). In practice, these two factors are interconnected, of course. Perceived usefulness (PU) is defined by Davis as how much a person believes that using a particular technology or technical system would enhance performance on their task. One can extend Davis’ thinking about technology acceptance to sonification by considering some of the specific contributors to usefulness, such as theoretical grounding for the use of the technology, scientific utility, scientific validity, replicability of results when using the technology, standardization of the use of a technology, educational utility, and accessibility or inclusivity for accomplishing a task.

Perceived ease-of-use (PEOU) is defined by Davis as how much a person believes that a technology would be user-friendly. Again, one can extend and update Davis’ thinking by considering factors such as the availability and prevalence of the (software) tools required to use a technology, the usability of the tools, the training that is available, the standardization of the technological solution, the existence of a community of users (and support), portability of the technology, integration of the technology into the work/school/science ecosystem, and the accessibility of the system.

5.2. Where Does Sonification Stand?

With the TAM framework, and an updated consideration of sonification, it is interesting to assess the current state of sonification, and provide a bit of a “report card”.

Perceived Utility «Report Card» for Sonification. In terms of theoretical grounding, there is a solid body of published research examining the core components of sonification, even though there may still not be an actual “theory of sonification” (and see Nees, 2019Nees, Michael A. (2019, June 23-27). Eight components of a design theory of sonification. Proceedings of the 25^thInternational Conference on Auditory Display (ICAD 2019) (pp. 176-183). Newcastle, UK. https://doi.org/10.21785/icad2019.048 , for a discussion of design theory for sonification). There is, for example, a Sonification Handbook (Hermann, Hunt, & Neuhoff, 2011Hermann, Thomas; Hunt, Andy y Neuhoff, John (2011). The Sonification Handbook. Berlin: Logos Publishing House. ), including a chapter on the Theory of Sonification (Walker & Nees, 2011Walker, Bruce N. y Nees, Michael A. (2011). Theory of Sonification. In T. Hermann, A. Hunt, & J. Neuhoff (Eds.). The Sonification Handbook (pp. 9-39). Berlin, Germany: Logos Publishing House. http://sonification.de/handbook/download/TheSonificationHandbook-chapter2.pdf ), providing a grounding for applications of sonification. There have been many findings made possible through sonification, ranging from early space science (e.g., NASA Cassini Mission¹³ https://solarsystem.nasa.gov/news/12580/sounds-of-cassini/ ) to recent cancer diagnosis (Dascalu et al., 2021Dascalu, A.; Walker, Bruce N.; Oron, Y. y David, E. O. (2021). Non-melanoma skin cancer diagnosis: a comparison between dermoscopic and smartphone images by unified visual and sonification deep learning algorithms. Journal of Cancer Research and Clinical Oncology, 148, 2497-2505. https://doi.org/10.1007/s00432-021-03809-x ), demonstrating scientific utility of sonification. The scientific validity has been less-thoroughly investigated, to date, with small sample sizes being typical and many generally unreplicated results. In terms of standardization, the basic concept of an auditory graph (x-y plot mapped onto time and pitch) has become a de facto standard, though there are many design differences amongst the countless implementations. Beyond simple auditory graphs, however, there is little standardization in sonification (in terms of the tools used, the designs, the deployment, and so on). The educational utility has started to emerge, especially as sonification is beginning to be used as part of the assistive technology used in Science, Technology, Engineering, and Mathematics (STEM) education. More needs to be done in this regard, since accessibility could be a huge area of impact for sonification.

Perceived Usability «Report Card» for Sonification. The sonification tools are quite readily available, with many options at various levels of sophistication, and using various underlying technology «stacks». It can be a challenge to know where to look, and there may be technical expertise required (e.g., programming in a particular programming language) in order to actually use some of the tools. The usability of the software tools varies greatly, from walk-up-and-use to experts-only, especially since many tools were not designed or developed for widespread deployment-they are often just built to assist a particular researcher to investigate a particular type of data. As previously discussed, more tools for sonification are becoming available with ease of use in mind (e.g., Highcharts Sonification Studio). Training has often not been very available for sonification tools, beyond the «readme» files that come with software packages, plus the limited details that can be gleaned from academic papers or technical reports. Recently there is an online course on the Coursera platform about the design of sonification (Moore, Tomlinson, & Walker, n.d.Moore, Emily; Tomlinson, Brianna J. y Walker, Bruce N. (n.d.). Sound and Sonification Design for Interactive Learning Tools. Online course on the Coursera platform. https://www.coursera.org/learn/sound-and-sonification-for-learning ), and countless emerging YouTube videos and channels¹⁴As just one example of a YouTube “how-to” resource channel: https://www.youtube.com/@HSS_How_To and websites to support novices in getting started with sonification. Standardization is minimal, as is portability, which makes it a challenge to use sonification. The emergence of file formats that can be exchanged across software applications (and shared from person to person) will help with portability, but this remains a fledgling concept. Thankfully, the community of users and developers in the field of sonification has been around for a few decades, but it remains relatively small and unfortunately a bit on the margins of many other fields (education, STEM, computer science). There is little or no financial support for sustainability of the community. Since sonification is often used by researchers in another field (e.g., astronomy), the sonification tools are often integrated into the ecosystem of data collection and analysis, though this also remains a work in progress, across the field. In education, however, sonification tools, when available, are still largely separate from the ecosystem of other educational technology and assistive technology. This is changing as sonification tools are now being built to play nice with file formats and data transfer protocols that are common. Finally, the heterogeneity of sonification tools also means that there is a range of accessibility and compliance. It is encouraging that many of the more recent sonification tools are «born accessible», often due to the involvement on the development team of a designer or developer or scientist with a disability.

Summary «Grades» for Sonification Acceptance. Overall, it seems fair to conclude that the field of sonification is doing well on perceived utility, though with room for improvement. In terms of the perceived usability, there is more work to be done. Sonification is already seeing adoption in science, perhaps because the «proof» of utility and usefulness has been delivered; and the technical sophistication of the typical users is higher, leading to a greater tolerance of usability and ease-of-use challenges. In STEM education, the promise of sonification is likely understood (especially in the accessibility domain), but the real and perceived challenges in usability hamper further adoption. The field of sonification researchers and developers need to work closely with the end users to make the case for utility, and build usability and accessibility into any and all new tools and methods for sonification.