Interfacing Sound with Screen Interfaces

Interfacing sound in screen-based music software is no simple task: traditionally the software tends to either follow linear scoring metaphors (piano rolls, traditional notation, tape tracks) that are useful for composition, or imitate hardware (sliders, knobs, buttons, cables, screens), allowing for a real-time manipulation of sound which is eventually “bounced down” to a fixed file. There have been myriads of other, more experimental approaches, that investigate how we can perform with screen-based musical interfaces. However, designing two- or three-dimensional representation of sound, where the physical interfaces might consist of a mouse, keyboard or touch screens, comes with some complications. Some of the design patterns that we find in the material world cannot easily be abstracted and represented in the digital domain. Such translations often become a process of transduction, where sounds or actions are transformed in the digital. Even when we attempt mimesis and aim to be true to the original object, we lose some of the unique (non-universal) characters of the individual instrument, the entropic qualities that often manifest in its behaviour, as well as the history and use of the particular object itself. A copy of software does not have a history in the same way as an individual object.

On the other hand, elements not found in acoustic instruments present themselves as natural properties within the digital, for example the possibility of timelines, looping techniques, learning mechanisms, or diverse mechanics of mapping gesture to sound. Here, screen-based instrument designers apply techniques from computer games, interface design, HCI, apps, installations, and computer networks. The metaphors abound, but they can be found in diverse areas of development, where techniques and user skills are reused in the new design. Commensurate with how the maker of hardware musical instruments seeks to enact the skills developed and incorporated into the motor memory of instrumentalists over the years, the designer of screen-based interfaces will apply techniques from diverse fields, such as drag and drop, shift-click for multiple selection, swipe for new screens, right click for menus, etc. Blackboxed design patterns are applied in the design of new instruments, consciously or not, and the user intuitively performs the system, learning its conceptual nature through interaction design that is already familiar.

< Prev   CONTENTS   Source   Next >