SpeakEase

Introduction

Augmentative communication (“aug com”) devices enable individuals who are otherwise unable to speak to communicate with the rest of the world. These devices are used by many different groups of people who are unable to verbally express themselves; from those who have lost their speaking facilities due to strokes to those with autism. Unfortunately, currently existing augmentative communication devices suffer from several deficiencies. The vast majority of current augmentative communication devices fit into one of two main categories: they are either affordable but difficult to expand or easily expandable but quite expensive. The first category includes devices that range from flipping through a series of cards (Figure 1) to pointing to a symbol on a specially designed belt. Although such devices are reasonably inexpensive, they are not very scaleable. When it comes time to add words to a user’s vocabulary, a new card or symbol must be created, then a user must learn to navigate through the symbols or cards to find the desired word. Imagine trying to search through a stack of hundreds of cards to express a single idea.

Figure 1. Examples of symbols that appear on augmentative communication cards (from Pictogram Symbol Reference Book)

Unfortunately, the majority of alternatives to these simple augmentative communication devices suffer from a serious and often insurmountable drawback. Most commercially available electronic alternatives are too expensive to be reasonable for many individuals to be able to afford. For instance, the “e-talk” systems have price tags ranging up to $5,500 (Figure 2).

Figure 2. An example of a commercial augmentative communication device, the e-talk 12.1

Motivation

One impetus for this project came from a personal contact, a friend who works as a speech and language therapist at an institute for children with autism and mental retardation in Belgium. As she was describing her interactions with the children, the discussion turned to the topic of communication. Because many of the children she works with have little or no verbal communication skills, she said that they use a primitive augmentative communication device. She described the device as a belt with a set of six symbols, representing various actions and emotions. These children would communicate by touching one of these symbols, which have meanings ranging from “I need to use the restroom” to “I am feeling angry.” This archaic and inflexible system needed improvement.

The second impetus came from professor Gary Bishop’s enabling technology course at the University of North Carolina. This class required a project involving the development of a technological device or advancement that would help to enable otherwise disabled individuals in the community. This became the perfect opportunity to help these children with autism. Mentioning the project to the Belgian teacher inspired a list of design considerations that she felt would be important. Over the course of the semester, the project was designed and implemented, resulting in an augmentative communication device that meets all of the design specifications. The final result, called Speak Ease, will aid not only the children with autism for whom it was originally designed, but also anybody else in need of an affordable, versatile way of communicating.

Design

The main motivation for the design of Speak Ease was to create a device that meets all user needs that are not currently addressed by existing systems. The financial problem is eased by the fact that the Speak Ease software can be installed for free on a commercially available cellular telephone, which can be purchased for around $100. The scalability is addressed by the fact that images and directories can be easily added to the program, allowing simple navigation among many nested directories. Speak Ease can also be considered scalable based on its ease of use on multiple telephones or other devices which can run .jar files. In its current form, Speak Ease on a cell phone is small and portable, unlike many heavy and unwieldy devices currently on the market.

In addition, rather than using conventional text to speech software, the words that are read aloud are playbacks of pre-recorded .wav files. This allows for Speak Ease to easily extend to foreign languages to which text-to-speech engines have not been fine-tuned. Additionally, the voice that is recorded can be one that is familiar to a user, rather than one that is unfamiliar and machine-generated.

As discussed in the motivation section, the design of Speak Ease targeted a group of children with autism who must rely on a set of symbols to communicate. With this audience in mind, the design was based on large, easy to see images and icons, uncomplicated buttons, simple navigation between images and straightforward selection of images. The underlying design goal was to keep everything as straightforward and understandable as possible, both with respect to the controls and the user interface on the screen.

When running in its most basic mode, navigation is controlled by left and right buttons, and an up button is used to select the desired image to be spoken. The controls are intuitive, as the left button returns to the previous image and the right button proceeds to the next image, as if flipping through pages of a book. The screen displays an image panel with a large image or icon representing the word along with a text panel with a text representation of the word and an indication of the current position in the list of images (Figure 3).

Figure 3. The Speak Ease device running in the most basic mode.

Additional features were added to this basic configuration to cater to users with more advanced vocabularies and more precise motor skills. In the advanced mode, the user can navigate through a series of directories, with each directory containing a combination of images and further subdirectories. In an effort to keep the design uniform across both modes, the subdirectories are presented by displaying a folder icon in the image panel and including the folder name in the text panel. This use of subdirectories allows for easy and custom organization of larger vocabularies. When running in this mode, the controls are slightly more complex. The left and right navigation buttons remain the same, however, the function of the up button is modified. The upward direction either moves the user to a previous directory (if we are currently looking at a directory) or reads the image (if we are currently looking at an image). The down arrow is used to return to a previous directory.

In an effort to keep the device uncomplicated for the user, Speak Ease selects whether to run in simple or advanced mode based on the content that the user inputs. If there are no subdirectories found, the device runs in simple mode. If there are subdirectories, the user is assumed to be more advanced, and the additional capabilities described above are added.

Through the use of straight-forward controls and a non-cluttered interface, the design of Speak Ease makes it inherently easy to use. New images and subdirectories can be painlessly created and added, allowing the device to expand and be updated as a user’s needs change. Finally, the ability to run in simple or advanced mode serves a larger group of potential users, maximizing the number of non-speakers that can be given the ability to communicate.

Implementation

The Speak Ease software was designed for use on a Motorola RAZR V3 cellular telephone. Although it has not been tested on other cellular telephones, it should be possible to be used on any phone that can play .jar files.

The software was designed using Sun’s J2ME (Java Micro Edition). J2ME provides many APIs that were useful in developing the Speak Ease software, particularly methods for loading images and playing sounds. Testing was done on a cellular phone emulator provide by Motorola.

For Speak Ease to work in its current form, the input images must be .png files, and the sound files must be 8 bit, 8 kilohertz mono .wav files, sharing names with the corresponding image files.

Conclusion

The Speak Ease augmentative communication device enables individuals with difficulty speaking to communicate with those around them. It is much more affordable than existing commercial devices and more scalable than its non-electrical alternatives. In addition, it is simple to use and update, and functions well for users of varying cognitive ability.

Future work on Speak Ease will include adding a desktop client to facilitate simpler transfer of images and sound files to the cellular phone. Another tool could be written incorporating a text to speech engine that could be used to automatically record audio files for playback on the device. A third tool could be written to allow for the automatic creation of directory icons based on images within the directory.

In addition to these additional tools, testing can be done to modify the buttons used for control to be more adaptive for individuals with poor cognitive skills. Speak Ease will also be tested on other cellular telephones and devices to determine if there is a more affordable alternative. Finally, sound files of different formats (.mp3 rather than .wav) can be used to decrease the size of the file.

Getting the files

Link to Java: link

Link to Motorola: link

Link to .java file: link

Link to sample audio files and images: link

Link to executable .jar file: link