Enabling Technology

StepNTime

Summary

My project for the semester is entitled ‘Step-N-Time’. It is an accessible version of Konami’s Dance-Dance Revolution intended for use by players with typical vision, limited vision or no vision. The program is written in Python and uses module files for music.

DDR Background

Dance-Dance Revolution (DDR) is an interactive game where players use a DDR dance pad to play the game. Stationary symbols at the top of the screen match the symbols on the dance pad. There are also symbols that scroll from the bottom of the screen to the top. The objective is to step on the corresponding dance pad symbol when the scrolling symbol overlaps the stationary one. Points are based on how close each step on the pad matches the overlapping of symbols on the screen. The entertainment factor is that music is playing in the background; symbols typically match up in beat to the music, thus stepping on the pads in time to music is very similar to dancing in beat to the music.

Purpose and Benefits

Dr. T. Carter Bethea, Clinical Fellow in the UNC Division of Child Psychiatry, gave a presentation on Using Technology to Undo the Incidental Harms Caused by Using New Technologies. He addressed America’s national obesity pandemic; one-fifth of the population is obese. Being obese, meaning one needs to lose 20–40 pounds, is worse than merely being overweight, meaning one needs to lose 10 pounds. The numbers of people overweight and obese has increased over the past few decades.

This trend matches the increase in fast food consumption, increase in TV watching, decrease in physical fitness activities at home and in school, increased urban sprawl and car culture. This obesity trend manifests itself across all age groups. Not only are there more 12–17 year olds and 8–12 year olds with obesity than in the 1980’s, but there are now substantial numbers of 2–4 year olds that are playing less and becoming more obese. Unfortunately, when kids are fat at a young age they tend to be fat when they are older, thus making them more susceptible to adverse health conditions and increased medical expenses.

TV watching is actually down, but that is due to competition from video games, the computer and taped programs such as DVD‘s and DVR. All of these have the same detrimental affect as TV watching; kids increased calorie intake is not offset by a parallel exercise increase. One approach proposed to help combat this is to have games that do increase exercise. With the large penetration of video game consoles and computers, a video and computer game could help combat that obesity pandemic.

Konami’s Dance-Dance Revolution is one such game. It has many of the hallmarks of a good game. It is easy to learn but hard to master. It crosses social paradigms; those that are sociable can invite several friends to play, while less sociable people can play by themselves or in small groups. It allows for single play, co-operative play and competitive play. It has an interactive lesson mode to teach the basic game moves. It has multiple levels, rewarding success with harder challenges for repeatable and extendable game play. Most importantly, it is a fun game that increase exercise in players. Unfortunately, it is an expensive game and is not easily accessible to those with low or no vision.

Exercise is just as important for kids with little or no vision as it is for those with typical vision. Exercise at a young age is the primary developer of mobility, agility and dexterity. The development of upper and lower body strength are needed for moving, walking and interacting with one’s surroundings. Kids with limited or no vision typically play in a more limited space and range of motion, due to fears from themselves or care-givers about their safety when running around. The goal of this project is to develop an adaptable game that allows younger kids with limited strength to develop that strength and allows older kids and kids with higher strength to be challenged, regardless of their vision of lack thereof.

Proposal

Step-N-Time, which also goes under the name Step In Time, is a video game in the tradition of DDR. It diverges from Konami’s Dance-Dance Revolution, Step Mania, Pydance, and other games of the genre in that it is accessible to kids irrespective of vision. This game addresses the cornerstone of the DDR genre, which also happens to be its limiting component; stepping on the dance pads to match the overlapping of the symbols on the screen. There are two ways this game interaction is addressed; one is by having audio voice commands tell the player when and where to step, the other is to let the player choose where and when to step. In this version, these modes of dance pad step communication will replace rather than supplement traditional game interaction. By telling players when to step, such as in a lessons setting, new players or those with limited mobility can learn how the game works, while experienced or more agile players can learn advanced moves. By allowing players to choose the step pattern, they take ownership of their game play experience. Points can assigned based on closeness to being on-beat to the music, step variety and step difficulty. Game play repeatability and extensibility can be enhanced through several means, such as restricted pad matching, match audio patterns or even creating audio patterns.

Technology

Step-N-Time is developed using Python, an extensible, interpreted, interactive, object-oriented programming language. It is very common for games, graphics, audio and networking applications. The graphical user interface uses WxPython, a Python GUI toolkit based on WxWidgets. Eclipse is the open source coding development platform and application framework, using the PyDev extension plug-in. Fmod is the high performance, cross platform library used to play the music files. PySonic is the python wrapper for the Fmod library. PyTTS is the Python text-to-speech wrapper for the Microsoft Speeach API, SAPI. Mark Hammond’s win32all extensions for Python are used to interact with the two wrappers. In the tradition of DDR style applications, it uses the DDR dance pad. The game is current written and tested on the Microsoft Windows XP operating system, since Microsoft is the most popular platform at homes and in schools.

The crux of any DDR style application is determining when a step is on-beat. Step-N-Time uses music module format files as the primary audio component. Music modules, or mods, are music created on the computer. More specifically, they are multi-channel format songs containing playback info for each instrument. Separate channels allowed independent playback capabilities, such as volume or special effects. Made popular by the Commodore Amiga, music mods allowed for simultaneous playback without detriment to the CPU. They have expanded from the original 4 channels and 15 instruments to 32 channels and 31 instruments per file, with several hundred instruments available in many mod instrument libraries. With the instrument sample included with the file, playback becomes independent of the sound card. Music modules are downloadable from the Internet; formats include .it (Impluse Tracker), .xm (Fast Tracker 2), .s3m (Scream Tracker 2) and .mod (Music Module).

Design & Focus

The original game design focused on the second and easier of the two game interactions; allowing players to choose when and where to step. With the guidance of Diane Braunner, education specialist, the focus was changed to the first and more difficult game interaction, voice commands delivered through a lessons setting. The game tested during Maze Days dealt nearly exclusively with the lessons setting for Step-N-Time.

There are two major obstacles to non-visual dance step interactions. First is determining when a player is on-time. Deciding the lateness of a step is easy, since it is merely the difference between when the step registered and when the instrument played. Deciding the earliness is harder. Either one must wait until an instrument plays after a step to determine the difference in time, or some type of prediction is needed to guess when the instrument should play next. Music module files already contain the information we need, instrument playback times. Step-N-Time pre-processes a song and catalogs each time and instrument plays, the next time it plays, the difference in time between playbacks, and the location of loop backs if a song repeats. Using the time an instrument played before the step, Step-N-Time uses the associated time information to calculate when it should play again. This gives us enough information to determine the lateness and earliness of a dance step.

The second major obstacle is the delivery of real-time feedback. Since we won’t know that a step has occurred until after the step, there will be some delay in the registration, calculation and determination of early or late for a step. The response therefore cannot be until after the step has occurred. With a visual feedback, the laws of physics allow us to exploit the time in calculation and distance between the screen and player to give feedback as the step occurs. Since the speed of sound is markedly slower than the speed of light, there may be a noticeable difference between when a player steps and when they hear the feedback. By using a low correlation sound, the feedback can have the appearance of being on-time rather than delayed. High correlation sounds are typically in the same key, octave, tone, etc of the background music, so they sound like a music note that is part of the music score. It can be hard to distinguish high correlation sounds from the actual music. Rather than music notes or sounds, voice generated feedback is used to designate the accuracy of a step. Since the correlation is low, the brain automatically handles the separation of music and feedback, allowing for simultaneous listening and response.

Game Play Modes

There are five proposed game play modes. Two are developed in detail here, while the other three fall under Unfinished Business. The five game modes are Step Lessons, Freestyle, Simon Says, Music Maker and Enhanced Freestyle. Game menu navigation is currently supported through the keyboard. Future menu controls are addressed in Unfinished Business.

Step Lessons are broken down into three levels. Each level has seven sub-levels. The same music is used for each sub-level within a stage. Each sub-level corresponds to a step type, which is a name and pattern of steps. Higher levels become progressively harder than their predecessor. Lessons are delivered in four parts; rules, step composition, practice and exam. The rules are repeated for each step, detailing what happens in the other three parts of the lesson. Step composition speaks the name of the pattern, along with where the steps should occur. The when is based on the time in the song the step direction is announced. Practice provides the player an opportunity to do the step with the step composition accompanying. Exam stores and calculates the players’ execution of the step, then reports their performance. Exams have step composition accompaniment on the lower levels, but at higher levels the player must also memorize the step composition. Currently, lesson only use the four cardinal game pad directions; up, down, left and right. Lessons using diagonals are a possible future enhancement, but are beyond the scope of most if not all current DDR style games.

Freestyle allows the player to step on any pad at any time. Accuracy is based on the best early/late match for all instruments. While the song is playing, freestyle rules monitor registers all instrument playbacks. When a step occurs, it checks the pad against the most recent playback for each instrument and the calculated next playback time, and then picks the one with the lowest time differential. The game pad is polled for steps every 5 milliseconds, or 200 Hz. For this mode of game play, it was determined that if a step is near the half way point of early and late, any inaccuracy due to polling will be negligible since the step is so far off from being on-beat in either direction. More points are allotted for higher precision; less accuracy generates fewer points, but there are no penalties for being off-beat.

Unfinished Business

Simon Says mode is based on the Milton Bradley game, in particular the 1984 release. Simon Says is based on the popular children’s game, where one person, referred to as ‘Simon’, instructs the other players on what to do. Players must only do commands proceeded by the words ‘Simon says’. Doing otherwise eliminates them from the game. Milton Bradley based their game on a variation, where everything ‘Simon’ says implicitly begins with ‘Simon says’, so players only need worry about copying the actions of ‘Simon’. The unfinished version for Step-N-Time was to be in the same fashion of the Milton Bradley adaptation. Each step on the game pad would have a unique sound. The computer would generate one of those sounds; the player must repeat that sound by stepping on the correct corresponding pad within a time limit. The next time, the computer plays the original sound plus one new sound, which the player must then repeat. This chain of sounds creates a sequence. The maximum number of sounds the computer generates for a sequence depends on the level of game play.

In Milton Bradley, the sounds had corresponding visual colors and location on the game system. Step-N-Time’s implementation, as an accessible game for typical, low and no vision players, would need to test sounds and sequence chains to determine what is appropriate. Basic music theory suggests that major or minor chords should be both pleasant and distinguishable enough to use for the pad notes. Possible options are to let the player choose which scale they want the notes played in or control the time limit. An extension to the game would be in two player mode. One player could determine the chain, with the other trying to match. They could also alternate, with one player adding a single note to the end of the current chain, if they play the current chain correctly. A higher level of difficulty is to match both the pattern and timing, rather than having a time limit. Another idea is to have the sequences be the notes to a song; the player gradually plays more and more of the song as the sequence chain lengthens.

Music Maker is the proposed content authoring mode for Step-N-Time. While a music track plays in the background, the player steps on different pads on the dance pad. Each pad maps to a particular note in the same key as the song. The player, controlling the step pattern they use, creates an accompaniment to the background music. This mode would allow for a unique expression and authoring of music, and arrangements could be saved for playback and sharing with others. Options could include allowing to player to choose which major, minor, diminished, etc chord and notes to use. As the music authoring option, there is probably enough for this to be its own separate game or project. One idea is to have the game auto determine the key of the song, which could then automate the other notes and chords using built in knowledge on the Circle of Fifths and other music theory. Several appropriate songs have already been found for this game play mode, but time did not permit further development of this game play mode.

Enhanced Freestyle mode places restrictions on the normal Freestyle mode. No longer can players match any pad to any instrument. Now, they can only match certain instruments using the pads. Further restrictions would be to only use certain pads for certain instruments; this would allow for game play where the left pads are for percussion like bass drum and snare, while the right pads are for electric string like the guitar. Another option is to allow normal freestyle game play, but with bonus points based on pad variation. By using the map of the game pad layout, the mode can generate a function based on what steps were pressed currently and previously, and use that to determine variation and difficulty. Either style within the mode could penalize for being off-beat.

There are a few miscellaneous items of unfinished business. One is using the DDR pad to control the menu, rather than just the keyboard. Another is support for multiple types of DDR pads, not just those using the EMS Dual Shooter PlayStation 2 to USB converter. In Freestyle mode, there was an idea to unlock music content based on scores on other music songs. This would necessitate persistence and possibly unique user ids. Menu playbacks and previews, where the music used in lower levels of the menu is played while browsing the menu needs to be added back in. Menu instructions were also a part of this menu playback feature. Both were removed during Maze Days (see below). Finally, since this game is for all vision types, more work needs to be done on the GUI to increase enjoyment for low and typical vision users. Ideas include having a grid on the screen during game play, with sections changing color based on the pad the player steps on. Song time playing and remaining would also be neat features to show, along with current score.

Maze Days: Ideas and Lessons Learned

Many types of people used Step-N-Time during Maze Days. Several ideas came from those players as well as Diane Braunner and Diane Pozefsky. Those with low mobility and dexterity are typically the younger children. Their experience was much different from older players or those in decent shape.

Maze Days exposed a problem with the lesson timing paradigm; how to correctly convey what stepping on-time or on-beat means. Those players that are naturally gifted with music, have exposure to dancing, or have played DDR have a better sense of what being on-time means. They realize that ideally one should anticipate the command, or at least react quickly. It is as if they say ‘since the command was issued on-beat, and the objective is to be on-beat, I must step at the same time as the command’. Other players were more reactionary, so it is as if they say ‘I hear the command, and will step in reaction, rather than anticipate, with no regard to being on-beat or off-beat’. Determination of on-time depends on when one steps, and my concept of being on-time matched those with exposure to music. I believe the instructions need to more clearly state which of those two modes of play is expected.

Younger players also exposed that lack of music awareness, rhythm, dancing ability or general physical fitness made even the lowest level too difficult. It was suggested that there be a more basic level, with the only audible noises the step instruction and a tone indicating when that step should be done. Timing for that level may need to be eliminated. Players get the next step instruction once they do the previous one. Later they can progress to a metronome with structured timing requirements, followed by actual music. Teaching them how to step was also difficult. Older and more agile players simply needed to know where up, left, down and back were, and could handle the rest themselves. Less mobile players needed help; Diane Braunner tried actually getting on the pad with the player and doing the step with them, providing another form of feedback on where and when to step. Diance Pozefsky tried a variation of this by physically helping to move the player’s legs. Several modalities are probably needed, depending on which one the player responds best to.

The Text-To-Speech was a common problem for all skill levels. Even though there was a low correlation between speech and music, hearing the voice command did not mean understanding what the command said. Using the voice of an actual person, perhaps saved in a wave file, would provide both the low correlation and high clarity needed to understand the voice commands and instructions.

Players exposed problems with scoring the lessons or rather exposed the need for a reinterpretation of the lesson scoring. For freestyle mode, we match the pad against the most recent and next expected time for all instruments and choose the closest. For lessons, this was generalized to matching pads to the most recent and next expected step time and choosing the closest. If the player was reactionary rather than anticipatory, they may have the correct pad but at the wrong time. The algorithm would match it to the next expected step time when it should have been matched to the previous expected step time. The correct interpretation might have been to catalog the times for each pad, then match the step to the closest expected time for that pad. There may be a need to check that a step is only matched to one expected pad time.

Overall, most players liked Step-N-Time. For younger players, it easily tired them out after a sub-level or two. Older and more agile players liked to skip directly to the higher levels. The name Step-N-Time received good feedback, as opposed to Dancing-Game or other accurate but perhaps less cool names. Players of various ages with typical, limited and low vision tried it. The most common complaint was not with the game but with understanding the text-to-speech. I think some of the shy players might have been more comfortable if they weren’t out for others to see and hear how they did. The set-up of the game for Maze Days did not allow for the less sociable side of the paradigm to be tested.

Source Code

To pre-process a song, use

sntPrepSongApp.py FileName ‘Song Name’

i.e.

sntPrepSongApp.py ModMusic\themoddepot.com\BILLYJEAN.MOD ‘Michael Jackson – Billy Jean’

To run the main application, use

stepNTime.py

Application Code with Step Lessons Song Files (318 KB)

Freestyle Song Files (48 MB)

Music Module File Links

Conclusion

Overall I had a great time working on this game. I’d really like to continue it and get it to the point where funding and user studies can be done using the game. I am not sure as to what type of research ideas from the computer science perspective are available. At worst case it would be a great sidebar project to continue with those in the medical field.