ProjectIdeas
Programming projects
Ideas from Marc Medwin
Ogg player that is accessible.
DVD player.
DVR.
Check out these Mozilla Accessibility Projects
http://www.mozilla.org/access/projects
Accessible Word Processor
OpenOffice can be automated like Microsoft Office using Python and a bridge called UNO. An interesting project might be to build a word processor, spreadsheet, etc. from the ground up by simply automating OO. Since OO is open source, free, and compatible with the MS office document formats, such a project would give blind users a great suite of office tools at no cost.
http://udk.openoffice.org/python/python-bridge.html
Accessible MP3 Player
Kids who are severely disabled have the same interests as any other kids. But their choices are limited because so many things most take for granted are unavailable to them. An MP3 player that is controllable using only one or two switches is a relatively simple project that would make a lot of people happy. Python or modify an existing open source player.
Audio Maze Game
Make a game that uses spatial sound (easy with libraries we have) to construct the audio version of the grid mazes that are popular with kids. You start at a point and find your way through the maze. The mazes should be automatically generated and of varying size, starting with super-simple 3×3 grids. The player will use the arrow keys to move around. The program will use sound to indicate when the player tries to move in a blocked direction (“umph” as in video games) and when he moves toward or away from the goal. Perhaps the program can have subtle spatial audio cues. It also might be nice to have players find their way back after going. Besides being fun, this game would teach spatial listening and mental mapping skills that are critical for survival of people who are blind.
Audio “map game”
Make a tactile overlay of a school —specifically Andrews Elementary in Burlington or the Governor Morehead School in Raleigh. (Diane can make the tactile map.) When the student taps his finger on a specific tactile marker, then the computer will activate the sound for that area. (Example: Velcro marker indicates the school office. When the student taps on the Velcro square, he will hear the office sounds (such as the telephone ringing). With Andrews, we could use the surround sounds that Gary recorded in that school. Next level would be for the student to “move” down the hallway using a joy stick or arrow keys. As the student moves, (possibly adding the “tick-tock” sound of the cane tapping when the student moves?) he will hear the sounds of the various rooms/intersections that he is passing. Basically, this would be a virtual environment – but one that is familiar to the student and one in which he can check his route using the tactile map. This has fantastic potential for teaching spatial concepts (real life and virtual) and would be much more motivating than typical map work! Project will use Intellikeys keyboard (looks like an ordinary keyboard to your program).
Audio/Video editing for kid with CP to do a radio show
He wants to interview NC musical artists and then edit the result into
a show with selections from their music. How to make an editing system
switch accessible?
Augmented Communication on a PDA (taken)
People who have cognitive or physical impairments that prevent them from speaking can sometimes be helped with an augmented communication device. For example see the DynaVox We could emulate most of the functions of these expensive devices ($7000) with a program running on an inexpensive PDA ($300) such as the Sharp Zaurus. Some discussion of using the open source text-to-speech engine flite on iPod Linux
Dance-Dance Revolution!
We are using DDR mats as the basis for games for children who are blind but there are tons of other possibilities. Ann Maloney from the med school is using DDR to help kids on medication avoid gaining weight. Problem is the commercial game is too hard for some kids and loses it attraction for others after a short time. If we had a programmable version of DDR we could change it up to meet the needs of more kids. A version available in source(?) appears to be here. And this one is in Python.
Here is a system and paper that attempts to automatically generate steps. http://www.monket.net/wiki/dancing-monkeys
Also, what could we do to make DDR accessible and fun for kids who are blind?
Finger-spelling enabled word processor
Karen Erickson has some kids who are hearing impaired, visually impaired, and have mobility disorders. She’d like a simple “word processor” that allows scanning through LARGE letters using one or two switches with displays of the American Sign Language “finger spelling” for each letter. This way kids could learn to associate the finger spelling with the symbols.
Joystick interface to Hark the Sound
Some kids can’t use the keyboard or an Intellikeys to control Hark the Sound. What could we do with a conventional game-controller type joystick.
Make a BATS map of Sitterson Hall
The goal is a lobby display with visual and audible maps of Sitterson. It would make both a cool demo of our work and an aid to visitors who are blind. We have building plans in machine readable form. So the first step would be to convert them into something readable by BATS. Recording sounds around the building to augment the map would be a cool addition.
Medievia
Brad Moore says:
There is a widely-used (300–500+ people at any moment) online text-based RPG game called Medievia. I use the term RPG (role-playing) loosely, players don’t actually “role play”, but it is a fantastic/medieval setting.
So, my idea for this project would be to make an accessible client for the game.
I think this might be a worthwhile project for the following reasons:
(1) The game is free, and the quality of the game is quite high (it is still being actively developed by a decently-sized group of professional programmers).
(2) There is no accessible client that I know of. The game is currently played by “sighted” players. An accessible client would allow blind players to have an equivalent game experience, and play along with sighted players.
(3) Play of the game has some real-time constraints. I’m not aware of exactly what interfaces are available for the blind, but I don’t think a simple screen reading approach would cut it. One approach would be to make use of multiple parsers to filter data (could this be useful in other applications? Kirstin said that some blind students know Perl?). I know that current players do make use of some scripting, and that current clients support very simple scripting. The game is text-based, but does make use of an ASCII map. Displaying the map data could also be a challenge.
The website for the game is http://www.medievia.com.
Raleigh Little Theatre Web site
Susan Corrin is looking for help to make their web site accessible. A good chance to learn about web standards and accessibility.
Sound Awareness
A system to help people who are deaf become aware of sounds around them while they are using a computer could be interesting. Perhaps a prototype could “decorates” the frame of the active window with a display giving information about environmental sounds. Or, perhaps the display is attached to the edge of the screen. Using a pair of microphones (stereo sound in) you could likely even indicate the direction to the sound.
Street Crossing Simulator.
Diane Brauner wants a street crossing simulator for kids who are blind. She helps train kids in six local school systems in how to get around their campus and community. Some of the kids do well until they start dealing with traffic. Some of them are too frightened by the traffic to learn. Diane believes that a simulator could help accustom kids to the situation in a safe place so they can learn better in the field. The resulting system should have nice spatial sound (easy with libraries we have in house), varying amounts of traffic, perhaps recorded sounds, and a simple user interface for the teacher and student. We found a web page about something similar to this in Oregon with “VR” hardware. We want a solution using commodity hardware; it should run on a Windows laptop or desktop computer. This is a continuation of a project from Spring 2004. They almost got there but one of their team dropped out of school late in the project and that crippled their final product. You can start with their code and provide some of the additional features that Diane would like or reimplement based on the lessons they learned. Don’t be afraid of the spatial audio component; we use the FMOD library that provides an easy to use programming interface. Diane Brauner has some students who would love to work with us on this.
Switch-accessible browser plugin
People with motor problems often can’t use a mouse or other pointing device. Sometimes all they can reliably do is operate a switch. This severely limits their access to web pages. What we need is a simple plugin for a web browser that sequentially scans the links by highlighting them one after another. When the desired link is highlighted the user presses a switch to select it.
Brett Clippingdale developed the Hawking Toolbar for Firefox last year. Start with his extension and make it into a complete product.
Switch Accessible Games
I got to meet Jake and his mom. We believe some interesting switch assessible games would be a great motivator for him to learn to use his devices. Ideas we came up with include:
- Castle of the Winds a simple “adventure” game.
- A basketball game with some simple physics. You set up your player, set the angle, how hard to throw the ball and then you get to see it shoot. 3D could be cool here but not mandatory.
- “Scorched Earth” the old classic game where you aim your cannon and fire hoping to destroy the computer’s cannon before it gets you.
- How about a bot for playing an online multiplayer shooter that allows Jake to exert control over what is happening but frees him from having to quickly aim and shoot? The robotics crowd calls this “supervisory control” and consider it for situations where the operator can’t control a remote robot in real time (for example on Mars). This could make a cool research project.
- Tic-Tac-Toe.
- Nibbles the classic “Snake” game where you steer a snake to “eat” while avoiding running into barriers or himself. Slow this down and it could be fun and help with learning to control the buttons.
Typing Tutor
Look at Typing the Dead for ideas for how to make a typing tutor fun.
Visual Neglect
Therapy/game for stroke victims with Visual Neglect.
Web Cam CCTV
People with impaired vision often use CCTV devices to enlarge printed text. How well could we simulate this with a cheap web cam, a milk crate, a strip light, and some software?
Essential Tremor Filter
Why is this mouse filter, described in the NY Times and here at the company website implemented in hardware? Shouldn’t this be doable in software?
Web-based games for kids who are VI
I read a nice paper about a project of the Swedish Library of Talking Books about their Flash games for kids who are visually impaired. How about a version of Hark like this? Would Flash or Java-Script be more portable?
Research Projects
Alternatives to Braille Embossers
Making a Braille embosser is really hard for CS types. We make software. So how can we use commodity devices and software to help children learn to read Braille? The method teachers use now is “make a letter—feel a letter”. The child writes by pressing the keys on the embosser and reads by feeling the result. The is comparable to sighted kids making the same letter over and over.
Could we turn that around? Perhaps we could use an Intellikeys keyboard or a cheap touch tablet with an embossed overlay. We could have an array of letters embossed on the overlay. In the software we know where the child is touching so we can say the letter being touched. Could we extend this to writing with some automatic guidance to help the child find the letters? The idea is for young children to have access to cheap and interesting Braille learning tools.
Augmented communication on a cell phone
Augcom devices sometimes are not used by people who need them because they make them stand out from their peers. Modern cell phones have all the computing and audio output capability necessary to make discrete and capable augcom devices.
GameBoy Advance
What could we do on the GameBoy Advance platform? An AugCom device? A game for kids with Autism? TEACCH schedule?
Games for kids with cognitive impairments
Most video games are too hard for kids with cognitive difficulties. About the only approach currently available is to use things like “Game Genie” to “cheat”. We’d like an interesting and visually attractive computer game that emphasizes memory and has variable levels of difficulty. This will require an imaginative team willing to do some experimentation and willing to work with potential game players to get ideas. Of course, there are many kinds of impairment and one game will certainly not work for everyone. Our goal will be to make a game that is fun for one or two kids and see if it appeals to a larger audience.
Joint Projects with Speech and Hearing Sciences
Karen Erickson is teaching a project course this semester for students in Speech and Hearing Sciences. There might be some possibilities for joint projects.
Making Computers More Usable
A fellow called looking for help for his father who is developing cognitive problems but still wants to use his computer. What kind of adaptive computer interfaces would help?
- more guidance?
- a simplified user interface?
- more visual “history”?
- are words or pictures better?
- kill accelerators?
- explain actions? “Why did you do that?” Enrichments?
- describe state and what we were doing?
He suggested a system that first tests the user’s ability and then adapts to their limitations. He also reminded me of the old Bob interface from Microsoft which was supposed to be simpler. Reminded me a little of Jackson Fox’s project.
There is a large and rapidly growing population for which this kind of thing will be important. Relates directly to MERL‘s interest in appliance control interfaces. Some people call this Cognitive Prothesis but that term also seems to apply to what we call Intelligence Amplification.
Mental Models
What kinds of models do people who have always been blind have for space, scale, projection, etc.? How might computer aids or specific lessons address changing that model? Research this topic and report on your findings. Develop a prototype if appropriate.
Phonetic Spell Checker
People often notice mistakes in their writing when they look at a word and notice it is ‘not quite right.’ What if you cannot see the word spelled out? How can you know a word is spelled incorrectly when a screen reader speaks things phonetically? For instance, ‘nock’ and ‘knock’ are pronounced the same by JAWS, but mean totally different things. In most cases, the first is most likely a spelling mistake. Research and build a spell checker that parses text based on phonetic spellings and indicates when a word might be spelled incorrectly.
Recursive Zooming
What does it mean to ‘zoom in’ without sight? Can users without vision effectively zoom in and out of parts of images, maps, and diagrams to get more and less information? Research methods for using the numeric keypad with its directional layout to zoom into and out of certain regions on the screen. Try to develop an interface that allows navigation both in and out of the screen. Such a development would be very useful in displaying maps to the blind.
Sample-based 3D Audio
I have just begun to investigate the possibility of doing for 3D audio what image-based rendering does for computer graphics. I have an 8-microphone array and can record sound simultaneously at up to 96kHz with 24 bits per sample. I’d like to process the recorded sound to produce the parameters of a random process that produces sound with similar spatial and temporal statistics.
SoundsLike
Develop a communication system that visually shows speech in some form to people who are deaf without relying on a full-blown and often faulty speech recognition system. One idea would be to break the speech into phonemes which could the be watched and assembled into words by the user.
Sound Synthesis
Pete says that he has read a few papers by Gaver (audio icon guy) about synthesizing natural sounds like breaking glass, water drops, impacts, etc. given physical parameters. Supposedly people (including him) put together software libraries that implement his ideas back in the early 90s, but I’ve yet to find one that still exists and is in working order. Having a library like this would be great for applications that want to have fine-tuned control over the sound being produced in order to convey more information. For instance, the density of rain drops hitting a surface might give information about how close a download is to completion. Likewise, the rate and rythmn of a machine sound might indicate a level of activity.
Having tools for this would be great for projects like our Street Crossing Simulator. We could synthesize the car sounds and thus have complete control over where they appear in space.
Read more about SoundSynthesis.
Staying Connected
A way for people with disabilities to be connected to the rest of the world. Young adult with CP in adult day care. How can we help him? How can he be productive?
Switch accessible concept mapping
Concept mapping programs are important literacy tools used by many schools. They are currently inaccessible to people who are only able to use one or two switches for input.
You can get a trial copy of concept mapping software at http://www2.smarttech.com/st/en-US/Products/SMART+Ideas/Free+trial.htm.
Switch accessible literacy software
One way the folks at CLD teach literacy is to have children rearrange letters to make words. For example “what letter do you have to change to make ‘pin’ into ‘pit’?” How to make this accessible to switch users? Talk to Karen and Sally about what might work.
Temporal user interfaces
Computer interfaces are mostly sequential. Consider telephone menu systems: enter 1 for parts, enter 2 for service, etc. As another example, when you kill an unresponsive program, Windows XP pops up a dialog asking me if you want to send an error report to MS. You must respond to it before proceeding. An alternative user interface strategy (for both sighted and blind) depends on asynchronous alerts and user responses. Think of the underlining of misspelled words in many editors; it occurs sometime after typing and can be corrected (or not) anytime. Emacspeak has some nice features like this. The presence of a footnote associated with a word is indicated by a audible signal played along with the speech for the word without stopping. The listener can respond to the signal by requesting the footnote be followed or ignore it. A project investigating what is known about asynchronous user interfaces and perhaps a prototype implementation would be really interesting and likely result in a paper.
Text Skimming
Skimming in text for people who are blind or reading help for people with learning disabilities. Automatic summarization or maybe Cliff Notes merged with a book so that you can read at the summary level or down at the detailed level. How can students who are blind skim for answers to questions? Can this help people with learning disabilities? Sort of Tell what you’re going to tell them, tell them, and tell them what you told them.
Universally accessible video games
Some forms of entertainment, video games for instance, cannot be accessed by persons with disabilities. Build a video game that can be enjoyed by anyone. A game that can be used in an educational setting for young children would be excellent. Multiplayer games would be interesting, especially ones that involve cooperation among kids with different abilities.
Web Cam Tracker
Could we make a very coarse resolution tracker using a web-cam and some software? Even sensing four directions would be very useful. How about gesture recognition?
Hardware Projects
Ceiling-display augmented communication device
Many people are permanently or temporarily unable to speak. Examples include:
- hospital patients who have a trachea tube or a broken jaw,
- people who have had a stroke, or
- people with CP, ALS, traumatic brain injury, etc.
It can be particularly stressful for a hospital patient who needs to participate in important health-care and quality-of-life decisions to be unable to communicate.
Devices have been available for many years to enable people to communicate without speech. These range from simple aids such as pointing to words or pictures on a message board and writing by hand to high-end systems with dynamic displays that produce synthetic or recorded voice messages when the user selects a word or symbol.
These devices are rarely available to people when they are lying down (e.g. hospital patients or anyone at night). They are hard to see, hold, and operate while prone.
We could combine a simple communication device with a display that projects onto the ceiling (like those clocks that display the time on the ceiling). In the simplest case it might display only “yes” and “no” and allow a user to select between them. Extension to multiple words, phrases, or even individual letters would be easy. Synthesized or pre-recorded speech and alert signals could be easily added.
This device is about the size of a clock radio and sits beside the bed. It has standard 1/8” mono plugs for connecting a variety of switches. The projector is normally off and is quiet when in operation. It displays single color, high-contrast text on the ceiling. In “scanning” mode it cycles among the choices until the user
makes a selection with a single switch. In “step” mode the user has two switches, one to move among choices and another to select one. Upon selection an audible or visual display is generated for the communication partner.
Cheap Wheel-based Braille Display
Most refreshable Braille displays are constructed with pin-grid arrays like these:
The pins stick and the refreshable cells are expensive.
Researchers at NIST have devised a solution that may be cheaper to make.
Could we make something even cheaper? Karen Erickson suggested a single disk with Braille characters embossed on the edge. This could be done using a manual label maker. Now all we have to do is quickly rotate the disk so that the correct character is on top where the user can feel it. It seems to me that a really cheap single character Braille display could be built this way. Little kids learning Braille could really benefit from such a thing.
Tactile Pen for a TabletPC
It would, I claim, be pretty easy to make the pen be a tactile display device by coupling either a tiny magnetic speaker or one of the vibrator motors to the top of it. The required wire might actually be a benefit to blind users because it connects the pen to the tablet so it can’t be dropped. A speaker could be driven with sound but we need the sound output for audio display. Easiest for a prototype would be to rip up one of the old iFeel mice to get the small vibrator and usb driver that is inside. Attaching this to the top of the pen should be easy for a quick prototype.
More to come