A Virtual Stringed Instrument With Haptic Feedback

Jeff Feasel

Submitted as a course project for Ming Lin’s COMP 259

Overview Of Project

    I used a digital waveguide model to simulate a plucked string.  I implemented the classes for the various filter components myself.  I used a C++ library called the Synthesis Toolkit -- see [STK] -- as a higher-level insterface to Microsoft DirectX's realtime sound output.  I created an environment for controling several of these string simulations that provides haptic feedback using a Phantom device.  I used the Ghost SDK for interfacing with the Phantom, using a general-purpose forcefield object to represent the forces of each string.  I used a the computer keyboard to control fretting and the Phantom to control plucking.  I created a crude, but functional display of the system using OpenGL.

Background

    The behavior of a string is guided by the wave equation which, in the ideal case, is a 2nd-order PDE over displacement.  Solution to this equation can be found by integration using standard finite-element methods, however sounds audible to the human ear range above 20kHz, requiring a sample rate of more than 40,000 samples per second to reconstruct frequencies components at the top of this range.  This requires a density of spacial samples that is prohibitively expensive to perform computations on in real time, and as a result physically based methods are not usually employed in commercial sound synthesis.

    A solution was proposed by Julius Smith (see [SMITH 92]), who noted that the wave equation is solved by any pair of identical waveforms that start in the same position and travel in opposite directions with a speed that is a certain function of the density of the string and the tension.  When these waveforms reach a rigid termination (i.e. a boundary condition of 0 at each end of the field) they are inverted and reflected back in the other direction.  Smith's idea was that, since we only care about the height at a few fixed points on the string over time, and since the wave components behave predictably between these points, we should only evaluate the wave at these fixed points and use filters to describe what happens to the waves between these points.  He models the system as a digital signal processing circuit, using digital delay lines (a queue-like structure used to model a fixed-delay component) and frequency-based decay filters.  Smith explains how this model can be used to include lossyness at the terminations and along the length of the string.  A description of the model can be found in my slides.
    Several improvements have been made on this method as it relates to plucked string synthesis.  [KVT 98] simplifies the DSP circuit by combining filters and in some cases eliminating those with a negligible effect on sound.  Another paper by the same authors, [TVK 00], describes extensions to model some of the non-linear behavior of the plucked string.

My Plucked String Model

    I started with Smith's waveguide model, including the extensions for the lossy wave equation.  The first diagram below shows the circuit for this model.  I commuted and combined the components betwen the excitation points ("in") through the fret termination and between the pickup points through the bridge termination.  This cuts the number of components in half, as shown in the second diagram below.

Original circuit

Simplified circuit

    The decay filters representing losses along the length of the string are labelled with "S" and the decay filters representing losses at the fret and bridge terminations (since they are non-rigid and therefore lossy) are labelled by "F" and "B".  The top portion of the circuit indicates right-travelling wave components and the bottom portion of the circuit indicates left-travelling components.  The sides of the circuit (in the upper figure) represent filtering due to the terminations.

    The initial conditions are fed as an input signal to the system, with half the initial waveform being sent leftward at the lower input position and the other half rightward at the upper input position.  The output is measured as the sum of the displacement waves at the two pickup points.

    To cut down on the amount of computation time spent with the signal processing, I made the losses in the middle two segments zero and lumped all the frequency dependent losses at the endpoints.  For the endpoints, I chose arbitrary low-pass filters.  These filters are implemented using small convolution kernels.

    In order to dampen the string, I placed a constant decay filter right before each of the pickup locations.  These filters are normally disabled, but when a "DampingOn" event is passed to the synthesis object, these filters are enabled.  The gain coefficient of these filters is only slightly less than 1, so this does not mute the wave immediately (which would cause an undesirable popping noise in the sounnd output) but allows it to decay smoothly.  The time to come to rest (or within inaudible tolerance to the resting position) is about 1/5 of a second, which is consistent with the behavior of a real guitar string.

Picking (The Right Hand)

    To excite the string, the right hand plucks it with a pick:  usually a small somewhat pliable triangular piece of plastic.  The string is drawn with this pick and the shape of the string just before it is released is the initial value of the system.  In my program, this is a triangular shape with a flat peak.  The width of this flat peak depends on the angle at which the user holds the pick.  I found that for peak widths varying near the width of the pick, there was not much perceptible variation in sound.  My choice of incident angle for the input parameter that controls peak width was somewhat arbitrary, but it doesn't appear to have much effect on the output anyway.

Pick-String Interaction

    The pick is held firmly between the fingers and drawn accross a string.  While being drawn the displaced string exerts a restorative force on the pick, opposite to the direction of displacement.  I assume this force to be spring-like:  proportional to displacement by a constant factor.  As shown in the diagram below, the component of the force that acts perpendicular to the pick causes the pick to rotate in the user's hand.  The pick's rotation causes some deformation of the user's fingers, which exert a force back on the pick to try to restore it to it's original orientation.  The other component of the force between the stiring on the pick -- that which runs along the length of the pick -- works to slide the string off the end of the pick.  Because the width of the pick's face varies at different locations, the friction coefficient varies as well.  Additionally, the pick itself bends under the pressure, making the string's release possible.

Simplified Pick-String Interaction

    The interaction between pick and string is rather complicated and involves forces being applied at both ends of the pick.  It also results in force feedback with both a rotational and translational component.  The version of the Phantom that I was working with does not supply rotational force feedback.  Though there is a Phantom device that does, the computation of the interaction is somewhat involved (and expensive).  Plus, a solution that uses the cheaper, more common 3DOF Phantom is probably more useful.
    The details of the pick and the string's interaction is important because it determines how far the string can be displaced before it slips off the end of the pick and is released.  This is important because variation of this so-called "release displacement" is necessary in creating the proper feel of the string, and it controls the amplitude of the excitation being produced.

   I pick out two factors that have the greatest bearing on this interaction and try to come up with a simplified model of it.  The "penetration depth", or how far down the tip of the pick extends beyond the string plane (the plane in which all the strings lie) is important because it determines how much of the pick the string must slide along before it slips off the tip.  I normalize this value to be in the range of 0 to 1 (1 being where the pick goes so far as to touch the instrument's body).  The incident angle (the angle at which the pick is incident to the string plane in the direction of the pick's movement).  I make the assumption that this is always between 0 and 90 degrees (any more and we would be "scooping" the string up away from the instrument body) and clamp that angle to this range.
       The release displacement is then calculated to be the product of the normalized penetration depth and the sine of the incident angle (thus yielding a value between 0 and 1) times the maximum displacement allowed for that string.  For a cheaper computation, one can easily use x/90 as an approximation to sin(x) in the range 0 to 90.
    This simple formula has one constant parameter, max displacement, which I set experimentally to a value that feels right, and allow the parameter to be changed in the UI.

    The true displacement is taken to be the distance between the string and the tip of the pick -- as if the string is somehow hooked upon the tip of the pick.  The displacement used in calculating the string's restorative force, however, is only the horizontal component of this force (the component in the direction of strumming).  The restorative force is applied in the direction of the true displacement, though.  This avoids the need to compute the location of the intersection of the pick with the string plane and the exact spot where the string touches the pick.  The horizontal displacement component is checked against the releasae displacement to determine if the string is relased or not.

Specific Issues With The Phantom Hardware

    The picking and strumming on a guitar mostly involves motion accross the strings -- rather than up and down the length of the string or towards and away from the fretboard.  So of the various orientation I tried, I found the side-to-side direction of the phantom to be best suited to the strumming direction.  The Phantom has the largest physical range in this dimension, thus making it suitable for very wide string spacing or numerous strings.  Motion in this direction also involves primarily the use of the first joint (the horizontal shoulder joint) of the phantom, which seems to be the smoothest of the joints and offer the least resistance, which is essential to playing accurately.

    The orientation of the stylus determines the orientation of the pick.  One of the drawbacks of the Phantom is that the orientation of the stylus is dependent on the position of the arm.  I couldn't find any means in the API to get the stylus's orientation relative to the global axes.  So to obtain the orientatation of the stylus, one would have to perform 3 rotations based on the state of the arm's 3 joints before being able to apply the angles that describe the stylus's state.  To avoid this, I assumed that the picking would be done mostly in a region located a fixed distance down the +z axis (which is perpendicular to the back wall of the device and towards the user) so that the arm is almost fully extended.  In this position the state of the 3 joints of the sylus happens to coincide with rotations relative to the global xyz axes.  Though there is a slight discrepency as the arm is moved away from this ideal position, the difference is negligible to the feel and sound of the instrument.

Fretting (The Left Hand)

    Fretting is pushing the string down against the fretboard at a certain location in order to change the length of the string.  There are a number of discrete locations along the string's length where this is done.  The act of pressing the string down at one of these locations does impart a small amount of energy to the string, but this kind of excitation is inaudible when done in conjunction with picking.  Only in very specialized styles of guitar playing is fretting alone used to excite the strings (this is called "tapping").  As a result, excitation due to fretting is ignored.  In this model, fretting simply changes the length of the string, and thus the length of the delay line.  The act of fretting usually mutes the string, so the delay lines in the sound synthesis circuit are zeroed out in this event.  However, in adapting the limited forms of left-hand input that I had available to make the instrument as easily playable as possible, I found that in some cases it was better to zero the delay lines when the pick strikes the string, rather than when the fret is chosen.

The Keyboard Interface

    The computer keyboard is a logical choice for choosing which fret to depress.  I wrote several versions of the fretting interface to test out different ways of mapping the keys to various fretting actions.  These interfaces can be divided into two styles:  Bass-style and Guitar-style.  A bass has 4 strings and typically plays only one note at a time.  So some versions of my bass-style interface would mute all other strings when a new string is struck.  I found that the best solution in this case was to map a single row of keys to the possible fretting locations along a string, and assume that all strings were fretted at the same length simultaneously even though only one string is being played at a time.
    The guitar has 6 strings which are often strummed simultaneously and fretted at different locations to form a chord.  This would require 6 times 15+ different fret locations, which is clearly unreasonable for a 1-to-1 mapping with the keyboard.  So I chose to assign a single key to each of the 12 Major and the 12 Minor chords.  Pressing a certain key causes each of the 6 strings to be fretted at the proper location to produce this chord.  I arranged these keys in the order of the "Circle Of 5ths", which provides an ordering of each set of 12 chords based on a certain harmonic relationship.  According to music Theory, the chord changes made in modern and traditional music have a high spacial locality within the Circle Of 5ths ordering.  Thus, successive chords in a song will likely correspond to nearby keys on the keyboard.  This prevents the user from having to "fish around" to find the right key.  This ordering is commonly found in accordians and other instruments in which a single key is used to play a whole chord.

The Outcome

    The resulting instrument is playable.  The feel of the strings is reasonably good.  Several people in the department (musicians and non-musicians alike) have tried this instrument out, and for everyone involved it took a few minutes to get the hang of it.  It is somewhat awkward to play -- personally I am a bass player and I still can't manage to play more than simple tunes on it yet.
    The big problem is latency.  When pressing the keyboard there is a delay of (I estimate) up to 0.25 seconds.  The update rate for the haptics remains high enough, and the sample rate of the sound being output remains steady at 22kHz.  This latency in the keyboard input is the result of a tradeoff between the time OpenGL gets and the time spent computing the sound output.  The sound output must be maintained steadily at 22kHz, so this is done at the expense of latency in the fretting.
    All the performance enhancements I've done so far have already helped bring the latency down steadily.  I'm sure eventually the latency will reach an acceptable  level.

Future Work

    Though the simplified pick-string model seems to produce very good results, I can't say for sure that a more accurate model -- even if not completely physically accurate -- would make this instrument easier to play and allow the user more control over the sound.

    There are several more advanced playing techniques that are commonly used among experienced musicians.  Sliding the fret finger along the fretboard to change the length of the string without damping it produces a somewhat continuous change in the pitch of a note that has already been struck.  There are several means of changing the tension of the string after a note has been struck, which change the velocity of the wave, thus changing the pitch of the sound.  Both of these techniques involve changing some properties of the string after the initial excitation has been made.  This is violates the time-invariance assumption of the waveguide model for sound synthesis.  However, I've done some experimenting with changing the delay line length while the waveform is still resonating in the string, and it produces an effect that sounds almost indistinguishable from real "sliding" or "pitch bending" techniques, despite the fact that my method of changing delay line lengths involves crudely dropping or adding samples to the delay quueue without regard to its effect on the spectrum of the waveform.  I haven't yet incorportated this pitch bend feature into the current Phantom-driven version of the system -- as it was one of my earlier experiments and the other subsystems have changed so much since then.  It poses somes issues of control that will take some time to solve.

    Rather than using the keyboard to select frets, I'd like to choose the fretting location based on where along the length of the string the user plucks.  I'm not sure if that would make the instrument more playable or not, but I think it would be an interesting variation to try.  I'm also interested in using the lateral position of plucking to change other properties of the sounds, such as the kind of decay filters that are applied in the circuit.  Basically, leaving aside physical realism, finding the parameters that have the most significant effect on the output, and mapping these parameters to some of the degrees of freedom of the input device, thus allowing the user as much control as possible.

    I think that arranging the instrument in a way that's more like a real guitar would help users become accustomed to it faster.  The virtual strings don't run in the same direction that a real guitar strings would run in relation to the body.  Arranging the phantom backwards and rotated 90-degrees would make its motion correspond to that of a real guitar.  Similarly, the keyboard would have to be somehow tilted up 90-degrees away from the user, and the user would reach around underneath it to do the fretting.  Also, trying other forms of input for the fretting might be a worthwhile endeavor.

    On the sound-synthesis end of things, I'd like to try implementing dispersion through the use of frequency dependent delay filters.  The sound I have now is pretty convincing without it, but I could still tell the difference between my instrument and its real-life counterpart.  I didn't do much experimentation with different damping filters at the terminations and along the string itself.  The values I picked -- somewhat arbitrarily -- ended up making the instrument sound like a nylon-stringed guitar.  This is fine, but I'd like to see what other kinds of timbres I can make with the waveguide system.

Acknowledgements

    Much thanks to Bill Baxter for his PHANToM expertise, and Sharif Razzaque for his practical advice.

Code

References