Interaction diagrams are models that describe how a group of objects collaborate in some behavior - typically a single use-case. The diagrams show a number of example objects and the messages that are passed between these objects within the use-case.
I'll illustrate the approach with the following simple use-case. In this behavior the order entry window sends a prepare message to an order. The order then sends prepare to each order line on the order. The order line first checks the stock item, and if the check returns true it removes stock from the stock item. If stock item falls below the reorder level it requests a new delivery.
Interaction diagrams come in two forms, both present in the UML.
The first form is the sequence diagram. In this form objects are
shown as vertical lines with the messages as horizontal lines
between them. This form was first popularized by Jacobson. The
diagram below shows this form in its UML notation. The sequence
of messages is indicated by reading down the page.
Figure 1: A sequence diagram.
The second form of the interaction diagram is the collaboration diagram. Here the example objects are shown as icons. Again arrows indicate the messages sent in the use case. This time the sequence is indicated by a numbering scheme. Simple collaboration diagrams simply number the messages in sequence. More complex schemes use a decimal numbering approach to indicate if messages are sent as part of the implementation of another message. In addition a letter can be used to show concurrent threads.
Figure 2: A collaboration diagram.
I have mixed feelings about these trends towards greater executability. To me, the beauty of interaction diagrams is their simplicity, and much of these additional notations lose this in their drive to computational completeness. Thus I would encourage you not to rush to the more complex forms of interaction diagrams, you may find that the simpler ones give you the best value.
Interaction diagrams should be used when you want to look at the behavior of several objects within a single use case. They are good at showing the collaborations between the objects, they are not so good at precise definition of the behavior.
If you want to look at the behavior of a single object across many use-cases, use a state transition diagram. If you want to look at behavior across many use cases or many threads, consider an activity diagram.
Most books on object-oriented modeling discuss some form of interaction
diagram. Notations are currently very varied, but they should
settle down to the UML in due course. A safe suggestion is that
of [Booch] which gives you a simple
outline, which is really all you need. [Buschmann et al]
use many of the extensions that are currently in the mix for the
UML, and thus should give you a good idea of what is in store.