Each IDE/ATA channel can support either one or two devices. IDE/ATA devices of course each contain their own integrated controllers, and so in order to maintain order on the channel, it is necessary to have some way of differentiating between the two devices. This is done by giving each device a designation as either master or slave, and then having the controller address commands and data to either one or the other. The drive that is the target of the command responds to it, and the other one ignores the command, remaining silent.
Note that despite the hierarchical-sounding names of "master" and "slave", the master drive does not have any special status compared to the slave one; they are really equals in most respects. The slave drive doesn't rely on the master drive for its operation or anything like that, despite the names (which are poorly-chosen--in the standards the master is usually just "drive 0" and the slave "drive 1"). The only practical difference between master and slave is that the PC considers the master "first" and the slave "second" in general terms. For example, DOS/Windows will assign drive letters to the master drive before the slave drive. If you have a master and slave on the primary IDE channel and each has only one regular, primary partition, the master will be "C:" and the slave "D:". This means that the master drive (on the primary channel) is the one that is booted, and not the slave.
Devices are designated as master or slave using jumpers, small connectors that fit over pairs of pins to program the drive through hardware. Each manufacturer uses a different combination of jumpers for specifying whether its drive is master or slave on the channel, though they are all similar. Some manufacturers put this information right on the top label of the drive itself, while many do not; it sometimes takes some hunting around to find where the jumper pins are on the drive even once you know how the jumpers are supposed to go. The manufacturers are better about this now than they have been in the past, and jumpering information is always available in the manual of the hard disk, or by checking the manufacturer's web site and searching for the model number. I describe (and illustrate) the jumpers on IDE/ATA disks in detail in the section on hard disk construction. For a fundamental description of what jumpers are, see here.
ATAPI devices such as optical, Zip and tape drives are jumpered in pretty much the same way as hard disks. They have the advantage of often having their jumpers much more clearly labeled than their hard disk counterparts. Most optical drives, for example, have three jumper blocks at the back, labeled "MA" (master), "SL" (slave) or "CS" (cable select).
If you are using two drives on a channel, it is important to ensure that they are jumpered correctly. Making both drives the master, or both the slave, will likely result in a very confused system. Note that in terms of configuration, it makes no difference which connector on the standard IDE cable is used in a standard IDE setup, because it is the jumpers that control master and slave, not the cable. This does not apply when cable select is being used, however. Also, there can be electrical signaling issues if one connects a single drive to only the middle connector on a cable, leaving the end connector unattached. In particular, the use of Ultra DMA is not supported in such a configuration; see the discussion of the 80-conductor Ultra DMA cable for more information.
As long as one drive is jumpered as master and the other as slave, any two IDE/ATA/ATAPI devices should work together on a single channel. Unfortunately, some older hard disks will fail to work properly when they are placed on a channel with another manufacturer's disk. One of the reasons why drives don't always "play nicely together" has to do with the Drive Active / Signal Present (/DASP) signal. This is an IDE/ATA interface signal carried on pin #39, which is used for two functions: indicating that a drive is active (during operation), and also indicating that a slave drive is present on the channel (at startup). Some early drives don't handle this signal properly, a residue of poor adherence to ATA standards many years ago. If an older slave drive won't work with a newer master, see if your master drive has an "SP" (slave present) jumper, and if so, enable it. This may allow the slave drive to be detected.
Drive compatibility problems can be extremely frustrating, and beyond the suggestion above, there usually is no solution, other than separating the drives onto different channels. Sometimes brand X won't work as a slave when brand Y is the master, but X will work as a master when Y is the slave! Modern drives adhere to the formal ATA standards and so as time goes on and more of these older "problem" drives fall out of the market, making all of this less and less of a concern. Any hard disk bought in the last five years should work just fine with any other of the same vintage or newer.
When using only a single drive on a channel, there are some considerations to be aware of. Some hard disks have only a jumper for master or slave; when the drive is being used solo on a channel it should be set to master. Other manufacturers, notably Western Digital, actually have three settings for their drives: master, slave, and single. The last setting is intended for use when the drive is alone on the channel. This type of disk should be set to single, and not master, when being used alone.
Also, a single device on an IDE channel "officially" should not be jumpered as a slave. In practice, this will often work despite being formally "illegal". Many ATAPI drives come jumpered by default as slave--because they are often made slaves to a hard disk's master on the primary IDE channel, this saves setup time. However, for performance reasons they are sometimes put on the secondary channel, and often the system assemblers don't bother to change the jumpers. It will work, but I don't recommend it; if nothing more, it's confusing to find a slave with no master when you or someone else goes back into the box a year or two later to upgrade.
For performance reasons, it is better to avoid mixing slower and faster devices on the same channel. If you are going to share a channel between a hard disk and an ATAPI device, it is generally a good idea to make the hard disk the master. In some situations there can be problems slaving a hard disk to an optical drive; it will usually work but it is non-standard, and since there is no advantage to making the ATAPI device the master, the configuration is best avoided.
There are many more performance considerations to take into account when deciding how to jumper your IDE devices, if you are using several different ones on more than one channel. Since only one of the master and slave can use any channel at a time, there are sometimes advantages to using more than one IDE/ATA channel even if not strictly necessary based on the number of devices you are trying to support. There can also be issues with using a drive that has support for a fast transfer mode like Ultra DMA with older devices that don't support these faster modes.