Historically, only expensive computers have this ability as it is very difficult to engineer correctly.
Machines that support hot swap need some ability to detect a component has been removed. In addition all electrical and mechanical connections need to be designed such that neither the part nor the user can be harmed by removing it. Lastly, other parts in the system must be designed such that the removal of a different part does not harm operation. (Often there is some automatic recovery process)
Simple implementations often require a component shut down procedure prior to removal. This simplifies implementation, but such devices are not robust in the case of part failure. Modern small computers often have USB and/or IEEE 1394 buses that support this simple hot swap. If a device is removed while it is being used, the operations to that device fail and the user is responsible for retrying if necessary. As this action is normally done by end users this is not considered a problem. These hot swap operations are generally used either to move a peripheral from one computer to another, or to allow a device to synchronize data with a computer.
Complex implementations may recommend that the part be shut down, but there is sufficient redundancy in the system such that if a part is removed without being shutdown operation continues. In these systems hot swap is normally used for regular maintenance to the computer, or to replace a broken part.