IPSec is a protocol suite (a set of protocols) consisting of protocols for securing packet flows, and of key exchange protocols being used for setting up those secure flows. Of the former there are two: Encapsulating Security Payload (ESP) for encrypting packet flows, and the rarely used Authentication Header (AH) which provides authentication and message integrity guarantees for such flows, but does not offer confidentiality. See Information security for definitions of these terms. Currently only one key exchange protocol is defined, the IKE protocol.
IPSec protocols operate at layer 3 of the OSI model, which makes them suitable for protecting UDP-based protocols when used alone. The down side is that compared with transport-layer protocols, such as SSL, the IPSec protocols need to deal with reliability and fragmentation issues, which are normally solved by TCP.
IPSec was intended to provide either (1) portal-to-portal communications security in which the security overhead is provided to several machines (even whole LANs) by a single node, or (2) end-to-end security in which the endpoint computers do the security processing. It can be used to construct Virtual Private Networks in either mode, and this is the dominant use.
End-to-end communication security use on an Internet-wide scale has been slower to develop than many had expected. Part of the reason is that no univeral, or universally trusted, [public key infrastructure]] has emerged (DNSSEC was originally envisioned for this), part is that many users (probably most) understand neither their needs nor the available options well enough to force adoption, and part is probably due to degradation of Net responsivity due to bandwidth loss from such things as spam.
The Free S/Wan project has developed an open source implementation of IPSec for GNU/Linux. IPSec is also bundled with newer versions of Windows, as well as several commercial flavors of Unix, e.g. Solaris. It is included in the 2.6 Linux kernel and so will be widely available as GNU/Linux distributions change over to 2.6.
IPSec protocols are defined by RFCs 2401-2409, currently (2003) these documents are slowly being replaced by newer versions.