IPSec is a security protocol that is used to secure end to end communication between two computers over any network. There are various ways to implement IPSec and because it operates at the network layer it has some advantages over SSL. IPSec uses two encryption protocols (AH & ESP) that can each be run in two different modes (Transport & Tunnel).
  • Authentication Header (AH)- AH assures the receiver that they are communicating with the right machine. It does not provide any confidentiality, it simply verifies that the sender is who they say they are. It also assures the receiver that the data has not been changed in transit by signing the entire packet. This secures the packets against any replay attacks.
  • Encapsulating Security Payload (ESP) - ESP encrypts the payload of the packet (the data being sent) and assures confidentiality. However, it does not sign the entire packet.
In most cases systems will use one protocol or another. Very rarely would both protocols be used. In the event that both protocols were used, two seperate Signing Authorities would be necessary.

Transport Mode- Transport mode means that only the payload will be encrypted. This will speed up the encryption/decryption process because the headers will not change. However, this would allow anyone with a packet sniffer to at least know who is sending and receiving the packets. Transport mode is used for host to host VPN's.

Tunnel Mode- Tunnel mode means that the entire packet will be encrypted. This limits the ability to find out any information about senders or receivers, however it slows the data transfer considerably. Tunnel mode is used any time a VPN goes through a gateway.

For IPSec to work both computers must know that they are supposed to use IPSec. The computers will be configured with a pre-shared key that will allow them to communicate securely. Before communicating, both computers will verify their identities by exchanging hashed values of the key. If the hashes are validated by each machine they will have established non-repudiation and begin their IPSec session. It is possible for IPSec to work without a pre-shared key. In such a case a Signing Authority would come into play.

A Signing Authority would negotiate the encryption key to be used for the IPSec session using Internet Key Exchange. This method may be preferred because the key is negotiated for a short time span and only when necessary.

IPSec is a secure way to establish a fast authentication VPN or a slow confidential VPN at the Network Layer. Since it operates at Layer three it can be easier to implement because it merely deals with packets and not applications. SSL can be harder to implement because applications will have to be configured to be compatible with SSL's protocol. However, IPSec is harder to distribute on a larger scale and many experts agree that it can become unmanageable.

Another downfall of IPSec from the administration standpoint is that IPSec grants full access to network resources once VPN access is granted. SSL is easier to administrate because it provides access to applications and not network resources.

As with any protocol IPSec has pluses and minuses and they should all be weighed before implementation.