The Internet Protocol (IP) was the first way of transferring datagram across users. However, it lacked the ability to authenticate the data being sent. This lead to the creation of IPsec, which is one of the major online security protocols used by organizations.

In this article, you will be introduced to what IPsec is all about and how it works. There are many technical bits and terms, but there will be short explanations to make you understand better as you read.


Internet Protocol Security (IPsec) is a set of protocols (or set conditions/agreement) that provides security between two communicating hosts. The host represents users like you and I. IPsec helps to provide data authentication and encryption.

Due to its ability to provide security, IPsec is sometimes used for setting up virtual private networks (VPNs). IPsec uses a different set of rules (protocols) to carry out its functions and duties. These protocols are the Authentication Header (AH) and Encapsulating Security Payloads (ESP).

IPsec standards

We now know that IPsec uses a collection of techniques and protocols to work. This makes it impossible to make it operate by a single standard. Instead, a collection of standards set by different publications authored by computer engineers and scientists.

These standards define the architecture, services, and specific protocols used in IPsec. It also helps the use of IPsec to stay uniform across the world. These collections of publications are officially known as “Request For Comments” (RFC).

IPsec mode of operation

Tunnel mode involves the total protection of the entire original IP packet by IPsec. IPsec covers the genuine packet (which is a set of data being transferred), encodes it, appends new information about the IP (this information is called the IP header) and sends it to the other side of the tunnel (IPsec peer) which is mostly a VPN tunnel which the other host is connected to. Tunnel mode is most regularly used between gateways, or at an end-station to a gateway.

Tunnel mode is used to secure the data exchange between secure IPsec gateways. In tunnel mode, an IPsec header which can be either AH or ESP is installed between the upper layer protocol and the IP header. ESP is most commonly used in the IPsec VPN tunnel configuration over AH. It is used if one of the Internet Key Exchange (a sort of basic IPsec rules and regulations) peers is a security gateway (a middleman) applying IPsec on behalf of another host.

Transport mode

This mode actually encloses just the IP payload (IP payloads) to ensure secure communication. Think of the information you send as a human, having a head, arms, torso, and legs. The same goes for your data. The tunnel model encloses the whole “human” while transport mode encloses just the body, leaving the head out. Therefore, when transport mode is used, the IP header reflects the original source and destination of the packet. Transport is most often used in a host-to-host scenario, where the data endpoints and the security endpoints are the same.

IPsec protocols

There are two main parts that do the heavy lifting when it comes to IPSec. These parts are referred to as the protocol, although they are not a standard standalone protocol as they cannot function on their own.

These protocols are:

  1. Authentication Header (AH): it provides verification services for IPsec. It allows the recipient of a message to verify that the supposed sender of a message was actually, in fact, the one that sent it. It also helps to discover disguise.
    In addition, it allows the recipient to authenticate that intermediary devices during transmission have not tampered with any of the data being transferred and also provides security against “replay attacks”, whereby information is read by an unauthorized host and resent. Ultimately, AH ensures the integrity of the data in the datagram, but not its privacy.
  2. Encapsulating Security Payload (ESP): this protocol helps to encrypt or encode your data as it is exchanged over the public network. Its main focus is the “body,” the data itself (IP payload).

Security Association (SA)

The Security Association (SA) concept works directly with the protocols discussed. An SA is a relationship between two or more hosts. This relationship describes how the hosts make use of security services (such as IPsec) to communicate securely.

IPsec provides many options for performing network encryption and verification. IPsec uses the security association to track all the details concerning a given security communication session. The good thing about this association is that it gives a host or computer the flexibility to choose any security service it desires.

Phases of IPsec

  1. Determine interesting traffic: Interesting Traffic means a data exchange that is worth protecting. The first phase of IPSec is to determine whether a particular connection is worth protecting or not.
  2. Internet Key Exchange (IKE) phase 1: in conjunction with the service of IPsec, a key management protocol standard is used. It creates a shared secret key, which is used to decipher the encrypted data. Imagine person A locking a book meant for person B in a box, A then sends B a key so only he can open.
  3. IKE phase 2: After IKE has established a secure tunnel, the IPsec policy and security association are established. The hosts must agree on a set of security protocols to use so that each one sends data in a format the other can understand.
  4. Transfer data: After all this has been done, each device must use all the parameters ( protocols, methods, and keys) previously agreed upon to encode, send and decode data across the network.
  5. On completing the data exchange, the tunnel of communication is immediately “torn down”.

Implementation of IPsec

Implementation is done in different ways because some feel IPsec should be installed on all hosts connected to a network and others feel it should be limited to specific hosts/routers.

  1. End-host implementation: This involves having IPsec on all devices and this provides the most flexibility and security, but it is tedious.
  2. Router implementation: this involves installing IPsec on specific routers which is a lot easier as only a few routers instead of hundreds or thousands of users need to be configured.

To get IPsec into a router or device, it also requires different methods. It can be installed in the layer of connection (IP stack) of an operating system. This procedure actually requires a modification of the source code. To carry this out you need to use this method for both the security gateway and the host. If you do not want the OS modified, you can use the bump-in-the-stack (BITS) implementation. Here, the IPsec is implemented between the network drivers and the IP stack.

Key functions or services of IPsec

We have been talking about the technicalities of IPsec, let us now properly consider the functions of IPsec:

  1. Confidentiality: IPsec helps to encrypt (to make a data only readable to authorized hosts) data so that only the desired hosts can read it.
  2. Data integrity: it helps to determine whether during transmission the data has been changed or modified in any way.
  3. Data authentication: is the sender/receiver who they say they are.
  4. Anti-replay: it helps to ensure each packet is unique.

Alternatives to IPSec

While IPSec is a very basic security protocol offering security at the network (IP) level which is the root of connection. There are also other security protocols, that can be used out there:

  1. Security at the connection level: this type of security enables tunnels to be set up when the hosts are prepped to be connected, unlike the IPsec that depends on the IP before establishing security. An example of this is the Point to Point Tunneling Protocol (PPTP) and Layer 2 forwarding protocol (L2F). These two protocols were later merged into the Layer 2 Tunneling Protocol. A major disadvantage of this protocol is the lack of security at the IP (packet) level.
  2. Security at higher levels: You can decide to go further and have your security at a higher level. This type of security usually does not require any change to your device. Example of the type of security in this category is Secure Sockets Layer (SSL), Secure/Multipurpose Internet Mail Extensions (S/MIME) and Pretty Good Privacy (PGP).

VPN and tunneling

A virtual private network (VPN) works by enabling users to exchange data across public networks as if the devices were directly linked to the virtual network. A VPN gives a user a different IP making them appear as if they are connected from another location.
IPsec can be used seamlessly with VPNs. this incorporation forms IPsec VPN. This refers to the process of creating and managing VPN connections using an IPsec protocol suite. You can also refer to it as VPN over IPsec.

Another important aspect of a VPN is the tunnel used in transmitting data. A VPN tunnel is an encrypted connection linking a device to a server. It works by covering (encapsulating) data in an encrypted data packet. IPsec naturally uses the tunnel mode for establishing VPN tunnels. IPsec provides an enhanced level of security on VPN connections by providing encryption, authentication and compression services at the network level of VPN.

Most times VPN service providers use IPsec in conjunction with other protocols to increase security.

Final thoughts

IPsec provides freedom in allowing different devices to decide how they want to implement security.

The IPsec method of administering security is more basic compared to other methods as it tackles security from the foundation, the IP itself.

This makes it still relevant even with more complex security protocols being developed regularly.

Overall, IP level security is quite solid and worth considering.