One of the most important parts of any VPN service is the VPN protocol. That, however, is a misnomer – there are lots of protocols, with completely different applications, which make the workings of VPNs possible. When we say “VPN protocol,” we’re really talking about the tunneling protocol (sometimes also called the “security protocol”).

Tunneling protocols are used to establish a “tunnel” between networks, allowing private communication across a public network. Differences between the various tunneling protocols used in VPN services determine the level of security, computational power requirements, speed, and more subtle characteristics – what types of devices they’re better for or which CPUs can handle them better.

For someone new to IT topics and VPNs, tunneling protocols can be completely incomprehensible. So, we decided to explain them to you as best we can.

Secure protocols

OpenVPN

  • Prevalence: the most common protocol in consumer VPNs
  • Security: high

Authored by James Yonan and released in 2001, OpenVPN is an open source VPN tunneling protocol used to provide secure point-to-point or site-to-site access. Due to its security and open source nature, OpenVPN has become the primary protocol used in commercial VPN solutions.

OpenVPN supports an array of operating systems including, but not limited to Windows, macOS, Android, and iOS.

How does OpenVPN work?

The protocol uses the OpenSSL library for encryption and authentication of data, which means it has access to all the ciphers in the OpenSSL library. The most commonly used in VPNs are AES, Blowfish, and ChaCha20.

OpenVPN uses UDP and TCP as common network protocols to create a transport tunnel. It allows users to get past firewalls and Network Address Translation (NAT).

Third-party plug-ins can be used to enable extensions established at defined entry points. This is necessary to enable authentication via usernames/passwords in OpenSSL. It is also important to extend the application with the ever-changing internet firewalls.

How to use OpenVPN

Most people will not use OpenVPN directly. Nowadays, this protocol is an integral part of most top VPN services, such as ExpressVPN, NordVPN, Private Internet Access, etc. That’s a good thing – many of these VPN providers combine the secure tunneling capabilities of OpenVPN and strong additional features to provide an all-around product.

With that said, there are situations where using the standalone app makes more sense. In those cases, how to use OpenVPN will differ based on your GUI. For more information, we will again refer you back to OpenVPN’s How To page.

Pros

  • Open source software. This means it possesses a source code open to the public and modifiable. This openness and flexibility ensures security
  • Top-notch encryption and cryptographic algorithms
  • Offers many configuration options
  • Available on many platforms

Cons

  • Not the easiest to configure or audit. One requires a deep understanding of the technology to reap its supplementary benefits, such as third-party plug-ins or scripts for username and password peer authentication
  • Could be more stable when switching network
  • Easy to identify using Deep Packet Inspection (DPI), therefore it’s not the best protocol for censorious countries, such as China

IKEv2/IPsec

  • Prevalence: common, especially in VPN mobile apps
  • Security: high

Internet Key Exchange Version 2 (IKEv2) is a protocol developed by Microsoft and Cisco (primarily) for mobile users. It was introduced as an updated version of IKEv1 in 2005. The IKEv2 MOBIKE (Mobility and Multihoming) protocol allows the client to maintain a VPN connection despite network switches, such as when leaving a wifi area for a mobile data area.

IKEv2 uses the IPsec protocol suite and works on most platforms, including some less-common ones. It also has native support on iOS, making the protocol particularly good for Apple’s mobile OS.

How does IKEv2 work?

IKEv2 is not a tunneling protocol per se. Whenever you read this title in association with VPN technology, you should assume that it actually means IKEv2/IPsec. In other words, the two protocols are used in combination, both performing different functions. IKEv2 uses UDP for transport, namely, UDP port 500 and 4500.

Just like OpenVPN, IKEv2 supports a variety of encryption ciphers, the most common of which are, again, AES, Blowfish, and ChaCha20.

IKEv2 is considered one of the fastest tunneling protocols, mainly because it doesn’t place much of a load on the CPU. Perhaps the most important criticism we can level at IKEv2 is that it’s not open source and is thus not independently auditable.

How to use IKEv2

You’re most likely going to use IKEv2 as part of a VPN app. As mentioned previously, it has benefits as part of a mobile VPN app – that’s where you’ll likely encounter it instead of OpenVPN.

It is also possible to use the protocol independently and many contemporary VPN services will include instructions on how to do exactly that. IKEv2 use is particularly prominent on iOS, where the protocol enjoys native support. This practice is seen at the very top of the VPN market – services like NordVPN or VyprVPN.

Pros

  • Very secure and compatible with lots of ciphers
  • One of the fastest protocols
  • Great for mobile due to ability to maintain connection despite network switches
  • Not very CPU-intensive

Cons

  • Not open source software – cannot be independently audited.

WireGuard

  • Prevalence: still under development, but increasingly being used
  • Security: high

WireGuard is at the cutting edge of VPN tunneling technology. The project was started in 2015 by Jason A. Donenfeld with the goal of creating an easily-implementable, easily-auditable, secure, and fast VPN protocol for the 21st century. Instead of taking some sort of framework and upgrading it, WireGuard was built from the ground up. This allowed the team to build something not plagued by many of the old preconceptions behind the most popular tunneling protocols in use today.

The result is widespread adoption or planned adoption within the VPN community. The software already supports most of the major platforms and operating systems.

How does WireGuard work?

Unlike the previously-mentioned protocols, WireGuard doesn’t offer a choice of encryption ciphers. It also doesn’t offer the most widely-used cipher today – AES. Instead, data encryption is handled by the more modern ChaCha20 cipher, which is a stream cipher rather than a block cipher. It is therefore easier on devices without CPUs that have native AES support.

Its developers call WireGuard a “connection-less protocol,” because the only state it keeps is a simple handshake, renegotiated every few minutes. This way, the protocol is able to ensure perfect forward secrecy.

Similarly to IKEv2, WireGuard should be particularly resilient to network changes, making it perfect for mobile devices.

How to use WireGuard

You can download WireGuard from the website, where you will also find some instructions on how to install/use it. However, it is also becoming increasingly available as part of consumer VPN services. A notable example is Mullvad VPN, who were one of the first adopters of WireGuard.

Pros

  • Very secure
  • Supports most mainstream platforms
  • Likely the fastest tunneling protocol
  • Easy to implement
  • Code is easy to audit
  • Open source

Cons

  • Still under development
  • Can‘t choose the encryption cipher
  • Doesn‘t support TCP (only UDP)

SSTP

  • Prevalence: not very common
  • Security: high (with some concerns)

The Secure Socket Tunneling Protocol was designed (and is still owned) by Microsoft and first introduced with Windows Server 2008. As such, it is only available on Windows (with some support on other operating systems) and has a potential security concern – that Microsoft may have a backdoor in place to decipher SSTP traffic.

Despite concerns over Microsoft‘s ownership of SSTP (and unlike its younger brother – PPTP) it is considered a secure protocol.

How does SSTP work?

In SSTP traffic is routed over TCP port 443 through an SSL/TLS channel. The traffic is thus able to bypass proxy servers and even firewalls. This makes SSTP a hard protocol to block and a good choice in countries that rely on DPI to block VPN traffic (like China).

SSTP works similarly to PPTP: it wraps up packets of data in a protective sheath. When using Secure Socket Tunneling Protocol, users are required to connect via a standard Transmission Control Port (TCP), which allows the target server to initiate authentication procedures. This involves sending a couple of encryption keys to the user’s system, which form the basis for the SSTP tunnel. When that’s done, the packets can be sent with relatively high levels of security directly to the server.

As with OpenVPN, SSTP uses OpenSSL, which means the same data encryption ciphers apply: AES, Blowfish, ChaCha20, etc. This makes the protocol cryptographically secure.

How to use SSTP

The protocol is natively supported by Windows, which means you can use it on this OS without any additional third-party software. On other operating systems, you’ll need some sort of app – for example, there’s an SSTP client for macOS called iSSTP.

In terms of using as part of a mainstream VPN client, you will still see the protocol available in some VPN suites (e.g. Astrill VPN), but there aren’t many.

Pros

  • Solid cryptographic security
  • Can pass through firewalls
  • Can work as an anti-DPI protocol

Cons

  • Owned by Microsoft (impossible to audit)
  • Limited support on operating systems outside of Windows

L2TP/IPsec

  • Prevalence: common, but decreasingly so
  • Security: high (with some concerns)

A combination of two protocols – the aforementioned IPsec and Layer 2 Tunneling Protocol (L2TP). L2TP originates from Cisco‘s (now outdated) L2F and Microsoft‘s (equally outdated) PPTP. While the popularity of this protocol combination is waning, it is still being used by many mainstream VPN services, particularly on iOS, where developers can‘t include OpenVPN as part of their apps.

L2TP/IPsec is considered secure, however, the Snowden leaks have revealed that the NSA may have found a way to compromise the protocol.

How does L2TP/IPsec work?

The basic premise of L2TP/IPsec is that L2TP is used to establish a tunnel and IPsec is responsible for encryption/authentication. L2TP does not provide any confidentiality in and of itself, which is why it is mostly used in conjunction with a protocol like IPsec. The data encryption ciphers on IPsec include the secure AES-CBC and AES-GCM, as well as the outdated TripleDES-CBC.

L2TP/IPsec enjoys native support on many operating systems, which means users don‘t need third-party software to use it. All that is required are configuration files, which you can get from your VPN service provider.

The connections in L2TP/IPsec are UDP, rather than TCP. In particular, it uses UDP Port 500 for the initial key exchange, which can sometimes cause issues with firewalls – have that in mind if your connection simply doesn‘t get through.

How to use L2TP/IPsec

Although you can use the protocol natively on most operating systems, regular folks will likely use L2TP/IPsec as part of some VPN app. Many of the top VPNs on the market still offer the protocol and some rely on it to do the heavy lifting on their iOS apps – CyberGhost is one such example.

Pros

  • Secure cryptography
  • Native support on most operating systems
  • Quite fast

Cons

  • Allegedly compromised by NSA
  • No TCP support
  • Not the best for mobile (lacks reliability in network switches)

Obsolete Protocols

PPTP

  • Prevalence: not very common
  • Security: low

The Point-to-Point Tunneling Protocol was developed by Microsoft and first implemented with Windows 95. It is supported by most operating systems in general, particularly all versions of Windows. While PPTP is not very reliable, it is nevertheless quite fast. This has a lot to do with its cryptography, which is very weak in relation to the capabilities of contemporary computers.

PPTP is still seen on mainstream VPN apps, but its prevalence is dwindling. And with good reason – it doesn‘t provide much security anymore.

How does PPTP work?

Like its more advanced brother SSTP, PPTP is routed over TCP (in this case, port 1723). The tunnel is established by using General Routing Encapsulation (GRE). At both ends of the tunnel, PPTP also authenticates the data packets being transferred, beginning and completing the process.

From the very beginning, using a PPTP server has been a risky idea for security-conscious web users. By 1998 (three years after the protocol was launched), hackers had published ways to extract password hashes from users employing the MS-CHAPv1 authentication protocol, which is part of the PPTP package.

Since then, PPTP’s list of unfixable security issues has grown to include vulnerabilities to cryptoanalysis, brute force attacks, and dictionary attacks. Multiple components of PPTP have been shown to be inherently susceptible to these: vulnerabilities plague MS-CHAPv1, MS-CHAPv2, as well as MPPE.

Therefore, despite its utility as a fast protocol that requires little computational power, PPTP should not be used where security is needed.

How to use PPTP

If you are a Windows, macOS, Linux, iOS, or Android user, you’ll be able to use PPTP without any additional software. It is one of the most straightforward protocols for setting up your own VPN server. However, a majority of users will find it integrated in their consumer VPN apps. PPTP is still part of many VPN services despite seriously lacking security.

Pros

  • Very fast
  • Requires little computational power
  • Natively supported by major operating systems

Cons

  • Outdated and insecure

UDP vs TCP

Tunneling protocols such as OpenVPN can routed over TCP (Transmission Control Protocol) or UDP (User Datagram Protocol). So what‘s the difference between the two and when should one be favored over the other?

Both TCP and UDP are used for the same general purpose – to send data from one point to another. Both are built on top of the internet protocol (hence, TCP/IP and UDP/IP) and both are ways of sending packets of data over the internet. There are more protocols serving the same function and these two are simply the most commonly used.

TCP is a connection-based protocol and its main benefits are error checking and reliability. Because in TCP exchanges are controlled – packets are numbered and discrepancies are compensated – data cannot be lost or corrupted on its way from one place to another. Naturally, this process slows connections down simply because it requires back-and-forth communication.

These characteristics of TCP are useful for file transfers as well as browsing the web.

Meanwhile, UDP is a so-called “connectionless“ protocol. There’s no error checking as seen in TCP – the packets are sent and they arrive or are lost. As you would expect, this makes UDP a faster protocol, but also less reliable and prone to in-transit file corruption.

The quickness and one-way nature of UDP is useful for situations where retrieving lost packets isn’t useful. This includes gaming, streaming, or VoIP.

When it comes to using TCP or UDP with OpenVPN, the choice should also be informed by the applications you intend to use it for. The security level doesn’t change in any meaningful way.