IPsec L2TP VPN server

Introduction
IPsec/L2TP is another commonly used VPN protocol used in Windows. All version of Windows since Windows 2000 have support built-in, not requiring an external client (like OpenVPN does) making it very convenient. However, it is significantly harder to set up on the server side on Linux, as there's at least 3 layers involved: IPsec, L2TP, and PPP.

This guide will not cover setting up DHCP, RADIUS, Samba or PKI or setting up a Linux client. It does cover some Windows configuration for the purposes of troubleshooting the server

For the purposes of this guide:
 * Domain is example.com
 * Server name is vpn.example.com
 * CA file is called
 * Server cert is
 * Server key is
 * Client cert is
 * Client key is

IPsec
The first layer - and most difficult one - to set up is IPsec. Note IPsec is a peer-to-peer, so in IPsec terminology, the client is the called the initiator and the server is called the responder

Windows uses IKEv1 for the process. There are 3 implementation of IPsec in Portage: ipsec-tools (racoon), openswan, and strongswan. If you are running, you need to add the appropriate entries to.

ipsec-tools (racoon)
ipsec-tools is the least featured one, but for those of you coming from *BSD, you might be more familar with it. However unlike *BSD, Linux does not use separate interface for ipsec

After installing ipsec-tools, we need to create a few files:

PSK setup
Each entry in the PSK file consists of an identifier and the key. Windows identifies itself according to its idea its FQDN. The key may be specified as either a string or a hex number (starting with 0x)

PKI Setup
Copy, , and to. Make sure is not world-readable or writable

Creating securiy policies and NAT
The should cause racoon to generate the appropriate Security Policies for us. However, in the presence of NAT (at least, if the server is behind NAT) it doesn't generate the policies quite the way one would hope. So, if no traffic flows over the tunnel, the policy will need to be defined manually:

Polices are created in the file:

Note that, while this policy says we want ALL L2TP traffic going to and from the server to be protected, if there's no Security Association, the traffic will be not protected and will travel the Internet in the usual way - it won't be dropped just because there's no Security Association.

Openswan
Openswan is 1 of the 2 forks of the original FreeSWAN,

First, we to create some files

PSK setup
First, we need to create a private key. They may either be specified by a quoted string or by a hex number

Then create

Without NSS
We need to copy the appropriate files to the right places

Edit ipsec.secrets

And then edit ipsec.conf

With NSS
First, create a PKCS12 file with the CA certificate, server certificate, and server key

Then we need to create a NSS database first, them import the certificates and keys into it. Leave the password blank, just hit ENTER twice. After importing,

After importing, Use certutil -L -d . and certutil -K -d . to see what the "friendly names" are.

The configuration above works just fine if the the server is behind NAT.

IPsec passthrough / broken NAT
strongSwan does not seem to work with IPsec passthough. It return "cannot respond to IPsec SA request because no connection is known" or (which heavy editing of the config file) INVALID_HASH_INFORMATION error.

Opening ports
2 port need to be open: UDP port 500 (for ISAKMP) and UDP port 4500 (for NAT Traversal). Forward those to your VPN server

Also IP Protocols (not ports!) 50 (ESP) and 51 (AH) need to be allowed as well, if the router has a per-IP protocol setting (most don't).

IPsec passthough / broken NAT
Many routers have an "IPsec passthrough" options, which can mean 1 of 2 things:
 * 1) Mangle IPsec packets in broken way not compatible with IPsec NAT Traversal
 * 2) Allow all IPsec packets through the router unmolested

If it means (1), DISABLE IPsec passthrough; if it means (2) ENABLE IPsec passthrough

Unfortunately, there are routers that will discard all IPsec traffic, even if the ports are forwarded, and only support method (1). For those with such a router, there are 3 options:
 * 1) Upgrade the firmware, if a newer version is available that behaves properly
 * 2) Open a bug/defect report with the make of the router, if its not EOL/Legacy
 * 3) Get a different router. Linksys and D-link routers are reported behave properly.

This tutorial was written with such a router (a Zyxel P-330W) - and (3) is the only option available.

Windows Vista/Server 2008 clients
These OSs do not automatically support IPSec/L2TP servers behind NAT. See http://support.microsoft.com/kb/926179/en-us for the registry edit to make them support it.

Limitation of Pre-shared keys (PSK)
There is no provision within the IPsec protocol to negotiate PSKs. The only information available to choose which key to use is based on the source and destination IP addresses. Since, in the usual scenario, the responder won't know the initiator's IP in advance, EVERYONE must use the same pre-shared key. Therefore, certificates (PKI) are highly recommended over pre-shared keys (PSK), even for only 1 connection. However generating certificates and creating a PKI is a rather complex process and out of scope of this document.

L2TP
The second layer, l2tp is much easier to setup. Like IPsec, l2tp is a peer-to-peer protocol. The client side called the L2TP Access Concentrator or LAC and the server side is called the L2TP Network Server or LNS

Use iptables to block all l2tp connection outside the ipsec layer:

Using xl2tpd
Unlike other L2TP servers, xl2tpd can maintain an IP address pool without a DHCP or RADIUS server. This is a layering violation, but for small setup its extremely convenient

If you want to use a RADIUS or DHCP server, leave off the "ip range" and "local ip" parts. If the connection is unstable, try adding "length bit = yes" to the "lns default" section. If you don't want to use PPP authentication, change "require authentication = yes" to "refuse authentication = yes"

We'll need to create the options file as well

If you are not using PPP authentication, change "auth" to "noauth".

Using rp-l2tp
Configuration of rp-l2tp is simple:

You'll need to specify the server IP as well in lns-pppd-opts (in the form a.b.c.d: ). You'll also need to setup pppd to use an IP address pool, either via DHCP or RADIUS.

No authentication
The easiest way to setup is no authentication at all. In that case, make sure "noauth" is specified. Useful for testing purposes, otherwise not recommended - especially if you use PSK. For certain PKI setups, such a configuration may be sensible - for example, all the client machines are under your control, or all the users are trusted and the keys are on machines no one else, besides the users' have access to, or all connections are unattended (using this method to connected multiple sites)

Authentication via chap.secrets
For small users (typically, those wanting to connect their home network from elsewhere), since add the information to chap.secrets

Authentication via Samba
If you're machine part of (or hosting) an MS Domain or ADS forest, and you are using winbind, Samba can do the authentication. Add to your ppp options. Setting up Samba and pppd to do this is beyond the scope of this document.

Authentication via RADIUS
If you are running a RADIUS server on the same machine, pppd can use RADIUS. Ensure the USE flag is set on. Then add to your PPP options. Setting up RADIUS and pppd to do this is beyond the scope of this document.

Authentication via EAP-TLS
If individual users have certificates (not the same as the machine certificate above), you can setup pppd to authenticate via EAP-TLS. You'll want to use this is users authenticate via smartcards or RSA secureID, Ensure the is set on. RADIUS needs to be setup (see above). You my need to include in your PPP options file as well. Setting up pppd to do this is beyond the scope of this document.

Windows: Correctly installing the certificate (for PKI users)
The certificate should be packaged in a PKCS12 package. You can do this using openssl or gnutls

Once you have the p12 file, you can import into Windows, however the method is not obvious. DO NOT double-click the key and follow the instructions, that won't work. That imports the key into your personal certificate store, but in Windows, its the local computer that need to so the authentication, so the certificate need to be added to the local computer's store. To do that, we need to use the Microsoft Management Console (mmc). Note you must have Administrator privileges for this to work

Start -> Run -> mmc File -> Add/Remove Snap-in -> Certificates -> Add Computer Account -> Local Computer -> Finish -> OK.

Expand the Certificates. Choose any folder (doesn't matter which), right-click, choose "All Tasks", then "Import". NOW follow the wizard, but on the last step, make sure you choose "Automatically select the certificate store based on the type of certificate"

Error 766: A certificate could not be found
This means you didn't import the certificate correctly. You have to import it though MMC, and not by double-clicking the file.

Error 810: VPN connection not complete
If you're using ipsec-tools (racoon) you may see in you syslog as well:

This means you didn't import the certificate correctly, or the p12 bundle is missing the CA certificate. You must do this though MMC, and you MUST choose "Automatically select the certificate store based on the type of certificate" at the end of the import process.

XP SP2 and above: Error 809: Server not responding (Server behind NAT)
Windows XP SP2 and Vista will not, by default, connect to server behind a NAT. A registry hack is required. Separate fixes are required for Windows XP and Windows Vista.

Vista: Error 835 Could not authenticate
This one occurs only when using PKI. It means the subjectAltName does not match the server you are connecting to. This often occurs when using dynamic DNS - the certificate has the internal name rather than the external name. Either add the external name to the certificate, or disable "Verify the Name and Usage attributes of the server's certificate" in Security -> Networking -> IPsec settings of the connection

Error 741: The local computer does not support required encryption type
Windows will try to negotiate MPPE a (weak) encryption. If you're not using PPP authentication, do not have a pppd with MPPE support, or MPPE supported not compiled into the kernel (or a module), you'll get this error. If you're using authentication, I recommend fixing the pppd or kernel, for minimal configuration problems, even though there's no point to double encryption. If you are not using PPP authentication, or don't want the double encryption, you can disable it by unchecking "Require data encryption" on the Security tab. Note the connection is still protected by IPsec encryption either way, this just disable the requirement for MPPE.

Mac OS X
Mac OS X clients appear to be picky on the proposals they will negotiate with. In particular:
 * dh_group must be "modp1024".
 * my_identifier must be an address, not an FQDN (address is the default, so just leave that line out from racoon.conf).

Mac OS X won't connect if subjectAltName does not match the server you are connecting to. Unlike Vista you cannot disable this check.

Also, Mac OS X won't connect if the server certificate contains any "Extended Key Usage" (EKU) fields (except for the deprecated ikeIntermediate one). In particular, if you are using certificates from the OpenVPN easy-rsa utility, it adds the "TLS WWW Server" or "TLS WWW Client" EKU, so such certificates will not work. However, you can use such certificates on the Mac OS X client, as it doesn't not care what on the client certificate - only the server.