Setting Up an IPSec L2TP VPN server on Ubuntu for Windows clients

UPDATE: This document was for Ubuntu 10.04 (Lucid), although it is probably basically the same steps for 12.04. For the latest, see this document for Debian 7.1 Wheezy.

We have a Windows network (Windows server and Windows clients) at work, with an Ubuntu server that has an external IP address. I wanted to allow certain users to be able to connect via a VPN. I also wanted to allow them to do it without any special software on their machines. Our Windows Server had a private IP address and only one network card, so the basic RRAS solution was not going to work for me.

Below is the step-by-step I went through to set up a "VPN server" that would work with the native Windows client.

Running on Ubuntu Server 10.04.3 LTS on our proxy server, so it has an external IP address and an internal IP address on two different NICs. It is assumed that you run the following commands as root when needed and that you are smart enough to know when that might be.

Windows VPN clients use either PPTP or IPSec L2TP. (Windows 7 also has IKEv2, but we won't cover it here since Openswan didn't yet support NAT for IKEv2.) In this guide, we'll use IPSec L2TP. We set things up first with Pre-shared Keys (PSK) since it's easier to test, then we step through using certificates with the default Windows Server Certificate Authority (CA). I also wanted to authenticate users off of their domain password and only if they were in a particular group.

Note: In this guide, the external IP of the machine is listed as The gateway IP is listed as The internal IP of the VPN server (since it has a NIC on both the inside and the outside) is in this example. Change these for your set up.

1) Install and set up the openswan package.

This will allow the Windows clients to create an IPSec SA between itself and the VPN server.

Unfortunately, the version of Openswan that comes with Lucid is now pretty out of date, and you may have trouble getting iPhones and iPads connected to this same setup if you use their package. I realize this guide is for Windows clients, but we try to be accommodating. If you want to try the regular Ubuntu packages, see the previous version of this document. As for the rest of us, we will continue on with the latest versions of things.

Download the latest version of Openswan to your computer.

Expand the file and enter the directory
tar -zxvpf openswan-2.6.35.tar.gz
cd openswan-2.6.35

In order to actually compile the program, we just need to install a few packages from the default Ubuntu repositories. Then we can compile Openswan. We do these with the commands below.
apt-get install libgmp3-dev bison flex
make programs

At this point, if there were no errors in the compiling, we can go ahead and install it. (If you have the Ubuntu version of Openswan installed, you can uninstall it first with apt-get remove openswan)
sudo make install

Congratulations! The latest version of Openswan is now installed. Now we can get on with the setup.

cp /etc/ipsec.d/examples/l2tp-psk.conf /etc/ipsec.d/l2tp-psk.conf
nano /etc/ipsec.d/l2tp-psk.conf

Set left= [should be set to your external IP address on the machine users will connect to]
leftnexthop= [set this to your external gateway]

nano /etc/ipsec.conf

Add: include /etc/ipsec.d/l2tp-psk.conf
Also, for Windows Vista to work properly, we need to tell it which private subnets are allowed, and which are not. In our example, since our company's internal subnet is, we disallow that (at the end of the line).

nano /etc/ipsec.secrets

Add: %any: "yourSharedPSK!"

/etc/init.d/ipsec restart

At this point, your VPN server should be listening on port 500/udp and 4500/udp for connections. You can check this using netstat -antu.

2) A quick test

This will check to see if the IPSec side of things is working properly.

You'll need to allow udp/500 and udp/4500 to your external interface through the firewall on your INPUT chain. I also added protocol 50. How this looks depends on your firewall implementation, but my iptables filter rules look like this:
-A INPUT -p 50 -j ACCEPT
-A INPUT -p udp -d --dport 500 -j ACCEPT
-A INPUT -p udp -d --dport 4500 -j ACCEPT

On a Windows XP client, we set things up for a quick test:
Control Panel > Network Connections > File > New connection...
Select Connect to the network at my workplace
Select Virtual Private Network connection
Company Name: Your Company
Select Do not dial the initial connection
Host name or IP address:
Properties > Security > IPSec Settings > Check Use pre-shared key for authentication
Pre-shared key: yourSharedPSK!
Properties > Network > Type of VPN: L2TP IPSec VPN
Whether you want to allow split tunneling is up to you: Properties > Networking > TCP/IP > Properties > Advanced... > General > Uncheck Use default gateway on remote network

Now, monitor /var/log/auth.log (perhaps with tail -f /var/log/auth.log) and connect with the Windows client.

In the end, the connection will fail, but you should see connection attempts on the VPN server with a STATE_QUICK_R2: IPsec SA established. This means the IPSec side of things is working with the pre-shared key.

3) Install xl2tpd

Since Windows default client is more than just IPSec, it uses L2TP inside of an IPSec SA, we need a daemon to handle that. Like Openswan, there have been a few bugs fixed since the version in the Ubuntu repository. Rather than compile it though, I just download a later version from a later version of Ubuntu.

After you have the .deb file downloaded, you can install it with a command such as: dpkg -i xl2tpd_1.2.8+dfsg-1_i386.deb.

Modify /etc/xl2tpd/xl2tpd.conf so it includes at least the following:

listen-addr =

[lns default]
ip range =
local ip =
;require chap = yes
refuse chap = yes
refuse pap = yes
require authentication = yes
ppp debug = yes
pppoptfile = /etc/ppp/options.xl2tpd
length bit = yes

The IP range specified above should be set to IP addresses of your internal network which can be given to your VPN clients. Don't worry much that we are refusing CHAP and PAP methods, because we will require MS-CHAP v2 next.

cp /etc/ppp/options /etc/ppp/options.xl2tpd
nano /etc/ppp/options.xl2tpd

Change noauth to auth.
Set name l2tpd. You can really set it to something other than l2tpd, but you have to match it in the next file.
Set mru 1280 and mtu 1280. I had some weird trouble with Vista's Remote Desktop not working over the VPN if these were left at their defaults of 1500. 1280 is chosen because that is the minimum required if the IPv6 protocol is to work as well (although that is not covered in this document).

Modify /etc/ppp/chap-secrets
#client	server		secret		IP addresses
username	l2tpd		"password"
l2tpd		username	"password"

Match the l2tpd with the name in the previous file. You can use this to test your CHAP authentication if you want... but you'd have to temporarily change the refuse chap = yes line above. I put it here just so you know how to test it if you want.

At this point, you need to add an extra rule to your firewall. Some of the sites I reference urge you to be security-minded here because if you open up this port to the whole world, then anyone may try to authenticate without IPSec. Basically, you want to allow connections to udp/1701, but they'd better be connected via IPSec. My filter rule looks like the following:
-A INPUT -m policy --dir in --pol ipsec -p udp --dport 1701 -j ACCEPT
This will allow L2TP traffic to connect to us ONLY if it shows up in an IPSec packet. The best information I've found about how IPSec (NETKEY) interacts with the iptables firewall was found in this post by Nigel Metheringham.

The last firewall modification we need to make for xl2tpd (which we could probably get more picky if you wanted). When an L2TP connection is made, it creates a ppp# interface on the VPN server, so we need to allow it to talk to the other interfaces.
-A FORWARD -i ppp+ -p all -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT

4) Join Ubuntu to the Active Directory domain

I wanted to have the authentication based off of the Windows AD rather than some /etc/ppp/chap-secrets file.

apt-get install smbclient
apt-get install winbind

Make sure /etc/resolv.conf points to your DNS servers that have your AD information. Add an A and PTR record for the machine if you don't already have that.

apt-get install krb5-user (includes krb5-config)

Kerberos servers for your realm: windowsserver.example.local
Administrative servers for your realm: windowsserver.example.local

Edit /etc/samba/smb.conf (if not specifically listed, I just left the defaults):

workgroup = EXAMPLE
interfaces = eth0 lo
bind interfaces only = true
security = ADS
password server = windowsserver.example.local
idmap uid = 10000-20000
idmap gid = 10000-20000

Edit /etc/krb5.conf (I believe this file is case-sensitive. I only list the lines here that I modified from the default.)
default-realm = EXAMPLE.LOCAL
		kdc = windowsserver.example.local
		admin_server = windowsserver.example.local
Run: /etc/init.d/winbind restart

Note that the clocks of the Windows server and the VPN server must be within 5 minutes of each other for the next commands:

net ads join -U Administrator

This joins the Ubuntu server to the Windows domain. On one machine, I had to make sure that the FQDN was listed in /etc/hosts before it let me join the domain.

net ads testjoin

This line tests to see if the VPN server was properly joined to the AD domain.

5) Update xl2tpd/ppp to work with the Windows domain authentication

Edit /etc/ppp/options.xl2tpd and add:

# We can enable MPPE for additional encryption, but all this should be coming over IPSec anyway
# The following lines let the authentication occur against the Windows domain, and require the user to be a member of the 'VPN Users' group on the 'EXAMPLE' domain.
ntlm_auth-helper '/usr/bin/ntlm_auth --helper-protocol=ntlm-server-1 --require-membership-of="EXAMPLE\\VPN Users"'

At this point, you should be able to connect using your Windows domain credentials rather than the chap-secrets file. For some people, this is all they really need and they skip step 6 and go to step 7.

6) Using Certificates instead of Pre-shared Keys

This assumes that you already have Certificate Services set up on your Windows Server. I don't go into a lot of detail here.

6.1) Getting a certificate for your VPN server

openssl req -new -out

Enter PEM pass phrase:
Country: US
State: State
Locality: City Name
Organization: Your Company Name
CN: E-mail:
Challenge password:
Optional company name:

mv /etc/ssl/private chmod 640 /etc/ssl/private/

Load up your Certification Authority on Windows. Right-click on the server, All Tasks > Submit new request... and give it the file you created.
Choose Pending Requests. Right-click the request, All Tasks > Issue
In Issued Certificates, Open the certificate
Details tab > Copy to File...
Choose DER encoded binary X.509 (.cer)
Export the certificate for the CA as well (not the private key!) using the Certificate snap-in of mmc.
Copy both to your VPN server.

openssl x509 -inform DER -in windowsserver.example.local.cer -outform PEM -out windowsserver.example.local.pem

This just converts the DER encoded file to a .PEM file.

cp windowsserver.example.local.pem /etc/ipsec.d/cacerts

IPSec needs to know the public key of the CA for verification purposes.

openssl x509 -inform DER -in -outform PEM -out cp /etc/ipsec.d/certs

Takes the generated certificate for our VPN server and lets openswan (IPSec) use it as needed.

6.2) Setting openswan to use certificates rather than PSKs

cp /etc/ipsec.d/examples/l2tp-cert.conf /etc/ipsec.d/l2tp-cert.conf
Edit /etc/ipsec.conf

Replace l2tp-psk.conf with l2tp-cert.conf.
Also, for Windows Vista to work properly, we need to tell it which private subnets are allowed, and which are not. In our example, since our company's internal subnet is, we disallow that (at the end of the line).

Edit /etc/ipsec.d/l2tp-cert.conf. Vista seems to like or need the leftid setting.


openssl req -new -keyout

PEM passphrase: passphraseToAccessFile
This tells openssl to spit out our private key into a file.

mv /etc/ipsec.d/private/

Move this private key to a place where IPSec (openswan) can have access to it.

We need to modify /etc/ipsec.secrets to make sure it can use the private key associated with our certificate:

Comment out the pre-shared key we put in there earlier and add the line (including the colon):
: RSA "passphraseToAccessFile"

/etc/init.d/ipsec restart

6.3) Getting the Windows client to work with certificates

First off, we need to get a certificate for the Windows machine. With Windows XP Professional attached to a domain, this is fairly straight-forward and I believe you can even push the certificates to the machines. It may go something like this for you:

Start > Run > mmc > File > Add/Remove Snap-in... > Add... > Certificates > Select Computer account > Local computer > Close > OK
Highlight Certificates > Personal, Right-click > All Tasks > Request New Certificate...

However, I was dealing with a bunch of Windows XP Home machines, unfortunately, so I had to go about things in a different way. I had to download Windows Server 2003 SP1 Administration Tools Pack (Adminpak) KB304718.
From there, one could take certreq.exe, certutil.exe, certcli.dll, and certadm.dll to a different machine.
Create req.inf:


Run: certreq.exe -new req.inf Request.pem
Take the Request.pem file to the CA, submit the request and issue the certificate. View the certificate details and Copy to File...
Transfer the certificate back to the original computer (we'll call it Issued.cer). Also transfer the certificate for the CA to the client machine (not the private key or anything like that).

We need to install both certificates in the Local Computer account store.
Command-line version looks like this:
certutil.exe -encode Issued.cer Issued.pem
certutil.exe -addstore "root" windowsserver.example.local.cer
certreq.exe -accept Issued.pem

This puts the Windows CA in the Trusted Root folder and accepts (into the Personal folder) the issued certificate that we requested earlier.

With Windows XP, you must run these commands as an administrative user. Windows Vista can use the same commands and needs to be running as an administrator as well. However, certutil.exe, etc. may need to be a different version between XP and Vista.

You should be able to connect to the VPN without using a PSK now!

7) Finished!

This setup should work even if the client is behind a NAT (using NAT-T on udp/4500). One caveat I've noticed however is that the client cannot be initially on the same subnet as your organization's internal IP range. Meaning, if your company is using and your user happens to be using the same IP range at home, he or she will not be able to connect. For one, this is because Windows will not know how to route the packets. Is a particular packet to meant to go over the VPN or stay local?

Another caveat that I noticed in previous setups is that I don't think two computers can connect to the server from behind the same NAT. I haven't tested this for a while, so it may be fixed in these later versions.

This is obviously a very brief setup guide and I don't go into lots of details on how it all works. You can read some of the sources at the bottom for more information. However, if you do see a better way to do things, let me know. My e-mail address can be deduced from the very bottom of the document.

Hope things work for you!

A Few Referenced Sources

The earlier version of this document
Compiling Openswan on Ubuntu

I hope this helps someone. Let me know if there are errors above and I'll update this document.
-W Gillespie (wgillespie,

Last updated: 2011-08-19