OpenSwan setup between NSS and Non-NSS Versions

This is not a complete instruction set for how to create an ipsec tunnel using OpenSwan, rather it is for those people who had working tunnels already until they upgraded one of the sides to OpenSwan newer than 2.6.23 where nss is required.

Errors:

CKAIDNSS keyword not found where expected in RSA key
STATE_MAIN_I2: sent MI2, expecting MR2
003 "connectionname" #1: Can't find the private key from the NSS CERT (err -12285)
108 "connectionname" #1: STATE_MAIN_I3: sent MI3, expecting MR3
003 "connectionname" #1: ignoring informational payload, type INVALID_KEY_INFORMATION msgid=00000000
ignoring unknown Vendor ID payload [494b457632]
003 "connectionname" #4: Signature check (on @ computer1 ) failed (wrong key?); tried *AQO29Jk1X
217 "connectionname" #4: STATE_MAIN_I3: INVALID_KEY_INFORMATION

Delete cert8.db from /etc/ipsec.d
Turn off FIPS to avoid certutil forcing you to use a password
modutil -fips false -dbdir /etc/ipsec.d
Create cert8.db
certutil -N -d /etc/ipsec.d
You should not have to enter a password. Press ENTER for the passwords so you don't have to create a nsspassword file
Create a machine certificate. This may not be required.
certutil -S -k rsa -n cacert1 -s "CN=cacert1" -v 12 -d . -t "C,C,C" -x -d /etc/ipsec.d
Create a host key ipsec newhostkey --bits 1024 --configdir /etc/ipsec.d --verbose --output /etc/ipsec.d/hostkey.secrets
Note: You can speed up the waiting period by logging on a second screen and making the computer work harder, such as using the find command or editing something. Otherwise this command can seem to hang.

Now you can grab the public key from the hostkey.secrets file and put it into the configuration file just like the prior versions and it should work.

Last Modified 11/08/2010
Lee Lofgren