Core 2.x IdP run book

This page is under construction

Executive Summary

Version 2 of the MIT core identity provider is based on version 2.1.x of Internet2's Shibboleth IdP package. Including the IdP software itself, the following major components are required:

Apache httpd 2.2 (from stock RHEL httpd RPM)
mod_ssl (from stock RHEL mod_ssl RPM)
mod_auth_kerb (from stock RHEL mod_auth_kerb RPM)
Tomcat 6.0
JDK 6.0 (from Sun, plus enhanced JCE and security policy)
Shibboleth IdP 2.1
terracotta 3.1
MySQL 5.0 (from stock RHEL RPMs)

In the configuration documented below, the Apache web server will listen on the following TCP ports:

80 (HTTP)
443 (SSL virtual host for HTTPS)
446 (SSL virtual host for authenticating via user certificates)
447 (SSL virtual host for certificate authentication using "SSLVerifyClient require")
8443 (SSL virtual host for SP's back-channel SOAP calls for attributes)

The terracotta server will listen on the following TCP ports:

9510 (client-to-server)
9530 (server-to-server)
Note these listeners only need to accept connections from peer servers in the cluster, so these ports should be configured accordingly in the firewall.

The following need to be created for use by Kerberos and SSL:

MIT SSL server certificate (CN idp.mit.edu)
HTTP keytab (i.e. HTTP/idp.mit.edu@ATHENA.MIT.EDU)

The following log files will be used:

Apache httpd log files in /var/log/httpd/:
- ssl_access_log
- ssl_request_log
- ssl_error_log
- idp-attr-query_access_log
- idp-attr-query_request_log
- idp-attr-query_error_log
- cert-authn-optional_access_log
- cert-authn-optional_request_log
- cert-authn-optional_error_log
- access_log
- error_log
Shibboleth IdP log files in /usr/local/shibboleth-idp/logs/:
- idp-process.log
- idp-access.log
- idp-audit.log
Tomcat logs in /usr/local/tomcat/logs/
- catalina.out
terracotta system logs in /usr/local/terracotta/logs/:
- terracotta.log
- run-dgc.cron.log
terracotta cluster logs in /usr/local/shibboleth-idp/cluster/:
- client/logs-127.0.0.1/terracotta-client.log
- server/logs/terracotta-server.log

Install and configure Apache httpd

Install needed RPMs

Use stock httpd RPM install (standard NIST install)

Install mod_ssl and mod_auth_kerb RPMs:

# yum install mod_ssl
# yum install mod_auth_kerb

Configure

Current versions of the various httpd configuration files can be obtained in the touchstone locker, in /mit/touchstone/config/idp2-core/httpd/.

Install the server certificate, key, and CA files in /etc/pki/tls/certs/ and /etc/pki/tls/private/, as appropriate, and make sure the paths are correct in ssl.conf, cert-authn.conf, and idp-attr-query.conf (see below). The key file should be readable by only the tomcat user, as the idp webapp also uses it.
In /etc/httpd/conf/httpd.conf, set ServerName:
```
ServerName idp.mit.edu:80
```
and set the UseCanonicalName option to On:
```
UseCanonicalName On
```
Disable the stock "Welcome" page, by commenting out the lines in /etc/httpd/conf.d/welcome.conf

In /etc/httpd/conf.d/ssl.conf, set the SSLRandomSeed options:

SSLRandomSeed startup file:/dev/urandom 1024
SSLRandomSeed connect file:/dev/urandom 1024

within the VirtualHost block, set the ServerName:

ServerName idp.mit.edu:443

set the SSL cipher suite:

SSLCipherSuite HIGH:MEDIUM:EXP:!aNULL:!SSLv2:+SHA1:+MD5:+HIGH:+MEDIUM:+EXP

Install the server certificate, key, and CA files in /etc/pki/tls/certs/ and /etc/pki/tls/private/, as appropriate, and set the paths in ssl.conf:

SSLCertificateFile /etc/pki/tls/certs/idp.mit.edu-cert.pem
SSLCertificateKeyFile /etc/pki/tls/private/idp.mit.edu-key.pem
SSLCertificateChainFile /etc/pki/tls/certs/EquifaxCA.pem
SSLCACertificateFile /etc/pki/tls/certs/mitCAclient.pem

set the SSL options:

SSLOptions +StrictRequire

configure custom logging:

CustomLog logs/ssl_request_log \
    "%t %h %{HTTPS}x %{SSL_PROTOCOL}x %{SSL_CIPHER}x %{SSL_CIPHER_USEKEYSIZE}x %{SSL_CLIENT_VERIFY}x \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\""

ensure that all access is via SSL:

<Directory />
    SSLRequireSSL
</Directory>

ensure that all rewrite rules are inherited:

RewriteEngine On
RewriteOptions inherit

Install an HTTP keytab in /etc/httpd/conf/keytab; it must be readable by (only) the apache user. This is used for authentication by HTTP/SPNEGO, as configured in /etc/httpd/conf.d/auth_kerb.conf.
Install the server's InCommon certificate and key files in /usr/local/shibboleth-idp/credentials/ – these are used for back-channel attribute queries by InCommon SPs, and are referenced in /etc/httpd/conf.d/idp-attr-query.conf. Make sure that they are readable by tomcat, as the idp webapp also accesses them. Also install the InCommon CA file in /etc/pki/tls/certs.
Install these additional conf files from the touchstone locker (/mit/touchstone/config/idp2-core/httpd) in /etc/httpd/conf.d:
- auth_kerb.conf
  This configures authentication by HTTP/SPNEGO (replaces version installed by mod_auth_kerb).
- cert-authn.conf
  This sets up the separate virtual hosts for certificate authentication on ports 446 and 447 (for certificate optional and required, respectively).
- idp-attr-query.conf
  This sets up the vhosts for back-channel attribute queries on ports 8443 and 8444 (from MIT and InCommon SPs, respectively).
- idp-rewrite.conf
  This adds various rewrite rules for compatibility, etc.
- proxy_ajp.conf
  Configures the AJP proxy module for the idp webapp (replaces version installed by httpd).
- ssl.conf (see above)
- welcome.conf (see above)
SELinux context mappings must be added for the non-standard ports configured for certificate authentication and attribute query, so that httpd can use them:
```
# semanage port -a -t http_port_t -p tcp 446
# semanage port -a -t http_port_t -p tcp 447
# semanage port -a -t http_port_t -p tcp 8444
```
Install /var/www/cgi-bin/negotiate-error-handler, from /mit/touchstone/cgi/. This script is used as the ErrorDocument 401 handler for HTTP/SPNEGO authentication, and is needed because handling the error behind the AJP proxy would break the Negotiate option.
Install our standard robots.txt and favicon.ico files in /var/www/html. The robots.txt should disallow all access:
```
User-agent: *
Disallow: /
```
Current versions of these files may be found in the touchstone locker, in /mit/touchstone/config/htdocs/.
Make sure httpd is started at boot time:
```
# chkconfig httpd on
```
Add the following settings to the stock /etc/logrotate.d/httpd configuration file:
```
daily
rotate 100
compress
delaycompress
```
This will cause the httpd log files in /var/log/httpd/ to be rotated daily and compressed, saving 100 days of old logs (in case we need them for quarterly metrics).
Install JDK and enhanced JCE

The IdP uses JDK 1.6; download and install the RPM from Sun, or use the version in the downloads directory in the touchstone locker (jdk-6uNN-linux-amd64.rpm, where NN is the update number):
```
# rpm -Uvh jdk-6uNN-linux-amd64.rpm
```

We installed the Bouncy Castle JCE in the 2.1.5 IdPs, but it should no longer be needed. To support additional cryptographic algorithms used by the IdP, download and install the Bouncy Castle JCE jar file (http://polydistortion.net/bc/index.html) in the lib/ext directory of the JRE (/usr/java/latest/jre/lib/ext/). For example:
```
# cd /usr/java/latest/jre/lib/ext
# cp /path/to/bcprov-jdk16-145.jar .
```
(Replace the file version number as needed).

Add it as a provider in in the JRE's lib/security/java.security, e.g.:

security.provider.9=org.bouncycastle.jce.provider.BouncyCastleProvider

(Replace 9 with the next sequential provider number as needed).

To support use of crypto key sizes larger than 2048 bits, we also add the Unlimited Strength Security Policy to the JVM. Download jce_policy-6.zip from the locker downloads directory, or from Sun (http://java.sun.com/javase/downloads/index.jsp, Other Downloads section at the bottom). Unzip the policy zip file and copy local_policy.jar and US_export_policy.jar into the JRE's lib/security directory (replacing the versions installed from the JDK RPM).
```
# cd /tmp
# unzip /path/to/jce_policy-6.zip
# cd jce
# cp *.jar /usr/java/latest/jre/lib/security/
```

The MIT CA certificates must be added to the certificate trust store. This can be done by downloading and adding them explicitly to the JRE cacerts trust store, as follows:

# wget 'http://ca.mit.edu.ezproxy.canberra.edu.au/mitca.crt'
# wget 'http://ca.mit.edu.ezproxy.canberra.edu.au/mitClient.crt'
# setenv JAVA_HOME /usr/java/latest
# cd $JAVA_HOME/jre/lib/security
# cp -p cacerts cacerts.orig
# $JAVA_HOME/bin/keytool -import -keystore cacerts -alias mitca -file /path/to/mitca.crt
# $JAVA_HOME/bin/keytool -import -keystore cacerts -alias mitclientca -file /path/to/mitClient.crt

The password for the trust store is "changeit". Answer "yes" to the "Trust this certificate?" prompt

An already-updated cacerts store is available in the touchstone locker, in /mit/touchstone/config/java.

For convenience, install shell profile scripts in /etc/profile.d that define JAVA_HOME, e.g. java.csh:

setenv JAVA_HOME /usr/java/default
if ( "${path}" !~ *${JAVA_HOME}/bin* ) then
    set path = ( ${JAVA_HOME}/bin $path )
endif

java.sh:

export JAVA_HOME=/usr/java/default
if ! echo $PATH | grep -q ${JAVA_HOME}/bin ; then
    export PATH=${JAVA_HOME}/bin:$PATH
fi

Install Tomcat

Download current Tomcat 6.0 binary distribution (tested with 6.0.20, available in /mit/touchstone/downloads/apache-tomcat-6.0.20.tar.gz, and install under /usr/local:
```
# cd /usr/local
# tar xzf /path/to/apache-tomcat-6.0.20.tar.gz
# rm -f tomcat
# ln -s apache-tomcat-6.0.20.tar.gz tomcat
```

Create the tomcat user, and change the ownership of the tomcat tree:

# groupadd -g 52 tomcat
# useradd -u 52 -g tomcat -c "Tomcat User" -d /usr/local/tomcat -M -s /sbin/nologin tomcat
# chown -R tomcat:tomcat /usr/local/apache-tomcat-6.0.20

Install our version of conf/server.xml (from /mit/touchstone/config/idp2-core/tomcat), which properly configures the AJP connector on port 8009, and disables the HTTP connector on port 8080.

Install the tomcat init script in /etc/init.d/, and make sure tomcat is started at boot time:
```
# chkconfig --add tomcat
```

Install Shibboleth IdP

Run the idp application installer from our customized binary distribution, available in /mit/touchstone/builds/NIST/idp2-mit/mit-shibboleth-identityprovider-2.x.y-bin.tgz, and the install script contained therein. For example:
```
# cd /tmp
# rm -rf shibboleth-identityprovider-2.*
# tar xzf /path/to/mit-shibboleth-identityprovider-2.1.5-bin.tgz
# cd shibboleth-identityprovider-2.1.5
# ./install.sh
[There should be no need to override the default responses to the installer's questions.]
```
By default (because of one of our customizations to the stock Internet2 distribution) this will install under /usr/local/shibboleth-idp/. The installer will not overwrite the configuration files of an existing installation. For a new installation, the installer will generate a keystore, and prompt for its password; currently we do not use this keystore, so the password does not matter. This distribution contains the standard shibboleth-identityprovider binary distribution, from the Internet2 zip file (http://shibboleth.internet2.edu/downloads/shibboleth/idp/latest/), plus the following customizations:
- MultiAuthnOptionLoginHandler
  This our custom login handler extension, which supports multiple authentication options (certificate, username/password, etc.). The current jar file for the extension is available in the locker, in /mit/touchstone/builds/NIST/shibboleth-idp-ext-MultiAuthnOptionLoginHandler-<version>.jar, where <version> is 1.0 for idp version 2.1.x, and 1.1 for idp version 2.3.x; it is copied into the lib/ subdirectory of the binary distribution.
- mitlogin
  This provides the custom login pages for MIT users, the web deployment descriptor, etc. It is available as a tarball (/mit/touchstone/builds/NIST/idp2-mit/mitlogin.tgz) which is unpacked into the top-level directory of the binary distribution.
- MySQL connector
  This is needed for accessing our targeted ID MySQL database. The connector is available in /mit/touchstone/downloads/mysql-connector-java-5.1.NN-bin.jar (current version is 5.1.14).
- vt-ldap-2.8.5.jar
  This is an updated version of the IdP's LDAP package. We currently use this in version 2.1.5 (instead of version 2.8.4 of the jar file) because the older version does not properly handle certain error conditions with the connection to the LDAP server. This should not be needed in future releases.
The installer will create and populate /usr/local/shibboleth-idp; the web application (war) file will be in /usr/local/shibboleth-idp/war/idp.war, but the current version of the idp.war will be available in the locker (/mit/touchstone/builds/NIST/idp2-mit/idp.war).
The idp application, running under Tomcat, needs full access to the install directory, so make sure it is owned by the tomcat user, e.g.:
```
# chown -R tomcat:tomcat /usr/local/shibboleth-idp
```
To ensure that we run the current version of the web application, download the latest idp.war file from the touchstone locker (/mit/touchstone/builds/NIST/idp2-mit/idp.war) and copy it into /usr/local/tomcat/webapps/:
```
# cp /path/to/idp.war /usr/local/tomcat/webapps/
# chown tomcat:tomcat /usr/local/tomcat/webapps/idp.war
```

Copy the idp's endorsed jar files to tomcat's endorsed dir:

# mkdir -p /usr/local/tomcat/endorsed
# cp -p /usr/local/shibboleth-idp/lib/endorsed/*.jar /usr/local/tomcat/endorsed/
# chown -R tomcat:tomcat /usr/local/tomcat/endorsed

Copy in the idp config files for the server, to the conf subdirectory (they are available in /mit/touchstone/config/idp2-core/shibboleth-idp/); these include:
- attribute-filter.xml
- attribute-resolver.xml.in
- authn-config.properties
- handler.xml
- internal.xml
- logging.xml
- login.config
- relying-party.xml
- service.xml
- tc-config.xml (for terracotta clustering)
  You must replace %%LDAPUID%% and %%LDAPPASSWORD%% in attribute-resolver.xml.in with the principal uid (e.g. touchstone-core-service) and passord (principalCredential attribute) for accessing our LDAP server, and save the resulting file as attribute-resolver.xml. Make sure the file is not world-readable.

Terracotta

(See https://spaces.internet2.edu/display/SHIB2/IdPCluster)

The terracotta software is used to cluster the IdP nodes. Each node must run the terracotta server, as well as the instrumented client (tomcat, in our case). The terracotta server operates in either the active or passive role; only one server should be in the "active/coordinator" state at a time.

Download the terracotta tarball; our current version is in the touchstone locker, in /mit/touchstone/downloads/terracotta-x.y.z.tar.gz. Extract it under /usr/local, create a logs directory for it, make it owned by the tomcat user, and symlink /usr/local/terracotta to it. For example (replace 3.1.1 with the appropriate terracotta version number):

# cd /usr/local
# tar xzf /path/to/terracotta-3.1.1.tar.gz
# mkdir -p terracotta-3.1.1/logs
# chown -R tomcat:tomcat terracotta-3.1.1
# rm -f terracotta
# ln -s terracotta-3.1.1 terracotta

The IdP requires the installation of a couple of Terracotta Integration Modules, and the generation of a boot jar file for Tomcat, which is specific to the Java version:

# setenv TC_HOME /usr/local/terracotta-3.1.1
# setenv TC_INSTALL_DIR $TC_HOME
# setenv JAVA_HOME /usr/java/default
# $TC_HOME/bin/tim-get.sh install tim-vector 2.5.1 org.terracotta.modules
# $TC_HOME/bin/tim-get.sh install tim-tomcat-6.0 2.0.1
# $TC_HOME/bin/make-boot-jar.sh -f /usr/local/shibboleth-idp/conf/tc-config.xml
# chown -R tomcat:tomcat /usr/local/terracotta-3.1.1

Be sure to regenerate this jar after installing a new JDK.

Install the init script from /mit/touchstone/maint/shibboleth-idp/terracotta/terracotta.init in /etc/init.d, and make sure it is configured to start at boot time. Note that terracotta must be started before tomcat.

# cp /path/to/terracotta.init /etc/init.d/terracotta
# chmod 755 /etc/init.d/terracotta
# chkconfig --add terracotta

To avoid performance impact during business hours, we disable automatic garbage collection of terracotta objects. Instead, we run a nightly cron job to do the garbage collection manually. Since this should only be done on the active/coordinator node, the script, run-dgc-if-active.sh, checks the server mode, then runs the garbage collector if and only if the server is the active node. Both the script and cron file can be obtained in /mit/touchstone/maint/shibboleth-idp/terracotta/; install as follows:

# cp /path/to/run-dgc-if-active.sh /usr/local/shibboleth-idp/bin/
# cp /path/to/run-dgc.cron /etc/cron.d/run-dgc

Targeted ID MySQL database

The core IdP uses a custom implementation supporting the generation of targeted (or persistent) IDs, backed by a MySQL database. We use the native Red Hat RPMs (5.0), part of the standard NIST install; the MySQL-python RPM is required for the synchronization daemon and supporting scripts.

MySQL initialization

Start up the daemon, and secure the installation:

# /etc/init.d/mysqld start
# mysql_secure_installation

Respond to the prompts to set the root password, remove anonymous users, disallow remote root logins, and remove the test database.

Make sure the daemon starts at boot time:

# chkconfig mysqld on

Make sure that you set a firewall rule which allows the peer IdP node to connect to the daemon (on TCP port 3306).

Create database users

Create the shib and (optionally) shibadmin database users, e.g.:

# mysql -u root -p
Enter password: [Supply the root password created above]

mysql> CREATE USER 'shib'@'localhost' IDENTIFIED BY 'PASSWORD';
Query OK, 0 rows affected (0.00 sec)

mysql> CREATE USER 'shib'@'idp-1.mit.edu' IDENTIFIED BY 'PASSWORD';
Query OK, 0 rows affected (0.00 sec)

mysql> CREATE USER 'shib'@'idp-2.mit.edu' IDENTIFIED BY 'PASSWORD';
Query OK, 0 rows affected (0.00 sec)

mysql> CREATE USER 'shibadmin'@'localhost' IDENTIFIED BY 'ADMINPASSWORD';
Query OK, 0 rows affected (0.00 sec)

mysql> CREATE USER 'shibadmin'@'idp-1.mit.edu' IDENTIFIED BY 'ADMINPASSWORD';
Query OK, 0 rows affected (0.00 sec)

mysql> CREATE USER 'shibadmin'@'idp-2.mit.edu' IDENTIFIED BY 'ADMINPASSWORD';
Query OK, 0 rows affected (0.00 sec)

Replace PASSWORD and ADMINPASSWORD with the passwords for the shib and shibadmin users, respectively. The shib user will be used by the targeted ID software to access the database. The shibadmin user can be used as an alternative to root to initialize, update, or backup the database. Note the shib password will need to be set in the idp attribute-resolver.xml file, as well as in the MySQL defaults file tid.cnf (see below). If the shibadmin account is used for the database backup, its password will need to be set in the admin.cnf MySQL defaults file.

Install the Targeted ID software

Install the scripts used to maintain the targeted ID database. This includes the tid-syncd daemon, which is used to synchronize the database between the machines in the cluster, an init script for the daemon, and a database backup script (run out of cron). The software is installed from the source tarball in /mit/touchstone/src/targeted-id-source.tgz.

# mkdir /tmp/targeted-id
# cd /tmp/targeted-id
# tar xzf /path/to/targeted-id-source.tgz
# make install

Create /usr/local/targeted-id/etc/tid.cnf, if necessary, and set the password for the shib database user (from above); you can copy tid.cnf.example in that directory, and simply set the password accordingly. The file should be readable only by root. Also create /usr/local/targeted-id/etc/admin.cnf, if necessary, which is used by the database backup script; the shibadmin MySQL user account can be used for the backup.

Initialize the Targeted ID database

You must either set up a new (empty) database, or initialize the database from a backup (e.g. a backup created on the peer system).

Initialize a new database

WARNING: THIS STEP WILL DROP THE TABLES OF ANY EXISTING DATABASE. Proceed to the next step to initialize the database from a backup.

To create a new (empty) database, process the schema file, e.g.:

# mysql -u root -p < /usr/local/targeted-id/etc/tid-init.sql

Initialize a Targeted ID database from backup

Perform this step to set up the database when adding a new machine to an existing cluster. You should obtain the backup from an existing machine in the cluster.

# mysql -u root -p < /path/to/most-recent-backup.sql

Load the targeted ID stored function into the database

The IdP resolver will call a stored function tid to generate and retrieve the targeted ID as the attribute source. The source for this function is in /usr/local/targeted-id/etc/tid.sql. Note that the function uses 2 hard-coded "secret" strings to randomize the generated IDs. It is imperative that you set these 2 strings in the function source before loading it, and that all nodes in a cluster always use the same 2 secret strings. To set the secret strings, make a copy of tid.sql (to, say, tid.sql.private), edit the copy, locate the declarations of mySecret1 and mySecret2, and replace the secret_1 and secret_2 string literals accordingly.

    DECLARE mySecret1 VARCHAR(255) DEFAULT 'secret_1';
    DECLARE mySecret2 VARCHAR(255) DEFAULT 'secret_2';

Make sure the resulting file is only readable by root. Once you have correctly set these strings, you can load the function as follows:

# mysql -u root -p targetedID < /usr/local/targeted-id/etc/tid.sql.private

Set up the grant tables

Once you have loaded the database tables and stored function, you must set up the grant tables for the shib and shibadmin database users. The file /usr/local/targeted-id/etc/tid-grants.sql contains the necessary grants for users on the local machine; you should modify this to add the same grants for the users on the peer machine.

# mysql -u root -p < /usr/local/targeted-id/etc/tid-grants.sql

Set up the tid-syncd daemon

In all but exceptional cases, a generated ID will be the consistent result of hashing the username, provider ID, and secret strings, so the ID for a (user, SP) pair will be the same, no matter which node in a cluster it is generated on. This may not be true, though, in the (highly unlikely) case of a hash collision, or if an ID needs to be revoked. To make sure that the databases in a cluster remain in synch, therefore, we employ a tid-syncd daemon which propagates new IDs to the peer(s). This daemon should run on all nodes, and be started at boot time. The daemon logs at the LOCAL5 syslog facility, so /etc/syslog.conf should be adjusted accordingly:

# targeted ID sync daemon
local5.*                                       /var/log/tid-syncd

Start the daemon, and make sure it is started at boot time:

# /etc/init.d/tid-sync start
# chkconfig --add tid-sync

Firewall

Make sure that the additional port used by the IdP are enabled in the firewall. Use the command "iptables --list -n --line-numbers" to determine the proper rule number; the following example assumes we are inserting rules beginning at number 36. Also replace 18.x.y.z with the appropriate IP address of the peer node in the cluster, not the local host.

# iptables --list -n --line-numbers
# iptables -I RH-Firewall-1-INPUT 36 -m state --state NEW -m tcp -p tcp --dport 8443 -j ACCEPT
# iptables -I RH-Firewall-1-INPUT 37 -m state --state NEW -m tcp -p tcp --dport 8444 -j ACCEPT
# iptables -I RH-Firewall-1-INPUT 38 -m state --state NEW -m tcp -p tcp --dport 446 -j ACCEPT
# iptables -I RH-Firewall-1-INPUT 39 -m state --state NEW -m tcp -p tcp --dport 447 -j ACCEPT
# iptables -I RH-Firewall-1-INPUT 40 -m state --state NEW -m tcp -p tcp -s 18.x.y.z --dport 9510 -j ACCEPT
# iptables -I RH-Firewall-1-INPUT 41 -m state --state NEW -m tcp -p tcp -s 18.x.y.z --dport 9530 -j ACCEPT
# iptables -I RH-Firewall-1-INPUT 42 -m state --state NEW -m tcp -p tcp -s 18.x.y.z --dport 3306 -j ACCEPT
# /etc/init.d/iptables save

Child pages