Threats 

The threats we need to protect against are:

1. An attacker exploiting an httpd or other service vulnerability to take over the apache account.
2. An attacker exploiting a web application vulnerability to take over a user account.
3. A user accessing privileged information belonging to another user, or changing data belonging to another user, or an attacker doing the same after succeeding at (2).
4. A user exploiting a local root vulnerability to take over the machine, or an attacker doing the same after succeeding at (1) or (2).

1. httpd exploit

By exploiting a remote vulnerability in httpd, an attacker could gain control of the apache account.  A similar attack might take place against any other service running on the machine, possibly resulting in immediate root access if the other service runs as root.

Such an attack would likely disrupt the functioning of the entire service while the vulnerability responsible for the attack is uncovered and the machine is reinstalled.  User data could be lost or corrupted and might need to be restored from backup, although the Apache account should not have access to most user content.

Mitigation: run as few services on the machine besides httpd as possible.  Be alert to publically known httpd vulnerabilities and patch them as quickly as possible.

 It may be possible to use SELinux to minimize the potential impact of an httpd or other service exploit.  More research is required.

 2. Web application exploits

Web applications are generally implemented in PHP or another scripting language and are not usually subject to buffer overflow attacks.  Nevertheless, they have traditionally contained a rich variety of vulnerabilities, generally of one of the following forms:

• Shell command injection: execution of shell commands containing unsanitized user input, allowing attackers to execute arbitrary shell code
• SQL injection: execution of SQL queries containing unsanitized user input, allowing attackers to execute arbitrary SQL code
• Inclusion vulnerabilities: inclusion of other scripting code via a variable path containing unsanitized user input, allowing attackers to execute arbitrary scripting code
• Cross-site scripting: display of unsanitized HTML provided by one user to another, allowing attackers to execute Javascript code on another user's browser in the security context of the web application

Moreover, these vulnerabilities are the subject of widespread and frequent exploits.  Mitigation of this risk will come in two forms:

1. Reduce the frequency of compromises.  To the extent that we foster development of custom-written web applications, educate developers about these attacks and how to prevent them.  For off-the shelf web applications, provide cookie-cutter installations, monitor security sites for known exploits, and have a framework for fixing the cookie-cutter installations users have performed.
2. Reduce the severity of compromises.  No matter what we do, user account compromises will be endemic to our service, so we need to provide good tools for recovering from attacks (frequent user-accessible backups, for example) and to ensure that attacks do not spread to other users of the service.
3. Make use of intrusion detection tools such as mod_security to prevent common attacks and to detect compromises so that we may disable compromised user accounts before they infect visiting clients or cause further damage.

3. User boundaries

This is a complicated topic and is best considered in pieces.  This is also an area where we are in conflict with the world of web hosting at large.  Most commercial web services do not insulate users from one another, and most web application installation instructions only concern themselves with instructing users how to break down what barriers exist, not how to work within them.

3a. Static content

Public static content (e.g HTML files) is not an issue as long as it is only writable by users other than the owner.  However, some number of users will naively set their static content to be group or other-writable.  It may be necessary to protect such users against themselves.

Private static content (anything protected by a .htaccess file) must be readable by the apache user if httpd is going to be able to serve it normally.  This means making those files world-readable, which is an issue if it allows other users to read the content without going through httpd access controls.  If all users are placed within the same group, the files can be protected against other users by making them not group-readable--assuming users have no ability to read the content from the apache account using dynamic content features (see below).  Again, it may be necessary to protect users against themselves.

Mitigation measures:

• Place all users in  the same group.
• Set the default homedir mode to 701 and the default public_html subdir mode to 705.
• Set the default umask to 072.
• Run a system-scanning process which masks all user file permissions with ~072 (removing all group permissions and world-write permissions) unless the user opts out by placing a magic flag file in the top level of the home directory.
• Do not allow users to execute dynamic content using the apache uid.

SIPB's scripts.mit.edu service has, for various reasons, opted to make static content read by the uid of the content owner, rather than by the apache accounts.  This decision comes at a hefty penalty in performance and complexity, and should not be necessary in our environment as long as we adhere to the above principles.

It may be possible to use SELinux to more robustly isolate user accounts from one another.  More research is required.

3b. Dynamic content

Dynamic content features in a web hosting environment include CGI or FastCGI scripts, scripting modules (mod_php, mod_perl, etc.), and server-side includes.  Most of these mechanisms are not limited in power; they allow dynamic content to take arbitrary actions including executing shell commands.  mod_php can be configured to be limited in power, but not in a robust manner, and generally only at the expense of compatibility with pre-existing web applications.

It is traditional in web hosting services to execute dynamic content as the apache account, possibly within the httpd worker process itself.  This practice is grossly insecure; although the apache account may be unable to modify any system files, all user dynamic content becomes undifferentiated in terms of what actions it can take.  If my dynamic content can read my database password from a file in my home directory, your dynamic content can do the same.  Some web hosting providers give the option of executing dynamic content as your own uid (using something like cgiwrap) so that you could protect files from other users, but any cookie-cutter installation scripts they provide generally do not set up this feature.

Our architecture will be to execute all dynamic content as the uid of the content owner, not as the httpd account.  There are a variety of technologies to do this with; so far the best option appears to be a combination of the built-in Apache httpd suexec mechanism and FastCGI.  In the future, httpd 2.x may provide more comprehensive support for this architecture, allowing httpd worker processes to adopt the uid of the content owner and then process content through standard mechanisms such as mod_php.  Idle worker processes could be reused if they match the target uid of a new request, eliminating the need for FastCGI.

There is one notable downside to executing dynamic content as the uid of the content owner: an attacker who compromises a web application will have write access to the content owner's entire account, and not just the files which must be writable to web applications.  To solve this problem, ideally there would be two uids per user: one for login access, the other for execution of dynamic web content.  The uids would live within the same group (otherwise unique) so that group permissions could be used to regulate read and write access for dynamic content.  Unfortunately, this model is probably too exotic to implement using standard tools.

3c. Cookie theft and cookie injection

A well-behaved web application will send cookies to clients containing a proper path value, so that the cookies are only sent back to the same web application which generated them.  Many web applications, however, set a cookie path of "/", which will cause the cookies to be transmitted to any other application in the same domain.  A malicious application merely has to inspect the cookies received from a client in order to steal any privileged information contained within.

In a similar vein, a well-behaved web application will check that all received cookies contain the expected domain and path values, but some applications do not check.  If an application does not check, it is easy for a malicious application in the same domain to inject a cookie containing a value controlled by the attacker.

There is no systemic mitigation for these risks other than encouraging the use of virtual domains.  They should be treated as application security issues.  Note that cookie injection is not a problem unique to applications running within the same domain; any attacker controlling content on a foo.mit.edu host can inject cookies into all .mit.edu hosts if the victim applications do not check cookie domains.

3d. Browser security

Browsers treat the protocol, host, and port of a URL as a protection domain, allowing a variety of attacks between multiple content owners within the same domain.  As one example: 

Basic auth is a rudimentary password protection system built into HTTP.  Most web applications do their own password management, but static content is sometimes protected with Basic auth.  Once a user enters a password for a given resource, the browser caches the password, so that the user does not have to reenter it each time the resource is accessed.  A Basic auth resource is identified by the domain part of the URL and a server-provided realm.  In a shared hosting environment, the normal Apache httpd mechanisms for doing Basic auth allow each content owner to pick the realm used for each resource.  An attacker can steal passwords from users of another content owner's resource by choosing the same realm name as the other content owner.

More generally, Javascript code can be used to attack content owned by someone else using the same domain.  To mitigate these attacks, we should use a separate domain for each content owner.  This can be easily achieved with a wildcard DNS record, so that user sites look like http://mv.ezproxy.com.ezproxy.canberra.edu.au/ instead of http://mv.ezproxy.com.ezproxy.canberra.edu.au/~username.

3e. PHP session hijacking

We should be sure to arrange for each user to use a separate PHP session directory, probably in a mode-700 subdir of the user's homedir, to avoid the possibility of users hijacking one another's sessions.

4. Local root vulnerabilities

A local root vulnerability could allow a regular user to take control of the entire hosting service, or could allow an attacker who has taken control of a user account (or the apache account) to take control of the entire service.  Recovering from a compromise of this sort would require disrupting the web hosting service, possibly for an extended period of time, while the server is reinstalled and user content is restored from backup.

Mitigation options:

• Audit the system for setuid programs and remove any unnecessary packages containing them.  Turn off the setuid bit on the remaining setuid programs which don't need to be setuid.
• Keep the system up to date.
• Treat kernel security updates as high-priority updates; schedule a reboot to install kernel updates on the evening of the upstream kernel package release at the latest.
• Use a Linux distribution which closely tracks the upstream kernel.org sources and run a custom-built kernel based on those sources, thus allowing us to apply kernel patches easily.
• Deploy a clustered environment, allowing individual servers to have their kernels upgraded immediately without a visible service outage.

It may be possible to use SELinux to reduce the likelihood of a local root privilege escalation.  More research is required.

References

apachesecurity.net -- the web site for an O'Reilly book on securing Apache installations.  A primary source of ideas and material for this document.  Solid material, though occasionally dated or slightly inaccurate (as can be expected with almost any technology book).

Honeynet report on web application threats -- a report by the Honeynet project on common attacks against web applications.

SANS Top 20 -- the web application section of a SANS report on security threats.  Contains recommendations on hardening web application servers.

owasp.org -- web site for the Open Web Application Security Project, an organization dedicated to improving web application security.  They have a guide to writing secure web application code.