New Test CGI Server

August–September 2021

CSE will soon replace the older www and cgi servers with a more modern configuration.

A prototype of this new configuration is now on the server Pepper, or: test-cgi.cse.unsw.edu.au

The new configuration is being reviewed so expect small, frequent changes to it as problems are found and fixed. This document will also be updated as the configuration is refined.

It is serving from the same directories (~/public_html/) as CSE's production CGI servers. Though it uses different, newer software so the way it serves content is not the same. While both old and new servers are working, there may be some changes you should make to your website configuration to allow both to work well.

There is documentation explaining how the older, production servers work in terms of CGI configuration and web authentication and authorisation. This document is intended to explain the key differences between the old and new server.

Logs of your Website

The logs of HTTP requests to your website can be seen here. The logs shown are only for the current day.

Your will need to authenticate to the server, and then your will only be able to see logs of requests for:

• your account website. Or,
• other accounts where your account is a member of the group associated with the other account.

HTTPS only

Content is only served over HTTPS by the new server. Requests on unencrypted HTTP are all redirected to HTTPS.

Old .htaccess configuration like the following is not needed and will be ignored by the new server: <IfModule !mod_ssl.c> RedirectMatch /(.*)$ https://cgi.cse.unsw.edu.au/$1 </IfModule>

Web queries will only be processed with mod_ssl enabled.

If you want to be extra sure queries only use HTTPS, use this directive in your .htaccess files: SSLRequireSSL

Apache Syntax Change

The webserver software for the new server is Apache version 2.4. This is a significant upgrade from the old servers running Apache v1.3.

This means the you may need to change the Apache directives you use in your .htaccess files. The Apache documentation includes a document explaining common changes.

Order, Allow, Deny

The Order, Allow and Deny directives were in the module mod_access. They are frequently used at CSE. They are deprecated syntax on the new server.

I have installed mod_access_compat in the new server. It allows use of the Order directive, but it is confusing to use it along with the new Require directive and doing so is not recommended.

If you have problems with your .htaccess configuration, you may be asked to disable instances of the Order directive for the new server before other problems are looked for.

Order deny,allow
Deny from all

Require all denied

Order allow,deny
Allow from all

Require all granted

Order Deny,Allow
Deny from all
Allow from 129.94.242.19

Require ip 129.94.242.19

The Require directive comes from mod_authz_core. The mod_authz_host module enhances that with "Require ip ..." and "Require host ..." access tests.

Example configuration that works on both servers: <IfModule !mod_authz_host.c> Order deny,allow Deny from all </IfModule> <IfModule mod_authz_host.c> Require all denied </IfModule>

Authentication and Authorisation

AuthYP no longer used

The old servers use the mod_auth_yp.c module with it's directives: AuthYP and AuthYPAuthoritative
The new server does not recognise those directives and will return a 500 Error page when it reads them.

The AuthYP module also provided this directive to check whether an authenticated user was a member of a NetGroup: Require group @netgroup_name Instead you will need to use: Require netgroup netgroup_name

Authentication

The new server uses mod_authn_sasl for authentication (checking user passwords).

If you share configuration between old and new webservers you can make the new webserver ignore old .htaccess config it by enclosing it like this:

Suggested change so both old and new servers work:

AuthName "CSE Users"
AuthType Basic
AuthYP On
Require valid-user

AuthName "CSE Users"
AuthType Basic
Require valid-user

AuthName "CSE Users"
AuthType Basic
<IfModule mod_auth_yp.c>
  AuthYP On
</IfModule>
Require valid-user

Authorisation

Authorisation (limiting access to certain groups) uses a custom module that is a wrapper for the innetgr function. In .htaccess files you would use it like this:

Suggested change so both old and new servers work:

AuthName "CSE users"
AuthType Basic
AuthYP On
require group @User

AuthName "CSE users"
AuthType Basic
require netgroup User

AuthName "CSE users"
AuthType Basic

<IfModule !mod_authz_netgroup.c>
  AuthYP On
  Require group @User
</IfModule>

<IfModule mod_authz_netgroup.c>
  Require netgroup User
</IfModule>

CGIwrapd replaced with suEXEC

Apache's standard suEXEC is used to run user CGI programs on the new server.

Files processed by the cgi-script content handler will be run using the account matching the home directory. In other words, your CGI scripts in your web directory will be run using your account.

On the old servers the CGIwrap daemon performed a similar task. There are unavoidable differences in the way these two systems operate.

To make a suEXEC CGI programs work:

• Associate the program file with the cgi-script handler. If you do not, the default-handler will likely process the file and return the contents of the file to the requester.
  (Likewise, the old servers would not run scripts as CGI unless their content-type was set to application/x-setuid-cgi.)

• File is owned by the account that will run it, and is not writable by another account. — Otherwise suEXEC will not run it and the webserver will instead return a 500 Error.
• Be executable by the account that will run them. (chmod u+rx …) Otherwise, suEXEC declines to run the script and the webserver returns a 500 Error.
• suEXEC will not run symbolic links as CGI. And so these will return a 500 Error.

• File not group-writeable nor in a directory that is group-writeable — or the webserver returns a 500 Error.
  … Possibly an overly cautious requirement, but that is how suEXEC is compiled — see point 14 in this reference.

  The earlier arrangement of setting directories with priv webonly will mean that suEXEC will not run the files inside as CGI.

We recommend that any file you want to run as CGI be set with restricted permissions. Eg: chmod 500 thing.cgi
This was the recommendation with the old servers also.

cgi-bin Subdirectory

All of ~/public_html/cgi-bin/ is set to use the cgi-script handler.

This includes directories, meaning that the directory index handler is not used and directory index pages are not returned by the server. — This may be a good thing.

Files which do not have the execute bit set will return a "Internal Server Error".

Outside cgi-bin

Outside your cgi-bin/ the webserver runs files as CGI if the filename ends with : .php .cgi (but not: .pl .py .sh)
All such files must have their execute permission set, or the new server will return an error page instead of running them. Other files, by default, do not run as CGI, but are served as-is to the client.

Your .htaccess may have set other files to run by associating them with handler application/x-setuid-cgi. For those, the new server returns an error. Instead, you will need to associate the file with the cgi-script handler for the new server.

See below for instructions on making your other files run as CGI.

PHP

Any file within ~/public_html with the extension .php and the execute-bit set (chmod u+x) is processed by the cgi-script handler and the php-cgi interpreter.

This is due to a setting for binfmt on the server which specifies /usr/bin/php-cgi as the interpreter for .php files. (Similar to the way #! at the start of a script specifies the interpreter for that script.)

Files with the extension .phps are currently set: Require all denied

Once people can use the new configuration for their websites we can test out PHP7 instead of PHP5.

application/x-setuid-cgi Handler

In the old servers these assigning these content-types triggered cgiwrap processing: application/x-setuid-cgi and application/x-setuid-cgid

The new server uses the cgi-script handler with suEXEC processing instead, and has not configured the old handler name. Instead the new server will return an error page where a resource is requested and it has an inappropriate content-type assigned, such as application/x-setuid-cgi.

Operation on both old and new servers can be configured in your .htaccess file like so:

<Files "thing.sh">
  SetHandler application/x-setuid-cgi
</Files>

<Files "thing.sh">
  SetHandler cgi-script
</Files>

<Files "thing.sh">
  <IfModule !mod_suexec.c>
    SetHandler application/x-setuid-cgi
  </IfModule>
  <IfModule mod_suexec.c>
    SetHandler cgi-script
  </IfModule>
</Files>

Stopping a file from being CGI

On the old CGI servers, you would set a file to not be content-type application/x-setuid-cgi. That would mean that it would not be associated with the Action /cgi-bin/php-cgiwrap and thus not be run by cgiwrapd. For example, to not run .py scripts in a directory but instead just serve the contents of those files you might have used this directive in a .htaccess file: AddType text/plain .cgi

The new server has a simpler arrangement to run CGI scripts. It associates the file extension directly with the web server handler cgi-script rather than through a media type. To make the new CGI server not treat files as CGI, you remove the association with the cgi-script handler: RemoveHandler .cgi

The two directives can be used together for both servers:

AddType text/plain .cgi

RemoveHandler .cgi

AddType text/plain .cgi
RemoveHandler .cgi

Server-Side Includes

Both the old and new CGI servers allow Server-Side Includes or HTML documents that the webserver inserts content into as it sends it to a client.

Simple webpage generation is possible with this without requiring more involved CGI scripting.

The New CGI server additionally allows use of the exec element, which can run scripts as the owning user account to generate content within a webpage.

Home directory mounting

The old webservers only mount /home/user/public_html at /web/user/. Web programs do not have access to the rest of the user home directory.

A prototype Autofs mounter is running on the test server to achieve a similar outcome. It only mounts the ~/public_html directory from user accounts in the normal home directory path. But not the whole homedir.

On the new server the suEXEC module only runs users' scripts that it can find at the correct user home directory path. (Within ~/public_html/)

Notes/Reference

Server Configuration

AddType application/x-setuid-cgi cgi pl py sh
AddType application/x-setuid-php php

<Directory "/web">
  AllowOverride All ≡ AuthConfig,FileInfo,Indexes,Limit,Options
  Options Indexes IncludesNOEXEC FollowSymLinks
  XBitHack Full
  Order deny,allow
  Allow from all
  Action application/x-setuid-php /cgi-bin/php-cgiwrap
  Action application/x-setuid-phpd /cgi-bin/php-cgiwrapd
  <Limit PUT>
   Order allow,deny
   Deny from all
  </Limit>
</Directory>

<Directory "/web/*/cgi-bin">
  DefaultType application/x-setuid-cgi
  AddType application/x-setuid-cgi .pl
  AddType application/x-setuid-cgi .py
  AddType application/x-setuid-cgi .sh
</Directory>

<FilesMatch ".+\.php$">
        SetHandler cgi-script
</FilesMatch>

# application/x-httpd-php-source                 phps
<FilesMatch ".+\.phps$">
        SetHandler application/x-httpd-php-source
        Require all denied
</FilesMatch>

<Directory "/web/*">

  AuthBasicProvider sasl

  AllowOverride AuthConfig FileInfo Indexes Limit Options
  Options Indexes Includes FollowSymLinks ExecCGI
  XBitHack Full
  AddHandler cgi-script .cgi
  AddHandler cgi-script .php
    # These are not run by CGI-Handler by default: .pl .py .sh
 <Limit PUT>
  Require all denied
 </Limit>

</Directory>

<Directory "/web/*/cgi-bin">

  AuthBasicProvider sasl

  Options -MultiViews +SymLinksIfOwnerMatch

  # Everything in cgi-bin is CGI.
  # Unfortunately that applies to directories which which would
  # otherwise be presented as webpages by mod_autoindex.
  Options +ExecCGI
  SetHandler cgi-script
</Directory>