m1tk4

ääliö, älä lyö, ööliä läikkyy!

Apache2 MPM-itk on RHEL/CentOS

Posted in Uncategorized by mitka on the May 11th, 2006

When you have to host several web applications on the same server, with each new one you probably feel less and less comfortable with all of them running as apache or nobody. This problem gets even bigger when some of your applications need to be able to write files to locations where they can later be executed by a http call, or even to be able to rewrite the application’s executable files themselves (to update, upgrade or install various “plugins”, for example).

suEXEC and What’s Wrong With It

The traditional solution to this problem was a small utility called suEXEC that ships with Apache. Here is how suEXEC works:

  1. You configure each virtual host to use its own user credentials, using SuexecUserGroup directive.
  2. Apache runs as always, but whenever it’s the time to call a CGI script, instead of calling it directly, Apache starts suEXEC.
  3. suEXEC checks around whether the request is “legit” and if it is, starts your CGI under the privileges specified in your SuexecUserGroup option. suEXEC can do it because it’s a setuid program. Apache can’t do it itself because in pre-forked multiprocess mode (which is the default for most Linux distributions) only the control process runs as root - all child processes run as apache.
  4. your CGI program now runs not as apache or nobody, but as some other user.

In its design, suEXEC favors security over flexibility, and that’s why it has several important shortcomings:

  • The privilege change (setuid) is only done when CGI’s are executed. When we have to serve static files (.jpeg’s, for example) they still have to be readable by apache user.
  • You are limited to CGI interface when running your applications. In some cases, this means much lower performance - for example, since your PHP code is run through the CGI/CLI PHP executable and not mod_php Apache module, you can say good-bye to eAccelerator, ionCube’s PHPAccelerator and the likes.
  • It does not solve the problem if your application is another Apache module. In addition to PHP/mod_php case, suEXEC won’t help you with mod_dav_svn (Subversion), for example.
  • Sometimes, you’ll find suEXEC’s security policy too restrictive. And it’s not just about being able to run your web applications as root (which, obviously, is not an option) - you typically can’t run anything with privileges of ANY of the system accounts (uid<500). Your CGI and the directory it resides in can’t be writeable by any other user. Your CGI has to be under /var/www or in /home//public_html and so on.

suPHP and What’s Wrong With It

Since PHP was mentioned several times already, here is another tool for isolating PHP applicaitons from each other: suPHP. The difference is that PHP files are now executed with user privileges of the file owner - a bit more convenient and straightforward setup for PHP. You will often find suPHP used by shared hosting providers, it’s even one of the options in cPanel/WHM and there is an RPM available through Fedora Extras.

However, the list of our grievances with suEXEC applies to suPHP almost 100%. It’s still a CGI-based solution.

There are several of other options available, namely the:

perchild and Metux

perchild MPM is a “Multi-Processing Module allowing for daemon processes serving requests to be assigned a variety of different userids”. Metux MPM is a “project evolution from the perchild module”.

I haven’t had much experience with either (didn’t even try them, to be exact;) for one very simple reason - both of them are threaded and PHP and threaded Apache don’t mix. If you don’t have to run PHP applications (and all your other applications and libraries used are thread-safe) you can try them out. Unfortunately, it’s not an option for me.

ITK MPM module

The solution I found to be the best so far is mpm-itk. In short, here is how it works:

  1. It’s based on prefork MPM so like your regular Apache, it has a parent managing process and several child processes serving the requests.
  2. Unlike the regular prefork model which has the managing process running as root and child processes as apache/nobody, ITK runs ALL processes as root.
  3. Whenever a request is served by a child process, the child process determines what user credentials are needed for a particular VirtualHost (based on your config file, of course), then the child process sets effective user ID and group ID on itself to these credentials, completes the request and sets them back.

What this means is this is next best thing to running a separate apache server per every user (and is actually better than that!). ALL your Apache modules still work (mod_php, mod_dav_svn, mod_python, etc.), they are restricted to a particular user specified in VirtualHost config and what’s best, your file READS in web applications are also confined to the privileges that specified user has.

Perfect!

Now, there’s gotta be some catch. And here it is (actually, two):

  • Strictly speaking, even though all capabilities that root processes have are dropped in the child processes, the child processes are still running as root until the target user/group is determined - i.e. while the HTTP request is parsed. This means that if there is ever a vulnerability in request parsing, you have your vulnerable application with higher privileges than normal. This is somehow offset by the fact that it’s not your normal Apache anymore, and a lot of attacks based on stack overflow and expecting to find themselves inside a normal Apache code will probably just crash instead of executing their payload. Whether this is an affordable risk for you depends entirely on your application, sensitivity of your data and so on. I find it acceptable.
  • Second - it’s a patch, not a standard Apache feature, so some extra elbow grease is needed to get it working.

While catch #1 is something we can’t do anything about, let’s work on catch #2.

Building Apache2 RPMs With MPM-itk Support

If you are like me, you like all your software packaged. I absolutely hate bash-based installers and ./configure - make - make install bruhaha. RPM is there for a reason.

I also like distribution-specific patches to my applications - they typically are there for a reason, too. This reason is usually somebody having a problem with something, reporting it to RedHat/SuSE/(inser your favorite) and getting the stuff fixed. Losing distro-specific patches is not wise, to say the least.

What I often do with 3-rd party patches like MPM-itk is I rebuild “stock” RPMs using them, and that’s exactly what we are going to do now. My distribution of choice is RedHat Enterprise Linux (version 4 in this example), and it’s cheap twin brother, CentOS. The directions are based on these 2 distributions, however your steps on SuSE/Mandriva should be very similar.

The Ingredients

We’ll need:

  • 1 source httpd RPM. In our case, this will be httpd-2.0.52-22.ent.src.rpm for RHEL4 or httpd-2.0.52-22.ent.centos4.src.rpm for CentOS4
  • 1 MPM-itk patch for it, either itk.RHEL.patch for RHEL4 or itk.CO4.patch for CentOS4.

Since you are getting the 2nd ingredient from me, I have to tell you what’s in it.

  • both patches from http://home.samfundet.no/~sesse/mpm-itk/ ;
  • a small fix for /etc/sysconfig/httpd that adds httpd.itk as a possible alternative;
  • a small fix for /etc/httpd/conf/httpd.conf that adds default settings for MPM-itk mode - we copy all performance parameters from prefork’s defaults and make sure the default host runs as apache/apache, because otherwise it would run as root;
  • a little fix for PHP session files: in prefork mode, they are all saved to /var/lib/php/session - a directory owned by apache we can not use anymore, we have to use /tmp, like in good old times;
  • a small fix for RPM spec file to make it build, install and package httpd.itk in addition to prefork httpd and worker-based httpd.worker

That’s all. Now let’s get to building it.

The Process

Repeat after me:

I WILL NOT BUILD RPMS AS ROOT. EVER.

That is, if you don’t want to accidentally execute rm -rf / , otherwise referred to as “rite of passage” by Unix old-timers. Let’s log in as some other, regular user and do this:

# cd # mkdir src # mkdir src/rpm # mkdir src/rpm/BUILD src/rpm/RPMS/ src/rpm/SOURCES src/rpm/SPECS src/rpm/SRPMS # echo -E %\_topdir /home/`whoami`/src > .rpmmacros

Now we are ready for some RPM building. Let’s install our source RPM:

# rpm -ivh httpd-2.0.52-22.ent.src.rpm

patch it:

# cp itk.RHEL.patch ~/src # cd ~/src # patch --strip=0 --backup --remove-empty-files --suffix=.itk

and rebuild it:

# rpmbuild -ba rpm/SPECS/httpd.spec

(example given for RHEL4, for CentOS4 build, replace the RPM name and patch name with the ones you need). If everything went well, your new httpd* RPMs are now in rpm/RPMS/i386 or x86_64. Grab them and install as usual.

Switching to MPM-itk

By default, these RPMs will still run Apache in “prefork” mode. To switch to mpm-itk mode,

  • stop Apache (important!)
  • edit /etc/sysconfig/httpd: uncomment (delete # in) the following line:#HTTPD=/usr/sbin/httpd.itk
  • start Apache

You can’t just edit /etc/sysconfig/httpd and restart httpd service - the httpd service script in sysv init depends on this config file for starting and stopping httpd.

That’s it, you should be all set.

If you feel better at this time, you should probably send a “thank you” note to Steinar H. Gunderson (http://home.samfundet.no/~sesse/mpm-itk/) who developed this great mod.

8 Responses to 'Apache2 MPM-itk on RHEL/CentOS'

Subscribe to comments with RSS

  1. 4/25/2009 10:59 am

    Speedboy said:

    Hello Mitka!

    have it all working fine on a server. But when I go to fix certain pages as https I suddenly find all directories are not writeable. As soon as I put the same path in with http everything is writeable again. is https not running as the same user or what? WHat do I have to do? there has to be a way.
    Speed

  2. 6/25/2009 9:33 pm

    fujipadam said:

    Hey Mitka,

    Does mpm-itk work with any of the opcode optimizers like APC, xcache or eaccelerator?

    Thanks,
    Fuji

  3. 6/25/2009 11:41 pm

    mitka said:

    Yes, it does.

  4. 8/8/2009 6:30 am

    andytson said:

    Sorry if I’m wrong, but in theory opcode optimizers wont work correctly.

    ITK forks a httpd process every request, and the fork child gets destroyed at the end of the request, and so any memory it allocated is copy-on-write, meaning any opcode cache in memory only lasts the duration of the request.

    File based opcode cache may or may not work correctly either. Does a file handle opened by root that gets forked contain necissary permissions? If the file handle is opened after the fork, you’re cache directory would supposedly need read/write permissions for the user your vhost is running as.

  5. 8/8/2009 9:38 am

    mitka said:

    andytson,

    ITK is a modification of Apache’s pre-forked model. PRE-forked means the child processes are not forked and killed on every httpd request, but forked and killed gradually as the server load increases and decreases. The default is to let a child process serve 10000 requests before killing it (see MaxRequestsPerChild).

    Forking and killing a process for every httpd request would be VERY inefficient, and I don’t think you can even make Apache work this way (besides setting MaxRequestsPerChild to 1 ;).

    ITK changes the effective user ID of already running process for the duration of every request, so your in-memory opcode cache is still there when another request for the same PHP file comes.

    File-based opcode cache works correctly too, because setuid/seteuid happens before any calls to PHP handler.

  6. 8/8/2009 12:10 pm

    andytson said:

    I had a look at the patches, and they do in fact fork their parent process every request.

    Basically, each parent is run as root, ready to be forked. On request, the process that gets the connection forks, and once it is able to establish the vhost, it setuid’s to that user.

    The problem is once you setuid a process, it cannot go back to root, and since only so many processes are run at once, they chose to terminate the process.

    It’s not as inefficient as you think. No memory is copied at first (as fork uses the copy-on-write I mentioned), unlike CGI where a whole new process is created and loaded.

    As far as I’m aware the only way to share memory between processes on linux is to use memory-mapped files, and that forked processes can’t write to the same memory space without the memory being copied, therefore unlinking the memory block from its parent’s one.

    Since PHP is loaded before the request, in the master process, opcode optimisers may in fact open the file handle to the file cache before setuid happens, although they may have chosen to open the file during the request.

  7. 8/8/2009 12:33 pm

    andytson said:

    The key is, itk handles almost the same as prefork, but the preforked processes fork again on each established connection. The parent of the fork waits for the child to complete, and after MaxRequestsPerChild, terminates.

    The child carries on as the same it would in prefork mpm, but when the vhost is established, setuid’s to the user. At the end of the connection it checks whether it ran setuid, and terminates if so, allowing the parent to continue.

  8. 5/1/2010 8:15 am

    lystor said:

    Increase Apache Vhost Security With mpm-itk In RHEL/CentOS 5

Leave a Reply

You must be logged in to post a comment.