Deploying and tuning Django

I used shared hosting for my Django-based Encyclopedia of Programming Languages (Russian version), but realized, that I need dedicated server for more performance and stability. For now, I chose cheap server with 1,8 GHz CPU and 512 Mb RAM, with CentOS 5 installed.

In this article I’ll share my experience for installing on this server Django with Apache mod_python, with memcached caching and lighttpd for serving static files.

So, we have a freshly installed CentOS 5 and want to host Django-based site (or several sites).

What should we install for this?

  1. Django itself
  2.  MySQL-python
  3.  mod_python
  4.  memcached
  5.  lighttpd

All steps excluding installing Django itself are optional (for example, you can use Django with fcgi instead of mod_python and PostgreSQL or other DBMS instead of MySQL), but steps 1-3 are what you likely want to use in typical environment and steps 4-6 are for gaining better performance.

Most of the things in this article are CentOS-oriented, but to some extent these instructions will be applicable to any other Linux distribution.

In this article if there is no rpm-package with program or library that I need in the official CentOS repositories, I install that program or library from source. You may prefer searching for third-party rpms, but it may lead to dependency problems and even security issues. The ideal option, I guess, is to roll rpms yourself, but it will require some extra time and knowledge.

Some security considerations: as this is a freshly installed system, I’m not afraid of screwing up things, so I may work as root user. But it’s better to run most commands as unprivileged user and use “sudo” for commands that require root privileges.

So, lets start.

Django itself

Usually, you should use the latest svn-version of Django framework, not so-called “release” version. Currently, “official” release version is too old and lacks many important features (for instance, unicode support and auto-escaping).

So you should go to command line and type

svn co http://code.djangoproject.com/svn/django/trunk/ django-trunk

But before that you should install svn

yum install subversion

After fetching files from svn run

python setup.py install

After installing Django you should get your project files (via svn or simply by copying via ssh or ftp; you may also want to create a new system user to own these files) and database dump, and adapt your database settings in settings.py file to new environment (it is assumed that MySQL is already installed and configured, database and user are created, all data is in place – these things are far beyond the scope of this article).

Then go to your project directory and try running Django development server

python manage.py runserver 127.0.0.1:8000

On CentOS 5 you’ll get error message “Error loading MySQLdb module”.

MySQL-python version in CentOS distribution is too old for the latest Django svn-version, so you need to build it from sources.

MySQL-python

cd /usr/local/src
wget http://garr.dl.sourceforge.net/sourceforge/mysql-python/MySQL-python-1.2.2.tar.gz
tar xvzf MySQL-python-1.2.2.tar.gz
cd MySQL-python-1.2.2

gcc, python-devel and mysql-devel packages are required to build MySQL-python.

yum install gcc
yum install python-devel
yum install mysql-devel

cd /usr/local/src/MySQL-python-1.2.2
python setup.py build
python setup.py install

Done, MySQL-python is installed as .egg file in your site-packages directory.

Than again run

python manage.py runserver 127.0.0.1:8000

and look for any errors. For example, I use ImageFields in my projects, so I had to additionally install Python Imaging Library (yum install python-imaging).

If there are no errors, it’s generally a good idea to change 127.0.0.1 in runserver command to your IP-address and check if site works OK with your browser, but to do this you need to open 8000 port on CentOS firewall or just temporarily disable firewall (or you can temporarily shut down apache and run Django development server on 80 port).

If there are no errors, it’s time to shut down django development server and configure real web-server.

mod_python

It’s easy to configure apache with mod_python to work with Django. Just follow instructions in Django official documentation.

Just one note: since we installed MySQL-python as .egg-package, it’s important to create temp directory (set 777 permissions for it) for .egg cache and make a reference to it in the httpd.conf file:

<Location "/">
    ...
    SetEnv PYTHON_EGG_CACHE /var/tmp/egg
</Location>

And don’t forget to restart apache after editing httpd.conf:

/etc/init.d/httpd restart

Now your site should work via apache on standard http port. Check it with your browser.

memcached

Enabling memcached caching is very important for performance. Unfortunately, memcached isn’t in CentOS repository, so again we have to install from source. This chapter is partially based on this howto.

First of all, you need to install libevent:

cd /usr/local/src
wget http://monkey.org/~provos/libevent-1.4.2-rc.tar.gz 
tar xvzf  libevent-1.4.2-rc.tar.gz
cd libevent-1.4.2-rc
./configure
make
make install


vi /etc/ld.so.conf.d/libevent.conf
##add a line containing:
/usr/local/lib/

ldconfig -v

And then memcached itself:

cd /usr/local/src
wget http://danga.com/memcached/dist/memcached-1.2.5.tar.gz
tar zxvf memcached-1.2.5.tar.gz
cd memcached-1.2.5
./configure
make
make install

cp scripts/memcached.sysv /etc/init.d/memcached

ln /usr/local/bin/memcached /usr/bin/memcached
/etc/init.d/memcached start

chkconfig memcached on

And finally we need to install libmemcache and cmemcache (python extension for libmemcache):

cd /usr/local/src/
wget http://people.freebsd.org/~seanc/libmemcache/libmemcache-1.4.0.rc2.tar.bz2
bunzip2 libmemcache-1.4.0.rc2.tar.bz2
tar xf libmemcache-1.4.0.rc2.tar
cd libmemcache-1.4.0.rc2
./configure
make 
make install

cd /usr/local/src/
wget http://gijsbert.org/downloads/cmemcache/cmemcache-0.91.tar.bz2
bunzip2 cmemcache-0.91.tar.bz2
tar xf cmemcache-0.91.tar
cd cmemcache-0.91
python setup.py install

Then enable memcached caching in your project settings.py file

CACHE_BACKEND = 'memcached://127.0.0.1:11211/'

(check the official Django documentation for more information) and restart apache. Now django stuff should be cached with memcached.

To get even more performance benefits we shouldn’t use bloated Apache for such simple task as serving static files, so let’s install lighttpd.

lighttpd

cd /usr/local/src
wget http://www.lighttpd.net/download/lighttpd-1.4.19.tar.bz2
bunzip2 lighttpd-1.4.19.tar.bz2
tar xf lighttpd-1.4.19.tar
cd lighttpd-1.4.19
yum install glib2-devel openssl-devel pcre-devel bzip2-devel gzip-devel
./configure
make 
make install

cp doc/rc.lighttpd.redhat /etc/init.d/lighttpd
chkconfig lighttpd on

mkdir /etc/lighttpd
cp doc/lighttpd.conf /etc/lighttpd/

And now we should configure apache and lighttpd so that static files will be served by lighttpd (on 80 port), and other requests will be handled by apache on 81 port (thanks to nolancaudill.com for solution):

  • tell apache to listen the 81 port: find Listen 80 and change it to Listen 127.0.0.1:81. Don’t forget to change port number in all VirtualHost directives, too. Then restart apache.

  • edit /etc/lighttpd/lighttpd.conf. Uncomment ‘mod_alias’ and ‘mod_proxy’ modules and add something like this:

    $HTTP["host"] == "yourdomain.com" {
    
        # here we are mapping /media/ for admin media
        # and /static/ for the standard media_url
        alias.url = (
            "/media/" => "/usr/lib/python2.4/site-packages/django/contrib/admin/media/",
            "/static/" => "/var/www/html/media.yourdomain.com/",
        )
    
    
        $HTTP["url"] !~ "^/(static|media)/" {
            proxy.server = ( "" => (("host" => "127.0.0.1", "port" => 81 )) )
    
        }
    
    }
    

Also change server.document-root to “/var/www/html/” and link your project media directory to ”/var/www/html/media.yourdomain.com/”.

  • Create directory for lighttpd log-file:

    mkdir /var/log/lighttpd/
    
  • One more little edit to /etc/init.d/lighttpd: change path from /usr/sbin/lighttpd to /usr/local/sbin/lighttpd. Start lighttpd:

    /etc/init.d/lighttpd start
    

And finally some other performance tips:

Add post to:   Delicious Reddit Slashdot Digg Technorati Google
Make comment

Pingbacks

links for 2008-04-28 &laquo; Mandarine @mandarine.wordpress.com 28.04.2008 0:35
Deploying and tuning Django
Linux Hosting Here: Linux or.. How to.. &raquo; Host News . biz @www.hostnews.biz 27.04.2008 17:18
Deploying and tuning Django In this article I’ll share my experience for installing on this server Django with Apache mod_python, with memcached caching and lighttpd for serving static files.
Deploying and tuning Django on CentOS | Server Tales - Ungureanu Ioan @blog.ro-design.net 26.04.2008 7:36
Article about installing Django with Apache mod_python, with memcached caching and lighttpd for serving static files on CentOS 5 server.
links for 2008-04-26 &laquo; Bloggitation @zhesto.wordpress.com 25.04.2008 20:33
Deploying and tuning Django

Comments

Glad the Django/lighttpd stuff was useful to someone!

Nice post and very thorough, though I must admit the Ubuntu/Debian style of ‘apt-get install whatever-i-need’ has me a little spoiled after looking through all the wgets and ‘../configure; make; make install;’ lines.

But to each their own :)

Nolan: I don’t think it’s so much the style of package management (deb/apt vs. rpm/yum) causing all the wgets and makes rather than the age of what’s in the repositories.

In Debian and Ubuntu the stuff in the repositories isn’t quite bleeding edge, but it’s still very new, some of which still occasionally needs to have some bugs stamped out.

CentOS is meant as a server distribution, and it’s nigh-legendary for how solid it is—which also means that the only software in the repos is extremely well-tested. “Well-tested,” of course, means it’s pretty old, and because this article wants to use a pretty newish version of Django, it’s unsurprising that the new dependencies aren’t in the CentOS repositories.

I also use CentOS for my production work. Although I prefer to install the latest Python (2.5.x) and build all the necessary libraries from source.

Personally I prefer using mod_wsgi over mod_python and the “so-called stable release” of Django. I have had a couple of bad experiences with using the SVN version in development and production. One time I was coding against an SVN checkout, when I deployed on production I checked out a more recent version which broke some of my code. Of course I could have avoided that by frequently checking out the SVN trunk during development. But that’s really against the tagline “perfectionists with deadlines”. When I’m on a deadline, I can’t simply code around every issue that crops up because of some changes in the SVN release.

It’s really a matter of personal preference, and I do prefer coding against a “stable” API. Sure I might miss a few important features, but it’s nothing totally groundbreaking. The only thing I really miss on the 0.96 release are the newforms updates, especially the working FileField.

With regards to lighttpd and memcached, have you tried using the RPMForge repository? The memcached and lighttpd packages are pretty much up to date.

If speed is your thing you should really checkout nginx and mod_wsgi

Personally I found nginx configuration to be lightyears better than lighttpd. With the bonus that the config settings actually do what the docs say which wasn’t true for lighttpd.

Excellent reference on how to build MySQL-python from source. 1and1 hosting uses CentOS 5 for its VPS package, and it was difficult finding the right resource on how to install the module.

Nice article. I have some projects where a CentOS server is used.

I also recently published a tutorial for setting up Django with Apache and Lighttpd on Ubuntu.

In my case Apache is running mod_proxy to send requests for media files to Lighty. I’m not too sure if Apache > Lighty or Lighty > Apache is better for the proxy part. Any of you have any ideas?