How I use Django, Virtualenv and Buildout together

I’ve been doing web development in Python for a little over two years now and thought I’d share how I typically set up my development environment for a new project. I’m going to talk about three Python technologies that each do a different job and together make it easy to create and deploy projects. First, let me cover what they are:

Django – If you’ve done Python, you know Django. It’s the most popular Python web framework. Now, that said I’ve used Tornado, Google’s App Engine framework (is it called webapp?) and Flask for other projects. They’re all good in their own ways, but I prefer Django because I can get it up and running quickly and, with the community’s support and my past experiences, I know when and where Django will eventually cause trouble. Hint: it’s not until you have serious usage on your site.

Virtualenv – I can’t describe virtualenv better than the site itself, so I’ll just quote it. “virtualenv is a tool to create isolated Python environments.” There are all sorts of reasons you’d want to work on a project in isolation of other projects, the first and most obvious is, what do you do if different projects rely on different versions of the same package? We’ll, I’ll tell you. You’re screwed unless you use virtualenv. It’s so easy to use, there’s really no reason not to. The docs are great, so start there and ask questions.

Buildout – Buildout is a build system in Python. There are a lot of great “recipes” prepackaged with it which will cover almost all use cases. Buildout solves the problem of “how do I package up the dependencies of my project so it’s easy to deploy?” Between the buildout configuration file and a setup.py file, you’re pretty much good to go.

First step: virtualenv

Alright, so here’s the setup: you’re about to start work on a new Django app. I’m going to walk through exactly how I get things setup and it starts with virtualenv. I’m working on a Mac and I typically stick all my projects in a subfolder of a “dev” directory in my home directory. So let’s say our app will be code named “fenway.” In Terminal, I’ll run the following:

sudo pip install virtualenv
cd ~/dev
virtualenv fenway --no-site-packages

What’s this doing? Well, first we make sure we have virtualenv installed. If you don’t know what pip is, it’s used for installing and managing packages in Python. If you don’t have it installed, go take care of that first. The second line takes me to where I put my projects and the last line creates a new virtualenv in a directory called “fenway” with no site packages. What does no site packages mean? Just that I don’t want to link to any globally installed site packages. This is good because it prevents us from having accidental dependencies.

Okay, so our virtualenv is ready to go, but we need to “activate” it so python in Terminal will point to the python in our virtualenv, not the system one. So now I run:

cd fenway
source bin/activate

Now we’re good to go. You might notice that the prompt changes (this’ll depend on your settings) and is now prefixed with the virtualenv you’re currently in. This is helpful for remembering where you are in case, you know, you get lost.

Setup Buildout

The next step is to get Buildout setup. In doing so, we’ll also create the basic directory structure for our project. To setup Buildout you might think, pip install buildout, but that’s not how we’ll do it. Instead, we’re going to fetch a bootstrap file and run that to get us setup:

wget http://svn.zope.org/\*checkout\*/zc.buildout/trunk/bootstrap/bootstrap.py

If you try to run python bootstrap.py to bootstrap Buildout, it’ll be looking for the configuration file: buildout.cfg. So let’s create that in this directory. I’m putting my template buildout.cfg below:

[buildout]
parts = django scripts
develop = .
eggs = fenway
eggs-directory = /opt/fenway/buildout/cache/eggs
download-cache = /opt/fenway/buildout/cache/download
download-directory = /opt/fenway/buildout/cache/download
unzip = true
 
[versions]
django = 1.4
 
[django]
recipe = djangorecipe
project = fenway
projectegg = fenway
settings = settings
test = app
wsgi = true
eggs = ${buildout:eggs}
 
# We add this extra path so the settings and urls files can be imported
# Maybe these belong somewhere else? Not sure of the best layout.
extra-paths = ${buildout:directory}/src
 
[scripts]
recipe = zc.recipe.egg:scripts
eggs = ${buildout:eggs}
extra-paths = ${buildout:directory}/src

Now let me pause to make a few comments on this:

  • The eggs-directory, download-cache, download-directory are locations for Buildout to cache packages it fetches. You don’t need those settings, but I strongly suggest having them because it they speed things when you re-run Buildout. Knowing those locations is also really useful if you ever want to look at the source for those packages and even add debugging. I can’t tell you the number of times I’ve run mate /opt/fenway/buildout/cache/eggs/some-package.1.2.3/ to have a look at the code. One of the big advantages of working with Python (or any scripted language) is you can take a look at the source of any dependencies whenever you want.
  • The unzip = true is also important because it’ll make sure to unpack eggs. This is really useful for the debugging I mentioned above.
  • Everywhere I have fenway, you’ll want to replace with whatever you’re code naming your project.
  • You’ll notice I have test = app. This is part of the djangorecipe for Buildout that’ll automatically create a test runner for the Django app inside our project. I tend to name my Django app “app” so I know which app is the main one I’m working on. This might be a bad idea if you know you’ll be splitting your project into multiple Django apps, but I’ve found that splitting my project into multiple Django apps early on causes more headaches than solves problems.
  • The last thing to point out is the comment where I add extra-paths=${buildout:directory}/src. This is actually a bit of a hack I’ve been unable to find a better solution for. The problem results from how I structure my projects and that the settings.py file can’t be found in the PYTHONPATH without this. If anyone has suggestions on a better way to do this, I’m all ears. Adding that directory to the path doesn’t cause any problems, it just “feels wrong.”

Okay so now we’re ready. Go ahead and run:

python bootstrap.py

Once that’s done, it’ll create a script called buildout in the bin directory. The last step is to scaffold some of files and directories we’ll need.

Your setup.py file

If you’re not familiar with setup.py files, they are how Python packages declare themselves and their dependencies. Your project is no different from any other Python package, so let’s create a basic setup.py file in this directory. Here’s an example:

#!/usr/bin/env python
 
from setuptools import setup
 
VERSION = '0.0.1'
 
setup(
  name='fenway',
  version=VERSION,
  description='Rebuilding Fenway Park in code',
  author='Andrew Bialecki',
  author_email='andrew.bialecki@example.com',
 
  # I'm not sure what's ideal, but I think we'd like to move these apps down a directory
  # so instead of "src/fenway/app," we'd have "src/app."
  # Anyway, for now the value in this is that you don't have to write "import fenway.app,"
  # you can write "import app."
  packages=['app',],
  package_dir={ '' : 'src/fenway' },
 
  install_requires=[
    'django == 1.4',
  ]
)

Short and sweet. Remember to change fenway and app to match your project and Django app names, respectifully. You’ll notice again I need to do a little work in defining the packages and package_dir so match the structure I’m going to use. Also note the only requirement is django == 1.4, which is the current version at the time of this writing.

Scaffolding our app

Last thing before we run Buildout, we need to create the Django project and app.  This part follows the Django tutorial closely, so you can look there for more information. To do this, we need Django installed directly in our virtualenv:

pip install django==1.4

Once that’s done, we just need to be careful to run commands in the right place. Here’s what you should do:

cd src
../bin/django-admin.py startproject fenway
mv fenway/fenway/* fenway
rmdir fenway/fenway
python manage.py startapp app

You’ll notice I moved some files around. With Django 1.4, they modified the default project structure. Personally, I liked the old way and it’s tighter, so I move things around to compensate for that. If you’d prefer to stick with Django’s defaults, that’s okay, just make sure you go back and modify the paths we set in earlier files to reflect the new locations.

Running Buildout

Okay, no more waiting, we’re good to go. Run:

cd ../..
bin/buildout

Once that finishes, you’ll notice there is now a django script in the bin directory. That’s how we’re going to execute django commands from now on. So instead of python manage.py command we’ll use bin/django command.

Directory structure

Let’s pause for a second and take a look at the structure of our project. It’s going to look like this:

bin/
  django
  buildout
bootstrap.py
buildout.cfg
setup.py
src/
  fenway/
    __init__.py
    manage.py
    settings.py
    urls.py
    app/
      __init__.py
      models.py
      urls.py
      tests.py
      views.py

If you’re familiar with Django, you should notice that what’s inside the src directory is just a regular Django project. I’m not going to cover what’s in there because the Django folks have already done a nice job of that. Everything else is files we’ve created or Buildout autogenerated for us.

You might notice that Buildout created a number of other directories like develop-eggs, build, include. You don’t have to worry about those, they’re automatically created and we’ll exclude them from our project repo when we create a git repo later.

Running the test server

Okay, let’s test this out. Try:

bin/django runserver

The test server should start and if you go to http://localhost:8000/ in your browser you’ll see the “It worked!” message. Congratulations! We’re all done.

Going further

Okay, that seemed like a lot of work. Why bother? There are lots of reasons, but let’s start with one. Say you now start working on your Django app and you decide you’d like to use South to manage database migrations. Okay, we need to install South, how do we do that?

Super easy. Edit your setup.py file and update the install_requires section like so:

install_requires=[
  'django == 1.4',
  'South == 0.7.5',
]

Save that file, run bin/buildout again and you’re done. You can now run bin/django schemamigration app --initial and it’ll generate a migration. You can then run that migration with bin/django migrate. Pretty simple, huh?

It gets even better when integrating this into your deployment process. I’ll cover how I typically do that next time. I hope that was helpful. If you have any builds let me know in the comments.

  • katezopp

    I’d be interested in knowing how you use this to deploy code to your servers. Do you use Fabric or something else?

    • bialecki

      I’m going to write a follow up to this post about how I do that. But to answer your question, I do use Fabric. Not sure if there’s a better tool, but works great for me.

  • http://lukeplant.me.uk/ spookylukey

    This seems like a strange layout: presumably you would want buildout.cfg under source control, and all of src/, but not bin/ and the rest of the virtualenv. Doesn’t putting the src/ dir inside the virtualenv make source control unnecessarily complicated?

    • bialecki

      I solve that problem in my .gitignore file (there are 4-5 directories to exclude) so in practice it ends up not being a big deal. That said, I’m sure there’s a better way. Care to sketch out how you’d organize stuff?

      • http://lukeplant.me.uk/ spookylukey

        I use virtualenvwrapper, which puts all your virtualenvs in one place (~/.virtualenvs/ by default), quite separate from your source dir, and has various other niceties. I was definitely interested in how buildout fits into the picture, so thanks for that.

      • bialecki

        Makes sense. I’ll have to experiment with virtualenvwrapper, does seem like it could be better for separating things.
        Sent from Bialecki’s iPhone

  • Pajju

    virtualenv fenway –no-site-packages

    –no-site-packages is not required from Virtual env 1.7version on-wards, its the default behavior.

  • http://twitter.com/vahid_r Vahid Rafiei

    Awesome Post .. I’m setting up a real world project for a start-up and it was exactly what I was looking for. Would you please mention some points on setting up Testing environment, specially with Django-nose as a test runner? FYI, I’m planning to use Nose, Selenuim, Webtest/test.Client()/Factory_Boy on Pinax/Django1.3 ..

  • http://www.domenkozar.com Domen Kožar

    Good introduction post. Few comments:

    – virtualenv is not needed with buildout (set “include-site-packages = false” in buildout section to achieve same goal)
    – you might want to use global buildout configuration for common settings, although community projects are adviced to have everything in version control http://paste.ubuntu.com/1041417/
    – it’s normally useful to generate python interpreter with all eggs, although django has shell command
    – there is buildout 2 on the way https://github.com/buildout/buildout

  • Pajju

    I did not understand the purpose of the complex Buildout configuration, when we can just run “pip install -r requirements.txt” in our virtualenv and list packages in a simple text file?

    What do we win using Buildout?

    So far a combination of pip + virtualenv + fabric has been more than sufficient for my deployment requirements.

    • Tom Willis

      with buildout you can run a recipe for example that can pull down the source for memcached, nginx, rabbitmq and postgresql, bulid all of it, and then write working config scripts based on templates and values from your buildout.cfg that run under supervidord. So you can roll a repeatable process for producing a usable environment that possibly mimics your ROFLSCALE infrastructure along with everything you need for your django applcation.

      pip doesn’t do that. and you may not need it. but some people find it invaluable. :)