Sophomore Dev

I’ve been doing web development in Python for a little over two years now and thought I’d share how I typically set up my development environment for a new project. I’m going to talk about three Python technologies that each do a different job and together make it easy to create and deploy projects. First, let me cover what they are:

Django – If you’ve done Python, you know Django. It’s the most popular Python web framework. Now, that said I’ve used Tornado, Google’s App Engine framework (is it called webapp?) and Flask for other projects. They’re all good in their own ways, but I prefer Django because I can get it up and running quickly and, with the community’s support and my past experiences, I know when and where Django will eventually cause trouble. Hint: it’s not until you have serious usage on your site.

Virtualenv – I can’t describe virtualenv better than the site itself, so I’ll just quote it. “virtualenv is a tool to create isolated Python environments.” There are all sorts of reasons you’d want to work on a project in isolation of other projects, the first and most obvious is, what do you do if different projects rely on different versions of the same package? We’ll, I’ll tell you. You’re screwed unless you use virtualenv. It’s so easy to use, there’s really no reason not to. The docs are great, so start there and ask questions.

Buildout – Buildout is a build system in Python. There are a lot of great “recipes” prepackaged with it which will cover almost all use cases. Buildout solves the problem of “how do I package up the dependencies of my project so it’s easy to deploy?” Between the buildout configuration file and a setup.py file, you’re pretty much good to go.

First step: virtualenv

Alright, so here’s the setup: you’re about to start work on a new Django app. I’m going to walk through exactly how I get things setup and it starts with virtualenv. I’m working on a Mac and I typically stick all my projects in a subfolder of a “dev” directory in my home directory. So let’s say our app will be code named “fenway.” In Terminal, I’ll run the following:

sudo pip install virtualenv
cd ~/dev
virtualenv fenway --no-site-packages

What’s this doing? Well, first we make sure we have virtualenv installed. If you don’t know what pip is, it’s used for installing and managing packages in Python. If you don’t have it installed, go take care of that first. The second line takes me to where I put my projects and the last line creates a new virtualenv in a directory called “fenway” with no site packages. What does no site packages mean? Just that I don’t want to link to any globally installed site packages. This is good because it prevents us from having accidental dependencies.

Okay, so our virtualenv is ready to go, but we need to “activate” it so python in Terminal will point to the python in our virtualenv, not the system one. So now I run:

cd fenway
source bin/activate

Now we’re good to go. You might notice that the prompt changes (this’ll depend on your settings) and is now prefixed with the virtualenv you’re currently in. This is helpful for remembering where you are in case, you know, you get lost.

Setup Buildout

The next step is to get Buildout setup. In doing so, we’ll also create the basic directory structure for our project. To setup Buildout you might think, pip install buildout, but that’s not how we’ll do it. Instead, we’re going to fetch a bootstrap file and run that to get us setup:

wget http://svn.zope.org/\*checkout\*/zc.buildout/trunk/bootstrap/bootstrap.py

If you try to run python bootstrap.py to bootstrap Buildout, it’ll be looking for the configuration file: buildout.cfg. So let’s create that in this directory. I’m putting my template buildout.cfg below:

[buildout]
parts = django scripts
develop = .
eggs = fenway
eggs-directory = /opt/fenway/buildout/cache/eggs
download-cache = /opt/fenway/buildout/cache/download
download-directory = /opt/fenway/buildout/cache/download
unzip = true
 
[versions]
django = 1.4
 
[django]
recipe = djangorecipe
project = fenway
projectegg = fenway
settings = settings
test = app
wsgi = true
eggs = ${buildout:eggs}
 
# We add this extra path so the settings and urls files can be imported
# Maybe these belong somewhere else? Not sure of the best layout.
extra-paths = ${buildout:directory}/src
 
[scripts]
recipe = zc.recipe.egg:scripts
eggs = ${buildout:eggs}
extra-paths = ${buildout:directory}/src

Now let me pause to make a few comments on this:

• The eggs-directory, download-cache, download-directory are locations for Buildout to cache packages it fetches. You don’t need those settings, but I strongly suggest having them because it they speed things when you re-run Buildout. Knowing those locations is also really useful if you ever want to look at the source for those packages and even add debugging. I can’t tell you the number of times I’ve run mate /opt/fenway/buildout/cache/eggs/some-package.1.2.3/ to have a look at the code. One of the big advantages of working with Python (or any scripted language) is you can take a look at the source of any dependencies whenever you want.
• The unzip = true is also important because it’ll make sure to unpack eggs. This is really useful for the debugging I mentioned above.
• Everywhere I have fenway, you’ll want to replace with whatever you’re code naming your project.
• You’ll notice I have test = app. This is part of the djangorecipe for Buildout that’ll automatically create a test runner for the Django app inside our project. I tend to name my Django app “app” so I know which app is the main one I’m working on. This might be a bad idea if you know you’ll be splitting your project into multiple Django apps, but I’ve found that splitting my project into multiple Django apps early on causes more headaches than solves problems.
• The last thing to point out is the comment where I add extra-paths=${buildout:directory}/src. This is actually a bit of a hack I’ve been unable to find a better solution for. The problem results from how I structure my projects and that the settings.py file can’t be found in the PYTHONPATH without this. If anyone has suggestions on a better way to do this, I’m all ears. Adding that directory to the path doesn’t cause any problems, it just “feels wrong.”

Okay so now we’re ready. Go ahead and run:

python bootstrap.py

Once that’s done, it’ll create a script called buildout in the bin directory. The last step is to scaffold some of files and directories we’ll need.

Your setup.py file

If you’re not familiar with setup.py files, they are how Python packages declare themselves and their dependencies. Your project is no different from any other Python package, so let’s create a basic setup.py file in this directory. Here’s an example:

#!/usr/bin/env python
 
from setuptools import setup
 
VERSION = '0.0.1'
 
setup(
  name='fenway',
  version=VERSION,
  description='Rebuilding Fenway Park in code',
  author='Andrew Bialecki',
  author_email='andrew.bialecki@example.com',
 
  # I'm not sure what's ideal, but I think we'd like to move these apps down a directory
  # so instead of "src/fenway/app," we'd have "src/app."
  # Anyway, for now the value in this is that you don't have to write "import fenway.app,"
  # you can write "import app."
  packages=['app',],
  package_dir={ '' : 'src/fenway' },
 
  install_requires=[
    'django == 1.4',
  ]
)

Short and sweet. Remember to change fenway and app to match your project and Django app names, respectifully. You’ll notice again I need to do a little work in defining the packages and package_dir so match the structure I’m going to use. Also note the only requirement is django == 1.4, which is the current version at the time of this writing.

Scaffolding our app

Last thing before we run Buildout, we need to create the Django project and app.  This part follows the Django tutorial closely, so you can look there for more information. To do this, we need Django installed directly in our virtualenv:

pip install django==1.4

Once that’s done, we just need to be careful to run commands in the right place. Here’s what you should do:

cd src
../bin/django-admin.py startproject fenway
mv fenway/fenway/* fenway
rmdir fenway/fenway
python manage.py startapp app

You’ll notice I moved some files around. With Django 1.4, they modified the default project structure. Personally, I liked the old way and it’s tighter, so I move things around to compensate for that. If you’d prefer to stick with Django’s defaults, that’s okay, just make sure you go back and modify the paths we set in earlier files to reflect the new locations.

Running Buildout

Okay, no more waiting, we’re good to go. Run:

cd ../..
bin/buildout

Once that finishes, you’ll notice there is now a django script in the bin directory. That’s how we’re going to execute django commands from now on. So instead of python manage.py command we’ll use bin/django command.

Directory structure

Let’s pause for a second and take a look at the structure of our project. It’s going to look like this:

bin/
  django
  buildout
bootstrap.py
buildout.cfg
setup.py
src/
  fenway/
    __init__.py
    manage.py
    settings.py
    urls.py
    app/
      __init__.py
      models.py
      urls.py
      tests.py
      views.py

If you’re familiar with Django, you should notice that what’s inside the src directory is just a regular Django project. I’m not going to cover what’s in there because the Django folks have already done a nice job of that. Everything else is files we’ve created or Buildout autogenerated for us.

You might notice that Buildout created a number of other directories like develop-eggs, build, include. You don’t have to worry about those, they’re automatically created and we’ll exclude them from our project repo when we create a git repo later.

Running the test server

Okay, let’s test this out. Try:

bin/django runserver

The test server should start and if you go to http://localhost:8000/ in your browser you’ll see the “It worked!” message. Congratulations! We’re all done.

Going further

Okay, that seemed like a lot of work. Why bother? There are lots of reasons, but let’s start with one. Say you now start working on your Django app and you decide you’d like to use South to manage database migrations. Okay, we need to install South, how do we do that?

Super easy. Edit your setup.py file and update the install_requires section like so:

install_requires=[
  'django == 1.4',
  'South == 0.7.5',
]

Save that file, run bin/buildout again and you’re done. You can now run bin/django schemamigration app --initial and it’ll generate a migration. You can then run that migration with bin/django migrate. Pretty simple, huh?

It gets even better when integrating this into your deployment process. I’ll cover how I typically do that next time. I hope that was helpful. If you have any builds let me know in the comments.