Category Archives: Uncategorized

How I Hacked a Reddit-powered Valentine’s Day Gift (and how to re-use it)

This year I decided do something a little different for Valentine’s Day. In addition to the usual flowers and sweets, I wanted to do something different. Something that leveraged my “nerd skills” for good that would spread some cheer automatically. My company, Klaviyo, had just released an API for rendering and send emails and I knew my girlfriend was a big fan of corgis and /r/corgi So I decided I’d put our new API to the test. I’d create a daily email of the top images from the Corgi subreddit that she’d get every morning. Here’s what the end result looks like:

Email Screenshot

If you’re interested in doing this or something similar, you need a few things to get started (all free):

  1. A free account with a service that sends emails. I’d recommend Mandrill or Sendgrid (note: I’m not an affiliate with either of those services).
  2. A free Klaviyo account. You need this to create an email template and to get an API key to make calls to render and send emails. You’ll also need to link your email service with Klaviyo so it can send emails.
  3. A script (mine is Python) to fetch the data and make the API call.

Then, conceptually, here’s how it works. I created an email template in Klaviyo with placeholders for the JSON data I’d be sending. I wrote a short Python script to fetch the top five Corgi images and captions from the Reddit API. I did a little data manipulation and made a request to the Quick Mail API to render my template and then send the email. Once I tested it out, I set it up as a cron task to schedule that script to run each morning and that’s was it. Here’s what the script looks like:

Valentine’s Day is over, but if you’ve got an hour and want to want to make yourself or someone else smile, give it a try. Happy hacking.

Hacking stubborn APIs: getting paginated results when there’s no paging option

First off, APIs are awesome. I remember back in 2006 when I was interning for a company in Redmond, WA and I didn’t even know what an API was or even what API stood for. Well, the situation is much different today. Making services and technologies talk to each other enables some amazing things. And it’s really great when APIs are well designed so they make the most common use cases easy and are extensible enough they can handle more nuanced situations as well.

Well, this is a short story about an API that was probably designed years ago and doesn’t give you a way to handle very common use: namely paginating results. In this case, we were doing some work to integrate Klaviyo, a great new approach to CRM, with Magento, an extremely popular open source, e-commerce platform. Magento has a SOAP API and here’s the documentation for retrieving a list of orders. (Side note: Magento just released a REST API in the past two months, so it’s too new for most Magento users and it also doesn’t support pagination.)

Start easy: try the docs

If you take a look at the docs, you’ll notice no there’s no way obvious to get, say, the first 100 orders. This is highly problematic if you’re dealing with a large e-commerce site. Iterating over all orders will, at best, be extremely slow, and, depending on the number of orders, might even cause server issues with the amount of memory required to construct and consume the response.

But if you look closely, there is a filters argument. Maybe I can do something to get pagination with a little “less than” and “greater than” magic. Let’s give it a go.

After setting up a Magento instance to play with, I tried making the following request to test it out (note I’m using Python and SUDS, but the ideas are the same in PHP, etc.):

from suds import client
client = Client('http://mymagentoinstance/api/v2_soap?wsdl=1')
session = client.service.login('username', 'password')
orders = client.service.salesOrderList(session, {
  'complex_filter' : [
    {
      'key' : 'order_id',
      'value' : {
        'key' : 'gteq',
        'value': 100,
      },
    },
    {
      'key' : 'order_id',
      'value' : {
        'key' : 'lt',
        'value': 200,
      },
    },
  ],
})

So what orders did I get? Well the first thing I noticed was I had 200 orders. Hmmm, not good. Well, looking at the Magento source code, here’s why:

// parse complex filter
if (isset($filters->complex_filter) && is_array($filters->complex_filter)) {
  foreach ($filters->complex_filter as $value) {
    if (is_object($value) && isset($value->key) && isset($value->value)) {
      $fieldName = $value->key;
      $condition = $value->value;
      if (is_object($condition) && isset($condition->key) && isset($condition->value)) {
        $this->formatFilterConditionValue($condition->key, $condition->value);
        $parsedFilters[$fieldName] = array($condition->key => $condition->value);
      }
    }
  }
}

Damn! Magento only allows one filter per field, so there goes any chance to do some less than/greater than trickery.

Next step: try the Google

Alright, well now I feel like I’ve given it a pretty serious go, let’s try the Google. Well, there are some results, but nothing that helps. As an aside, I have to give this solution an A for effort for trying to chunk things by the first character. I think (I hope) I can do better.

Roll up sleeves: read the source and find a way

Digging through the Magento source code for the list of available filters, I found this:

$conditionKeyMap = array(
  'eq' => "{{fieldName}} = ?",
  'neq' => "{{fieldName}} != ?",
  'like' => "{{fieldName}} LIKE ?",
  'nlike' => "{{fieldName}} NOT LIKE ?",
  'in' => "{{fieldName}} IN(?)",
  'nin' => "{{fieldName}} NOT IN(?)",
  'is' => "{{fieldName}} IS ?",
  'notnull' => "{{fieldName}} IS NOT NULL",
  'null' => "{{fieldName}} IS NULL",
  'gt' => "{{fieldName}} > ?",
  'lt' => "{{fieldName}} < ?",
  'gteq' => "{{fieldName}} >= ?",
  'lteq' => "{{fieldName}} <= ?",
  'finset' => "FIND_IN_SET(?, {{fieldName}})",
  'regexp' => "{{fieldName}} REGEXP ?",
  'from' => "{{fieldName}} >= ?",
  'to' => "{{fieldName}} <= ?",
  'seq' => null,
  'sneq' => null
);

Interesting, there’s a regexp filter. Okay, this isn’t going to be pretty, but it might just do the trick. What if I just created a regexp that would match only numbers in a certain range? Here’s the Python code to generate that regular expression:

def regexp_for_pagination(self, from_, to):
  return '^(%s)$' % '|'.join(map(str, xrange(from_, to)))

so regexp_for_pagination(0, 10) would produce "^(0|1|2|3|4|5|6|7|8|9)$". And then a test call:

orders = client.service.salesOrderList(session, {
  'complex_filter' : [
    {
      'key' : 'order_id',
      'value' : {
        'key' : 'regexp',
        'value': regexp_for_pagination(0, 10),
      },
    },
  ],
})

and voila, pagination when they said I couldn’t have it!

Is this a hack? Yep. Am I proud of it? You better believe it.

What APIs have you hacked to do things they “weren’t suppposed to do?”

Creating my first complete Django project on Snow Leopard

Surely there were going to be some issues getting Django working on my Mac (I mean aren’t there are always some problems?), so I decided to crack open a text document and keep track of the things that happened so that in the future when I do this, I don’t run into any problems.

Just as a bit of context, I’ve been playing with Google App Engine recently and it’s been my first attempt at writing Python code. Python and Django? Love ‘em. App Engine? Eh, good at what it does, but I don’t really like working in Google’s sandbox. It just feels so restrictive. So for my next project I decided to skip the App Engine bit and try to build a site still using Django, but with a MySQL backend. Because of my work with App Engine, I already had installed Django and obviously Python 2.6 and 2.6 come with a standard Snow Leopard install, so I had those as well.

So from there I set off. The first thing I knew I would need was MySQL. I’ve used MySQL before, but since doing a clean install of Snow Leopard a few months ago, I hadn’t put it back on, so first to handle that.

Finding the binary installer for Snow Leopard was easy, but here I made a mistake.  I figured I’d install the 32-bit version which ended up causing me some headaches.  What I didn’t realize is that Snow Leopard installs most programs as 64-bit if your machine is 64-bit.  How do you find out if your Mac is 64-bit or not?  See this short post on how to tell from simply looking at your processor.  So I mistakenly installed the 32-bit version and was on my way.

The next thing you need to do is install a MySQL Python adapter.  The source for that is here.  I chose to sudo python setup.py build/install, but you could use easy_install if you want.  Once I installed, I tried to import the MySQLdb package and got the following error:

File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/db/backends/mysql/base.py", line 13, in
    raise ImproperlyConfigured("Error loading MySQLdb module: %s" % e)

This post and others indicated it was related to the build architecture of Python and MySQL. Bah, so this is when I realized I needed to go back and install the 64-bit version of MySQL.  Okay, so I downloaded the 64-bit installer and ran that.  (See further below how that worked, but not completely and I eventually did a clean install.)

Then I ran the following to try to re-install the MySQLdb adapter stuff:

cd MySQL-python-1.2.3c1
ARCHFLAGS='-arch x86_64' python setup.py build
ARCHFLAGS='-arch x86_64' python setup.py install

No dice, still not working. Poking through the comments of the post above, I came across this indicating that re-building doesn’t actually do it from scratch and you need to go into the build directory and manually rm *.  Once I did that, then I was in business — the import MySQLdb statement now worked.

Following the Django tutorial, I created by app, code-named “sanddollar.” So far so good until I got to:

python mange.py syncdb

and crossed my fingers. Forgot to create the database, but after that I still have a problem. I’m getting an error related to MySQL being able to create/write to a file:

File "build/bdist.macosx-10.6-universal/egg/MySQLdb/connections.py", line 36, in defaulterrorhandler
_mysql_exceptions.InternalError: (1, "Can't create/write to file '/usr/local/mysql/data/sanddollar/auth_permission.MYI' (Errcode: 2)")

This was a tough one.  It spawned this Stack Overflow question, but after some poking around I noticed that despite the fact I was able to create databases through the MySQL Workbench, they weren’t creating new database directories under the mysql/data folder.  My suspicion was that multiple installs on MySQL had gotten MySQL in a messed up state.  Solution: remove all traces of MySQL and then do a clean install.

This sounds pretty straightforward, but it wasn’t.  In the end, it took two sources (here and here) to get all the instructions for wiping MySQL where the second set of instructions is specific to Snow Leopard.  Alright, once I had done that, I used the same .pkg installer to install the 64-bit version of MySQL one more time.

Then I tried to connect to MySQL, and I get the following error:

ERROR 2002 (HY000): Can’t connect to local MySQL server through socket ‘/tmp/mysql.sock’ (2)

Fairly straightforward fix.  No idea where the actual mysql.sock file is, but because I could connect via MySQL Workbench over TCP, someone in the IRC channel tipped me off to run:

SELECT @@socket

which outputted the location of the mysql.sock file.  In my case, it was located at /var/lib/mysql/mysql.sock.  So then I ran the following from a bash shell:

$sudo ln -s /var/lib/mysql/mysql.sock /tmp/mysql.sock

and voila, now I can connect from Terminal.  Almost there.  I created a new user for the “sanddollar” database named “sanddollar” and granted the user privileges for basic CRUD operations at host 127.0.0.1 (for some reason localhost wouldn’t work).  Tried to connect through Terminal, worked great. Then tried the syncdb command again and this time paydirt.

Final thoughts: 32-bit/64-bit is complicated, more complicated than it feels like it should be.  This makes working with Snow Leopard as a development environment a pain.  However, I learned about the “file [filename]” unix command to determine the architecture of programs.  No idea what it means but opened it up on Stack Overflow with this question. In the end, not too bad for getting things started.  Feels like it could be simpler, but less than a day isn’t that bad, so onwards!