The Python Tourist #1: Passing Mutable Objects as Default Args

Many modern languages allow for default function arguments. Default arguments can be a great thing - they allow you to use sensible defaults for the common cases, without taking away the power to add more functionality as needed. You can arbitrarily expand a function definition without breaking existing code by providing sensible defaults for the new parameters.

Here is a fun example of how Python will bite you if you try to use a mutable object (like a list, dict, or object instance) as a default argument. The following is an example function I'm trying to write ...



At first glance this appears to do what is intended. If the user doesn't pass a list to append items to, then a new list is started. Or at least that appears to be what will happen. Lets try running it:



When you run this, you will get the following output:



Whoa! That was unexpected. When you use a mutable object as a default arg, Python creates a single object, and reuses that same object on every call. I find this non-intuitive, personally, but that's the way Python works. I think it has something to do with passing args to lambda functions, but I could be completely wrong.

Anyways, here is the correct way to code the function to acheive the result I want:



Now when you run the program you will get the expected result:



Note that not every instance of using a mutable type as a default arg will cause problems. In fact, you'll probably be OK most of the time. The danger is in those times that you aren't expecting trouble, and can't figure out why your code isn't working. I find it tends to happen more with recursive functions.

At any rate, my personal rule of thumb is to never use a mutable type as a default arg, and instead to always us None as shown above, and then set the variable inside the function body (where a new empty object will always be created).

It's simple, neat, takes one line, and can save you a marathon debugging session trying to figure out what is wrong (speaking from painful experience!)

The Python Tourist #2: Taking Exception

Exception handling in Python is a tremendously useful feature. I grew up on C programming, where, to write a really "correct" program, it seems like you have to spend half your code on mundane error checking.

True, C++ has exception handling, but if you are calling standard C-library functions, then you have to go back to the mundane way.

Rewriting the above mess into an equivalent Python block gives:

Notice that I wrote "be careful here" under the last except clause. The "be careful" part is what I want to cover in this article. The topics I'm going to cover are:

1. Globally catching unhandled errors.
2. Catching exceptions, with details.
3. Catching multiple exceptions.
4. When it is bad/good to use "bare" exceptions.
   
   
   

Globally catching unhandled errors.

There is a special hook, sys.excepthook, that (in my experience) is one of the best debugging tools available in Python. I have a custom module that I always include in my projects that handles this automatically. The bulk of the work is done by a module (in the standard Python library) called cgitb. It is called "cgitb" because its original intent was to show a traceback of errors that occurred in CGI scripts, but it is just as easy to use in any other kind of program. Here is my custom error handling module, if you'd like to cut and paste: Now, anytime you have a "thinko" error, it will automatically be caught and verbose debugging information will be saved to a file. Here is an example: Now lets see what happens when I run it ... The dumpfile shows what happened, with tons of detail. Notice how it shows the function parameters (do_thing_1(value=100)) as well as the value of the local variables when the error occurred. Normally, you'd have to rerun the test case and step through the loop to see what these values were. Now, immediately after the crash, you have a snapshot of the program state. Of course, in a final product, you wouldn't want the program to crash so abruptly. I have a more polished GUI version that I use in situations where an end user might see the error. I just gave a bare-bones version here so you can see the important parts, and can customize the error module to suit your application.

As a beginning Python programmer, I was originally tempted to put try ... except clauses around everything. After a while, I realized that it was much better to only have try .. except clauses in places where there was some sort of state information that needed to be cleaned up or rolled back after the error. Other errors (like the "thinko" cases) are better left to the global handler. You can actually lose information by being too greedy with your except clauses.

That leads nicely into the next topic ...

Catching exceptions, with details.

When you catch an exception as shown below, you are only getting part of the available information: Here is a little example of how to provide and use more information in your except clauses.

As you can see, even without writing a try ... except clause, you are already getting more information from the txt string. Now let's see how to capture the exception and access all of its attributes.



Conveniently, that leads us into the next topic ...

Catching multiple exceptions.

Sometimes it is convenient to be able to catch multiple exceptions with a single except clause. In the example below, I'm going to check for bad types being passed to a function, and raise a per-type exception if an error is detected.



In an example like this, where all exceptions have common attributes, it makes sense to derive all exceptions from a single base class. Rewriting the classes to derive from a common class APIError: Now the exceptions can be caught in a more compact way:

Hopefully it is clearer now why you cannot use except (IOError,info) as a substitute for except IOError, info. If you tried the first form, you would be trying to catch an exception of class info, which isn't what was intended (and if info is undefined, Python will raise another exception).

When it is bad/good to use "bare" exceptions.

When I was first learning about exceptions, it seemed like a good idea to write code blocks like this:

This initially seemed robust to me because you are guaranteed to catch all errors in the block. There are three problems with this:

1. No exception type is specified, so you have no idea what sort of error occurred.
2. No info parameter is given, so you are throwing away any extra information that was present.
3. You are catching not only the errors you expect to occur, but are in essense masking out the errors that you didn't expect to occur.
   
   

Here is a brief example to demonstrate:

Now, I run this example and get the following:

So I start debugging. Expecting only an IOError, I make a list of what could have happened:

1. t.py does not exist
2. t.py is not readable
3. I cannot create fileout.txt because of insufficient privileges or the disk is out of space
   
   

I check and recheck, and there is "no" reason that the code should fail, yet it is telling me that there was a file error. The problem is that I was not specific enough in my except clause, and have (essentially) masked out the real failure.

Rewriting the except clause shows the real problem:

Ah! Of course, I forgot to import to import re before using it.

I think this highlights why you should only catch those specific errors you are prepared to handle, and let the others float up to the top level (where you can catch them, i.e. with the "global hook" described eariler).

Sometimes, unqualified "excepts" are okay!

Now, I'm going to immediately contradict myself and state that there are times when unqualified except clauses are okay, and even desired. One example I run across all the time is in GUI code, when I've set the mouse cursor to a "busy" state before starting a long operation. If you crash in the middle, you don't want to leave the cursor in the busy state forever since that would be confusing to the user. Here is the typical way I code that situation (using wxPython here): That final line is critical: When you use a bare raise statement, it will re-raise the original exception, so the error will propogate back to the caller with no loss of information. I've also used unqualified except clauses in situations involving database transactions:

This is very nice because the caller will know that an error has occurred, but doesn't have to worry about the database state because it has already been cleaned up.

One final example of where I find unqualified excepts useful is in threaded programs where I'm locking around a set of global data:

The Python Tourist #3: Forgetting how cmp() works ... no problem!

The cmp() function is the basis for sorting in Python. It has a simple definition but I'm always forgetting the sense of the return values. For reference:

cmp(a, b) returns:

• -1, if a < b
• 0 if a == b
• 1 if a > b
  
  

Now, cmp() only knows how to sort built-in values, like integers, strings, etc. If you try to sort a list of user-defined objects, you'll find that Python has no idea how to sort them (though it will do something). What you are supposed to do is define a __cmp__ function for your custom classes. Then, when your objects are sorted, Python calls your __cmp__ to tell it how to order them.

Every time I create a class that needs a __cmp__, I'm tempted to go scrambling to the Python docs for a refresher on what the three return values of cmp mean. But really, there is no need to worry about this. You can simply use the cmp() function to do it for you.

I'll demonstrate with an example: This works, but is extremely cumbersome and error prone. However, there is no need to go to all this trouble. All you need to do is to split your object into items that cmp() knows how to handle. In other words, cmp() knows perfectly well how to sort the .first, .last, and .state values, you just have to split them up and pass them in the correct order: True, the "or" short-circuit logic relies on the fact that cmp(a,b) == 0 when a == b, but that's well-defined and gives much cleaner code in comparison to the bloated mess of doing it yourself.

The Python Tourist #4: None, empty, and nothing.

In learning Python, I had read in several places that you should always use a test like if obj is None if you wanted to check for the None value. For some reason, I tend to ignore blanket statements that are presented without supporting rationale. If the underlying rationale isn't stated, I generally assume it's some sort of esoteric thing that doesn't really matter. Only after it bites me and I can understand the logic will I pay attention.

Here are a couple of cases where not explicitly testing for None has gotten me into trouble. Maybe these can help someone else avoid the same headaches.

Look at this sample: Looks correct enough. The if parse_file(...) should be True if I get a list, and the else should be True if I get None. There is one little problem though. Look at the following snippet: This will always print "False". Coming from a C background, I want to think of None as "the absence of something", like a NULL pointer. Unfortunately, Python treats empty objects as False values. To me, an empty object is still something as opposed to None which (I think) should be nothing, so this is confusing.

I think the thing to do is recognize that this function has three exit states:

1. None, indicating an error.
2. An empty list, indicating an empty file.
3. A non-empty list, holding tags.
   
   

The correct test is then: We can make this worse and give it four exit states, with the same functionality: Although it looks like the same logic, I've introduced a (sort of) hidden fourth state: If the "end-of-file" tag isn't found, the for loop will exit without returning a value. When you don't return a value, None is returned. For example: Of course, the code sample above is buggy, I shouldn't let it fall out of the loop. Once again, my C background gave me a false sense of security. A C compiler will tell you when you exit a routine in different ways (with and without a return value), so things like this won't happen if you pay attention to the compiler warnings. The dynamic nature of Python means that it really can't do that kind of checking, since it would be impractical to run through every branch inside the function to see if the return values match.

Anyways, disregarding the buggy code for the moment, recognize that the above function has four distinct exit states:

1. None, indicating an error.
2. An empty list, indicating an empty file.
3. A non-empty list, holding tags.
4. None, indicating no return value.
   
   

The first and last cases bother me a little bit. I don't like that None can have two meanings:

1. The value None.
2. The absence of a value.
   
   

In my "C thinking" of the first example, I was assuming None meant "the absence of a value", so was surprised to find that an empty list was (apparently) the same as nothing. Of course, that isn't the case, it's just that an empty list evaluates to the same boolean value as None.

I appreciate that Python is a practical language. An impractical language could "fix" this by forcing you to only use (exactly) True or False in boolean expressions. Python tends to loosen the rules as much as practical, without going overboard. (Some languages like perl go overboard in their coercion rules, which I think leads to even harder to understand code.). I wish that empty lists didn't evaluate to False, but that's the way it is, so you just have to keep it in mind. One final note: The correct test for None is if obj is None, not if obj == None. The reason not to use == is that an object can define its own __eq__ function, and might implement __eq__ in a way that would cause it to be equal to (even if not the same as) None. The "is" operator means "the same object", so is the more correct test here.

The Python Tourist #5: Replacing sys.version_info with pyconfig

If you've spent any time writing Python code that is meant to be portable across multiple versions of Python, you've most likely written a few statements like this: The problem I see with this is that it is ultra-verbose, and doesn't actually tell you what capability you require here. Although you can chop down the verbosity with a statement like: That doesn't take care of the fact that you are relying on a hardcoded version number. There are a few downsides to this:

• If you came back to this code later, and wanted to improve the support for Python < 2.2, it isn't clear what capability is missing in Python < 2.2.
• With the creation of alternative Python implementations (like Jython) it isn't so clear what "2.2" means. Jython (currently) has a mix of 2.2 and 2.3 features, so a simple version check doesn't tell the whole story.
• If the format of version_info changes, your code breaks. It is better to work at a higher level of abstraction.
  
  

For the sake of argument, lets say that IronPython has a different feature set than CPython and Jython. Picture writing code like this: After a while, it begins to feel like writing C-style #ifdefs instead of Python.

The dynamic nature of Python means you can do all sort of neat introspective things, including introspection of runtime capabilities. Want to know if the current Python understands a particular piece of code? Run it and see!

There is a little module called pyconfig that I wrote while working on xml.pickle (part of Gnosis_Utils). It is sort of like an autoconf for Python, except it works at runtime. It is bundled with Gnosis_Utils (since it uses it internally), but can be used as a stand-alone module, as there are no external dependencies.

The pyconfig module provides a set of prewritten tests to let you check for capabilities of the Python interpreter, instead of relying on version numbers.

Compare the following two code segments: Compare to the pyconfig-based code: In the second case, it is clear exactly what capability is needed. Also, the code is now robust across any nonstandard versions of Python that it might be running on.

If you import pyconfig as from pyconfig import pyconfig, then all test results will be automatically cached. This allows you to use the tests inline with a minimum speed penalty.

pyconfig is written as set of small tests, with the reusable parts modularized. This makes it very easy to write any new tests you need. If nothing else, the source code to pyconfig is an interesting historical reference of the various PEPs that have been included over the evolution of Python.

Getting pyconfig

As mentioned, pyconfig comes bundled with Gnosis_Utils: Gnosis_Utils

Or if you prefer, you can grab it as a separate module: pyconfig.py