The Python Tourist #2: Taking Exception

Exception handling in Python is a tremendously useful feature. I grew up on C programming, where, to write a really "correct" program, it seems like you have to spend half your code on mundane error checking.

for(i=0; i<NR; ++i) {
    result = do_thing_1();
    if (result < 0) {
        if (result == IO_ERROR) {
             /* handle error */
        }
        else if (result == API_ERROR) {
            /* handle error */
        }
        else {
            /* handle unknown error */
        }
    }
    result = do_thing_2();

    /* sigh ... I have to code another huge error block ... */
    ...

/* finally, the loop ends! */
}

True, C++ has exception handling, but if you are calling standard C-library functions, then you have to go back to the mundane way.

Rewriting the above mess into an equivalent Python block gives:

try:
    # uninterrupted program logic here, no need to break up
    # the natural flow with error checking
    for i in range(NR):
        do_thing_1()
        do_thing_2()

# errors can all be handled out-of-line
except IOERROR:
    # handle IO error
except API_ERROR:
    # handle API error
except:
    # handle unknown error
    # ** be careful here -- keep reading! **

Notice that I wrote "be careful here" under the last except clause. The "be careful" part is what I want to cover in this article. The topics I'm going to cover are:

Globally catching unhandled errors.
Catching exceptions, with details.
Catching multiple exceptions.
When it is bad/good to use "bare" exceptions.

Globally catching unhandled errors.

There is a special hook, sys.excepthook, that (in my experience) is one of the best debugging tools available in Python. I have a custom module that I always include in my projects that handles this automatically. The bulk of the work is done by a module (in the standard Python library) called cgitb. It is called "cgitb" because its original intent was to show a traceback of errors that occurred in CGI scripts, but it is just as easy to use in any other kind of program. Here is my custom error handling module, if you'd like to cut and paste:

Module errors.py

import sys, cgitb
from datetime import datetime

def catch_errors():
    sys.excepthook = my_except_hook

def my_except_hook(etype, evalue, etraceback):
    do_verbose_exception( (etype,evalue,etraceback) )

def do_verbose_exception(exc_info=None):
    if exc_info is None:
        exc_info = sys.exc_info()

    txt = cgitb.text(exc_info)

    d = datetime.now()
    p = (d.year, d.month, d.day, d.hour, d.minute, d.second)
    filename = "ErrorDump-%d%02d%02d-%02d%02d%02d.txt" % p

    open(filename,'w').write(txt)
    print "** EXITING on unhandled exception - See %s" % filename
    sys.exit(1)

Now, anytime you have a "thinko" error, it will automatically be caught and verbose debugging information will be saved to a file. Here is an example:

"Thinko" bug that will be caught by error handler.

# enable catching of unhandled exceptions
import errors
errors.catch_errors()

def do_thing_1(value):

    # just coding along, not expecting anything to go wrong ...
    for i in range(20):
        print "ratio is %d" % (value/(10-i))

do_thing_1(100)

Now lets see what happens when I run it ...

ratio is 10
ratio is 11
ratio is 12
ratio is 14
ratio is 16
ratio is 20
ratio is 25
ratio is 33
ratio is 50
ratio is 100
** EXITING on unhandled exception - See ErrorDump-20060305-110304.txt

The dumpfile shows what happened, with tons of detail. Notice how it shows the function parameters (do_thing_1(value=100)) as well as the value of the local variables when the error occurred. Normally, you'd have to rerun the test case and step through the loop to see what these values were. Now, immediately after the crash, you have a snapshot of the program state.

Contents of "ErrorDump-20060305-110304.txt"

ZeroDivisionError
Python 2.4.2: /usr/bin/python
Sun Mar  5 11:03:04 2006

A problem occurred in a Python script.  Here is the sequence of
function calls leading up to the error, in the order they occurred.

/var/www/localhost/htdocs/python/tourist/t.py
    9         print "ratio is %d" % (value/(10-i))
   10
   11 do_thing_1(100)
   12
do_thing_1 = <function do_thing_1>

/var/www/localhost/htdocs/python/tourist/t.py in do_thing_1(value=100)
    7     # just coding along, not expecting anything to go wrong ...
    8     for i in range(20):
    9         print "ratio is %d" % (value/(10-i))
   10
   11 do_thing_1(100)
value = 100
i = 10
ZeroDivisionError: integer division or modulo by zero
    __doc__ = 'Second argument to a division or modulo operation was zero.'
    __getitem__ = <bound method ZeroDivisionError.__getitem__ of <exceptions.ZeroDivisionError instance>>
    __init__ = <bound method ZeroDivisionError.__init__ of <exceptions.ZeroDivisionError instance>>
    __module__ = 'exceptions'
    __str__ = <bound method ZeroDivisionError.__str__ of <exceptions.ZeroDivisionError instance>>
    args = ('integer division or modulo by zero',)

The above is a description of an error in a Python program.  Here is
the original traceback:

Traceback (most recent call last):
  File "t.py", line 11, in ?
    do_thing_1(100)
  File "t.py", line 9, in do_thing_1
    print "ratio is %d" % (value/(10-i))
ZeroDivisionError: integer division or modulo by zero

Of course, in a final product, you wouldn't want the program to crash so abruptly. I have a more polished GUI version that I use in situations where an end user might see the error. I just gave a bare-bones version here so you can see the important parts, and can customize the error module to suit your application.

As a beginning Python programmer, I was originally tempted to put try ... except clauses around everything. After a while, I realized that it was much better to only have try .. except clauses in places where there was some sort of state information that needed to be cleaned up or rolled back after the error. Other errors (like the "thinko" cases) are better left to the global handler. You can actually lose information by being too greedy with your except clauses.

That leads nicely into the next topic ...

Catching exceptions, with details.

When you catch an exception as shown below, you are only getting part of the available information:

try:
    ... do some stuff ...

except API_ERROR:
    #
    # OK, I know an API_ERROR occurred, but have no other details!
    #

Here is a little example of how to provide and use more information in your except clauses.

class ParseError(Exception):
    """
    Custom exception class to capture details on a
    parsing error:

          txt = Will be shown by default exception handler.
          filename,line,col = Where the error occurred.
    """
    def __init__(self, txt, filename, line, col):
        Exception.__init__(self, txt)
        self.filename = filename
        self.line = line
        self.col = col

def parsefile(filename):

    for line in open(filename,'r'):
        # ... parsing code ...

        # Say I find an error in line 20, column 10 ...
        line = 20
        col = 10
        raise ParseError('Parse Error, file=%s line=%d,col=%d' % \
                (filename,line,col),
                filename, 20, 10)

# If I do nothing, Python will show the 'txt' as the error.
parsefile('t.py')

Output

Traceback (most recent call last):
  File "t.py", line 27, in ?
    parsefile('t.py')
  File "t.py", line 24, in parsefile
    filename, 20, 10)
__main__.ParseError: Parse Error, file=t.py line=20,col=10

As you can see, even without writing a try ... except clause, you are already getting more information from the txt string. Now let's see how to capture the exception and access all of its attributes.

# Catch it so I can access .filename, .line, and .col
try:
    parsefile('t.py')

except ParseError, info:
    # Now I can do whatever I want to with the detailed info
    print "CAUGHT! Parse error in %s at line=%d, column=%d" % \
        (info.filename, info.line, info.col)

Output

CAUGHT! Parse error in t.py at line=20, column=10

WARNING

The except clause must be exactly except ParseError, info. If you try to use except (ParseError,info) or except [ParseError,info] it will not work.

Conveniently, that leads us into the next topic ...

Catching multiple exceptions.

Sometimes it is convenient to be able to catch multiple exceptions with a single except clause. In the example below, I'm going to check for bad types being passed to a function, and raise a per-type exception if an error is detected.

"""
As in the previous example, I will place useful info into
the 'txt' parameter to the base Exception class. This way
the caller can see exactly what happened without having
to catch the exception and look at the 'info' parameter.
"""
class ErrNeedList(Exception):
    def __init__(self, parm):
        Exception.__init__(self, "Need a list for '%s'" % parm)
        self.parm = parm
        self.usage = "Need a list"

class ErrNeedDict(Exception):
    def __init__(self, parm):
        Exception.__init__(self, "Need a dictionary for '%s'" % parm)
        self.parm = parm
        self.usage = "Need a dictionary"

class ErrNeedString(Exception):
    def __init__(self, parm):
        Exception.__init__(self, "Need a string for '%s'" % parm)
        self.parm = parm
        self.usage = "Need a string"

def test_function(a_list, a_dict, a_string):
    # check for type errors
    if not isinstance(a_list, list):
        raise ErrNeedList('a_list')

    if not isinstance(a_dict, dict):
        raise ErrNeedDict('a_dict')

    if not isinstance(a_string, str):
        raise ErrNeedString('a_string')

# cause errors and catch them

try:
    test_function( 1,2,3)
#------------------------------------------------------
# here I can test for all errors at once - since each
# has a .parm and .usage attribute, I can treat them
# the same way
#------------------------------------------------------
except (ErrNeedList, ErrNeedDict, ErrNeedString), info:
    print "CAUGHT API ERROR in parameter: %s - %s" % (info.parm, info.usage)

try:
    test_function( [],2,3)
except (ErrNeedList, ErrNeedDict, ErrNeedString), info:
    print "CAUGHT API ERROR in parameter: %s - %s" % (info.parm, info.usage)

try:
    test_function( [],{},3)
except (ErrNeedList, ErrNeedDict, ErrNeedString), info:
    print "CAUGHT API ERROR in parameter: %s - %s" % (info.parm, info.usage)

Output

CAUGHT API ERROR in parameter: a_list - Need a list
CAUGHT API ERROR in parameter: a_dict - Need a dictionary
CAUGHT API ERROR in parameter: a_string - Need a string

In an example like this, where all exceptions have common attributes, it makes sense to derive all exceptions from a single base class. Rewriting the classes to derive from a common class APIError:

"Base class"
class APIError(Exception):
    def __init__(self, txt, parm, usage):
        Exception.__init__(self, txt)
        self.parm = parm
        self.usage = usage

class ErrNeedList(APIError):
    def __init__(self, parm):
        APIError.__init__(self, "Need a list for '%s'" % parm,
                            parm, "Need a list")

class ErrNeedDict(APIError):
    def __init__(self, parm):
        APIError.__init__(self, "Need a dictionary for '%s'" % parm,
                            parm, "Need a dictionary")

class ErrNeedString(APIError):
    def __init__(self, parm):
        APIError.__init__(self, "Need a string for '%s'" % parm,
                            parm, "Need a string")

Now the exceptions can be caught in a more compact way:

try:
    test_function( [],{},3)

#
# Now I can just catch the baseclass, and it will catch
# all subclasses as well!
#
except APIError, info:
    print "CAUGHT API ERROR in parameter: %s - %s" % (info.parm, info.usage)

WARNING

You must use a tuple when catching multiple exceptions, i.e. except (Err1,Err2,Err3). If you try to use a list, i.e. except [Err1,Err2,Err3] it will not catch the exception, and what's worse, Python will not flag it as a syntax error.

Hopefully it is clearer now why you cannot use except (IOError,info) as a substitute for except IOError, info. If you tried the first form, you would be trying to catch an exception of class info, which isn't what was intended (and if info is undefined, Python will raise another exception).

When it is bad/good to use "bare" exceptions.

When I was first learning about exceptions, it seemed like a good idea to write code blocks like this:

try:
... do something ...

except:
print "Got an error!"

This initially seemed robust to me because you are guaranteed to catch all errors in the block. There are three problems with this:

No exception type is specified, so you have no idea what sort of error occurred.
No info parameter is given, so you are throwing away any extra information that was present.
You are catching not only the errors you expect to occur, but are in essense masking out the errors that you didn't expect to occur.

Here is a brief example to demonstrate:

try:
    # I could get an IOError here if "filein.txt" doesn't exist,
    # or is not readable.
    fin = open('t.py','r')

    # I could get an IOError here if I do not have write-permissions
    # in the current directory, or the disk is out of space.
    fout = open('fileout.txt','w')

    # I don't expect anything bad to happen here ...
    for line in fin:
        # filter out comment lines
        if re.match('^\s*#$', line):
            continue

except:
    # The only errors I *expect* are IOErrors, so obviously
    # I can say this ... or can I??
    print "A file error occurred."

Now, I run this example and get the following:

A file error occurred.

So I start debugging. Expecting only an IOError, I make a list of what could have happened:

t.py does not exist
t.py is not readable
I cannot create fileout.txt because of insufficient privileges or the disk is out of space

I check and recheck, and there is "no" reason that the code should fail, yet it is telling me that there was a file error. The problem is that I was not specific enough in my except clause, and have (essentially) masked out the real failure.

Rewriting the except clause shows the real problem:

try:
    # I could get an IOError if "filein.txt"  doesn't exist,
    # or is not readable.
    fin = open('t.py','r')

    # Same here, if I cannot create "fileout.txt"
    fout = open('fileout.txt','w')

    for line in fin:
        # filter out comment lines
        if re.match('^\s*#$', line):
            continue

# Only catch what I am PREPARED to handle!
except IOError:
    print "A file error occurred."

Output

Traceback (most recent call last):
File "t.py", line 11, in ?
if re.match('^s*$', line):
NameError: name 're' is not defined

Ah! Of course, I forgot to import to import re before using it.

I think this highlights why you should only catch those specific errors you are prepared to handle, and let the others float up to the top level (where you can catch them, i.e. with the "global hook" described eariler).

Sometimes, unqualified "excepts" are okay!

Now, I'm going to immediately contradict myself and state that there are times when unqualified except clauses are okay, and even desired. One example I run across all the time is in GUI code, when I've set the mouse cursor to a "busy" state before starting a long operation. If you crash in the middle, you don't want to leave the cursor in the busy state forever since that would be confusing to the user. Here is the typical way I code that situation (using wxPython here):

#
# I'm about to perform a long operation, so set the
# cursor to the "busy" (hourglass) state
#
wx.BeginBusyCursor()

try:
    # A long operation begins here ...

    ...
    ...

    # when I'm finished, exit the busy state
    wx.EndBusyCursor()

except:
    # cancel the busy state - don't leave the user hanging!
    wx.EndBusyCursor()

    # re-raise original error to caller
    raise

That final line is critical: When you use a bare raise statement, it will re-raise the original exception, so the error will propogate back to the caller with no loss of information.

WARNING

You do not want to do it like this:

try:
    ... stuff ...

except Exception, exc:
    raise exc

As this will cause you to lose information from the original exception.

I've also used unqualified except clauses in situations involving database transactions:

# "pseudo-SQL" code, just to give the idea ...
try:
    # run entire transaction inside of "try"
    sql.run("begin transaction")

    sql.run("insert into ...")
    sql.run("insert into ...")
    sql.run("update ...")

    sql.run("commit transaction")

except:
    # undo any changes on error
    sql.run("rollback transaction")

    # propagate original error
    raise

This is very nice because the caller will know that an error has occurred, but doesn't have to worry about the database state because it has already been cleaned up.

One final example of where I find unqualified excepts useful is in threaded programs where I'm locking around a set of global data:

from threading import Lock
DATA_LOCK = Lock()

try:
    DATA_LOCK.acquire()

    .. perform operations on global data ...

    DATA_LOCK.release()

except:
    # unlock on error
    DATA_LOCK.release()

    # propagate original error
    raise

Written in WikklyText.

Comments

September 24, 2007 - 7:07am — Anonymous (not verified)

I ran your errors.py script

I ran your errors.py script example and it didn't create the dump file and exceptions are being printed on the python shell rather than into the file. I am running python 2.4.3. Dump file is not created at all, and I looked for the dump file in same folder where the script was and dump file is not there. Any idea why that may be?

September 24, 2007 - 10:31am — frank

Updated errors.py

I noticed that when I cut & pasted errors.py directly from the box above that the continuation character in the filename = ... line was messed up. I rewrote it to remove the continuation.

Could you try cutting & pasting errors.py again and let me know if that fixes it?

June 10, 2007 - 5:32pm — Liam Clarke (not verified)

Just saw this on

Just saw this on programming.reddit - great post, I just learnt something very useful about Python. :)

March 12, 2007 - 8:19am — Jim (not verified)

try - finally

In your last few examples, you use a generic except statement to tidy up loose ends (such as database transactions and busy cursors). Sometimes, the finally keyword is more suited to this task. The finally keyword makes sure that a section of code is always executed, even if the function ends via an exception or a return statement.

I'll attempt some code in demonstration, using a rewrite of your final example, though I'm not sure how successful it will turn out due to the formatting on these comments:

from threading import Lock DATA_LOCK = Lock()


try:

    DATA_LOCK.acquire()

.. perform operations on global data ... finally: # always unlock DATA_LOCK.release()

March 12, 2007 - 7:50pm — frank

Thanks - I need to update

Thanks - I need to update the article to include "try ... finally". Python 2.5 has some new features in this regard that I need to add as well.