I l@ve RuBoard Next Section

• | Table of Contents
• | Index
• | Reviews
• | Examples
• | Reader Reviews
• | Errata
Python Standard Library
By Fredrik Lundh
Publisher | : O'Reilly
Pub Date | : May 2001
ISBN | : 0-596-00096-0
Pages | : 300

Python Standard Library, an essential guide for serious Python programmers, delivers accurate, author-tested documentation of all the modules in the Python Standard Library, along with over 300 annotated example scripts using the modules. This version of the book covers all the new modules and related information for Python 2.0, the first major release of Python in four years.

I l@ve RuBoard Next Section


I l@ve RuBoard

Copyright © 2001 O'Reilly & Associates, Inc. All rights reserved.

Printed in the United States of America.

Published by O'Reilly & Associates, Inc., 101 Morris Street, Sebastopol, CA 95472.

Nutshell Handbook, the Nutshell Handbook logo, and the O'Reilly logo are registered trademarks of O'Reilly & Associates, Inc. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this book, and O'Reilly & Associates, Inc. was aware of a trademark claim, the designations have been printed in caps or initial caps. The association between the image of a harvest mouse and the Python programming language is a trademark of O'Reilly & Associates.

While every precaution has been taken in the preparation of this book, the publisher assumes no responsibility for errors or omissions, or for damages resulting from the use of the information contained herein.
    I l@ve RuBoard


Chapter 1. Core Modules

"Since the functions in the C runtime library are not part of the Win32 API, we believe the number of applications that will be affected by this bug to be very limited."

—Microsoft, January 1999


1.1 Introduction

Python's standard library covers a wide range of modules. It includes everything from modules that are as much a part of the Python language as the types and statements defined by the language specification, to obscure modules that are probably useful only to a small number of programs.

This chapter describes a number of fundamental standard library modules. Any larger Python program is likely to use most of these modules, either directly or indirectly.

1.1.1 Built-in Functions and Exceptions

The following two modules are even more basic than all other modules combined: the _ _builtin_ _ module, which defines built-in functions (like len, int, and range), and the exceptions module, which defines all built-in exceptions.

Python imports both modules when it starts up, and makes their content available for all programs.

1.1.2 Operating System Interface Modules

There are a number of modules modeled after the POSIX standard API and the standard C library that provide platform-independent interfaces to the underlying operating system.

The modules in this group include os, which provides file and process operations, os.path, which offers a platform-independent way to pull apart and put together filenames, and time, which provides functions to work with dates and times.

To some extent, networking and thread support modules could also belong in this group, but they are not supported by all Python implementations.

1.1.3 Type Support Modules

Several built-in types have support modules in the standard library. The string module implements commonly used string operations, the math module provides math operations and constants, and the cmath module does the same for complex numbers.

1.1.4 Regular Expressions

The re module provides regular expressions support for Python. Regular expressions are string patterns written in a special syntax, which can be used to match strings and extract substrings.

1.1.5 Language Support Modules

sys gives you access to various interpreter variables, such as the module search path, and the interpreter version. operator provides functional equivalents to many built-in operators. copy allows you to copy objects. And finally, gc gives you more control over the garbage collector facilities in Python 2.0.


1.2 The _ _builtin_ _ Module

This module contains built-in functions that are automatically available in all Python modules. You usually don't have to import this module; Python does that for you when necessary.

1.2.1 Calling a Function with Arguments from a Tuple or Dictionary

Python allows you to build function argument lists on the fly. Just put all the arguments in a tuple, and call the built-in apply function, as illustrated in Example 1-1.

Example 1-1. Using the apply Function
File: builtin-apply-example-1.py

def function(a, b):
    print a, b

apply(function, ("whither", "canada?"))
apply(function, (1, 2 + 3))

whither canada?
1 5

To pass keyword arguments to a function, you can use a dictionary as the third argument to apply, as shown in Example 1-2.

Example 1-2. Using the apply Function to Pass Keyword Arguments
File: builtin-apply-example-2.py

def function(a, b):
    print a, b

apply(function, ("crunchy", "frog"))
apply(function, ("crunchy",), {"b": "frog"})
apply(function, (), {"a": "crunchy", "b": "frog"})

crunchy frog
crunchy frog
crunchy frog

One common use for apply is to pass constructor arguments from a subclass on to the base class, especially if the constructor takes a lot of arguments. See Example 1-3.

Example 1-3. Using the apply Function to Call Base Class Constructors
File: builtin-apply-example-3.py

class Rectangle:
    def _ _init_ _(self, color="white", width=10, height=10):
        print "create a", color, self, "sized", width, "x", height

class RoundedRectangle(Rectangle):
    def _ _init_ _(self, **kw):
        apply(Rectangle._ _init_ _, (self,), kw)

rect = Rectangle(color="green", height=100, width=100)
rect = RoundedRectangle(color="blue", height=20)

create a green <Rectangle instance at 8c8260> sized 100 x 100
create a blue <RoundedRectangle instance at 8c84c0> sized 10 x 20

Python 2.0 provides an alternate syntax. Instead of apply, you can use an ordinary function call, and use * to mark the tuple, and ** to mark the dictionary.

The following two statements are equivalent:

result = function(*args, **kwargs)
result = apply(function, args, kwargs)

1.2.2 Loading and Reloading Modules

If you've written a Python program larger than just a few lines, you know that the import statement is used to import external modules (you can also use the from-import version). What you might not know already is that import delegates the actual work to a built-in function called _ _import_ _.

The trick is that you can call this function directly. This can be handy if you have the module name in a string variable, as in Example 1-4, which imports all modules whose names end with "-plugin":

Example 1-4. Using the _ _import_ _ Function to Load Named Modules
File: builtin-import-example-1.py

import glob, os

modules = []

for module_file in glob.glob("*-plugin.py"):
    try:
        module_name, ext = os.path.splitext(os.path.basename(module_file))
        module = _ _import_ _(module_name)
        modules.append(module)
    except ImportError:
        pass # ignore broken modules

# say hello to all modules
for module in modules:
    module.hello()

example-plugin says hello

Note that the plug-in modules have hyphens. This means that you cannot import such a module using the ordinary import command, since you cannot have hyphens in Python identifiers.

Example 1-5 shows the plug-in used in Example 1-4.

Example 1-5. A Sample Plug-in
File: example-plugin.py

def hello():
    print "example-plugin says hello"

Example 1-6 shows how to get a function object, given that the module and function name are strings.

Example 1-6. Using the _ _import_ _ Function to Get a Named Function
File: builtin-import-example-2.py

def getfunctionbyname(module_name, function_name):
    module = _ _import_ _(module_name)
    return getattr(module, function_name)

print repr(getfunctionbyname("dumbdbm", "open"))

<function open at 794fa0>

You can also use this function to implement lazy module loading. In Example 1-7, the string module is imported when it is first used.

Example 1-7. Using the _ _import_ _ Function to Implement Lazy Import
File: builtin-import-example-3.py

class LazyImport:
    def _ _init_ _(self, module_name):
        self.module_name = module_name
        self.module = None
    def _ _getattr_ _(self, name):
        if self.module is None:
            self.module = _ _import_ _(self.module_name)
        return getattr(self.module, name)

string = LazyImport("string")

print string.lowercase

abcdefghijklmnopqrstuvwxyz

Python provides some basic support for reloading modules that you've already imported. Example 1-8 loads the hello.py file three times.

Example 1-8. Using the reload Function
File: builtin-reload-example-1.py

import hello
reload(hello)
reload(hello)

hello again, and welcome to the show
hello again, and welcome to the show
hello again, and welcome to the show

reload uses the module name associated with the module object, not the variable name. Even if you've renamed the original module, reload can still find it.

Note that when you reload a module, it is recompiled, and the new module replaces the old one in the module dictionary. However, if you have created instances of classes defined in that module, those instances will still use the old implementation.

Likewise, if you've used from-import to create references to module members in other modules, those references will not be updated.

1.2.3 Looking in Namespaces

The dir function returns a list of all members of a given module, class, instance, or other type. It's probably most useful when you're working with an interactive Python interpreter, but can also come in handy in other situations. Example 1-9 shows the dir function in use.

Example 1-9. Using the dir Function
File: builtin-dir-example-1.py

def dump(value):
    print value, "=>", dir(value)

import sys

dump(0)
dump(1.0)
dump(0.0j) # complex number
dump([]) # list
dump({}) # dictionary
dump("string")
dump(len) # function
dump(sys) # module

0 => []
1.0 => []
0j => ['conjugate', 'imag', 'real']
[] => ['append', 'count', 'extend', 'index', 'insert',
    'pop', 'remove', 'reverse', 'sort']
{} => ['clear', 'copy', 'get', 'has_key', 'items',
    'keys', 'update', 'values']
string => []
<built-in function len> => ['_ _doc_ _', '_ _name_ _', '_ _self_ _']
<module 'sys' (built-in)> => ['_ _doc_ _', '_ _name_ _',
    '_ _stderr_ _', '_ _stdin_ _', '_ _stdout_ _', 'argv',
    'builtin_module_names', 'copyright', 'dllhandle',
    'exc_info', 'exc_type', 'exec_prefix', 'executable',
...

In Example 1-10, the getmember function returns all class-level attributes and methods defined by a given class.

Example 1-10. Using the dir Function to Find All Members of a Class
File: builtin-dir-example-2.py

class A:
    def a(self):
        pass
    def b(self):
        pass

class B(A):
    def c(self):
        pass
    def d(self):
        pass

def getmembers(klass, members=None):
    # get a list of all class members, ordered by class
    if members is None:
        members = []
    for k in klass._ _bases_ _:
        getmembers(k, members)
    for m in dir(klass):
        if m not in members:
            members.append(m)
    return members

print getmembers(A)
print getmembers(B)
print getmembers(IOError)

['_ _doc_ _', '_ _module_ _', 'a', 'b']
['_ _doc_ _', '_ _module_ _', 'a', 'b', 'c', 'd']
['_ _doc_ _', '_ _getitem_ _', '_ _init_ _', '_ _module_ _', '_ _str_ _']

Note that the getmembers function returns an ordered list. The earlier a name appears in the list, the higher up in the class hierarchy it's defined. If order doesn't matter, you can use a dictionary to collect the names instead of a list.

The vars function is similar, but it returns a dictionary containing the current value for each member. If you use vars without an argument, it returns a dictionary containing what's visible in the current local namespace, as shown in Example 1-11.

Example 1-11. Using the vars Function
File: builtin-vars-example-1.py

book = "library2"
pages = 250
scripts = 350

print "the %(book)s book contains more than %(scripts)s scripts" % vars()

the library book contains more than 350 scripts

1.2.4 Checking an Object's Type

Python is a dynamically typed language, which means that a given variable can be bound to values of different types on different occasions. In the following example, the same function is called with an integer, a floating point value, and a string:

def function(value):
    print value
function(1)
function(1.0)
function("one")

The type function (shown in Example 1-12) allows you to check what type a variable has. This function returns a type descriptor, which is a unique object for each type provided by the Python interpreter.

Example 1-12. Using the type Function
File: builtin-type-example-1.py

def dump(value):
    print type(value), value

dump(1)
dump(1.0)
dump("one")

<type 'int'> 1
<type 'float'> 1.0
<type 'string'> one

Each type has a single corresponding type object, which means that you can use the is operator (object identity) to do type testing (as shown in Example 1-13).

Example 1-13. Using the type Function with Filenames and File Objects
File: builtin-type-example-2.py

def load(file):
    if isinstance(file, type("")):
        file = open(file, "rb")
    return file.read()

print len(load("samples/sample.jpg")), "bytes"
print len(load(open("samples/sample.jpg", "rb"))), "bytes"

4672 bytes
4672 bytes

The callable function, shown in Example 1-14, checks if an object can be called (either directly or via apply). It returns true for functions, methods, lambda expressions, classes, and class instances that define the _ _call_ _ method.

Example 1-14. Using the callable Function
File: builtin-callable-example-1.py

def dump(function):
    if callable(function):
        print function, "is callable"
    else:
        print function, "is *not* callable"

class A:
    def method(self, value):
        return value

class B(A):
    def _ _call_ _(self, value):
        return value

a = A()
b = B()

dump(0) # simple objects
dump("string")
dump(callable)
dump(dump) # function

dump(A) # classes
dump(B)
dump(B.method)

dump(a) # instances
dump(b)
dump(b.method)

0 is *not* callable
string is *not* callable
<built-in function callable> is callable
<function dump at 8ca320> is callable
A is callable
B is callable
<unbound method A.method> is callable
<A instance at 8caa10> is *not* callable
<B instance at 8cab00> is callable
<method A.method of B instance at 8cab00> is callable

Note that the class objects (A and B) are both callable; if you call them, they create new objects. However, instances of class A are not callable, since that class doesn't have a _ _call_ _ method.

You'll find functions to check if an object is of any of the built-in number, sequence, or dictionary types in the operator module. However, since it's easy to create a class that implements (for example, the basic sequence methods), it's usually a bad idea to use explicit type testing on such objects.

Things get even more complicated when it comes to classes and instances. Python doesn't treat classes as types per se; instead, all classes belong to a special class type, and all class instances belong to a special instance type.

This means that you cannot use type to test if an instance belongs to a given class; all instances have the same type! To solve this, you can use the isinstance function, which checks if an object is an instance of a given class (or of a subclass to it). Example 1-15 illustrates the isinstance function.

Example 1-15. Using the isinstance Function
File: builtin-isinstance-example-1.py

class A:
    pass

class B:
    pass

class C(A):
    pass

class D(A, B):
    pass

def dump(object):
    print object, "=>",
    if isinstance(object, A):
        print "A",
    if isinstance(object, B):
        print "B",
    if isinstance(object, C):
        print "C",
    if isinstance(object, D):
        print "D",
    print

a = A()
b = B()
c = C()
d = D()

dump(a)
dump(b)
dump(c)
dump(d)
dump(0)
dump("string")

<A instance at 8ca6d0> => A
<B instance at 8ca750> => B
<C instance at 8ca780> => A C
<D instance at 8ca7b0> => A B D
0 =>
string =>

The issubclass function is similar, but it instead checks whether a class object is the same as a given class, or is a subclass of it. The issubclass function is shown in Example 1-16.

Note that while isinstance accepts any kind of object, the issubclass function raises a TypeError exception if you use it on something that is not a class object.

Example 1-16. Using the issubclass Function
File: builtin-issubclass-example-1.py

class A:
    pass

class B:
    pass

class C(A):
    pass

class D(A, B):
    pass

def dump(object):
    print object, "=>",
    if issubclass(object, A):
        print "A",
    if issubclass(object, B):
        print "B",
    if issubclass(object, C):
        print "C",
    if issubclass(object, D):
        print "D",
    print

dump(A)
dump(B)
dump(C)
dump(D)
dump(0)
dump("string")

A => A
B => B
C => A C
D => A B D
0 =>
Traceback (innermost last):
  File "builtin-issubclass-example-1.py", line 29, in ?
  File "builtin-issubclass-example-1.py", line 15, in dump
TypeError: arguments must be classes

1.2.5 Evaluating Python Expressions

Python provides several ways to interact with the interpreter from within a program. For example, the eval function evaluates a string as if it were a Python expression. You can pass it a literal, simple expression, or use built-in functions, as shown in Example 1-17.

Example 1-17. Using the eval Function
File: builtin-eval-example-1.py

def dump(expression):
    result = eval(expression)
    print expression, "=>", result, type(result)

dump("1")
dump("1.0")
dump("'string'")
dump("1.0 + 2.0")
dump("'*' * 10")
dump("len('world')")

1 => 1 <type 'int'>
1.0 => 1.0 <type 'float'>
'string' => string <type 'string'>
1.0 + 2.0 => 3.0 <type 'float'>
'*' * 10 => ********** <type 'string'>
len('world') => 5 <type 'int'>

If you cannot trust the source from which you got the string, you may get into trouble using eval. For example, someone might use the built-in _ _import_ _ function to load the os module, and then remove files on your disk (as shown in Example 1-18).

Example 1-18. Using the eval Function to Execute Arbitrary Commands
File: builtin-eval-example-2.py

print eval("_ _import_ _('os').getcwd()")
print eval("_ _import_ _('os').remove('file')")

/home/fredrik/librarybook
Traceback (innermost last):
 File "builtin-eval-example-2", line 2, in ?
 File "<string>", line 0, in ?
os.error: (2, 'No such file or directory')

Note that you get an os.error exception, which means that Python actually tried to remove the file!

Luckily, there's a way around this problem. You can pass a second argument to eval, which should contain a dictionary defining the namespace in which the expression is evaluated. Let's pass in an empty namespace:

>>> print eval("_ _import_ _('os').remove('file')", {})
Traceback (innermost last):
  File "<stdin>", line 1, in ?
  File "<string>", line 0, in ?
os.error: (2, 'No such file or directory')

Hmm. We still end up with an os.error exception.

The reason for this is that Python looks in the dictionary before it evaluates the code, and if it doesn't find a variable named _ _builtins_ _ in there (note the plural form), it adds one:

>>> namespace = {}
>>> print eval("_ _import_ _('os').remove('file')", namespace)
Traceback (innermost last):
  File "<stdin>", line 1, in ?
  File "<string>", line 0, in ?
os.error: (2, 'No such file or directory')
>>> namespace.keys()
['_ _builtins_ _']

If you print the contents of the namespace variable, you'll find that they contain the full set of built-in functions.

The solution to this little dilemma isn't far away: since Python doesn't add this item if it is already there, simply add a dummy item called _ _builtins_ _ to the namespace before calling eval, as shown in Example 1-19.

Example 1-19. Using the eval Function to Evaluate Arbitrary Strings Safely
File: builtin-eval-example-3.py

print eval("_ _import_ _('os').getcwd()", {})
print eval("_ _import_ _('os').remove('file')", {"_ _builtins_ _": {}})

/home/fredrik/librarybook
Traceback (innermost last):
  File "builtin-eval-example-3.py", line 2, in ?
  File "<string>", line 0, in ?
NameError: _ _import_ _

Note that this doesn't protect you from CPU or memory-resource attacks (for example, something like eval("'*'*1000000*2*2*2*2*2*2*2*2*2") will most likely cause your program to run out of memory after a while).

1.2.6 Compiling and Executing Code

The eval function only works for simple expressions. To handle larger blocks of code, use the compile and exec functions (as demonstrated in Example 1-20).

Example 1-20. Using the compile Function to Check Syntax
File: builtin-compile-example-1.py

NAME = "script.py"

BODY = """
prnt 'owl-stretching time'
"""

try:
    compile(BODY, NAME, "exec")
except SyntaxError, v:
    print "syntax error:", v, "in", NAME

# syntax error: invalid syntax in script.py

When successful, the compile function returns a code object, which you can execute with the exec statement, as in Example 1-21.

Example 1-21. Compiling and Executing Compiled Code
File: builtin-compile-example-2.py

BODY = """
print 'the ant, an introduction'
"""

code = compile(BODY, "<script>", "exec")

print code

exec code

<code object ? at 8c6be0, file "<script>", line 0>
the ant, an introduction

To generate code on the fly, use the class shown in the Example 1-22. Use the write method to add statements, and the methods indent and dedent to add structure. The class will take care of the rest.

Example 1-22. A Simple Code Generator Tool
File: builtin-compile-example-3.py

import sys, string

class CodeGeneratorBackend:
    "Simple code generator for Python"

    def begin(self, tab="\t"):
        self.code = []
        self.tab = tab
        self.level = 0

    def end(self):
        self.code.append("") # make sure there's a newline at the end
        return compile(string.join(self.code, "\n"), "<code>", "exec")

    def write(self, string):
        self.code.append(self.tab * self.level + string)

    def indent(self):
        self.level = self.level + 1
        # in 2.0 and later, this can be written as: self.level += 1

    def dedent(self):
        if self.level == 0:
            raise SyntaxError, "internal error in code generator"
        self.level = self.level - 1
        # or: self.level -= 1

#
# try it out!

c = CodeGeneratorBackend()
c.begin()
c.write("for i in range(5):")
c.indent()
c.write("print 'code generation made easy!'")
c.dedent()
exec c.end()

code generation made easy!
code generation made easy!
code generation made easy!
code generation made easy!
code generation made easy!

Python also provides a function called execfile, a shortcut for loading code from a file, compiling it, and executing it. Example 1-23 shows how to use and emulate this function.

Example 1-23. Using the execfile Function
File: builtin-execfile-example-1.py

execfile("hello.py")

def EXECFILE(filename, locals=None, globals=None):
    exec compile(open(filename).read(), filename, "exec") in locals, globals

EXECFILE("hello.py")

hello again, and welcome to the show
hello again, and welcome to the show

The contents of the hello.py file used Example 1-23 are shown in Example 1-24.

Example 1-24. The hello.py Script
File: hello.py

print "hello again, and welcome to the show"

1.2.7 Overloading Functions from the _ _builtin_ _ Module

Since Python does not look among the built-in functions until after it has checked the local and module namespace, there may be situations in which you need to explicitly refer to the _ _builtin_ _ module. For instance, the script in Example 1-25 overloads the open function with a version that opens an ordinary file and checks that it starts with a "magic" string. To be able to use the original open function, the script explicitly refers to the function using the module name.

Example 1-25. Explicitly Accessing Functions in the _ _builtin_ _ Module
File: builtin-open-example-1.py

def open(filename, mode="rb"):
    import _ _builtin_ _
    file = _ _builtin_ _.open(filename, mode)
    if file.read(5) not in("GIF87", "GIF89"):
        raise IOError, "not a GIF file"
    file.seek(0)
    return file

fp = open("samples/sample.gif")
print len(fp.read()), "bytes"

fp = open("samples/sample.jpg")
print len(fp.read()), "bytes"

3565 bytes
Traceback (innermost last):
  File "builtin-open-example-1.py", line 12, in ?
  File "builtin-open-example-1.py", line 5, in open
IOError: not a GIF file


1.3 The exceptions Module

The exceptions module provides the standard exception hierarchy. It's automatically imported when Python starts, and the exceptions are added to the _ _builtin_ _ module. In other words, you usually don't need to import this module.

This is a Python module in 1.5.2, and a built-in module in 2.0 and later.

The following standard exceptions are defined by this module:

  • Exception is used as a base class for all exceptions. It's strongly recommended (but not yet required) that user exceptions are derived from this class too.

  • SystemExit(Exception) is raised by the sys.exit function. If it propagates to the top level without being caught by a try-except clause, the interpreter is terminated without a traceback message.

  • StandardError(Exception) is used as a base class for all standard exceptions (except SystemExit, that is).

  • KeyboardInterrupt(StandardError) is raised when the user presses Control-C (or any other interrupt key). Note that this may cause strange errors if you use "catch all" try-except statements.

  • ImportError(StandardError) is raised when Python fails to import a module.

  • EnvironmentError is used as a base class for exceptions that can be caused by the interpreter's environment (that is, they're usually not caused by bugs in the program).

  • IOError(EnvironmentError) is used to flag I/O-related errors.

  • OSError(EnvironmentError) is used to flag errors by the os module.

  • WindowsError(OSError) is used to flag Windows-specific errors from the os module.

  • NameError(StandardError) is raised when Python fails to find a global or local name.

  • UnboundLocalError(NameError) is raised if your program attempts to access a local variable before it has been assigned a value. This exception is only used in 2.0 and later; earlier versions raise a plain NameError exception instead.

  • AttributeError(StandardError) is raised when Python fails to find (or assign to) an instance attribute, a method, a module function, or any other qualified name.

  • SyntaxError(StandardError) is raised when the compiler stumbles upon a syntax error.

  • (2.0 and later) IndentationError(SyntaxError) is raised for syntax errors caused by bad indentation. This exception is only used in 2.0 and later; earlier versions raise a plain SyntaxError exception instead.

  • (2.0 and later) TabError(IndentationError) is raised by the interpreter when the -tt option is used to check for inconsistent indentation. This exception is only used in 2.0 and later; earlier versions raise a plain SyntaxError exception instead.

  • TypeError(StandardError) is raised when an operation cannot be applied to an object of the given type.

  • AssertionError(StandardError) is raised when an assert statement fails (if the expression is false, that is).

  • LookupError(StandardError) is used as a base class for exceptions raised when a sequence or dictionary type doesn't contain a given index or key.

  • IndexError(LookupError) is raised by sequence objects when the given index doesn't exist.

  • KeyError(LookupError) is raised by dictionary objects when the given key doesn't exist.

  • ArithmeticError(StandardError) is used as a base class for exceptions that are math-related.

  • OverflowError(ArithmeticError) is raised when an operations overflows (for example, when an integer is too large to fit in the given type).

  • ZeroDivisionError(ArithmeticError) is raised when you try to divide a number by zero.

  • FloatingPointError(ArithmeticError) is raised when a floating point operation fails.

  • ValueError(StandardError) is raised if an argument has the right type, but an invalid value.

  • (2.0 and later) UnicodeError(ValueError) is raised for type problems related to the Unicode string type. This is only used in 2.0 and later.

  • RuntimeError(StandardError) is used for various run-time problems, including attempts to get outside the box when running in restricted mode, unexpected hardware problems, etc.

  • NotImplementedError(RuntimeError) can be used to flag functions that hasn't been implemented yet, or methods that should be overridden.

  • SystemError(StandardError) is raised if the interpreter messes up, and knows about it. The exception value contains a more detailed description (usually something cryptic, like "eval_code2: NULL globals" or so). I cannot recall ever seeing this exception in over five years of full-time Python programming, but maybe that's just me.

  • MemoryError(StandardError) is raised when the interpreter runs out of memory. Note that this only happens when the underlying memory allocation routines complain; you can often send your poor computer into a mindless swapping frenzy before that happens.

You can create your own exception classes. Just inherit from the built-in Exception class (or a proper standard exception), and override the constructor and/or _ _str_ _ method as necessary. Example 1-26 shows the exceptions module.

Example 1-26. Using the exceptions Module
File: exceptions-example-1.py

# python imports this module by itself, so the following
# line isn't really needed
# import exceptions

class HTTPError(Exception):
    # indicates an HTTP protocol error
    def _ _init_ _(self, url, errcode, errmsg):
        self.url = url
        self.errcode = errcode
        self.errmsg = errmsg
    def _ _str_ _(self):
        return (
            "<HTTPError for %s: %s %s>" %
            (self.url, self.errcode, self.errmsg)
            )

try:
    raise HTTPError("http://www.python.org/foo", 200, "Not Found")
except HTTPError, error:
    print "url", "=>", error.url
    print "errcode", "=>", error.errcode
    print "errmsg", "=>", error.errmsg
    raise # reraise exception

url => http://www.python.org/foo
errcode => 200
errmsg => Not Found
Traceback (innermost last):
  File "exceptions-example-1", line 16, in ?
HTTPError: <HTTPError for http://www.python.org/foo: 200 Not Found>


1.4 The os Module

The os module provides a unified interface to many operating system functions.

Most of the functions in this module are implemented by platform-specific modules, such as posix or nt. The os module automatically loads the right implementation module when it is first imported.

1.4.1 Working with Files

The built-in open function lets you create, open, and modify files, as shown in Example 1-27. This module adds those extra functions you need to rename and remove files.

Example 1-27. Using the os Module to Rename and Remove Files
File: os-example-3.py

import os
import string

def replace(file, search_for, replace_with):
    # replace strings in a text file

    back = os.path.splitext(file)[0] + ".bak"
    temp = os.path.splitext(file)[0] + ".tmp"

    try:
        # remove old temp file, if any
        os.remove(temp)
    except os.error:
        pass

    fi = open(file)
    fo = open(temp, "w")

    for s in fi.readlines():
        fo.write(string.replace(s, search_for, replace_with))

    fi.close()
    fo.close()

    try:
        # remove old backup file, if any
        os.remove(back)
    except os.error:
        pass

    # rename original to backup...
    os.rename(file, back)

    # ...and temporary to original
    os.rename(temp, file)

#
# try it out!

file = "samples/sample.txt"

replace(file, "hello", "tjena")
replace(file, "tjena", "hello")

1.4.2 Working with Directories

The os module also contains many functions that work on entire directories.

The listdir function returns a list of all filenames in a given directory, as shown in Example 1-28. The current and parent directory markers used on Unix and Windows (. and ..) are not included in this list.

Example 1-28. Using the os Module to List the Files in a Directory
File: os-example-5.py

import os

for file in os.listdir("samples"):
    print file

sample.au
sample.jpg
sample.wav
...

The getcwd and chdir functions are used to get and set the current directory, as shown in Example 1-29.

Example 1-29. Using the os Module to Change the Working Directory
File: os-example-4.py

import os

# where are we?
cwd = os.getcwd()
print "1", cwd

# go down
os.chdir("samples")
print "2", os.getcwd()

# go back up
os.chdir(os.pardir)
print "3", os.getcwd()

1 /ematter/librarybook
2 /ematter/librarybook/samples
3 /ematter/librarybook

The makedirs and removedirs functions are used to create and remove directory hierarchies, as shown in Example 1-30.

Example 1-30. Using the os Module to Create and Remove Multiple Directory Levels
File: os-example-6.py

import os

os.makedirs("test/multiple/levels")

fp = open("test/multiple/levels/file", "w")
fp.write("inspector praline")
fp.close()

# remove the file
os.remove("test/multiple/levels/file")

# and all empty directories above it
os.removedirs("test/multiple/levels")

Note that removedirs removes all empty directories along the given path, starting with the last directory in the given pathname. In contrast, the mkdir and rmdir functions can only handle a single directory level, as shown in Example 1-31.

Example 1-31. Using the os Module to Create and Remove Directories
File: os-example-7.py

import os

os.mkdir("test")
os.rmdir("test")

os.rmdir("samples") # this will fail

Traceback (innermost last):
  File "os-example-7", line 6, in ?
OSError: [Errno 41] Directory not empty: 'samples'

To remove non-empty directories, you can use the rmtree function in the shutil module.

1.4.3 Working with File Attributes

The stat function fetches information about an existing file, as demonstrated in Example 1-32. It returns a 9-tuple which contains the size, inode change timestamp, modification timestamp, and access privileges.

Example 1-32. Using the os Module to Get Information About a File
File: os-example-1.py

import os
import time

file = "samples/sample.jpg"

def dump(st):
    mode, ino, dev, nlink, uid, gid, size, atime, mtime, ctime = st
    print "- size:", size, "bytes"
    print "- owner:", uid, gid
    print "- created:", time.ctime(ctime)
    print "- last accessed:", time.ctime(atime)
    print "- last modified:", time.ctime(mtime)
    print "- mode:", oct(mode)
    print "- inode/dev:", ino, dev

#
# get stats for a filename

st = os.stat(file)

print "stat", file
dump(st)
print

#
# get stats for an open file

fp = open(file)

st = os.fstat(fp.fileno())

print "fstat", file
dump(st)

stat samples/sample.jpg
- size: 4762 bytes
- owner: 0 0
- created: Tue Sep 07 22:45:58 1999
- last accessed: Sun Sep 19 00:00:00 1999
- last modified: Sun May 19 01:42:16 1996
- mode: 0100666
- inode/dev: 0 2

fstat samples/sample.jpg
- size: 4762 bytes
- owner: 0 0
- created: Tue Sep 07 22:45:58 1999
- last accessed: Sun Sep 19 00:00:00 1999
- last modified: Sun May 19 01:42:16 1996
- mode: 0100666
- inode/dev: 0 0

Some fields don't make sense on non-Unix platforms; for example, the (inode, dev) tuple provides a unique identity for each file on Unix, but may contain arbitrary data on other platforms.

The stat module contains a number of useful constants and helper functions for dealing with the members of the stat tuple. Some of these are shown in the examples that follow.

You can modify the mode and time fields using the chmod and utime functions, as shown in Example 1-33.

Example 1-33. Using the os Module to Change a File's Privileges and Timestamps
File: os-example-2.py

import os
import stat, time

infile = "samples/sample.jpg"
outfile = "out.jpg"

# copy contents
fi = open(infile, "rb")
fo = open(outfile, "wb")

while 1:
    s = fi.read(10000)
    if not s:
        break
    fo.write(s)

fi.close()
fo.close()

# copy mode and timestamp
st = os.stat(infile)
os.chmod(outfile, stat.S_IMODE(st[stat.ST_MODE]))
os.utime(outfile, (st[stat.ST_ATIME], st[stat.ST_MTIME]))

print "original", "=>"
print "mode", oct(stat.S_IMODE(st[stat.ST_MODE]))
print "atime", time.ctime(st[stat.ST_ATIME])
print "mtime", time.ctime(st[stat.ST_MTIME])

print "copy", "=>"
st = os.stat(outfile)
print "mode", oct(stat.S_IMODE(st[stat.ST_MODE]))
print "atime", time.ctime(st[stat.ST_ATIME])
print "mtime", time.ctime(st[stat.ST_MTIME])

original =>
mode 0666
atime Thu Oct 14 15:15:50 1999
mtime Mon Nov 13 15:42:36 1995
copy =>
mode 0666
atime Thu Oct 14 15:15:50 1999
mtime Mon Nov 13 15:42:36 1995

1.4.4 Working with Processes

The system function runs a new command under the current process, and waits for it to finish, as shown in Example 1-34.

Example 1-34. Using the os Module to Run an Operating System Command
File: os-example-8.py

import os

if os.name == "nt":
    command = "dir"
else:
    command = "ls -l"

os.system(command)

-rwxrw-r--   1 effbot  effbot        76 Oct  9 14:17 README
-rwxrw-r--   1 effbot  effbot      1727 Oct  7 19:00 SimpleAsyncHTTP.py
-rwxrw-r--   1 effbot  effbot       314 Oct  7 20:29 aifc-example-1.py
-rwxrw-r--   1 effbot  effbot       259 Oct  7 20:38 anydbm-example-1.py
...

The command is run via the operating system's standard shell, and returns the shell's exit status. Under Windows 95/98, the shell is usually command.com, whose exit status is always 0.

Since os.system passes the command on to the shell as is, it can be dangerous to use if you don't check the arguments carefully (consider running os.system("viewer %s" % file) with the file variable set to "sample.jpg; rm -rf $HOME"). When unsure, it's usually better to use exec or spawn instead (explained later).

The exec function starts a new process, replacing the current one ( "go to process," in other words). In Example 1-35, note that the "goodbye" message is never printed.

Example 1-35. Using the os Module to Start a New Process
File: os-exec-example-1.py

import os
import sys

program = "python"
arguments = ["hello.py"]

print os.execvp(program, (program,) +  tuple(arguments))
print "goodbye"

hello again, and welcome to the show

Python provides a whole bunch of exec functions, with slightly varying behaviors. Example 1-35 uses execvp, which searches for the program along the standard path, passes the contents of the second argument tuple as individual arguments to that program, and runs it with the current set of environment variables. See the Python Library Reference for more information on the other seven ways to call this function.

Under Unix, you can call other programs from the current one by combining exec with two other functions, fork and wait, as shown in Example 1-36. The fork function makes a copy of the current process, and the wait function waits for a child process to finish.

Example 1-36. Using the os Module to Run Another Program (Unix)
File: os-exec-example-2.py

import os
import sys

def run(program, *args):
    pid = os.fork()
    if not pid:
        os.execvp(program, (program,) +  args)
    return os.wait()[0]

run("python", "hello.py")

print "goodbye"

hello again, and welcome to the show
goodbye

The fork returns zero in the new process (the return from fork is the first thing that happens in that process!), and a non-zero process identifier in the original process. Or in other words, "not pid" is true only if we're in the new process.

The fork and wait functions are not available on Windows, but you can use the spawn function instead, as shown in Example 1-37. Unfortunately, there's no standard version of spawn that searches for an executable along the path, so you have to do that yourself.

Example 1-37. Using the os Module to Run Another Program (Windows)
File: os-spawn-example-1.py

import os
import string

def run(program, *args):
    # find executable
    for path in string.split(os.environ["PATH"], os.pathsep):
        file = os.path.join(path, program) + ".exe"
        try:
            return os.spawnv(os.P_WAIT, file, (file,) + args)
        except os.error:
            pass
    raise os.error, "cannot find executable"

run("python", "hello.py")

print "goodbye"

hello again, and welcome to the show
goodbye

You can also use spawn to run other programs in the background. Example 1-38 adds an optional mode argument to the run function; when set to os.P_NOWAIT, the script doesn't wait for the other program to finish. The default flag value os.P_WAIT tells spawn to wait until the new process is finished.

Other flags include os.P_OVERLAY, which makes spawn behave like exec, and os.P_DETACH, which runs the new process in the background, detached from both console and keyboard.

Example 1-38. Using the os Module to Run Another Program in the Background (Windows)
File: os-spawn-example-2.py

import os
import string

def run(program, *args, **kw):
    # find executable
    mode = kw.get("mode", os.P_WAIT)
    for path in string.split(os.environ["PATH"], os.pathsep):
        file = os.path.join(path, program) + ".exe"
        try:
            return os.spawnv(mode, file, (file,) + args)
        except os.error:
            pass
    raise os.error, "cannot find executable"

run("python", "hello.py", mode=os.P_NOWAIT)
print "goodbye"

goodbye
hello again, and welcome to the show

Example 1-39 provides a spawn method that works on either platform.

Example 1-39. Using Either spawn or fork/exec to Run Another Program
File: os-spawn-example-3.py

import os
import string

if os.name in ("nt", "dos"):
    exefile = ".exe"
else:
    exefile = ""

def spawn(program, *args):
    try:
        # possible 2.0 shortcut!
        return os.spawnvp(program, (program,) + args)
    except AttributeError:
        pass
    try:
        spawnv = os.spawnv
    except AttributeError:

        # assume it's unix
        pid = os.fork()
        if not pid:
            os.execvp(program, (program,) + args)
        return os.wait()[0]
    else:
        # got spawnv but no spawnp: go look for an executable
        for path in string.split(os.environ["PATH"], os.pathsep):
            file = os.path.join(path, program) + exefile
            try:
                return spawnv(os.P_WAIT, file, (file,) + args)
            except os.error:
                pass
        raise IOError, "cannot find executable"

#
# try it out!

spawn("python", "hello.py")

print "goodbye"

hello again, and welcome to the show
goodbye

Example 1-39 first attempts to call a function named spawnvp. If that doesn't exist (it doesn't, in 2.0 and earlier), the function looks for a function named spawnv and searches the path all by itself. As a last resort, it falls back on exec and fork.

1.4.5 Working with Daemon Processes

On Unix, you can also use fork to turn the current process into a background process (a "daemon"). Basically, you need to fork off a copy of the current process, and terminate the original process, as shown in Example 1-40.

Example 1-40. Using the os Module to Run as Daemon (Unix)
File: os-example-14.py

import os
import time

pid = os.fork()
if pid:
    os._exit(0) # kill original

print "daemon started"
time.sleep(10)
print "daemon terminated"

It takes a bit more work to create a real daemon, however. First, call setpgrp to make the new process a "process group leader." Otherwise, signals sent to a (by that time) unrelated process group might cause problems in your daemon:

os.setpgrp()

It's also a good idea to remove the user mode mask, to make sure files created by the daemon actually get the mode flags specified by the program:

os.umask(0)

Then, you should redirect the stdout/stderr files, instead of just closing them (if you don't do this, you may get unexpected exceptions the day some of your code tries to write something to the console via stdout or stderr).

class NullDevice:
    def write(self, s):
        pass
sys.stdin.close()
sys.stdout = NullDevice()
sys.stderr = NullDevice()

In other words, while Python's print and C's printf/fprintf won't crash your program if the devices have been disconnected, sys.stdout.write() happily throws an IOError exception when the application runs as a daemon. But your program works just fine when running in the foreground...

By the way, the _exit function used in the previous examples terminates the current process. In contrast to sys.exit, this works also if the caller happens to catch the SystemExit exception, as shown in Example 1-41.

Example 1-41. Using the os Module to Exit the Current Process
File: os-example-9.py

import os
import sys

try:
    sys.exit(1)
except SystemExit, value:
    print "caught exit(%s)" % value

try:
    os._exit(2)
except SystemExit, value:
    print "caught exit(%s)" % value

print "bye!"

caught exit(1)


1.5 The os.path Module

The os.path module contains functions that deal with long filenames (pathnames) in various ways. To use this module, import the os module, and access this module as os.path.

1.5.1 Working with Filenames

The os.path module contains a number of functions that deal with long filenames in a platform independent way. In other words, you won't have to deal with forward and backward slashes, colons, and whatnot. Let's look at Example 1-42.

Example 1-42. Using the os.path Module to Handle Filename
File: os-path-example-1.py

import os

filename = "my/little/pony"

print "using", os.name, "..."
print "split", "=>", os.path.split(filename)
print "splitext", "=>", os.path.splitext(filename)
print "dirname", "=>", os.path.dirname(filename)
print "basename", "=>", os.path.basename(filename)
print "join", "=>", os.path.join(os.path.dirname(filename),
                                 os.path.basename(filename))

using nt ...
split => ('my/little', 'pony')
splitext => ('my/little/pony', '')
dirname => my/little
basename => pony
join => my/little\pony

Note that split only splits off a single item.

The os.path module also contains a number of functions that allow you to quickly figure out what a filename represents, as shown in Example 1-43.

Example 1-43. Using the os.path Module to Check What a Filename Represents
File: os-path-example-2.py

import os

FILES = (
    os.curdir,
    "/",
    "file",
    "/file",
    "samples",
    "samples/sample.jpg",
    "directory/file",
    "../directory/file",
    "/directory/file"
    )

for file in FILES:
    print file, "=>",
    if os.path.exists(file):
        print "EXISTS",
    if os.path.isabs(file):
        print "ISABS",
    if os.path.isdir(file):
        print "ISDIR",
    if os.path.isfile(file):
        print "ISFILE",
    if os.path.islink(file):
        print "ISLINK",
    if os.path.ismount(file):
        print "ISMOUNT",
    print

. => EXISTS ISDIR
/ => EXISTS ISABS ISDIR ISMOUNT
file =>
/file => ISABS
samples => EXISTS ISDIR
samples/sample.jpg => EXISTS ISFILE
directory/file =>
../directory/file =>
/directory/file => ISABS

The expanduser function treats a username shortcut in the same way as most modern Unix shells (it doesn't work well on Windows), as shown in Example 1-44.

Example 1-44. Using the os.path Module to Insert the Username into a Filename
File: os-path-expanduser-example-1.py

import os

print os.path.expanduser("~/.pythonrc")

# /home/effbot/.pythonrc

The expandvars function inserts environment variables into a filename, as shown in Example 1-45.

Example 1-45. Using the os.path Module to Insert Variables into a Filename
File: os-path-expandvars-example-1.py

import os

os.environ["USER"] = "user"

print os.path.expandvars("/home/$USER/config")
print os.path.expandvars("$USER/folders")

/home/user/config
user/folders

1.5.2 Traversing a Filesystem

The walk function helps you find all files in a directory tree (as Example 1-46 demonstrates). It takes a directory name, a callback function, and a data object that is passed on to the callback.

Example 1-46. Using the os.path Module to Traverse a Filesystem
File: os-path-walk-example-1.py

import os

def callback(arg, directory, files):
    for file in files:
        print os.path.join(directory, file), repr(arg)

os.path.walk(".", callback, "secret message")

./aifc-example-1.py 'secret message'
./anydbm-example-1.py 'secret message'
./array-example-1.py 'secret message'
...
./samples 'secret message'
./samples/sample.jpg 'secret message'
./samples/sample.txt 'secret message'
./samples/sample.zip 'secret message'
./samples/articles 'secret message'
./samples/articles/article-1.txt 'secret message'
./samples/articles/article-2.txt 'secret message'
...

The walk function has a somewhat obscure user interface (maybe it's just me, but I can never remember the order of the arguments). The index function in Example 1-47 returns a list of filenames instead, which lets you use a straightforward for-in loop to process the files.

Example 1-47. Using os.listdir to Traverse a Filesystem
File: os-path-walk-example-2.py

import os

def index(directory):
    # like os.listdir, but traverses directory trees
    stack = [directory]
    files = []
    while stack:
        directory = stack.pop()
        for file in os.listdir(directory):
            fullname = os.path.join(directory, file)
            files.append(fullname)
            if os.path.isdir(fullname) and not os.path.islink(fullname):
                stack.append(fullname)
    return files

for file in index("."):
    print file

.\aifc-example-1.py
.\anydbm-example-1.py
.\array-example-1.py
...

If you don't want to list all files (for performance or memory reasons), Example 1-48 uses a different approach. Here, the DirectoryWalker class behaves like a sequence object, returning one file at a time:

Example 1-48. Using DirectoryWalker to Traverse a Filesystem
File: os-path-walk-example-3.py

import os

class DirectoryWalker:
    # a forward iterator that traverses a directory tree

    def _ _init_ _(self, directory):
        self.stack = [directory]
        self.files = []
        self.index = 0

    def _ _getitem_ _(self, index):
        while 1:
            try:
                file = self.files[self.index]
                self.index = self.index + 1
            except IndexError:
                # pop next directory from stack
                self.directory = self.stack.pop()
                self.files = os.listdir(self.directory)
                self.index = 0
            else:
                # got a filename
                fullname = os.path.join(self.directory, file)
                if os.path.isdir(fullname) and not os.path.islink(fullname):
                    self.stack.append(fullname)
                return fullname

for file in DirectoryWalker("."):
    print file

.\aifc-example-1.py
.\anydbm-example-1.py
.\array-example-1.py
...

Note the DirectoryWalker class doesn't check the index passed to the _ _getitem_ _ method. This means that it won't work properly if you access the sequence members out of order.

Finally, if you're interested in the file sizes or timestamps, Example 1-49 demonstrates a version of the class that returns both the filename and the tuple returned from os.stat. This version saves one or two stat calls for each file (both os.path.isdir and os.path.islink uses stat), and runs quite a bit faster on some platforms.

Example 1-49. Using DirectoryStatWalker to Traverse a Filesystem
File: os-path-walk-example-4.py

import os, stat

class DirectoryStatWalker:
    # a forward iterator that traverses a directory tree, and
    # returns the filename and additional file information

    def _ _init_ _(self, directory):
        self.stack = [directory]
        self.files = []
        self.index = 0

    def _ _getitem_ _(self, index):
        while 1:
            try:
                file = self.files[self.index]
                self.index = self.index + 1
            except IndexError:
                # pop next directory from stack
                self.directory = self.stack.pop()
                self.files = os.listdir(self.directory)
                self.index = 0
            else:
                # got a filename
                fullname = os.path.join(self.directory, file)
                st = os.stat(fullname)
                mode = st[stat.ST_MODE]
                if stat.S_ISDIR(mode) and not stat.S_ISLNK(mode):
                    self.stack.append(fullname)
                return fullname, st

for file, st in DirectoryStatWalker("."):
    print file, st[stat.ST_SIZE]

.\aifc-example-1.py 336
.\anydbm-example-1.py 244
.\array-example-1.py 526


1.6 The stat Module

The stat module, shown in Example 1-50, contains a number of constants and test functions that can be used with the os.stat function.

Example 1-50. Using the stat Module
File: stat-example-1.py

import stat
import os, time

st = os.stat("samples/sample.txt")

print "mode", "=>", oct(stat.S_IMODE(st[stat.ST_MODE]))

print "type", "=>",
if stat.S_ISDIR(st[stat.ST_MODE]):
    print "DIRECTORY",
if stat.S_ISREG(st[stat.ST_MODE]):
    print "REGULAR",
if stat.S_ISLNK(st[stat.ST_MODE]):
    print "LINK",
print

print "size", "=>", st[stat.ST_SIZE]

print "last accessed", "=>", time.ctime(st[stat.ST_ATIME])
print "last modified", "=>", time.ctime(st[stat.ST_MTIME])
print "inode changed", "=>", time.ctime(st[stat.ST_CTIME])

mode => 0664
type => REGULAR
size => 305
last accessed => Sun Oct 10 22:12:30 1999
last modified => Sun Oct 10 18:39:37 1999
inode changed => Sun Oct 10 15:26:38 1999


1.7 The string Module

The string module contains a number of functions to process standard Python strings, as shown in Example 1-51.

Example 1-51. Using the string Module
File: string-example-1.py

import string

text = "Monty Python's Flying Circus"

print "upper", "=>", string.upper(text)
print "lower", "=>", string.lower(text)
print "split", "=>", string.split(text)
print "join", "=>", string.join(string.split(text), "+")
print "replace", "=>", string.replace(text, "Python", "Java")
print "find", "=>", string.find(text, "Python"), string.find(text, "Java")
print "count", "=>", string.count(text, "n")

upper => MONTY PYTHON'S FLYING CIRCUS
lower => monty python's flying circus
split => ['Monty', "Python's", 'Flying', 'Circus']
join => Monty+Python's+Flying+Circus
replace => Monty Java's Flying Circus
find => 6 -1
count => 3

In Python 1.5.2 and earlier, the string module uses functions from the strop implementation module where possible.

In Python 1.6 and later, most string operations are made available as string methods as well, as shown in Example 1-52. Many of the functions in the string module are simply wrapper functions that call the corresponding string method.

Example 1-52. Using string Methods Instead of string Module Functions
File: string-example-2.py

text = "Monty Python's Flying Circus"

print "upper", "=>", text.upper()
print "lower", "=>", text.lower()
print "split", "=>", text.split()
print "join", "=>", "+".join(text.split())
print "replace", "=>", text.replace("Python", "Perl")
print "find", "=>", text.find("Python"), text.find("Perl")
print "count", "=>", text.count("n")

upper => MONTY PYTHON'S FLYING CIRCUS
lower => monty python's flying circus
split => ['Monty', "Python's", 'Flying', 'Circus']
join => Monty+Python's+Flying+Circus
replace => Monty Perl's Flying Circus
find => 6 -1
count => 3

In addition to the string-manipulation capabilities offered by string, the module also contains a number of functions that convert strings to other types (as Example 1-53 demonstrates).

Example 1-53. Using the string Module to Convert Strings to Numbers
File: string-example-3.py

import string

print int("4711"),
print string.atoi("4711"),
print string.atoi("11147", 8), # octal
print string.atoi("1267", 16), # hexadecimal
print string.atoi("3mv", 36) # whatever...

print string.atoi("4711", 0),
print string.atoi("04711", 0),
print string.atoi("0x4711", 0)

print float("4711"),
print string.atof("1"),
print string.atof("1.23e5")

4711 4711 4711 4711 4711
4711 2505 18193
4711.0 1.0 123000.0

In most cases (especially if you're using 1.6 or later), you can use the int and float functions instead of their string module counterparts.

The atoi function takes an optional second argument, which specifices the number base. If the base is 0, the function looks at the first few characters before attempting to interpret the value: if "0x," the base is set to 16 (hexadecimal), and if "0," the base is set to 8 (octal). The default is base 10 (decimal), just as if you hadn't provided an extra argument.

In 1.6 and later, int also accepts a second argument, just like atoi. Unlike the string versions, int and float accept Unicode strings.


1.8 The re Module

"Some people, when confronted with a problem, think 'I know, I'll use regular expressions.' Now they have two problems."

—Jamie Zawinski, on comp.lang.emacs

The re module provides a set of powerful regular expression facilities, which allows you to quickly check whether a given string matches a given pattern (using the match function), or contains such a pattern (using the search function). A regular expression is a string pattern written in a compact (and quite cryptic) syntax.

The match function attempts to match a pattern against the beginning of the given string, as shown in Example 1-54. If the pattern matches anything at all (including an empty string, if the pattern allows that!), match returns a match object. The group method can be used to find out what matched.

Example 1-54. Using the re Module to Match Strings
File: re-example-1.py

import re

text = "The Attila the Hun Show"

# a single character
m = re.match(".", text)
if m: print repr("."), "=>", repr(m.group(0))

# any string of characters
m = re.match(".*", text)
if m: print repr(".*"), "=>", repr(m.group(0))

# a string of letters (at least one)
m = re.match("\w+", text)
if m: print repr("\w+"), "=>", repr(m.group(0))

# a string of digits
m = re.match("\d+", text)
if m: print repr("\d+"), "=>", repr(m.group(0))

 '.' => 'T'
'.*' => 'The Attila the Hun Show'
'\\w+' => 'The'

You can use parentheses to mark regions in the pattern. If the pattern matched, the group method can be used to extract the contents of these regions, as shown in Example 1-55. group(1) returns the contents of the first group, group(2) returns the contents of the second, and so on. If you pass several group numbers to the group function, it returns a tuple.

Example 1-55. Using the re Module to Extract Matching Substrings
File: re-example-2.py

import re

text ="10/15/99"

m = re.match("(\d{2})/(\d{2})/(\d{2,4})", text)
if m:
    print m.group(1, 2, 3)

('10', '15', '99')

The search function searches for the pattern inside the string, as shown in Example 1-56. It basically tries the pattern at every possible character position, starting from the left, and returns a match object as soon it has found a match. If the pattern doesn't match anywhere, it returns None.

Example 1-56. Using the re Module to Search for Substrings
File: re-example-3.py

import re

text = "Example 3: There is 1 date 10/25/95 in here!"

m = re.search("(\d{1,2})/(\d{1,2})/(\d{2,4})", text)

print m.group(1), m.group(2), m.group(3)

month, day, year = m.group(1, 2, 3)
print month, day, year

date = m.group(0)
print date

10 25 95
10 25 95
10/25/95

The sub function used in Example 1-57 can be used to replace patterns with another string.

Example 1-57. Using the re Module to Replace Substrings
File: re-example-4.py

import re

text = "you're no fun anymore..."

# literal replace (string.replace is faster)
print re.sub("fun", "entertaining", text)

# collapse all non-letter sequences to a single dash
print re.sub("[^\w]+", "-", text)

# convert all words to beeps
print re.sub("\S+", "-BEEP-", text)

you're no entertaining anymore...
you-re-no-fun-anymore-
-BEEP- -BEEP- -BEEP- -BEEP-

You can also use sub to replace patterns via a callback function. Example 1-58 shows how to precompile patterns.

Example 1-58. Using the re Module to Replace Substrings via the callback Function
File: re-example-5.py

import re
import string

text = "a line of text\\012another line of text\\012etc..."

def octal(match):
    # replace octal code with corresponding ASCII character
    return chr(string.atoi(match.group(1), 8))

octal_pattern = re.compile(r"\\(\d\d\d)")

print text
print octal_pattern.sub(octal, text)

a line of text\012another line of text\012etc...
a line of text
another line of text
etc...

If you don't compile, the re module caches compiled versions for you, so you usually don't have to compile regular expressions in small scripts. In Python 1.5.2, the cache holds 20 patterns. In 2.0, the cache size has been increased to 100 patterns.

Finally, Example 1-59 matches a string against a list of patterns. The list of patterns are combined into a single pattern, and precompiled to save time.

Example 1-59. Using the re Module to Match Against One of Many Patterns
File: re-example-6.py

import re, string

def combined_pattern(patterns):
    p = re.compile(
        string.join(map(lambda x: "("+x+")", patterns), "|")
        )
    def fixup(v, m=p.match, r=range(0,len(patterns))):
        try:
            regs = m(v).regs
        except AttributeError:
            return None # no match, so m.regs will fail
        else:
            for i in r:
                if regs[i+1] != (-1, -1):
                    return i
    return fixup

#
# try it out!

patterns = [
    r"\d+",
    r"abc\d{2,4}",
    r"p\w+"
]

p = combined_pattern(patterns)

print p("129391")
print p("abc800")
print p("abc1600")
print p("python")
print p("perl")
print p("tcl")

0
1
1
2
2
None


1.9 The math Module

The math module implements a number of mathematical operations for floating-point numbers. The functions are generally thin wrappers around the platform C library functions of the same name, so results may vary slightly across platforms in normal cases, or vary a lot in exceptional cases. Example 1-60 demonstrates the use of the math module.

Example 1-60. Using the math Module
File: math-example-1.py

import math

print "e", "=>", math.e
print "pi", "=>", math.pi
print "hypot", "=>", math.hypot(3.0, 4.0)

# and many others...

e => 2.71828182846
pi => 3.14159265359
hypot => 5.0

See the Python Library Reference for a full list of functions.


1.10 The cmath Module

The cmath module shown in Example 1-61 contains a number of mathematical operations for complex numbers.

Example 1-61. Using the cmath Module
File: cmath-example-1.py

import cmath

print "pi", "=>", cmath.pi
print "sqrt(-1)", "=>", cmath.sqrt(-1)

pi => 3.14159265359
sqrt(-1) => 1j

See the Python Library Reference for a full list of functions.


1.11 The operator Module

The operator module provides a "functional" interface to the standard operators in Python. The functions in this module can be used instead of some lambda constructs, when processing data with functions like map and filter. They are also quite popular among people who like to write obscure code, for obvious reasons. The operator module is demonstrated in Example 1-62.

Example 1-62. Using the operator Module
File: operator-example-1.py

import operator

sequence = 1, 2, 4

print "add", "=>", reduce(operator.add, sequence)
print "sub", "=>", reduce(operator.sub, sequence)
print "mul", "=>", reduce(operator.mul, sequence)
print "concat", "=>", operator.concat("spam", "egg")
print "repeat", "=>", operator.repeat("spam", 5)
print "getitem", "=>", operator.getitem(sequence, 2)
print "indexOf", "=>", operator.indexOf(sequence, 2)
print "sequenceIncludes", "=>", operator.sequenceIncludes(sequence, 3)

add => 7
sub => -5
mul => 8
concat => spamegg
repeat => spamspamspamspamspam

getitem => 4
indexOf => 1
sequenceIncludes => 0

Example 1-63 shows some operator functions that can be used to check object types.

Example 1-63. Using the operator Module for Type Checking
File: operator-example-2.py

import operator
import UserList

def dump(data):
    print type(data), "=>",
    if operator.isCallable(data):
        print "CALLABLE",
    if operator.isMappingType(data):
        print "MAPPING",
    if operator.isNumberType(data):
        print "NUMBER",
    if operator.isSequenceType(data):
        print "SEQUENCE",
    print

dump(0)
dump("string")
dump("string"[0])
dump([1, 2, 3])
dump((1, 2, 3))
dump({"a": 1})
dump(len) # function
dump(UserList) # module
dump(UserList.UserList) # class
dump(UserList.UserList()) # instance

<type 'int'> => NUMBER
<type 'string'> => SEQUENCE
<type 'string'> => SEQUENCE
<type 'list'> => SEQUENCE
<type 'tuple'> => SEQUENCE
<type 'dictionary'> => MAPPING
<type 'builtin_function_or_method'> => CALLABLE
<type 'module'> =>
<type 'class'> => CALLABLE
<type 'instance'> => MAPPING NUMBER SEQUENCE

Note that the operator module doesn't handle object instances in a normal fashion. Be careful when you use the isNumberType, isMappingType, and isSequenceType functions. It's easy to make your code less flexible than it has to be.

Also, note that a string sequence member (a character) is also a sequence. If you're writing a recursive function that uses isSequenceType to traverse an object tree, you better not pass it an ordinary string (or anything containing one).


1.12 The copy Module

The copy module contains two functions that are used to copy objects, as shown in Example 1-64.

copy(object) => object creates a "shallow" copy of the given object. In this context, shallow means that the object itself is copied, but if the object is a container, the members will still refer to the original member objects.

Example 1-64. Using the copy Module to Copy Objects
File: copy-example-1.py

import copy

a = [[1],[2],[3]]
b = copy.copy(a)

print "before", "=>"
print a
print b

# modify original
a[0][0] = 0
a[1] = None

print "after", "=>"
print a
print b

before =>
[[1], [2], [3]]
[[1], [2], [3]]
after =>
[[0], None, [3]]
[[0], [2], [3]]

You can also make shallow copies of lists using the [:] syntax (full slice), and you can make copies of dictionaries using the copy method.

In contrast, deepcopy(object) => object creates a "deep" copy of the given object, as shown in Example 1-65. If the object is a container, all members are copied as well, recursively.

Example 1-65. Using the copy Module to Copy Collections
File: copy-example-2.py

import copy

a = [[1],[2],[3]]
b = copy.deepcopy(a)

print "before", "=>"
print a
print b

# modify original
a[0][0] = 0
a[1] = None

print "after", "=>"
print a
print b

before =>
[[1], [2], [3]]
[[1], [2], [3]]
after =>
[[0], None, [3]]
[[1], [2], [3]]


1.13 The sys Module

The sys module provides a number of functions and variables that can be used to manipulate different parts of the Python runtime environment.

1.13.1 Working with Command-line Arguments

The argv list contains the arguments that were passed to the script, when the interpreter was started, as shown in Example 1-66. The first item contains the name of the script itself.

Example 1-66. Using the sys Module to Get Script Arguments
File: sys-argv-example-1.py

import sys

print "script name is", sys.argv[0]

if len(sys.argv) > 1:
    print "there are", len(sys.argv)-1, "arguments:"
    for arg in sys.argv[1:]:
        print arg
else:
    print "there are no arguments!"

script name is sys-argv-example-1.py
there are no arguments!

If you read the script from standard input (like "python < sys-argv-example-1.py"), the script name is set to an empty string. If you pass in the program as a string (using the -c option), the script name is set to "-c."

1.13.2 Working with Modules

The path list contains a list of directory names in which Python looks for extension modules (Python source modules, compiled modules, or binary extensions). When you start Python, this list is initialized from a mixture of built-in rules, the contents of the PYTHONPATH environment variable, and the registry contents (on Windows). But since it's an ordinary list, you can also manipulate it from within the program, as Example 1-67 shows.

Example 1-67. Using the sys Module to Manipulate the Module Search Path
File: sys-path-example-1.py

import sys

print "path has", len(sys.path), "members"

# add the sample directory to the path
sys.path.insert(0, "samples")
import sample

# nuke the path
sys.path = []
import random # oops!

path has 7 members
this is the sample module!
Traceback (innermost last):
  File "sys-path-example-1.py", line 11, in ?
    import random # oops!
ImportError: No module named random

Example 1-68 demonstrates the builtin_module_names list, which contains the names of all modules built into the Python interpreter.

Example 1-68. Using the sys Module to Find Built-in Modules
File: sys-builtin-module-names-example-1.py

import sys

def dump(module):
    print module, "=>",
    if module in sys.builtin_module_names:
        print "<BUILTIN>"
    else:
        module = _ _import_ _(module)
        print module._ _file_ _

dump("os")
dump("sys")
dump("string")
dump("strop")
dump("zlib")

os => C:\python\lib\os.pyc
sys => <BUILTIN>
string => C:\python\lib\string.pyc
strop => <BUILTIN>
zlib => C:\python\zlib.pyd

The modules dictionary contains all loaded modules. The import statement checks this dictionary before it actually loads something from disk.

As you can see from Example 1-69, Python loads quite a bunch of modules before handing control over to your script.

Example 1-69. Using the sys Module to Find Imported Modules
File: sys-modules-example-1.py

import sys

print sys.modules.keys()

['os.path', 'os', 'exceptions', '_ _main_ _', 'ntpath', 'strop', 'nt',
'sys', '_ _builtin_ _', 'site', 'signal', 'UserDict', 'string', 'stat']

1.13.3 Working with Reference Counts

The getrefcount function (shown in Example 1-70) returns the reference count for a given object—that is, the number of places where this variable is used. Python keeps track of this value, and when it drops to 0, the object is destroyed.

Example 1-70. Using the sys Module to Find the Reference Count
File: sys-getrefcount-example-1.py

import sys

variable = 1234

print sys.getrefcount(0)
print sys.getrefcount(variable)
print sys.getrefcount(None)

50
3
192

Note that this value is always larger than the actual count, since the function itself hangs on to the object while determining the value.

1.13.4 Checking the Host Platform

Example 1-71 shows the platform variable, which contains the name of the host platform.

Example 1-71. Using the sys Module to Find the Current Platform
File: sys-platform-example-1.py

import sys

#
# emulate "import os.path" (sort of)...

if sys.platform == "win32":
    import ntpath
    pathmodule = ntpath
elif sys.platform == "mac":
    import macpath
    pathmodule = macpath
else:
    # assume it's a posix platform
    import posixpath
    pathmodule = posixpath

print pathmodule

Typical platform names are win32 for Windows 9X/NT and mac for Macintosh. For Unix systems, the platform name is usually derived from the output of the "uname -r" command, such as irix6, linux2, or sunos5 (Solaris).

1.13.5 Tracing the Program

The setprofiler function allows you to install a profiling function. This is called every time a function or method is called, at every return (explicit or implied), and for each exception. Let's look at Example 1-72.

Example 1-72. Using the sys Module to Install a Profiler Function
File: sys-setprofiler-example-1.py

import sys

def test(n):
    j = 0
    for i in range(n):
        j = j + i
    return n

def profiler(frame, event, arg):
    print event, frame.f_code.co_name, frame.f_lineno, "->", arg

# profiler is activated on the next call, return, or exception
sys.setprofile(profiler)

# profile this function call
test(1)

# disable profiler
sys.setprofile(None)

# don't profile this call
test(2)

call test 3 -> None
return test 7 -> 1

The profile module provides a complete profiler framework, based on this function.

The settrace function in Example 1-73 is similar, but the trace function is called for each new line:

Example 1-73. Using the sys Module to Install a trace Function
File: sys-settrace-example-1.py

import sys

def test(n):
    j = 0
    for i in range(n):
        j = j + i
    return n

def tracer(frame, event, arg):
    print event, frame.f_code.co_name, frame.f_lineno, "->", arg
    return tracer

# tracer is activated on the next call, return, or exception
sys.settrace(tracer)

# trace this function call
test(1)

# disable tracing
sys.settrace(None)

# don't trace this call
test(2)

call test 3 -> None
line test 3 -> None
line test 4 -> None
line test 5 -> None
line test 5 -> None
line test 6 -> None
line test 5 -> None
line test 7 -> None
return test 7 -> 1

The pdb module provides a complete debugger framework, based on the tracing facilities offered by this function.

1.13.6 Working with Standard Input and Output

The stdin, stdout, and stderr variables contain stream objects corresponding to the standard I/O streams. You can access them directly if you need better control over the output than print can give you. You can also replace them, if you want to redirect output and input to some other device, or process them in some non-standard way, as shown in Example 1-74.

Example 1-74. Using the sys Module to Redirect Output
File: sys-stdout-example-1.py

import sys
import string

class Redirect:

    def _ _init_ _(self, stdout):
        self.stdout = stdout

    def write(self, s):
        self.stdout.write(string.lower(s))

# redirect standard output (including the print statement)
old_stdout = sys.stdout
sys.stdout = Redirect(sys.stdout)

print "HEJA SVERIGE",
print "FRISKT HUM\303\226R"

# restore standard output
sys.stdout = old_stdout

print "M\303\205\303\205\303\205\303\205L!"

heja sverige friskt hum\303\266r
M\303\205\303\205\303\205\303\205L!

An object that implements the write method is all it takes to redirect output.

(Unless it's a C type instance, that is: Python uses an integer attribute called softspace to control spacing, and adds it to the object if it isn't there. You don't have to bother if you're using Python objects, but if you need to redirect to a C type, you should make sure that type supports the softspace attribute.)

1.13.7 Exiting the Program

When you reach the end of the main program, the interpreter is automatically terminated. If you need to exit in midflight, you can call the sys.exit function, which takes an optional integer value that is returned to the calling program. It is demonstrated in Example 1-75.

Example 1-75. Using the sys Module to Exit the Program
File: sys-exit-example-1.py

import sys

print "hello"

sys.exit(1)

print "there"

hello

It may not be obvious, but sys.exit doesn't exit at once. Instead, it raises a SystemExit exception. This means that you can trap calls to sys.exit in your main program, as Example 1-76 shows.

Example 1-76. Catching the sys.exit Call
File: sys-exit-example-2.py

import sys

print "hello"

try:
    sys.exit(1)
except SystemExit:
    pass

print "there"

hello
there

If you want to clean things up after yourself, you can install an "exit handler," which is a function that is automatically called on the way out. This is shown in Example 1-77.

Example 1-77. Catching the sys.exit Call Another Way
File: sys-exitfunc-example-1.py

import sys

def exitfunc():
    print "world"

sys.exitfunc = exitfunc

print "hello"
sys.exit(1)
print "there" # never printed

hello
world

In Python 2.0, you can use the atexit module to register more than one exit handler.


1.14 The atexit Module

(2.0 only) The atexit module allows you to register one or more functions that are called when the interpreter is terminated.

To register a function, simply call the register function, as shown in Example 1-78. You can also add one or more extra arguments, which are passed as arguments to the exit function.

Example 1-78. Using the atexit Module
File: atexit-example-1.py

import atexit

def exit(*args):
    print "exit", args

# register two exit handler
atexit.register(exit)
atexit.register(exit, 1)
atexit.register(exit, "hello", "world")

exit ('hello', 'world')
exit (1,)
exit ()

This module is a straightforward wrapper for the sys.exitfunc hook.


1.15 The time Module

The time module provides a number of functions that deal with dates and the time within a day. It's a thin layer on top of the C runtime library.

A given date and time can either be represented as a floating-point value (the number of seconds since a reference date, usually January 1, 1970), or as a time tuple.

1.15.1 Getting the Current Time

Example 1-79 shows how you can use the time module to get the current time.

Example 1-79. Using the time Module to Get the Current Time
File: time-example-1.py

import time

now = time.time()

print now, "seconds since", time.gmtime(0)[:6]
print
print "or in other words:"
print "- local time:", time.localtime(now)
print "- utc:", time.gmtime(now)

937758359.77 seconds since (1970, 1, 1, 0, 0, 0)

or in other words:
- local time: (1999, 9, 19, 18, 25, 59, 6, 262, 1)
- utc: (1999, 9, 19, 16, 25, 59, 6, 262, 0)

The tuple returned by localtime and gmtime contains the year, month, day, hour, minute, second, day of the week, day of the year, daylight savings flag. The year number is four digits, the day of week begins with 0 for Monday, and January 1 is day number 1.

1.15.2 Converting Time Values to Strings

You can of course use standard string-formatting operators to convert a time tuple to a string, but the time module also provides a number of standard conversion functions, as Example 1-80 illustrates.

Example 1-80. Using the time Module to Format Dates and Times
File: time-example-2.py

import time

now = time.localtime(time.time())

print time.asctime(now)
print time.strftime("%y/%m/%d %H:%M", now)
print time.strftime("%a %b %d", now)
print time.strftime("%c", now)
print time.strftime("%I %p", now)
print time.strftime("%Y-%m-%d %H:%M:%S %Z", now)

# do it by hand...
year, month, day, hour, minute, second, weekday, yearday, daylight = now
print "%04d-%02d-%02d" % (year, month, day)
print "%02d:%02d:%02d" % (hour, minute, second)
print ("MON", "TUE", "WED", "THU", "FRI", "SAT", "SUN")[weekday], yearday

Sun Oct 10 21:39:24 1999
99/10/10 21:39
Sun Oct 10
Sun Oct 10 21:39:24 1999
09 PM
1999-10-10 21:39:24 CEST
1999-10-10
21:39:24
SUN 283

1.15.3 Converting Strings to Time Values

On some platforms, the time module contains a strptime function, which is pretty much the opposite of strftime. Given a string and a pattern, it returns the corresponding time tuple, as shown in Example 1-81.

Example 1-81. Using the time.strptime Function to Parse Dates and Times
File: time-example-6.py

import time

# make sure we have a strptime function!
try:
    strptime = time.strptime
except AttributeError:
    from strptime import strptime

print strptime("31 Nov 00", "%d %b %y")
print strptime("1 Jan 70 1:30pm", "%d %b %y %I:%M%p")

The time.strptime function is currently only made available by Python if it's provided by the platform's C libraries. For platforms that don't have a standard implementation (this includes Windows), Example 1-82 offers a partial replacement.

Example 1-82. A strptime Implementation
File: strptime.py

import re
import string

MONTHS = ["Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug",
          "Sep", "Oct", "Nov", "Dec"]

SPEC = {
    # map formatting code to a regular expression fragment
    "%a": "(?P<weekday>[a-z]+)",
    "%A": "(?P<weekday>[a-z]+)",
    "%b": "(?P<month>[a-z]+)",
    "%B": "(?P<month>[a-z]+)",
    "%C": "(?P<century>\d\d?)",
    "%d": "(?P<day>\d\d?)",
    "%D": "(?P<month>\d\d?)/(?P<day>\d\d?)/(?P<year>\d\d)",
    "%e": "(?P<day>\d\d?)",
    "%h": "(?P<month>[a-z]+)",
    "%H": "(?P<hour>\d\d?)",
    "%I": "(?P<hour12>\d\d?)",
    "%j": "(?P<yearday>\d\d?\d?)",
    "%m": "(?P<month>\d\d?)",
    "%M": "(?P<minute>\d\d?)",
    "%p": "(?P<ampm12>am|pm)",
    "%R": "(?P<hour>\d\d?):(?P<minute>\d\d?)",
    "%S": "(?P<second>\d\d?)",
    "%T": "(?P<hour>\d\d?):(?P<minute>\d\d?):(?P<second>\d\d?)",
    "%U": "(?P<week>\d\d)",
    "%w": "(?P<weekday>\d)",
    "%W": "(?P<weekday>\d\d)",
    "%y": "(?P<year>\d\d)",
    "%Y": "(?P<year>\d\d\d\d)",
    "%%": "%"
}

class TimeParser:
    def _ _init_ _(self, format):
        # convert strptime format string to regular expression
        format = string.join(re.split("(?:\s|%t|%n)+", format))
        pattern = []
        try:
            for spec in re.findall("%\w|%%|.", format):
                if spec[0] == "%":
                    spec = SPEC[spec]
                pattern.append(spec)
        except KeyError:
            raise ValueError, "unknown specificer: %s" % spec
        self.pattern = re.compile("(?i)" + string.join(pattern, ""))
    def match(self, daytime):
        # match time string
        match = self.pattern.match(daytime)
        if not match:
            raise ValueError, "format mismatch"
        get = match.groupdict().get
        tm = [0] * 9
        # extract date elements
        y = get("year")
        if y:
            y = int(y)
            if y < 68:
                y = 2000 + y
            elif y < 100:
                y = 1900 + y
            tm[0] = y
        m = get("month")
        if m:
            if m in MONTHS:
                m = MONTHS.index(m) + 1
            tm[1] = int(m)
        d = get("day")
        if d: tm[2] = int(d)
        # extract time elements
        h = get("hour")
        if h:
            tm[3] = int(h)
        else:
            h = get("hour12")
            if h:
                h = int(h)
                if string.lower(get("ampm12", "")) == "pm":
                    h = h + 12
                tm[3] = h
        m = get("minute")
        if m: tm[4] = int(m)
        s = get("second")
        if s: tm[5] = int(s)
        # ignore weekday/yearday for now
        return tuple(tm)

def strptime(string, format="%a %b %d %H:%M:%S %Y"):
    return TimeParser(format).match(string)

if _ _name_ _ == "_ _main_ _":
    # try it out
    import time
    print strptime("2000-12-20 01:02:03", "%Y-%m-%d %H:%M:%S")
    print strptime(time.ctime(time.time()))

(2000, 12, 20, 1, 2, 3, 0, 0, 0)
(2000, 11, 15, 12, 30, 45, 0, 0, 0)

1.15.4 Converting Time Values

Converting a time tuple back to a time value is pretty easy, at least as long as we're talking about local time. Just pass the time tuple to the mktime function, as shown in Example 1-83.

Example 1-83. Using the time Module to Convert a Local Time Tuple to a Time Integer
File: time-example-3.py

import time

t0 = time.time()
tm = time.localtime(t0)

print tm

print t0
print time.mktime(tm)

(1999, 9, 9, 0, 11, 8, 3, 252, 1)
936828668.16
936828668.0

Unfortunately, there's no function in the 1.5.2 standard library that converts UTC time tuples back to time values (neither in Python nor in the underlying C libraries). Example 1-84 provides a Python implementation of such a function, called timegm.

Example 1-84. Converting a UTC Time Tuple to a Time Integer
File: time-example-4.py

import time

def _d(y, m, d, days=(0,31,59,90,120,151,181,212,243,273,304,334,365)):
    # map a date to the number of days from a reference point
    return (((y - 1901)*1461)/4 + days[m-1] + d +
        ((m > 2 and not y % 4 and (y % 100 or not y % 400)) and 1))

def timegm(tm, epoch=_d(1970,1,1)):
    year, month, day, h, m, s = tm[:6]
    assert year >= 1970
    assert 1 <= month <= 12
    return (_d(year, month, day) - epoch)*86400 + h*3600 + m*60 + s

t0 = time.time()
tm = time.gmtime(t0)

print tm

print t0
print timegm(tm)

(1999, 9, 8, 22, 12, 12, 2, 251, 0)
936828732.48
936828732

In 1.6 and later, a similar function is available in the calendar module, as calendar.timegm.

1.15.5 Timing Things

The time module can be used to time the execution of a Python program, as Example 1-85 demonstrates. You can measure either "wall time" (real world time), or "process time" (amount of CPU time the process has consumed, thus far).

Example 1-85. Using the time Module to Benchmark an Algorithm
File: time-example-5.py

import time

def procedure():
    time.sleep(2.5)

# measure process time
t0 = time.clock()
procedure()
print time.clock() - t0, "seconds process time"

# measure wall time
t0 = time.time()
procedure()
print time.time() - t0, "seconds wall time"

0.0 seconds process time
2.50903499126 seconds wall time

Not all systems can measure the true process time. On such systems (including Windows), clock usually measures the wall time since the program was started.

The process time has limited precision. On many systems, it wraps around after just over 30 minutes.

Also, see the timing module, which measures the wall time between two events.


1.16 The types Module

The types module contains type objects for all object types defined by the standard interpreter, as Example 1-86 demonstrates. All objects of the same type share a single type object. You can use is to test if an object has a given type.

Example 1-86. Using the types Module
File: types-example-1.py

import types

def check(object):
    print object,

    if type(object) is types.IntType:
        print "INTEGER",
    if type(object) is types.FloatType:
        print "FLOAT",
    if type(object) is types.StringType:
        print "STRING",
    if type(object) is types.ClassType:
        print "CLASS",
    if type(object) is types.InstanceType:
        print "INSTANCE",
    print

check(0)
check(0.0)
check("0")

class A:
    pass

class B:
    pass

check(A)
check(B)

a = A()
b = B()

check(a)
check(b)

0 INTEGER
0.0 FLOAT
0 STRING
A CLASS
B CLASS
<A instance at 796960> INSTANCE
<B instance at 796990> INSTANCE

Note that all classes have the same type, as do all instances. To test what class hierarchy a class or an instance belongs to, use the built-in issubclass and isinstance functions.

The types module destroys the current exception state when it is first imported. In other words, don't import it (or any module that imports it!) from within an exception handler.


1.17 The gc Module

(Optional, 2.0 and later) The gc module provides an interface to the built-in cyclic garbage collector.

Python uses reference counting to keep track of when to get rid of objects; as soon as the last reference to an object goes away, the object is destroyed.

Starting with Version 2.0, Python also provides a cyclic garbage collector, which runs at regular intervals. This collector looks for data structures that point to themselves, and attempts to break the cycles. Example 1-87 shows this.

You can use the gc.collect function to force full collection. This function returns the number of objects destroyed by the collector.

Example 1-87. Using the gc Module to Collect Cyclic Garbage
File: gc-example-1.py

import gc

# create a simple object that links to itself
class Node:

    def _ _init_ _(self, name):
        self.name = name
        self.parent = None
        self.children = []

    def addchild(self, node):
        node.parent = self
        self.children.append(node)

    def _ _repr_ _(self):
        return "<Node %s at %x>" % (repr(self.name), id(self))

# set up a self-referencing structure
root = Node("monty")

root.addchild(Node("eric"))
root.addchild(Node("john"))
root.addchild(Node("michael"))

# remove our only reference
del root

print gc.collect(), "unreachable objects"
print gc.collect(), "unreachable objects"

12 unreachable objects
0 unreachable objects

If you're sure that your program doesn't create any self-referencing data structures, you can use the gc.disable function to disable collection. After calling this function, Python 2.0 works exactly like 1.5.2 and earlier.


2.1 Overview

This chapter describes a number of modules that are used in many Python programs. It's perfectly possible to write large Python programs without using them, but they can help you save a lot of time and effort.

2.1.1 Files and Streams

The fileinput module makes it easy to write different kinds of text filters. This module provides a wrapper class, which lets you use a simple for-in statement to loop over the contents of one or more text files.

The StringIO module (and the cStringIO variant) implements an in-memory file object. You can use StringIO objects in many places where Python expects an ordinary file object.

2.1.2 Type Wrappers

UserDict, UserList, and UserString are thin wrappers on top of the corresponding built-in types. Unlike the built-in types, these wrappers can be subclassed. This can come in handy if you need a class that works almost like a built-in type, but has one or more extra methods.

2.1.3 Random Numbers

The random module provides a number of different random number generators. The whrandom module is similar, but it also allows you to create multiple generator objects.

2.1.4 Digests and Encryption Algorithms

The md5 and sha modules are used to calculate cryptographically strong message signatures (so-called "message digests").

The crypt module implements a DES-style one-way encryption. This module is usually only available on Unix systems.

The rotor module provides simple two-way encryption.


2.10 The UserString Module

(New in 2.0) The UserString module contains two classes, UserString and MutableString. The former is a wrapper for the standard string type that can be subclassed, and the latter is a variation that allows you to modify the string in place.

Note that MutableString is not very efficient. Most operations are implemented using slicing and string concatenation. If performance is important, use lists of string fragments or the array module. Example 2-17 shows the UserString module.

Example 2-17. Using the UserString Module
File: userstring-example-1.py

import UserString

class MyString(UserString.MutableString):

    def append(self, s):
        self.data = self.data + s

    def insert(self, index, s):
        self.data = self.data[index:] + s + self.data[index:]

    def remove(self, s):
        self.data = self.data.replace(s, "")

file = open("samples/book.txt")
text = file.read()
file.close()

book = MyString(text)

for bird in ["gannet", "robin", "nuthatch"]:
    book.remove(bird)

print book

...
C: The one without the !
P: The one without the -!!! They've ALL got the !! It's a
Standard British Bird, the , it's in all the books!!!
...


2.11 The traceback Module

The traceback module in Example 2-18 allows you to print exception tracebacks inside your programs, just like the interpreter does when you don't catch an exception yourself.

Example 2-18. Using the traceback Module to Print a Traceback
File: traceback-example-1.py

# note! importing the traceback module messes up the
# exception state, so you better do that here and not
# in the exception handler
import traceback

try:
    raise SyntaxError, "example"
except:
    traceback.print_exc()

Traceback (innermost last):
  File "traceback-example-1.py", line 7, in ?
SyntaxError: example

Example 2-19 uses the StringIO module to put the traceback in a string.

Example 2-19. Using the traceback Module to Copy a Traceback to a String
File: traceback-example-2.py

import traceback
import StringIO

try:
    raise IOError, "an i/o error occurred"
except:
    fp = StringIO.StringIO()
    traceback.print_exc(file=fp)
    message = fp.getvalue()

    print "failure! the error was:", repr(message)

failure! the error was: 'Traceback (innermost last):\012  File
"traceback-example-2.py", line 5, in ?\012IOError: an i/o error
occurred\012'

To format the traceback in a nonstandard way, use the extract_tb function to convert a traceback object to a list of stack entries, as Example 2-20 demonstrates.

Example 2-20. Using the traceback Module to Decode a Traceback Object
File: traceback-example-3.py

import traceback
import sys

def function():
    raise IOError, "an i/o error occurred"

try:
    function()
except:
    info = sys.exc_info()
    for file, lineno, function, text in traceback.extract_tb(info[2]):
        print file, "line", lineno, "in", function
        print "=>", repr(text)
    print "** %s: %s" % info[:2]

traceback-example-3.py line 8 in ?
=> 'function()'
traceback-example-3.py line 5 in function
=> 'raise IOError, "an i/o error occurred"'
** exceptions.IOError: an i/o error occurred


2.12 The errno Module

The errno module defines a number of symbolic error codes, such as ENOENT ( "no such directory entry") and EPERM ( "permission denied"). It also provides a dictionary mapping from platform-dependent numerical error codes to symbolic names. Example 2-21 shows how to use errno.

In most cases, the IOError exception provides a 2-tuple with the numerical error code and an explanatory string. If you need to distinguish between different error codes, use the symbolic names where possible.

Example 2-21. Using the errno Module
File: errno-example-1.py

import errno

try:
    fp = open("no.such.file")
except IOError, (error, message):
    if error == errno.ENOENT:
        print "no such file"
    elif error == errno.EPERM:
        print "permission denied"
    else:
        print message

no such file

Example 2-22 is a bit contrived, but it shows how to use the errorcode dictionary to map from a numerical error code to the symbolic name.

Example 2-22. Using the errorcode Dictionary
File: errno-example-2.py

import errno

try:
    fp = open("no.such.file")
except IOError, (error, message):
    print error, repr(message)
    print errno.errorcode[error]

# 2 'No such file or directory'
# ENOENT


2.13 The getopt Module

The getopt module used in Example 2-23 contains functions to extract command-line options and arguments. It can handle both short and long option formats.

The second argument specifies the short options that should be allowed. A colon (:) after an option name means that option must have an additional argument.

Example 2-23. Using the getopt Module
File: getopt-example-1.py

import getopt
import sys

# simulate command-line invocation
sys.argv = ["myscript.py", "-l", "-d", "directory", "filename"]

# process options
opts, args = getopt.getopt(sys.argv[1:], "ld:")

long = 0
directory = None

for o, v in opts:
    if o == "-l":
        long = 1
    elif o == "-d":
        directory = v

print "long", "=", long
print "directory", "=", directory
print "arguments", "=", args

long = 1
directory = directory
arguments = ['filename']

To make getopt look for long options, as in Example 2-24, pass a list of option descriptors as the third argument. If an option name ends with an equals sign (=), that option must have an additional argument.

Example 2-24. Using the getopt Module to Handle Long Options
File: getopt-example-2.py

import getopt
import sys

# simulate command-line invocation
sys.argv = ["myscript.py", "--echo", "--printer", "lp01", "message"]

opts, args = getopt.getopt(sys.argv[1:], "ep:", ["echo", "printer="])

# process options
echo = 0
printer = None

for o, v in opts:
    if o in ("-e", "--echo"):
        echo = 1
    elif o in ("-p", "--printer"):
        printer = v

print "echo", "=", echo
print "printer", "=", printer
print "arguments", "=", args

echo = 1
printer = lp01
arguments = ['message']


2.14 The getpass Module

The getpass module provides a platform-independent way to enter a password in a command-line program, as Example 2-25 shows.

getpass(prompt) prints the prompt string, switches off keyboard echo, and reads a password. If the prompt argument is omitted, it prints "Password:".

getuser() gets the current username, if possible.

Example 2-25. Using the getpass Module
File: getpass-example-1.py

import getpass

usr = getpass.getuser()

pwd = getpass.getpass("enter password for user %s: " % usr)

print usr, pwd

enter password for user mulder:
mulder trustno1


2.15 The glob Module

The glob module generates lists of files matching given patterns, just like the Unix shell.

File patterns are similar to regular expressions, but simpler. An asterisk (*) matches zero or more characters, and a question mark (?) matches exactly one character. You can also use brackets to indicate character ranges, such as [0-9] for a single digit. All other characters match themselves.

glob(pattern) returns a list of all files matching a given pattern. The glob module is demonstrated in Example 2-26.

Example 2-26. Using the glob Module
File: glob-example-1.py

import glob

for file in glob.glob("samples/*.jpg"):
    print file

samples/sample.jpg

Note that glob returns full pathnames, unlike the os.listdir function. glob uses the fnmatch module to do the actual pattern matching.


2.16 The fnmatch Module

The fnmatch module matches filenames against a pattern, as Example 2-27 shows.

The pattern syntax is the same as that used in Unix shells. An asterisk (*) matches zero or more characters, and a question mark (?) matches exactly one character. You can also use brackets to indicate character ranges, such as [0-9] for a single digit. All other characters match themselves.

Example 2-27. Using the fnmatch Module to Match Files
File: fnmatch-example-1.py

import fnmatch
import os

for file in os.listdir("samples"):
    if fnmatch.fnmatch(file, "*.jpg"):
        print file

sample.jpg

In Example 2-28, the translate function converts a file pattern to a regular expression.

Example 2-28. Using the fnmatch Module to Convert a Pattern to a Regular Expression
File: fnmatch-example-2.py

import fnmatch
import os, re

pattern = fnmatch.translate("*.jpg")

for file in os.listdir("samples"):
    if re.match(pattern, file):
        print file

print "(pattern was %s)" % pattern

sample.jpg
(pattern was .*\.jpg$)

The fnmatch module is used by the glob and find modules.


Chapter 10. Data Storage

"Unlike mainstream component programming, scripts usually do not introduce new components but simply 'wire' existing ones. Scripts can be seen as introducing behavior but no new state ... Of course, there is nothing to stop a 'scripting' language from introducing persistent state — it then simply turns into a normal programming language."

—Clemens Szyperski, in Component Software


10.1 Overview

Python comes with drivers for a number of very similar database managers, all modeled after Unix's dbm library. These databases behave like ordinary dictionaries, except that you can only use strings for keys and values (the shelve module can handle any kind of value).


10.2 The anydbm Module

The anydbm module provides a unified interface to the simple database drivers supported by Python.

The first time it is imported, the anydbm module looks for a suitable database driver, testing for dbhash, gdbm, dbm, or dumbdbm, in that order. If no such module is found, it raises an ImportError exception.

In Example 10-1, the open function is used to open or create a database, using the chosen database handler.

Example 10-1. Using the anydbm Module
File: anydbm-example-1.py

import anydbm

db = anydbm.open("database", "c")
db["1"] = "one"
db["2"] = "two"
db["3"] = "three"
db.close()

db = anydbm.open("database", "r")
for key in db.keys():
    print repr(key), repr(db[key])

'2' 'two'
'3' 'three'
'1' 'one'


10.3 The whichdb Module

The whichdb module can be used to figure out which database handler was used for a given database file, as shown in Example 10-2.

Example 10-2. Using the whichdb Module
File: whichdb-example-1.py

import whichdb

filename = "database"

result = whichdb.whichdb(filename)

if result:
    print "file created by", result
    handler = _ _import_ _(result)
    db = handler.open(filename, "r")
    print db.keys()
else:
    # cannot identify data base
    if result is None:
        print "cannot read database file", filename
    else:
        print "cannot identify database file", filename
    db = None

Example 10-2 uses the _ _import_ _ function to import a module with the given name.


10.4 The shelve Module

The shelve module, shown in Example 10-3, uses the database handlers to implement persistent dictionaries. A shelve object uses string keys, but the value can be of any datatype, as long as it can be handled by the pickle module.

Example 10-3. Using the shelve Module
File: shelve-example-1.py

import shelve

db = shelve.open("database", "c")
db["one"] = 1
db["two"] = 2
db["three"] = 3
db.close()

db = shelve.open("database", "r")
for key in db.keys():
    print repr(key), repr(db[key])

'one' 1
'three' 3
'two' 2

Example 10-4 shows how to use the shelve module with a given database driver.

Example 10-4. Using the shelve Module with a Given Database
File: shelve-example-3.py

import shelve
import gdbm

def gdbm_shelve(filename, flag="c"):
    return shelve.Shelf(gdbm.open(filename, flag))

db = gdbm_shelve("dbfile")


10.5 The dbhash Module

(Optional) The dbhash module provides a dbm-compatible interface to the bsddb database handler. Example 10-5 uses this module.

Example 10-5. Using the dbhash Module
File: dbhash-example-1.py

import dbhash

db = dbhash.open("dbhash", "c")
db["one"] = "the foot"
db["two"] = "the shoulder"
db["three"] = "the other foot"
db["four"] = "the bridge of the nose"
db["five"] = "the naughty bits"
db["six"] = "just above the elbow"
db["seven"] = "two inches to the right of a very naughty bit indeed"
db["eight"] = "the kneecap"
db.close()

db = dbhash.open("dbhash", "r")
for key in db.keys():
    print repr(key), repr(db[key])


10.6 The dbm Module

(Optional) The dbm module provides an interface to the dbm database handler (available on many Unix platforms). This is shown in Example 10-6.

Example 10-6. Using the dbm Module
File: dbm-example-1.py

import dbm

db = dbm.open("dbm", "c")
db["first"] = "bruce"
db["second"] = "bruce"
db["third"] = "bruce"
db["fourth"] = "bruce"
db["fifth"] = "michael"
db["fifth"] = "bruce" # overwrite
db.close()

db = dbm.open("dbm", "r")
for key in db.keys():
    print repr(key), repr(db[key])

'first' 'bruce'
'second' 'bruce'
'fourth' 'bruce'
'third' 'bruce'
'fifth' 'bruce'


10.7 The dumbdbm Module

The dumbdbm module, shown in Example 10-7, is a very simple database implementation, similar to dbm and friends, but written in pure Python. It uses two files: a binary file (.dat), which contain the data, and a text file (.dir), which contains data descriptors.

Example 10-7. Using the dumbdbm Module
File: dumbdbm-example-1.py

import dumbdbm

db = dumbdbm.open("dumbdbm", "c")
db["first"] = "fear"
db["second"] = "surprise"
db["third"] = "ruthless efficiency"
db["fourth"] = "an almost fanatical devotion to the Pope"
db["fifth"] = "nice red uniforms"
db.close()

db = dumbdbm.open("dumbdbm", "r")
for key in db.keys():
    print repr(key), repr(db[key])

'first' 'fear'
'third' 'ruthless efficiency'
'fifth' 'nice red uniforms'
'second' 'surprise'
'fourth' 'an almost fanatical devotion to the Pope'


10.8 The gdbm Module

(Optional) The gdbm module provides an interface to the GNU dbm database handler, as Example 10-8 shows.

Example 10-8. Using the gdbm Module
File: gdbm-example-1.py

import gdbm

db = gdbm.open("gdbm", "c")
db["1"] = "call"
db["2"] = "the"
db["3"] = "next"
db["4"] = "defendant"
db.close()

db = gdbm.open("gdbm", "r")

keys = db.keys()
keys.sort()
for key in keys:
    print db[key],

call the next defendant


Chapter 11. Tools and Utilities

The standard library comes with a number of modules that can be used both as modules and as command-line utilities.


11.1 The dis Module

The dis module is the Python disassembler. It converts byte codes to a format that is slightly more appropriate for human consumption.

You can run the disassembler from the command line. It compiles the given script and prints the disassembled byte codes to the terminal:

$ dis.py hello.py

          0 SET_LINENO              0

          3 SET_LINENO              1
          6 LOAD_CONST              0 ('hello again, and welcome to the show')
          9 PRINT_ITEM
         10 PRINT_NEWLINE
         11 LOAD_CONST              1 (None)
         14 RETURN_VALUE

You can also use dis as a module. The dis function takes a class, method, function, or code object as its single argument. Example 11-1 uses the module.

Example 11-1. Using the dis Module
File: dis-example-1.py

import dis

def procedure():
    print 'hello'

dis.dis(procedure)

          0 SET_LINENO          3

          3 SET_LINENO          4
          6 LOAD_CONST          1 ('hello')
          9 PRINT_ITEM
         10 PRINT_NEWLINE
         11 LOAD_CONST          0 (None)
         14 RETURN_VALUE


11.2 The pdb Module

The pdb module is the standard Python debugger. It is based on the bdb debugger framework.

You can run the debugger from the command line (type n [or next] to go to the next line and help to get a list of available commands):

$ pdb.py hello.py
> hello.py(0)?()
(Pdb) n
> hello.py()
(Pdb) n
hello again, and welcome to the show
--Return--
> hello.py(1)?()->None
(Pdb)

Example 11-2 shows how to start the debugger from inside a program.

Example 11-2. Using the pdb Module
File: pdb-example-1.py

import pdb

def test(n):
    j = 0
    for i in range(n):
        j = j + i
    return n

db = pdb.Pdb()
db.runcall(test, 1)

> pdb-example-1.py(3)test()
-> def test(n):
(Pdb) s
> pdb-example-1.py(4)test()
-> j = 0
(Pdb) s
> pdb-example-1.py(5)test()
-> for i in range(n):
...


11.3 The bdb Module

The bdb module provides a framework for debuggers. You can use this to create your own custom debuggers, as Example 11-3 shows.

To implement custom behavior, subclass the Bdb class, and override the user methods (which are called whenever the debugger stops). To control the debugger, use the various set methods.

Example 11-3. Using the bdb Mdule
File: bdb-example-1.py

import bdb
import time

def spam(n):
    j = 0
    for i in range(n):
        j = j + i
    return n

def egg(n):
    spam(n)
    spam(n)
    spam(n)
    spam(n)

def test(n):
    egg(n)

class myDebugger(bdb.Bdb):

    run = 0

    def user_call(self, frame, args):
        name = frame.f_code.co_name or "<unknown>"
        print "call", name, args
        self.set_continue() # continue

    def user_line(self, frame):
        if self.run:
            self.run = 0
            self.set_trace() # start tracing
        else:
            # arrived at breakpoint
            name = frame.f_code.co_name or "<unknown>"
            filename = self.canonic(frame.f_code.co_filename)
            print "break at", filename, frame.f_lineno, "in", name
        print "continue..."
        self.set_continue() # continue to next breakpoint

    def user_return(self, frame, value):
        name = frame.f_code.co_name or "<unknown>"
        print "return from", name, value
        print "continue..."
        self.set_continue() # continue

    def user_exception(self, frame, exception):
        name = frame.f_code.co_name or "<unknown>"
        print "exception in", name, exception
        print "continue..."
        self.set_continue() # continue

db = myDebugger()
db.run = 1
db.set_break("bdb-example-1.py", 7)
db.runcall(test, 1)

continue...
call egg None
call spam None
break at C:\ematter\librarybook\bdb-example-1.py 7 in spam
continue...
call spam None
break at C:\ematter\librarybook\bdb-example-1.py 7 in spam
continue...
call spam None
break at C:\ematter\librarybook\bdb-example-1.py 7 in spam
continue...
call spam None
break at C:\ematter\librarybook\bdb-example-1.py 7 in spam
continue...


11.4 The profile Module

The profile module is the standard Python profiler.

Like the disassembler and the debugger, you can run the profiler from the command line:

$ profile.py hello.py

hello again, and welcome to the show

         3 function calls in 0.785 CPU seconds

  Ordered by: standard name

  ncalls  tottime  percall  cumtime  percall filename:lineno(function)
       1  0.001    0.001    0.002    0.002 <string>:1(?)
       1  0.001    0.001    0.001    0.001 hello.py:1(?)
       1  0.783    0.783    0.785    0.785 profile:0(execfile('hello.py'))
       0  0.000             0.000          profile:0(profiler)

It can also be used to profile part of a program, as Example 11-4 shows.

Example 11-4. Using the profile Module
File: profile-example-1.py

import profile

def func1():
    for i in range(1000):
        pass

def func2():
    for i in range(1000):
        func1()

profile.run("func2()")

        1003 function calls in 2.380 CPU seconds

  Ordered by: standard name

  ncalls  tottime  percall  cumtime  percall filename:lineno(function)
       1  0.000    0.000    2.040    2.040 <string>:1(?)
    1000  1.950    0.002    1.950    0.002 profile-example-1.py:3(func1)
       1  0.090    0.090    2.040    2.040 profile-example-1.py:7(func2)
       1  0.340    0.340    2.380    2.380 profile:0(func2())
       0  0.000             0.000          profile:0(profiler)

You can modify the report to suit your needs, via the pstats module.


11.5 The pstats Module

The pstats module is a tool that analyzes data collected by the Python profiler, as Example 11-5 shows.

Example 11-5. Using the pstats Module
File: pstats-example-1.py

import pstats
import profile

def func1():
    for i in range(1000):
        pass

def func2():
    for i in range(1000):
        func1()

p = profile.Profile()
p.run("func2()")

s = pstats.Stats(p)
s.sort_stats("time", "name").print_stats()

         1003 function calls in 1.574 CPU seconds

   Ordered by: internal time, function name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     1000    1.522    0.002    1.522    0.002 pstats-example-1.py:4(func1)
        1    0.051    0.051    1.573    1.573 pstats-example-1.py:8(func2)
        1    0.001    0.001    1.574    1.574 profile:0(func2())
        1    0.000    0.000    1.573    1.573 <string>:1(?)
        0    0.000             0.000          profile:0(profiler)


11.6 The tabnanny Module

(New in 2.0) The tabnanny module checks Python source files for ambigous indentation. If a file mixes tabs and spaces in a way that throws off indentation, no matter what tab size you're using, the nanny complains.

In the badtabs.py file used in the following examples, the first line after the if statement uses four spaces followed by a tab. The second uses spaces only.

$ tabnanny.py -v samples/badtabs.py
';samples/badtabs.py': *** Line 3: trouble in tab city! ***
offending line:         print "world"

indent not equal e.g. at tab sizes 1, 2, 3, 5, 6, 7, 9

Since the Python interpreter reads a tab as eight spaces, the script will run correctly. It will also display correctly in any editor that assumes that a tab is either eight or four spaces. That's not enough to fool the tab nanny, of course.

Example 11-6 shows how you can also use tabnanny from your own programs.

Example 11-6. Using the tabnanny Module
File: tabnanny-example-1.py

import tabnanny

FILE = "samples/badtabs.py"

file = open(FILE)
for line in file.readlines():
    print repr(line)

# let tabnanny look at it
tabnanny.check(FILE)

'if 1:\012'
'    \011print "hello"\012'
'        print "world"\012'
samples/badtabs.py 3 '        print "world"'\012'

To capture the output, you can redirect sys.stdout to a StringIO object.


Chapter 12. Platform-Specific Modules

    Section 12.1. Overview

    Section 12.2. The fcntl Module

    Section 12.3. The pwd Module

    Section 12.4. The grp Module

    Section 12.5. The nis Module

    Section 12.6. The curses Module

    Section 12.7. The termios Module

    Section 12.8. The tty Module

    Section 12.9. The resource Module

    Section 12.10. The syslog Module

    Section 12.11. The msvcrt Module

    Section 12.12. The nt Module

    Section 12.13. The _winreg Module

    Section 12.14. The posix Module


12.1 Overview

This chapter describes some platform-specific modules. I've emphasized modules that are available on entire families of platforms (such as Unix or the Windows family).


12.2 The fcntl Module

(Unix only) The fcntl module provides an interface to the ioctl and fcntl functions on Unix. They are used for "out of band" operations on file handles and I/O device handles. This includes things like reading extended attributes, controlling blocking, modifying terminal behavior, and so on.

Exactly how to use these functions is highly platform dependent. For more information on what you can do on your platform, check the corresponding Unix manpages.

This module also provides an interface to Unix's file locking mechanisms. Example 12-1 uses the flock function to place an advisory lock on the file, while it is being updated.

The output shown later was obtained by running three instances of the program in parallel, like this (all on one command line):

python fcntl-example-1.py& python fcntl-example-1.py&
  python fcntl-example-1.py&
If you comment out the call to flock, the counter will not be updated properly.

Example 12-1. Using the fcntl Module
File: fcntl-example-1.py

import fcntl, FCNTL
import os, time

FILE = "counter.txt"

if not os.path.exists(FILE):
    # create the counter file if it doesn't exist
    file = open(FILE, "w")
    file.write("0")
    file.close()

for i in range(20):
    # increment the counter
    file = open(FILE, "r+")
    fcntl.flock(file.fileno(), FCNTL.LOCK_EX)
    counter = int(file.readline()) + 1
    file.seek(0)
    file.write(str(counter))
    file.close() # unlocks the file
    print os.getpid(), "=>", counter
    time.sleep(0.1)

30940 => 1
30942 => 2
30941 => 3
30940 => 4
30941 => 5
30942 => 6


12.3 The pwd Module

(Unix only) The pwd module provides an interface to the Unix password "database" (/etc/passwd and friends). This database (usually a plain-text file) contains information about the user accounts on the local machine. Example 12-2 demonstrates pwd.

Example 12-2. Using the pwd Module
File: pwd-example-1.py

import pwd
import os

print pwd.getpwuid(os.getgid())
print pwd.getpwnam("root")

('effbot', 'dsWjk8', 4711, 4711, 'eff-bot', '/home/effbot', '/bin/bosh')
('root', 'hs2giiw', 0, 0, 'root', '/root', '/bin/bash')

The getpwall function returns a list of database entries for all available users. This can be useful if you want to search for a user.

If you have to look up many names, you can use getpwall to preload a dictionary, as shown in Example 12-3.

Example 12-3. Using the pwd Module
File: pwd-example-2.py

import pwd
import os

# preload password dictionary
_pwd = {}
for info in pwd.getpwall():
    _pwd[info[0]] = _pwd[info[2]] = info

def userinfo(uid):
    # name or uid integer
    return _pwd[uid]

print userinfo(os.getuid())
print userinfo("root")

('effbot', 'dsWjk8', 4711, 4711, 'eff-bot', '/home/effbot', '/bin/bosh')
('root', 'hs2giiw', 0, 0, 'root', '/root', '/bin/bash')


12.4 The grp Module

(Unix only) The grp module provides an interface to the Unix group database (/etc/group). The getgrgid function returns data for a given group identity (see Example 12-4), and getgrnam returns data for a group name.

Example 12-4. Using the grp Module
File: grp-example-1.py

import grp
import os

print grp.getgrgid(os.getgid())
print grp.getgrnam("wheel")

('effbot', '', 4711, ['effbot'])
('wheel', '', 10, ['root', 'effbot', 'gorbot', 'timbot'])

The getgrall function returns a list of database entries for all available groups.

If you're going to do a lot of group queries, you can save some time by using getgrall to copy all the (current) groups into a dictionary. The groupinfo function in Example 12-5 returns the information for either a group identifier (an integer) or a group name (a string).

Example 12-5. Using the grp Module to Cache Group Information
File: grp-example-2.py

import grp
import os

# preload password dictionary
_grp = {}
for info in grp.getgrall():
    _grp[info[0]] = _grp[info[2]] = info

def groupinfo(gid):
    # name or gid integer
    return _grp[gid]

print groupinfo(os.getgid())
print groupinfo("wheel")

('effbot', '', 4711, ['effbot'])
('wheel', '', 10, ['root', 'effbot', 'gorbot', 'timbot'])


12.5 The nis Module

(Unix only, Optional) The nis module provides an interface to the NIS (yellow pages) services, as Example 12-6 shows. It can be used to fetch values from a NIS database, if available.

Example 12-6. Using the nis Module
File: nis-example-1.py

import nis
import string

print nis.cat("ypservers")
print string.split(nis.match("bacon", "hosts.byname"))

{'bacon.spam.egg': 'bacon.spam.egg'}
['194.18.155.250', 'bacon.spam.egg', 'bacon', 'spam-010']


12.6 The curses Module

(Unix only, Optional) The curses module gives you better control of the text terminal window, in a terminal-independent way. Example 12-7 shows its use.

Example 12-7. Using the curses Module
File: curses-example-1.py

import curses

text = [
    "a very simple curses demo",
    "",
    "(press any key to exit)"
]

# connect to the screen
screen = curses.initscr()

# setup keyboard
curses.noecho() # no keyboard echo
curses.cbreak() # don't wait for newline

# screen size
rows, columns = screen.getmaxyx()

# draw a border around the screen
screen.border()

# display centered text
y = (rows - len(text)) / 2

for line in text:
    screen.addstr(y, (columns-len(line))/2, line)
    y = y + 1

screen.getch()

curses.endwin()


12.7 The termios Module

(Unix only, Optional) The termios module provides an interface to the Unix terminal control facilities. It can be used to control most aspects of the terminal communication ports.

In Example 12-8, this module is used to temporarily disable keyboard echo (which is controlled by the ECHO flag in the third flag field).

Example 12-8. Using the termios Module
File: termios-example-1.py

import termios, TERMIOS
import sys

fileno = sys.stdin.fileno()

attr = termios.tcgetattr(fileno)
orig = attr[:]

print "attr =>", attr[:4] # flags

# disable echo flag
attr[3] = attr[3] & ~TERMIOS.ECHO

try:
    termios.tcsetattr(fileno, TERMIOS.TCSADRAIN, attr)
    message = raw_input("enter secret message: ")
    print
finally:
    # restore terminal settings
    termios.tcsetattr(fileno, TERMIOS.TCSADRAIN, orig)

print "secret =>", repr(message)

attr => [1280, 5, 189, 35387]
enter secret message:
secret => 'and now for something completely different'


12.8 The tty Module

(Unix only) The tty module contains some utility functions for dealing with tty devices. Example 12-9 shows how to switch the terminal window over to "raw" mode, and back again.

Example 12-9. Using the tty Module
File: tty-example-1.py

import tty
import os, sys

fileno = sys.stdin.fileno()

tty.setraw(fileno)
print raw_input("raw input: ")

tty.setcbreak(fileno)
print raw_input("cbreak input: ")

os.system("stty sane") # ...

raw input: this is raw input
cbreak input: this is cbreak input


12.9 The resource Module

(Unix only, Optional) The resource module is used to query or modify the system resource current settings limits. Example 12-10 shows how to use the module to query, and Example 12-11 shows how to modify resource limits.

Example 12-10. Using the resource Module to Query Current Settings
File: resource-example-1.py

import resource

print "usage stats", "=>", resource.getrusage(resource.RUSAGE_SELF)
print "max cpu", "=>", resource.getrlimit(resource.RLIMIT_CPU)
print "max data", "=>", resource.getrlimit(resource.RLIMIT_DATA)
print "max processes", "=>", resource.getrlimit(resource.RLIMIT_NPROC)
print "page size", "=>", resource.getpagesize()

usage stats => (0.03, 0.02, 0, 0, 0, 0, 75, 168, 0, 0, 0, 0, 0, 0, 0, 0)
max cpu => (2147483647, 2147483647)
max data => (2147483647, 2147483647)
max processes => (256, 256)
page size => 4096
Example 12-11. Using the resource Module to Limit Resources
File: resource-example-2.py

import resource

resource.setrlimit(resource.RLIMIT_CPU, (0, 1))

# pretend we're busy
for i in range(1000):
    for j in range(1000):
        for k in range(1000):
            pass

CPU time limit exceeded


12.10 The syslog Module

(Unix only, Optional) The syslog module sends messages to the system logger facility (syslogd). Exactly what happens to these messages is system-dependent, but they usually end up in a log file named /var/log/messages, /var/adm/syslog, or some variation thereof. (If you cannot find it, check with your system administrator.) Example 12-12 demonstrates.

Example 12-12. Using the syslog Module
File: syslog-example-1.py

import syslog
import sys

syslog.openlog(sys.argv[0])

syslog.syslog(syslog.LOG_NOTICE, "a log notice")
syslog.syslog(syslog.LOG_NOTICE, "another log notice: %s" % "watch out!")

syslog.closelog()


12.11 The msvcrt Module

(Windows/DOS only) The msvcrt module gives you access to a number of functions in the Microsoft Visual C/C++ Runtime Library (MSVCRT).

Example 12-13 demonstrates the getch function reading a single keypress from the console.

Example 12-13. Using the msvcrt Module to Get Key Presses
File: msvcrt-example-1.py

import msvcrt

print "press 'escape' to quit..."

while 1:
    char = msvcrt.getch()
    if char == chr(27):
        break
    print char,
    if char == chr(13):
        print

press 'escape' to quit...
h e l l o

The kbhit function returns true if a key has been pressed (which means that getch won't block), as shown in Example 12-14.

Example 12-14. Using the msvcrt Module to Poll the Keyboard
File: msvcrt-example-2.py

import msvcrt
import time

print "press SPACE to enter the serial number"

while not msvcrt.kbhit() or msvcrt.getch() != " ":
    # do something else
    print ".",
    time.sleep(0.1)

print

# clear the keyboard buffer
while msvcrt.kbhit():
    msvcrt.getch()

serial = raw_input("enter your serial number: ")

print "serial number is", serial

press SPACE to enter the serial number
. . . . . . . . . . . . . . . . . . . . . . . .
enter your serial number: 10
serial number is 10

The locking function in Example 12-15 can be used to implement cross-process file locking under Windows.

Example 12-15. Using the msvcrt Module for File Locking
File: msvcrt-example-3.py

import msvcrt
import os

LK_UNLCK = 0 # unlock the file region
LK_LOCK = 1 # lock the file region
LK_NBLCK = 2 # non-blocking lock
LK_RLCK = 3 # lock for writing
LK_NBRLCK = 4 # non-blocking lock for writing

FILE = "counter.txt"

if not os.path.exists(FILE):
    file = open(FILE, "w")
    file.write("0")
    file.close()

for i in range(20):
    file = open(FILE, "r+")
    # look from current position (0) to end of file
    msvcrt.locking(file.fileno(), LK_LOCK, os.path.getsize(FILE))
    counter = int(file.readline()) + 1
    file.seek(0)
    file.write(str(counter))
    file.close() # unlocks the file
    print os.getpid(), "=>", counter
    time.sleep(0.1)

208 => 21
208 => 22
208 => 23
208 => 24
208 => 25
208 => 26


12.12 The nt Module

(Implementation, Windows only) The nt module is an implementation module used by the os module on Windows platforms. There's hardly any reason to use this module directly; use os instead. Example 12-16 shows its use.

Example 12-16. Using the nt Module
File: nt-example-1.py

import nt

# in real life, use os.listdir and os.stat instead!
for file in nt.listdir("."):
    print file, nt.stat(file)[6]

aifc-example-1.py 314
anydbm-example-1.py 259
array-example-1.py 48


12.13 The _winreg Module

(Windows only, New in 2.0) The _winreg module provides a basic interface to the Windows registry database. Example 12-17 demonstrates the module.

Example 12-17. Using the _winreg Module
File: winreg-example-1.py

import _winreg

explorer = _winreg.OpenKey(
    _winreg.HKEY_CURRENT_USER,
    "Software\\Microsoft\\Windows\CurrentVersion\\Explorer"
    )

#list values owned by this registry key
try:
    i = 0
    while 1:
      name, value, type= _winreg.EnumValue(explorer, i)
      print repr(name),
      i += 1
except WindowsError:
    print

value, type = _winreg.QueryValueEx(explorer, "Logon User Name")

print
print "user is", repr(value)

'Logon User Name' 'CleanShutdown' 'ShellState' 'Shutdown Setting'
'Reason Setting' 'FaultCount' 'FaultTime' 'IconUnderline'...

user is u'Effbot'


12.14 The posix Module

(Implementation, Unix/POSIX only) The posix module is an implementation module used by the os module on Unix and other POSIX systems. While everything in here can be (and should be) accessed via the os module, you may wish to explicitly refer to this module in situations where you want to make it clear that you expect POSIX behavior. Example 12-18 demonstrates.

Example 12-18. Using the posix Module
File: posix-example-1.py

import posix

for file in posix.listdir("."):
    print file, posix.stat(file)[6]

aifc-example-1.py 314
anydbm-example-1.py 259
array-example-1.py 48


Chapter 13. Implementation Support
Modules

    Section 13.1. The dospath Module

    Section 13.2. The macpath Module

    Section 13.3. The ntpath Module

    Section 13.4. The posixpath Module

    Section 13.5. The strop Module

    Section 13.6. The imp Module

    Section 13.7. The new Module

    Section 13.8. The pre Module

    Section 13.9. The sre Module

    Section 13.10. The py_compile Module

    Section 13.11. The compileall Module

    Section 13.12. The ihooks Module

    Section 13.13. The linecache Module

    Section 13.14. The macurl2path Module

    Section 13.15. The nturl2path module

    Section 13.16. The tokenize Module

    Section 13.17. The keyword Module

    Section 13.18. The parser Module

    Section 13.19. The symbol Module

    Section 13.20. The token Module


13.1 The dospath Module

The dospath module, shown in Example 13-1, provides os.path functionality on DOS platforms. You can also use it to handle DOS paths on other platforms.

Example 13-1. Using the dospath Module
File: dospath-example-1.py

import dospath

file = "/my/little/pony"

print "isabs", "=>", dospath.isabs(file)
print "dirname", "=>", dospath.dirname(file)
print "basename", "=>", dospath.basename(file)
print "normpath", "=>", dospath.normpath(file)
print "split", "=>", dospath.split(file)
print "join", "=>", dospath.join(file, "zorba")

isabs => 1
dirname => /my/little
basename => pony
normpath => \my\little\pony
split => ('/my/little', 'pony')
join => /my/little/pony\zorba

Note that Python's DOS support can use both forward ( / ) and backward slashes ( \ ) as directory separators.


13.2 The macpath Module

The macpath module (see Example 13-2) provides os.path functionality on Macintosh platforms. You can also use it to handle Macintosh paths on other platforms.

Example 13-2. Using the macpath Module
File: macpath-example-1.py

import macpath

file = "my:little:pony"

print "isabs", "=>", macpath.isabs(file)
print "dirname", "=>", macpath.dirname(file)
print "basename", "=>", macpath.basename(file)
print "normpath", "=>", macpath.normpath(file)
print "split", "=>", macpath.split(file)
print "join", "=>", macpath.join(file, "zorba")

isabs => 1
dirname => my:little
basename => pony
normpath => my:little:pony
split => ('my:little', 'pony')
join => my:little:pony:zorba


13.3 The ntpath Module

The ntpath module (see Example 13-3) provides os.path functionality on Windows platforms. You can also use it to handle Windows paths on other platforms.

Example 13-3. Using the ntpath Module
File: ntpath-example-1.py

import ntpath

file = "/my/little/pony"

print "isabs", "=>", ntpath.isabs(file)
print "dirname", "=>", ntpath.dirname(file)
print "basename", "=>", ntpath.basename(file)
print "normpath", "=>", ntpath.normpath(file)
print "split", "=>", ntpath.split(file)
print "join", "=>", ntpath.join(file, "zorba")

isabs => 1
dirname => /my/little
basename => pony

normpath => \my\little\pony
split => ('/my/little', 'pony')
join => /my/little/pony\zorba

Note that this module treats both forward slashes ( / ) and backward slashes ( \ ) as directory separators.


13.4 The posixpath Module

The posixpath module, shown in Example 13-4, provides os.path functionality on Unix and other POSIX-compatible platforms. You can also use it to handle POSIX paths on other platforms. In addition, it can be used to process URLs.

Example 13-4. Using the posixpath Module
File: posixpath-example-1.py

import posixpath

file = "/my/little/pony"

print "isabs", "=>", posixpath.isabs(file)
print "dirname", "=>", posixpath.dirname(file)
print "basename", "=>", posixpath.basename(file)
print "normpath", "=>", posixpath.normpath(file)
print "split", "=>", posixpath.split(file)
print "join", "=>", posixpath.join(file, "zorba")

isabs => 1
dirname => /my/little
basename => pony
normpath => /my/little/pony
split => ('/my/little', 'pony')
join => /my/little/pony/zorba


13.5 The strop Module

(Obsolete) The strop is a low-level module that provides fast C implementations of most functions in the string module. It is automatically included by string, so there's seldom any need to access it directly.

However, one reason to use this module is if you need to tweak the path before you start loading Python modules. Example 13-5 demonstrates the module.

Example 13-5. Using the strop Module
File: strop-example-1.py

import strop
import sys

# assuming we have an executable named ".../executable", add a
# directory named ".../executable-extra" to the path

if strop.lower(sys.executable)[-4:] == ".exe":
    extra = sys.executable[:-4] # windows
else:
    extra = sys.executable

sys.path.insert(0, extra + "-extra")

import mymodule

In Python 2.0 and later, you should use string methods instead of strop. In Example 13-5, replace "strop.lower(sys.executable)" with "sys.executable.lower()."


13.6 The imp Module

The imp module contains functions that can be used to implement your own import behavior. Example 13-6 overloads the import statement with a version that logs from where it gets the modules.

Example 13-6. Using the imp Module
File: imp-example-1.py

import imp
import sys

def my_import(name, globals=None, locals=None, fromlist=None):
    try:
        module = sys.modules[name] # already imported?
    except KeyError:
        file, pathname, description = imp.find_module(name)
        print "import", name, "from", pathname, description
        module = imp.load_module(name, file, pathname, description)
    return module

import _ _builtin_ _
_ _builtin_ _._ _import_ _ = my_import

import xmllib

import xmllib from /python/lib/xmllib.py ('.py', 'r', 1)
import re from /python/lib/re.py ('.py', 'r', 1)

import sre from /python/lib/sre.py ('.py', 'r', 1)
import sre_compile from /python/lib/sre_compile.py ('.py', 'r', 1)
import _sre from /python/_sre.pyd ('.pyd', 'rb', 3)

Note that the alternative version shown here doesn't support packages. For a more extensive example, see the sources for the knee module.


13.7 The new Module

(Optional in 1.5.2) The new modules is a low-level module that allows you to create various kinds of internal objects, such as class objects, function objects, and other kinds that are usually created by the Python runtime system. Example 13-7 demonstrates this module.

Note that if you're using 1.5.2, you may have to rebuild Python to use this module; it isn't enabled by the default on all platforms. In 2.0 and later, however, it's always available.

Example 13-7. Using the new Module
File: new-example-1.py

import new

class Sample:

    a = "default"

    def _ _init_ _(self):
        self.a = "initialised"

    def _ _repr_ _(self):
        return self.a

#
# create instances

a = Sample()
print "normal", "=>", a

b = new.instance(Sample, {})
print "new.instance", "=>", b

b._ _init_ _()
print "after _ _init_ _", "=>", b

c = new.instance(Sample, {"a": "assigned"})
print "new.instance w. dictionary", "=>", c

normal => initialised
new.instance => default
after _ _init_ _ => initialised
new.instance w. dictionary => assigned


13.8 The pre Module

(Implementation) The pre module, used in Example 13-8, is a low-level implementation module for the 1.5.2 re module. There's usually no need to use this module directly (and code using it may stop working in future releases).

Example 13-8. Using the pre Module
File: pre-example-1.py

import pre

p = pre.compile("[Python]+")

print p.findall("Python is not that bad")

['Python', 'not', 'th', 't']


13.9 The sre Module

(Implementation) The sre module, used in Example 13-9, is a low-level implementation module for the 2.0 re module. There's usually no need to use this module directly (and code using it may stop working in future releases).

Example 13-9. Using the sre Module
File: sre-example-1.py

import sre

text = "The Bookshop Sketch"

# a single character
m = sre.match(".", text)
if m: print repr("."), "=>", repr(m.group(0))

# and so on, for all 're' examples...

'.' => 'T'


13.10 The py_compile Module

The py_compile module, shown in Example 13-10, allows you to explicitly compile Python modules to bytecode. It behaves like Python's import statement, but takes a filename, not a module name.

Example 13-10. Using the py_compile Module
File: py-compile-example-1.py

import py_compile

# explicitly compile this module
py_compile.compile("py-compile-example-1.py")

The compileall module can be used to compile all Python files in an entire directory tree.


13.11 The compileall Module

The compileall module (see Example 13-11) contains functions to compile all Python scripts in a given directory (or along the Python path) to byte code. It can also be used as a script (on Unix platforms, it's automatically run when Python is installed).

Example 13-11. Using the compileall Module to Compile All Scripts in a Directory
File: compileall-example-1.py

import compileall

print "This may take a while!"

compileall.compile_dir(".", force=1)

This may take a while!
Listing . ...
Compiling .\SimpleAsyncHTTP.py ...
Compiling .\aifc-example-1.py ...
Compiling .\anydbm-example-1.py ...
...


13.12 The ihooks Module

The ihooks module, shown in Example 13-12, provides a framework for import replacements. The idea is to allow several alternate import mechanisms to coexist.

Example 13-12. Using the ihooks Module
File: ihooks-example-1.py

import ihooks, imp, os

def import_from(filename):
    "Import module from a named file"

    loader = ihooks.BasicModuleLoader()
    path, file = os.path.split(filename)
    name, ext  = os.path.splitext(file)
    m = loader.find_module_in_dir(name, path)
    if not m:
        raise ImportError, name
    m = loader.load_module(name, m)
    return m

colorsys = import_from("/python/lib/colorsys.py")

print colorsys

<module 'colorsys' from '/python/lib/colorsys.py'>


13.13 The linecache Module

The linecache module in Example 13-13 is used to read lines from module source code. It caches recently visited modules (the entire source file, actually).

Example 13-13. Using the linecache module
File: linecache-example-1.py

import linecache

print linecache.getline("linecache-example-1.py", 5)

print linecache.getline("linecache-example-1.py", 5)

This module is used by the traceback module.


13.14 The macurl2path Module

(Implementation) The macurl2path module, shown in Example 13-14, contains code to map between URLs and Macintosh filenames. It should not be used directly; use the mechanisms in urllib instead.

Example 13-14. Using the macurl2path Module
File: macurl2path-example-1.py

import macurl2path

file = ":my:little:pony"

print macurl2path.pathname2url(file)
print macurl2path.url2pathname(macurl2path.pathname2url(file))

my/little/pony
:my:little:pony


13.15 The nturl2path module

(Implementation) The nturl2path module, shown in Example 13-15, contains code to map between URLs and Windows filenames.

Example 13-15. Using the nturl2path Module
File: nturl2path-example-1.py

import nturl2path

file = r"c:\my\little\pony"

print nturl2path.pathname2url(file)
print nturl2path.url2pathname(nturl2path.pathname2url(file))

///C|/my/little/pony
C:\my\little\pony

This module should not be used directly; for portability, access these functions via the urllib module instead, as shown in Example 13-16.

Example 13-16. Using the nturl2path Module via the urllib Module
File: nturl2path-example-2.py

import urllib

file = r"c:\my\little\pony"

print urllib.pathname2url(file)
print urllib.url2pathname(urllib.pathname2url(file))

///C|/my/little/pony
C:\my\little\pony


13.16 The tokenize Module

The tokenize module splits a Python source file into individual tokens. It can be used for syntax highlighting or for various kinds of code-analysis tools.

In Example 13-17, we simply print the tokens.

Example 13-17. Using the tokenize Module
File: tokenize-example-1.py

import tokenize

file = open("tokenize-example-1.py")

def handle_token(type, token, (srow, scol), (erow, ecol), line):
    print "%d,%d-%d,%d:\t%s\t%s" % \
        (srow, scol, erow, ecol, tokenize.tok_name[type], repr(token))

tokenize.tokenize(
    file.readline,
    handle_token
    )

1,0-1,6:     NAME    'import'
1,7-1,15:    NAME    'tokenize'
1,15-1,16:   NEWLINE '\012'
2,0-2,1:     NL      '\012'
3,0-3,4:     NAME    'file'
3,5-3,6:     OP      '='
3,7-3,11:    NAME    'open'
3,11-3,12:   OP      '('
3,12-3,35:   STRING  '"tokenize-example-1.py"'
3,35-3,36:   OP      ')'
3,36-3,37:   NEWLINE '\012'
...

Note that the tokenize function takes two callable objects: the first argument is called repeatedly to fetch new code lines, and the second argument is called for each token.


13.17 The keyword Module

The keyword module (see Example 13-18) contains a list of the keywords used in the current version of Python. It also provides a dictionary with the keywords as keys, and a predicate function that can be used to check if a given word is a Python keyword.

Example 13-18. Using the keyword Module
File: keyword-example-1.py

import keyword

name = raw_input("Enter module name: ")

if keyword.iskeyword(name):
    print name, "is a reserved word."
    print "here's a complete list of reserved words:"
    print keyword.kwlist

Enter module name: assert
assert is a reserved word.
here's a complete list of reserved words:
['and', 'assert', 'break', 'class', 'continue', 'def', 'del',
'elif', 'else', 'except', 'exec', 'finally', 'for', 'from',
'global', 'if', 'import', 'in', 'is', 'lambda', 'not', 'or',
'pass', 'print', 'raise', 'return', 'try', 'while']


13.18 The parser Module

(Optional) The parser module provides an interface to Python's built-in parser and compiler.

Example 13-19 compiles a simple expression into an abstract syntax tree (AST), turns the AST into a nested list, dumps the contents of the tree (where each node contains either a grammar symbol or a token), increments all numbers by one, and, finally, turns the list back into a code object. At least that's what I think it does.

Example 13-19. Using the parser Module
File: parser-example-1.py

import parser
import symbol, token

def dump_and_modify(node):
    name = symbol.sym_name.get(node[0])
    if name is None:
        name = token.tok_name.get(node[0])
    print name,
    for i in range(1, len(node)):
        item = node[i]
        if type(item) is type([]):
            dump_and_modify(item)
        else:
            print repr(item)
            if name == "NUMBER":
                # increment all numbers!
                node[i] = repr(int(item)+1)

ast = parser.expr("1 + 3")

list = ast.tolist()

dump_and_modify(list)

ast = parser.sequence2ast(list)

print eval(parser.compileast(ast))

eval_input testlist test and_test not_test comparison
expr xor_expr and_expr shift_expr arith_expr term factor
power atom NUMBER '1'
PLUS '+'
term factor power atom NUMBER '3'
NEWLINE ''
ENDMARKER ''
6


13.19 The symbol Module

The symbol module, used in Example 13-20, contains a listing of non-terminal symbols from the Python grammar. It's probably only useful if you're dabbling with the parser module.

Example 13-20. Using the symbol Module
File: symbol-example-1.py

import symbol

print "print", symbol.print_stmt
print "return", symbol.return_stmt

print 268
return 274


13.20 The token Module

The token module, shown in Example 13-21, contains a list of all tokens used by the standard Python tokenizer.

Example 13-21. Using the token Module
File: token-example-1.py

import token

print "NUMBER", token.NUMBER
print "PLUS", token.STAR
print "STRING", token.STRING

NUMBER 2
PLUS 16
STRING 3


Chapter 14. Other Modules

    Section 14.1. Overview

    Section 14.2. The pyclbr Module

    Section 14.3. The filecmp Module

    Section 14.4. The cmd Module

    Section 14.5. The rexec Module

    Section 14.6. The Bastion Module

    Section 14.7. The readline Module

    Section 14.8. The rlcompleter Module

    Section 14.9. The statvfs Module

    Section 14.10. The calendar Module

    Section 14.11. The sched Module

    Section 14.12. The statcache Module

    Section 14.13. The grep Module

    Section 14.14. The dircache Module

    Section 14.15. The dircmp Module

    Section 14.16. The cmp Module

    Section 14.17. The cmpcache Module

    Section 14.18. The util Module

    Section 14.19. The soundex Module

    Section 14.20. The timing Module

    Section 14.21. The posixfile Module

    Section 14.22. The bisect Module

    Section 14.23. The knee Module

    Section 14.24. The tzparse Module

    Section 14.25. The regex Module

    Section 14.26. The regsub Module

    Section 14.27. The reconvert Module

    Section 14.28. The regex_syntax Module

    Section 14.29. The find Module


14.1 Overview

This chapter describes a number of less-common modules. Some are useful, others are quite obscure, and some are just plain obsolete.


14.2 The pyclbr Module

The pyclbr module, shown in Example 14-1, contains a basic Python class parser.

In 1.5.2, the module exports a single function, readmodule, which parses a given module, and returns a list of all classes defined at the module's top level.

Example 14-1. Using the pyclbr Module
File: pyclbr-example-1.py

import pyclbr

mod = pyclbr.readmodule("cgi")

for k, v in mod.items():
    print k, v

MiniFieldStorage <pyclbr.Class instance at 7873b0>
InterpFormContentDict <pyclbr.Class instance at 79bd00>
FieldStorage <pyclbr.Class instance at 790e20>
SvFormContentDict <pyclbr.Class instance at 79b5e0>
StringIO <pyclbr.Class instance at 77dd90>
FormContent <pyclbr.Class instance at 79bd60>
FormContentDict <pyclbr.Class instance at 79a9c0>

In 2.0 and later, there's also an alternative interface, readmodule_ex, which returns global functions as well. This is shown in Example 14-2.

Example 14-2. Using the pyclbr Module to Read Classes and Functions
File: pyclbr-example-3.py

import pyclbr

# 2.0 and later
mod = pyclbr.readmodule_ex("cgi")

for k, v in mod.items():
    print k, v

MiniFieldStorage <pyclbr.Class instance at 00905D2C>
parse_header <pyclbr.Function instance at 00905BD4>
test <pyclbr.Function instance at 00906FBC>
print_environ_usage <pyclbr.Function instance at 00907C94>
parse_multipart <pyclbr.Function instance at 00905294>
FormContentDict <pyclbr.Class instance at 008D3494>
initlog <pyclbr.Function instance at 00904AAC>
parse <pyclbr.Function instance at 00904EFC>
StringIO <pyclbr.Class instance at 00903EAC>
SvFormContentDict <pyclbr.Class instance at 00906824>
...

To get more information about each class, use the various attributes in the Class instances, as Example 14-3 shows.

Example 14-3. Using the pyclbr Module
File: pyclbr-example-2.py

import pyclbr
import string

mod = pyclbr.readmodule("cgi")

def dump(c):
    # print class header
    s = "class " + c.name
    if c.super:
        s = s +  "(" + string.join(map(lambda v: v.name, c.super), ", ") + ")"
    print s + ":"
    # print method names, sorted by line number
    methods = c.methods.items()
    methods.sort(lambda a, b: cmp(a[1], b[1]))
    for method, lineno in methods:
        print "  def " + method
    print

for k, v in mod.items():
    dump(v)

class MiniFieldStorage:
  def _ _init_ _
  def _ _repr_ _

class InterpFormContentDict(SvFormContentDict):
  def _ _getitem_ _
  def values
  def items

...


14.3 The filecmp Module

(New in 2.0) The filecmp module, shown in Example 14-4, contains functions to compare files and directories.

Example 14-4. Using the filecmp Module
File: filecmp-example-1.py

import filecmp

if filecmp.cmp("samples/sample.au", "samples/sample.wav"):
    print "files are identical"
else:
    print "files differ!"

# files differ!

In 1.5.2 and earlier, you can use the cmp and dircmp modules instead.


14.4 The cmd Module

The cmd module (see Example 14-5) provides a simple framework for command-line interfaces (CLI). This is used by the pdb debugger module, but you can also use it for your own programs.

To implement your own command-line interface, subclass the Cmd class, and define do and help methods. The base class automatically turns all do methods into commands and uses the help methods to show help information.

Example 14-5. Using the cmd Module
File: cmd-example-1.py

import cmd
import string, sys

class CLI(cmd.Cmd):

    def _ _init_ _(self):
        cmd.Cmd._ _init_ _(self)
        self.prompt = '> '

    def do_hello(self, arg):
        print "hello again", arg, "!"

    def help_hello(self):
        print "syntax: hello [message]",
        print "-- prints a hello message"

    def do_quit(self, arg):
        sys.exit(1)

    def help_quit(self):
        print "syntax: quit",
        print "-- terminates the application"

    # shortcuts
    do_q = do_quit

#
# try it out

cli = CLI()
cli.cmdloop()

> help

Documented commands (type help <topic>):
========================================
hello           quit

Undocumented commands:
======================
help            q

> hello world
hello again world !
> q


14.5 The rexec Module

The rexec module, shown in Example 14-6, provides versions of exec, eval, and import, which execute code in a restricted execution environment. In this environment, functions that can damage resources on the local machine are no longer available.

Example 14-6. Using the rexec Module
File: rexec-example-1.py

import rexec

r = rexec.RExec()
print r.r_eval("1+2+3")
print r.r_eval("_ _import_ _('os').remove('file')")

6
Traceback (innermost last):
  File "rexec-example-1.py", line 5, in ?
    print r.r_eval("_ _import_ _('os').remove('file')")
  File "/usr/local/lib/python1.5/rexec.py", line 257, in r_eval
    return eval(code, m._ _dict_ _)
  File "<string>", line 0, in ?
AttributeError: remove


14.6 The Bastion Module

The Bastion module, shown in Example 14-7, allows you to control how a given object is used. It can be used to pass objects from unrestricted parts of your application to code running in restricted mode.

To create a restricted instance, simply call the Bastion wrapper. By default, all instance variables are hidden, as well as all methods that start with an underscore.

Example 14-7. Using the Bastion Module
File: bastion-example-1.py

import Bastion

class Sample:
    value = 0

    def _set(self, value):
        self.value = value

    def setvalue(self, value):
        if 10 < value <= 20:
            self._set(value)
        else:
            raise ValueError, "illegal value"

    def getvalue(self):
        return self.value

#
# try it

s = Sample()
s._set(100) # cheat
print s.getvalue()

s = Bastion.Bastion(Sample())
s._set(100) # attempt to cheat
print s.getvalue()

100
Traceback (innermost last):
...
AttributeError: _set

You can control which functions to publish. In Example 14-8, the internal method can be called from outside, but the getvalue no longer works.

Example 14-8. Using the Bastion Module with a Non-Standard Filter
File: bastion-example-2.py

import Bastion

class Sample:
    value = 0

    def _set(self, value):
        self.value = value

    def setvalue(self, value):
        if 10 < value <= 20:
            self._set(value)
        else:
            raise ValueError, "illegal value"

    def getvalue(self):
        return self.value

#
# try it

def is_public(name):
    return name[:3] != "get"

s = Bastion.Bastion(Sample(), is_public)
s._set(100) # this works
print s.getvalue() # but not this

100
Traceback (innermost last):
...
AttributeError: getvalue


14.7 The readline Module

(Optional) The readline module, shown in Example 14-9, activates input editing on Unix, using the GNU readline library (or compatible).

Once imported, this module provides improved command-line editing, as well as command history. It also enhances the input and raw_input functions.

Example 14-9. Using the readline Module
File: readline-example-1.py

import readline # activate readline editing


14.8 The rlcompleter Module

(Optional, Unix only) The rlcompleter module provides word completion for the readline module.

To enable word completion, just import this module. By default, the completion function is bound to the Esc key. Press Esc twice to finish the current word. To change the key, you can use something like:

import readline
readline.parse_and_bind("tab: complete")

The script in Example 14-10 shows how to use the completion functions from within a program.

Example 14-10. Using the rlcompleter Module to Expand Names
File: rlcompleter-example-1.py

import rlcompleter
import sys

completer = rlcompleter.Completer()

for phrase in "co", "sys.p", "is":
    print phrase, "=>",
    # emulate readline completion handler
    try:
        for index in xrange(sys.maxint):
            term = completer.complete(phrase, index)
            if term is None:
                break
            print term,
    except:
        pass
    print

co => continue compile complex coerce completer
sys.p => sys.path sys.platform sys.prefix
is => is isinstance issubclass


14.9 The statvfs Module

The statvfs module, used in Example 14-11, contains a number of constants and test functions that can be used with the optional os.statvfs function, which returns information about a filesystem.

Example 14-11. Using the statvfs Module
File: statvfs-example-1.py

import statvfs
import os

st = os.statvfs(".")

print "preferred block size", "=>", st[statvfs.F_BSIZE]
print "fundamental block size", "=>", st[statvfs.F_FRSIZE]
print "total blocks", "=>", st[statvfs.F_BLOCKS]
print "total free blocks", "=>", st[statvfs.F_BFREE]
print "available blocks", "=>", st[statvfs.F_BAVAIL]
print "total file nodes", "=>", st[statvfs.F_FILES]
print "total free nodes", "=>", st[statvfs.F_FFREE]
print "available nodes", "=>", st[statvfs.F_FAVAIL]
print "max file name length", "=>", st[statvfs.F_NAMEMAX]

preferred block size => 8192
fundamental block size => 1024
total blocks => 749443
total free blocks => 110442
available blocks => 35497
total file nodes => 92158
total free nodes => 68164
available nodes => 68164
max file name length => 255


14.10 The calendar Module

The calendar module is a Python reimplementation of the Unix cal command. It simply prints the calendar for any given month or year to standard output.

In Example 14-12, prmonth(year, month) prints the calendar for a given month.

Example 14-12. Using the calendar Module
File: calendar-example-1.py

import calendar
calendar.prmonth(1999, 12)

    December 1999
Mo Tu We Th Fr Sa Su
       1  2  3  4  5
 6  7  8  9 10 11 12
13 14 15 16 17 18 19
20 21 22 23 24 25 26
27 28 29 30 31

In Example 14-13, prcal(year) prints the calendar for a given year.

Example 14-13. Using the calendar Module
File: calendar-example-2.py

import calendar
calendar.prcal(2000)

                                  2000

       January                  February                    March
Mo Tu We Th Fr Sa Su      Mo Tu We Th Fr Sa Su      Mo Tu We Th Fr Sa Su
                1  2          1  2  3  4  5  6             1  2  3  4  5
 3  4  5  6  7  8  9       7  8  9 10 11 12 13       6  7  8  9 10 11 12
10 11 12 13 14 15 16      14 15 16 17 18 19 20      13 14 15 16 17 18 19
17 18 19 20 21 22 23      21 22 23 24 25 26 27      20 21 22 23 24 25 26
24 25 26 27 28 29 30      28 29                     27 28 29 30 31
31

        April                      May                      June
Mo Tu We Th Fr Sa Su      Mo Tu We Th Fr Sa Su      Mo Tu We Th Fr Sa Su
                1  2       1  2  3  4  5  6  7                1  2  3  4
 3  4  5  6  7  8  9       8  9 10 11 12 13 14       5  6  7  8  9 10 11
10 11 12 13 14 15 16      15 16 17 18 19 20 21      12 13 14 15 16 17 18
17 18 19 20 21 22 23      22 23 24 25 26 27 28      19 20 21 22 23 24 25
24 25 26 27 28 29 30      29 30 31                  26 27 28 29 30

        July                     August                   September
Mo Tu We Th Fr Sa Su      Mo Tu We Th Fr Sa Su      Mo Tu We Th Fr Sa Su
                1  2          1  2  3  4  5  6                   1  2  3
 3  4  5  6  7  8  9       7  8  9 10 11 12 13       4  5  6  7  8  9 10
10 11 12 13 14 15 16      14 15 16 17 18 19 20      11 12 13 14 15 16 17
17 18 19 20 21 22 23      21 22 23 24 25 26 27      18 19 20 21 22 23 24
24 25 26 27 28 29 30      28 29 30 31               25 26 27 28 29 30
31

       October                  November                  December
Mo Tu We Th Fr Sa Su      Mo Tu We Th Fr Sa Su      Mo Tu We Th Fr Sa Su
                   1             1  2  3  4  5                   1  2  3
 2  3  4  5  6  7  8       6  7  8  9 10 11 12       4  5  6  7  8  9 10
 9 10 11 12 13 14 15      13 14 15 16 17 18 19      11 12 13 14 15 16 17
16 17 18 19 20 21 22      20 21 22 23 24 25 26      18 19 20 21 22 23 24
23 24 25 26 27 28 29      27 28 29 30               25 26 27 28 29 30 31
30 31

Note that the calendars are printed using European conventions; in other words, Monday is the first day of the week.

This module contains a number of support functions that can be useful if you want to output calendars in other formats. It's probably easiest to copy the entire file, and tweak it to suit your needs.


14.11 The sched Module

The sched module is a simple event scheduler for non-threaded environments. Example 14-14 demonstrates.

Example 14-14. Using the sched Module
File: sched-example-1.py

import sched
import time, sys

scheduler = sched.scheduler(time.time, time.sleep)

# add a few operations to the queue
scheduler.enter(0.5, 100, sys.stdout.write, ("one\n",))
scheduler.enter(1.0, 300, sys.stdout.write, ("three\n",))
scheduler.enter(1.0, 200, sys.stdout.write, ("two\n",))

scheduler.run()

one
two
three


14.12 The statcache Module

The statcache module, shown in Example 14-15, contains a function that returns information about files. It's an extension of the os.stat function in that it keeps a cache with recently collected information.

Example 14-15. Using the statcache Module
File: statcache-example-1.py

import statcache
import os, stat, time

now = time.time()
for i in range(1000):
    st = os.stat("samples/sample.txt")
print "os.stat", "=>", time.time() - now

now = time.time()
for i in range(1000):
    st = statcache.stat("samples/sample.txt")
print "statcache.stat", "=>", time.time() - now

print "mode", "=>", oct(stat.S_IMODE(st[stat.ST_MODE]))
print "size", "=>", st[stat.ST_SIZE]
print "last modified", "=>", time.ctime(st[stat.ST_MTIME])

os.stat => 0.371000051498
statcache.stat => 0.0199999809265
mode => 0666
size => 305
last modified => Sun Oct 10 18:39:37 1999


14.13 The grep Module

The grep module provides different ways to search for text in text files, as Example 14-16 shows.

Example 14-16. Using the grep Module
File: grep-example-1.py

import grep
import glob

grep.grep("\<rather\>", glob.glob("samples/*.txt"))

# 4: indentation, rather than delimiters, might become


14.14 The dircache Module

(Obsolete) The dircache module contains a function to get a list of files in a directory. It's an extension of the os.listdir function in that it keeps a cache to avoid rereading a directory that hasn't been modified. Example 14-17 demonstrates this.

Example 14-17. Using the dircache Module
File: dircache-example-1.py

import dircache

import os, time

#
# test cached version

t0 = time.clock()

for i in range(100):
    dircache.listdir(os.sep)

print "cached", time.clock() - t0

#
# test standard version

t0 = time.clock()

for i in range(100):
    os.listdir(os.sep)

print "standard", time.clock() - t0

cached 0.0664509964968
standard 0.5560845807


14.15 The dircmp Module

(Obsolete, Only in 1.5.2) The dircmp module provides a class that can be used to compare the contents of two disk directories, as Example 14-18 shows.

Example 14-18. Using the dircmp Module
File: dircmp-example-1.py

import dircmp

d = dircmp.dircmp()
d.new("samples", "oldsamples")
d.run()
d.report()

diff samples oldsamples
Only in samples : ['sample.aiff', 'sample.au', 'sample.wav']
Identical files : ['sample.gif', 'sample.gz', 'sample.jpg', ...]

In Python 2.0 and later, this module has been replaced by filecmp.


14.16 The cmp Module

(Obsolete, Only in 1.5.2) The cmp module contains a function to compare two files, as Example 14-19 demonstrates.

Example 14-19. Using the cmp Module
File: cmp-example-1.py

import cmp

if cmp.cmp("samples/sample.au", "samples/sample.wav"):
    print "files are identical"
else:
    print "files differ!"

files differ!

In Python 2.0 and later, this module has been replaced by the filecmp module.


14.17 The cmpcache Module

(Obsolete, Only in 1.5.2) The cmpcache module contains a function that compares two files. It's an extension of the cmp module in that it keeps a cache over recently made comparisons. Example 14-20 shows the module's use.

Example 14-20. Using the cmpcache Module
File: cmpcache-example-1.py

import cmpcache

if cmpcache.cmp("samples/sample.au", "samples/sample.wav"):
    print "files are identical"
else:
    print "files differ!"

files differ!

In Python 2.0 and later, this module has been replaced by the filecmp module.


14.18 The util Module

(Obsolete, Only in 1.5.2) The util module is included for backward-compatibility only. New code should use the replacement constructs shown in Examples 14-21 through 14-23.

Example 14-21 shows how remove(sequence, item) removes the given item, if found in the sequence.

Example 14-21. Emulating the util Module's remove Function
File: util-example-1.py

def remove(sequence, item):
    if item in sequence:
        sequence.remove(item)

Example 14-22 shows how readfile(filename) => string reads the contents of a text file as a single string.

Example 14-22. Emulating the util Module's readfile Function
File: util-example-2.py

def readfile(filename):
    file = open(filename, "r")
    return file.read()

Example 14-23 shows how readopenfile(file) => string returns the contents of an open file (or other file object).

Example 14-23. Emulating the util Module's readopenfile Function
File: util-example-3.py

def readopenfile(file):
    return file.read()


14.19 The soundex Module

(Optional, Only 1.5.2) The soundex module implements a simple hash algorithm, which converts words to 6-character strings based on their English pronunciation.

As of Version 2.0, this module is no longer included.

get_soundex(word) returns the soundex string for the given word. Words that sound similar should give the same soundex string. sound_similar(word1, word2) returns true if the two words have the same soundex. Example 14-24 uses both functions.

Example 14-24. Using the soundex Module
File: soundex-example-1.py

import soundex

a = "fredrik"
b = "friedrich"

print soundex.get_soundex(a), soundex.get_soundex(b)

print soundex.sound_similar(a, b)

F63620 F63620
1


14.20 The timing Module

(Obsolete, Unix only) The timing module can be used to time the execution of a Python program. Example 14-25 demonstrates.

Example 14-25. Using the timing Module
File: timing-example-1.py

import timing
import time

def procedure():
    time.sleep(1.234)

timing.start()
procedure()
timing.finish()

print "seconds:", timing.seconds()
print "milliseconds:", timing.milli()
print "microseconds:", timing.micro()

seconds: 1
milliseconds: 1239
microseconds: 1239999

The script in Example 14-26 shows how you can emulate this module using functions in the standard time module.

Example 14-26. Emulating the timing Module
File: timing-example-2.py

import time

t0 = t1 = 0

def start():
    global t0
    t0 = time.time()

def finish():
    global t1
    t1 = time.time()

def seconds():
    return int(t1 - t0)

def milli():
    return int((t1 - t0) * 1000)

def micro():
    return int((t1 - t0) * 1000000)

You can use time.clock() instead of time.time() to get CPU time, where supported.


14.21 The posixfile Module

(Obsolete, Unix only) The posixfile module provides a file-like object with support for file locking, as Example 14-27 shows. New programs should use the fcntl module instead.

Example 14-27. Using the posixfile Module
File: posixfile-example-1.py

import posixfile
import string

filename = "counter.txt"

try:
    # open for update
    file = posixfile.open(filename, "r+")
    counter = int(file.read(6)) + 1
except IOError:
    # create it
    file = posixfile.open(filename, "w")
    counter = 0

file.lock("w|", 6)

file.seek(0) # rewind
file.write("%06d" % counter)

file.close() # releases lock


14.22 The bisect Module

The bisect module provides functions to insert items in sorted sequences.

insort(sequence, item) inserts an item into the sequence, keeping it sorted. The sequence can be any mutable sequence object that implements _ _getitem_ _ and insert; Example 14-28 demonstrates.

Example 14-28. Using the bisect Module to Insert Items in an Ordered List
File: bisect-example-1.py

import bisect

list = [10, 20, 30]

bisect.insort(list, 25)
bisect.insort(list, 15)

print list

[10, 15, 20, 25, 30]

In Example 14-29, bisect(sequence, item) => index returns the index where the item should be inserted. The sequence is not modified.

Example 14-29. Using the bisect Module to Find Insertion Points
File: bisect-example-2.py

import bisect

list = [10, 20, 30]

print list
print bisect.bisect(list, 25)
print bisect.bisect(list, 15)

[10, 20, 30]
2
1


14.23 The knee Module

The knee module is a Python reimplementation of the package import mechanism that was introduced in Python 1.5. Since this is already supported by the standard interpreter, this module is mainly provided to show how things are done in there. It does work, though. Just import the module to enable it, as Example 14-30 shows.

Example 14-30. Using the knee Module
File: knee-example-1.py

import knee

# that's all, folks!


14.24 The tzparse Module

(Obsolete) The (highly incomplete) tzparse module contains a parser for time zone specifications. When you import this module, it parses the content of the TZ environment variable; Example 14-31 demonstrates.

Example 14-31. Using the tzparse Module
File: tzparse-example-1.py

import os
if not os.environ.has_key("TZ"):
    # set it to something...
    os.environ["TZ"] = "EST+5EDT;100/2,300/2"

# importing this module will parse the TZ variable
import tzparse

print "tzparams", "=>", tzparse.tzparams
print "timezone", "=>", tzparse.timezone
print "altzone", "=>", tzparse.altzone
print "daylight", "=>", tzparse.daylight
print "tzname", "=>", tzparse.tzname

tzparams => ('EST', 5, 'EDT', 100, 2, 300, 2)
timezone => 18000
altzone => 14400
daylight => 1
tzname => ('EST', 'EDT')

In addition to the variables shown in Example 14-31, this module contains a number of time manipulation functions that use the defined time zone.


14.25 The regex Module

(Obsolete) The regex module, shown in Example 14-32, is the old (pre-1.5) regular expression machinery. New code should use re where possible.

Note that regex is faster than the re module used in Python 1.5.2, but slower than the new version used in 1.6 and later.

Example 14-32. Using the regex Module
File: regex-example-1.py

import regex

text = "Man's crisis of identity in the latter half of the 20th century"

p = regex.compile("latter") # literal
print p.match(text)
print p.search(text), repr(p.group(0))

p = regex.compile("[0-9]+") # number
print p.search(text), repr(p.group(0))

p = regex.compile("\<\w\w\>") # two-letter word
print p.search(text), repr(p.group(0))

p = regex.compile("\w+$") # word at the end
print p.search(text), repr(p.group(0))

-1
32 'latter'
51 '20'
13 'of'
56 'century'


14.26 The regsub Module

(Obsolete) The regsub module, shown in Example 14-33, provides string replacements, based on regular expressions. New code should use the re module's replace function instead.

Example 14-33. Using the regsub Module
File: regsub-example-1.py

import regsub

text = "Well, there's spam, egg, sausage, and spam."

print regsub.sub("spam", "ham", text) # just the first
print regsub.gsub("spam", "bacon", text) # all of them

Well, there's ham, egg, sausage, and spam.
Well, there's bacon, egg, sausage, and bacon.


14.27 The reconvert Module

(Obsolete) The reconvert module, shown in Example 14-34, converts old-style regular expressions as used by the regex module to the new style used by the re module. It can also be used as a command-line tool.

Example 14-34. Using the reconvert Module
File: reconvert-example-1.py

import reconvert

for pattern in "abcd", "a\(b*c\)d", "\<\w+\>":
    print pattern, "=>", reconvert.convert(pattern)

abcd => abcd
a\(b*c\)d => a(b*c)d
\<\w+\> => \b\w+\b


14.28 The regex_syntax Module

(Obsolete) The regex_syntax module, shown in Example 14-35, contains a bunch of flags that can be used to change the behavior of the regex regular expression module.

Example 14-35. Using the regex_syntax Module
File: regex-syntax-example-1.py

import regex_syntax
import regex

def compile(pattern, syntax):
    syntax = regex.set_syntax(syntax)
    try:
        pattern = regex.compile(pattern)
    finally:
        # restore original syntax
        regex.set_syntax(syntax)
    return pattern

def compile_awk(pattern):
    return compile(pattern, regex_syntax.RE_SYNTAX_AWK)

def compile_grep(pattern):
    return compile(pattern, regex_syntax.RE_SYNTAX_GREP)

def compile_emacs(pattern):
    return compile(pattern, regex_syntax.RE_SYNTAX_EMACS)


14.29 The find Module

(Only 1.5.2, Obsolete) The find module provides a single function, with the same name as the module: find(pattern, directory) => list scans a given directory and all its subdirectories for files matching a given pattern, as Example 14-36 shows.

For more information on the pattern syntax, see the fnmatch module.

Example 14-36. Using the find Module
File: find-example-1.py

import find

# find all JPEG files in or beneath the current directory
for file in find.find("*.jpg", "."):
    print file

.\samples\sample.jpg


I l@ve RuBoard Previous Section

Colophon

Our look is the result of reader comments, our own experimentation, and feedback from distribution channels. Distinctive covers complement our distinctive approach to technical topics, breathing personality and life into potentially dry subjects.

The animals on the cover of Python Standard Library are harvest mice. Many species of harvest mice populate North American grasslands and marshes, while only one species-Micromys minutus, the Old World harvest mouse-resides in the grasslands and farmlands of Europe and Asia.

Smaller than the common house mouse, the harvest mouse sports prominent ears and a very long hairy tail, and its hind feet have an opposable fifth toe for grasping and climbing stems. Behaviorally, harvest mice set themselves apart from other mice species by building breeding nests suspended in high grasses. These nests are baseball-sized globes of woven grass with small entrance holes and are lined with soft plant material, such as dandelion fluff, to keep the young warm. The young are born in litters of three to six, completely dependent on the mother (the father is not allowed in the nest). By the time they are five weeks old, however, they are independent and sexually mature. Overall, harvest mice typically live for six to eighteen months in the wild-enough time for a female to produce one to six litters in her lifetime. These numbers are much higher for mice in captivity.

The harvest mouse is a "cover dependent" species, as it relies on brush and vegetation to hide its small, brown body from predators as it forages for seeds and insect larvae. It moves slowly and adopts a still "camouflage posture" as further defense; overall, it is much more calm than the common house mouse.

The Western, Eastern, and Fulvous harvest mice (Reithrodontomys megalotis, Reithrodontomys humulis, and Reithrodontomys fulvescens, respectively) currently populate various regions of the United States and Canada with relative success, challenged somewhat by habitat loss due to crop farming, cattle grazing, and urbanization. However, their cousin the Saltmarsh harvest mouse (Reithrodontomys raviventris) suffers severe threat due to the filling in of its dwindling marshland home in the San Francisco Bay Area. The only endangered harvest mouse species, its members number in only the hundreds to the few thousands.

Catherine Morris was the production editor and proofreader, and Linley Dolby was the copyeditor for Python Standard Library. Emily Quill, Matt Hutchinson, and Claire Cloutier provided quality control. Joe Wizda wrote the index. Interior composition was done by Gabe Weiss, Matt Hutchinson, and Catherine Morris.

Hanna Dyer designed the cover of this book, based on a series design by Edie Freedman. The cover image is a 19th-century engraving from the Dover Pictorial Archive. Emma Colby produced the cover layout with Quark™XPress 4.1 using Adobe's ITC Garamond font. Emma Colby also designed the CD label.

David Futato designed the interior layout based on a series design by Nancy Priest. The print version of this book was created by translating the DocBook SGML markup of its source files into a set of gtroff macros using a filter developed at O'Reilly & Associates by Norman Walsh. Steve Talbott designed and wrote the underlying macro set on the basis of the GNU troff -gs macros; Lenny Muellner adapted them to SGML and implemented the book design. The GNU groff text formatter Version 1.11.1 was used to generate PostScript output. The text and heading fonts are ITC Garamond Light and Garamond Book; the code font is Constant Willison. This colophon was written by Sarah Jane Shangraw.

The online edition of this book was created by the Safari production group (John Chodacki, Becki Maisch, and Madeleine Newell) using a set of Frame-to-XML conversion and cleanup tools written and maintained by Erik Ray, Benn Salter, John Chodacki, and Jeff Liggett.
    I l@ve RuBoard Previous Section


[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]


[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

% string formatting operator
&rArr; object
& (ampersand)
> (greater than)
< (less then), in markup languages
<P> (paragraph) tag
* (asterisk)
apply function, using in place of
fnmatch module and
glob module and
.dat (binary files), using dumbdbm module
.dir (text files), using dumbdbm module
\ (backward slashes), as directory separators
\N{} excapes, mapping unicode characters
/ (forward slash), as directory separators
? (question mark)
fnmatch module and
glob module and
[ ... ] (brackets)
fnmatch module and
glob module and
[:] syntax (full slice)


[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

abstract syntax tree (AST)
AbstractFormatter class
AbstractWriter class
accept loop
advisory lock, using fcntl module for locking files
aifc module
ampersand (&)
anydbm module
Apache, using mimetypes
apply function, building function argument lists
archive formats
gzip module and
zipfile module and
arguments, calling a function with
ArithmeticError(StandardError)
array module
article method
AST (abstract syntax tree)
asterisk (*)
apply function, using in place of
fnmatch module and
glob module and
asynchat module
asyncore module
atexit module
atoi function
AttributeError(StandardError)
attributes
AU (Sun) audio files
audio files
audiodev module
AutoList class


[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

backward slashes (\), as directory separators
base64 module
BaseHTTPServer module
bass constructors, using apply function
Bastion module
bdb module
begin lines
binary data
converting
binary files (.dat), using dumbdbm module
binascii module
binhex module
bisect module
bmp file formats
<body> tag
brackets ([ ... ])
fnmatch module and
glob module and
bsddb database handler
_ _builtin_ema_ module
exceptions module and
_ _builtin_ _ module
overloading functions from
builtin_module_names list
byte code
compiling
converting


[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

calendar module
_ _call_ _ method
callable function
callback function
case-sensitivity in markup languages
cgi module
CGIHTTPServer module
character entities in markup languages
chdir function
chmod function
CJK ideograph properties, in Python 2.0
CLIs (command-line interfaces)
cmath module
cmd module
cmp module
cmpcache module
code module
code, compiling/executing
colorsys module
command-line arguments
command-line interfaces (CLIs)
commands module
compile function
compile_command function
compileall module
compiling to byte code
compression modules
ConfigParser module 2nd
configuration files
connect method, connecting sockets
conversions of data
cookie module
copy function
copy module
copy_reg module
copytree function [See ]
core modules
cPickle module
crypt module 2nd
cStringIO module
curses module


[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

daemon processes
data representation
data storage
dates, using time module
dbhash module
anydbm module and
dbm module
anydbm module and
dbm-compatible interface
debugger (Python)
declarations in SGML
decodestring function
dedent methods, adding structure
deepcopy()
dictionaries, calling a function with arguments from
digests
dir function
dircache module
dircmp module
directories, using os module
DirectoryWalker class, traversing filesystems
dis module
disassembler (Python)
dispatcher class
dispatcher_with_send class
do methods
<document> tag
document type description (DTD)
DOS path, handling
dospath module
downloading files using the ftplib module
DTD (document type description)
dumbdbm module
anydbm module and
DumbWriter class


[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

echo flag, using termios module
elements, in markup languages
encoded binary data
encodestring function
encoding directives, in XML
encrypting
algorithms
end tags, in markup languages
end_tag method, using XML parsers
ENDENT error code
entity references in markup languages
EnvironmentError
EPERM error code
errno module
error condition in socket signals
eval function
exceptions module
creating classes
exec function 2nd
execfile function
exit handlers
expanduser function
expandvars function
expressions
evaluating
regular
Extensible Markup Language (XML)
external data representation (XDR)
extract_tb function


[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

FancyURLopener class, using the urllib module
fcntl module
file formats
File Transfer Protocol (FTP), using the ftplib module
filecmp module
fileinput module 2nd
filenames, working with os.path module
files
attributes, working with
os module and
filesystems, traversing
find module
float functions
FloatingPointError(ArithmeticError)
flock function, placing advisory locks on files
fnmatch module
fork functions
formatter module
forward slashes (/), as directory separators
FTP (File Transfer Protocol), using the ftplib module
ftplib module
full slice syntax ([:])
functions
calling with arguments


[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

gauss function
gdbm module
anydbm module and
GET request handler
get value method
getcwd function
getgrall function, using grp module
getgrgid function, using grp module
getgrnam function, using grp module
getitem function
_ _getitem_ _ method
getmembers function
getopt module
getpall function, using pwd module
getpass module
getrefcount function
gettime function
ghostscript viewer, using mailcap module
gif file formats
glob module
global interpreter lock
GNU dbm database handler, using gdbm module
gopherlidb module
grep module
groupinfo function
grp module
GZIP formats
gzip module and
support for
gzip module


[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

handle_accept method
handle_close method
handle_connect method
handle_error(type, value, traceback) method
handle_expt method
handle_read method
handle_write method
Hangul syllable properties, in Python 2.0
HEAD request handler
<header> tag
help methods
helper class for MimeWriter module
hexadecimal strings, converting to/from binary data
HLS color value, using functions for converting to
host platforms, checking
HSV color values, using functions for converting to
HTML (Hypertext Markup Language)
htmllib module 2nd
htmllib parser
HTTP (Hypertext Transfer Protocol)
HTTP servers
BaseHTTPServer module and
CGIHTTPServer module and
posting data to
httplib module


[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

I/O streams, working with
ihooks module
IMAP (Internet Message Access Protocol)
imaplib module
imghdr module
imNumberType function, using operator module
imp module
import behaviors, implementing with imp module
_ _import_ _ function 2nd
import statement
ImportError(StandardError)
indent methods, adding structure
IndentationError(SyntaxError)
index function
IndexError(LookupError)
infolist method
inMappingType function, using operator module
inplace keyword
input function
input/output, working with
int function
InteracticeConsole class
internationalization
Internet Message Access Protocol (IMAP)
Internet Time Protocol
IOError(EnvironmentError)
is operator
isinstance function, checking objects
ISO Latin-1 characters, translating to a XML string
isSequenceType function, using operator module
issubclass function, checking objects


[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

James Clark's Expat XML parser
jpeg file formats


[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

kbhit function, using the msvcrt module and
keyboard echo, disabling
KeyboardInterrupt(StandardError)
keyword module
knee module


[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

language support modules
linecache module
listdir function
listen call, accepting connections
locale module
lock objects
threading module and
log messages, overriding
LookupError(StandardError)
loops, using in threads


[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

Macintosh computers
binhex module and
macpath module and
mapping between files and URLs
macpath module
macurl2path module
madedirs function
mail message processing
mailbox module
mailcap module
markup languages
marshal module 2nd
match function
math module
md5 module 2nd
MemoryError(StandardError)
Message class (rfc822 module)
Message object, downloading messages
messages, using nntplib module
Microsoft Visual C/C++ Runtime Library (MSVCRT)
Microsoft WAV audio files
MIME (Multipurpose Internet mail Extensions)
mimetools module
mimetypes module
mimewriter module
mimify module
mmap modules
modules 2nd
aifc
anydbm
array
asynchat
asyncore
atexit
audiodev
base64
BaseHTTPServer
Bastion
bdb
binascii
binhex
bisect
_ _builtin_ _ module
calendar
cgi
CGIHTTPServer
cmath
cmd
cmp
cmpcache
code
colorsys
command
compileall
ConfigParser
cookie
copy
copy_reg
cPickle
crypt
cStringIO
curses
dbhash
dbm
dircache
dircmp
dis
dospath
dumbdbm
error codes and
exceptions
fcntl
filecmp
fileinput
find
fnmatch
formatter
ftplib
gdbm
getopt
getpass
glob
gopherlib
grep
grp
gzip
htmllib 2nd
httplib
ihooks
imaplib
imghdr
imp
keyword
knee
linecache
loading and reloading
locale
macpath
macurl2path
mailbox
mailcap
marshal 2nd
math
md5
mimetools
mimetypes
MimeWriter
mimify
mmap
msvcrt
multifile
netrc 2nd
new
nis
nntplib
nt
ntpath
nturl2path
operator
os
os.path
packmail
parser
pdb
pickle 2nd
sheleve module and
pipes
popen2
poplib
posix 2nd
posixfile
pprint
pre
profile
pstats
pwd
py_compile
pyclbr
queue
quopri
random
random numbers and
re
readline
reconvert
regex
regex_syntax
regsub
repr
resource
rexec
rfc833
rlcompleter
robotparser
rotor
sched
select
sgmllib
sha
shelve
shlex
shutil
signal
SimpleHTTPServer
smtplib
sndhdr
socket
SocketServer
soundex
sre
stat
statcache
statvfs
string
StringIO
strop
struct
sunau
sunaudio
symbol
sys
syslog
tabnanny
telnetlib
tempfile
termios
thread
threading
time
timing
token
tokenize
traceback
tty
type support
types
tzparse
ucnhash
unicodedata
urllib
urlparse
UserDict 2nd
UserList 2nd
UserString 2nd
util
uu
wave
webbbrowser
whatsound
whichdb
whrandom
_winreg
winsound
xdrlib
xml.parsers.expat
xmllib 2nd
zipfiles
zlib
msvcrt module
multifile module
multimedia
Multipurpose Internet Mail Extensions (MIME)
MutableString class


[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

NameError(StandardError)
namelist method
namespaces
nesting in markup languages
netrc module 2nd
Netscape configuration files, using mimetypes module
Network News Transfer Protocol (NNTP)
network protocols
new module
news message processing
NIS (yellow pages) services
nis module
NNTP (Network News Transfer Protocol)
nntplib module
non-blocking mode in sockets
nt module
ntpath module
nturl2path module
NullWriter class


[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

objects, checking types
open function 2nd
creating a database with
open method, using the urllib module
operating system interface modules
operator module
checking objects
os module 2nd
os.listdir function, using dircache module
os.path module
OSError(EnvironmentError)
output formatting
OverflowError(ArithmeticError)


[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

packmail module
paragraph (<P>) tag
parser module
parsers
htmllib module and 2nd
pyclbr module and
rfc833 module and
sgmllib module and
xml.parsers.expat module and
xmllib module and 2nd
passwords
getpass module and
netrc module and
path list
pathnames, working with os.path module
pdb debugger module
pdb module
pdm file formats
pgm file formats
pickle module 2nd
sheleve module and
pilview, using mailcap module
pipes module
platform-specific modules
plug-in modules
png file formats
popen2 module
poplib module
posix module
posixfile module
posixpath module
POST command
PostScript documents, using mailcap module
ppm file formats
pprint module
pre module
processes 2nd
daemon
profile module
profiler (Python)
programs
exiting
tracing
pstats module
pwd module
py_compile module
pyclbr module
Python expressions, evaluating
PYTHONPATH environment variable


[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

question mark (?)
fnmatch module and
glob module and
queue module
quopri module


[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

random module 2nd
rast (Sun raster) file formats
raw_input function
re module 2nd
readline module
reconvert module
recv method
recvfrom method
reference counts
regex module
regex_syntax module
regsub module
regular expressions
reload function
remote procedure call (RPC)
removedirs function
repr module 2nd
resource module
retr function, downloading files
rexec module
rfc822 module
rgb (SGI) file formats
RGB color values, using functions for converting to
rlcompleter module
rmtree function 2nd
robotparser module
rotor module 2nd
RPC (remote procedure call)
RuntimeError(StandardError)


[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

sched module
search function
select module
sendto methods
server sockets, creating
servers
HTTP 2nd
news, listing messages with nntplip module
setblocking method
setpgrp function
setprofiler function
SGML (Standard Generalized Markup Language)
sgmllib module 2nd
sha module 2nd
shell archives (Unix), creating with packmail module
shelve module
shlex module
reading configuration files
shutil module
removing directories
signal module
Simple Mail Transfer Protocol (SMTP)
SMTP (Simple Mail Transfer Protocol)
SimpleHTTPServer module
smtplib module
sndhdr module
socket module
SocketServer module
softspace attribute
sound files
soundex module
spawn method
sre module
Standard Generalized Markup Language (SGML)
sgmllib module and
StandardError(Exception)
start tags
start_tag method, using XML parsers
stat module
statcache module
statvfs module
stderr variable
stdin variable
stdout variable
streams
string module
StringIO module 2nd
StringIO objects
strings
match functions and
time values and
strop module
struct module 2nd
Sun AU audio files
sunau module
sunaudio module
symbol module
symbolic error codes
SyntaxError(StandardError)
sys module
sys.exit function
syslog module
system function
system logger facility (syslogd)
SystemError(StandardError)
SystemExit(Exception)


[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

TabError(IndentationError)
tabnanny module
telnetlib module
tempfile module
TemporaryFile function
termios module
text files (.dir), using dumbdbm module
thread module
threading module
threads
tiff file formats
time module
connecting time value to/from strings
execution of programs, timing with
time tuple to time values
timeclient module
timing module
token module
tokenize module
tools
traceback module
tty module
tuples, calling a function with arguments from
type codes in the marshal module
type function
type support modules
type wrappers
TypeError(StandardError)
types module
tzparse module


[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

ucnhash module
UnboundLocalError(NameError)
Unicode string, encoding XML files
unicodedata module
UnicodeError(ValueError)
Unix
command module and
dbm module and
packmail modules and
platform-specific modules
terminal control facilities and
Unix shell archives, creating with packmail module
unpickler
urllib module
urlopen function, using the urllib module
urlparse module
URLs, mapping between files
UserDict module 2nd
UserList module 2nd
UserString module 2nd
UTF-8, encoding XML file
util module
utilities
utime function
uu module


[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

ValueError(StandardError)


[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

wait function
walk function
WAV (Microsoft) audio files
wave module
Wave sound files
webbrowser module
well-formed tags in markup languages
what function, using sndhdr module
whatsound module
whichdb module
whrandom module 2nd
Windows
mapping between files and URLs
mmap module and
platform-specific modules
Wave sound files on
WindowsError(OSError)
_winreg module
winsound module
write method 2nd
write method, adding statements
Writer class


[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

xbm file formats
XDR (external data representation)
xdrlib module 2nd
XML (Extensible markup Language)
xml.parsers.expat module
xmllib module 2nd


[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

yellow pages (NIS) services
YIQ (video) color values, using functions for converting to


[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Y] [Z]

ZeroDivisionError(ArithmeticError)
ZIP formats
support for
zipfile module and
zipfile module
ZipInfo instances
zlib data compression module
zlib module