4.25 File Locking Using a Cross-Platform API

Credit: Jonathan Feinberg, John Nielsen

4.25.1 Problem

You need to lock files in a cross-platform way between NT and Posix, but the Python standard library offers only platform-specific ways to lock files.

4.25.2 Solution

When the Python standard library itself doesn't offer a cross-platform solution, it's often possible to implement one ourselves:

import os

# needs win32all to work on Windows
if os.name == 'nt':
    import win32con, win32file, pywintypes
    LOCK_EX = win32con.LOCKFILE_EXCLUSIVE_LOCK
    LOCK_SH = 0 # the default
    LOCK_NB = win32con.LOCKFILE_FAIL_IMMEDIATELY
    _ _overlapped = pywintypes.OVERLAPPED(  )

    def lock(file, flags):
        hfile = win32file._get_osfhandle(file.fileno(  ))
        win32file.LockFileEx(hfile, flags, 0, 0xffff0000, _ _overlapped)

    def unlock(file):
        hfile = win32file._get_osfhandle(file.fileno(  ))
        win32file.UnlockFileEx(hfile, 0, 0xffff0000, _ _overlapped)
elif os.name == 'posix':
    from fcntl import LOCK_EX, LOCK_SH, LOCK_NB

    def lock(file, flags):
        fcntl.flock(file.fileno(  ), flags)

    def unlock(file):
        fcntl.flock(file.fileno(  ), fcntl.LOCK_UN)
else:
    raise RuntimeError("PortaLocker only defined for nt and posix platforms")

4.25.3 Discussion

If you have multiple programs or threads that may want to access a shared file, it's wise to ensure that accesses are synchronized, so that two processes don't try to modify the file contents at the same time. Failure to do so could corrupt the entire file in some cases.

This recipe supplies two functions, lock and unlock, that request and release locks on a file, respectively. Using the portalocker.py module is a simple matter of calling the lock function and passing in the file and an argument specifying the kind of lock that is desired:

LOCK_SH

A shared lock (the default value). This denies all processes write access to the file, including the process that first locks the file. All processes can read the locked file.

LOCK_EX

An exclusive lock. This denies all other processes both read and write access to the file.

LOCK_NB

A nonblocking lock. If this value is specified, the function returns immediately if it is unable to acquire the requested lock. Otherwise, it waits. LOCK_NB can be ORed with either LOCK_SH or LOCK_EX.

For example:

import portalocker
file = open("somefile", "r+")
portalocker.lock(file, portalocker.LOCK_EX)

The implementation of the lock and unlock functions is entirely different on Unix-like systems (where they can rely on functionality made available by the standard fcntl module) and on Windows systems (where they must use the win32file module, part of the very popular win32all package of Windows-specific extensions to Python, authored by Mark Hammond). But the important thing is that, despite the differences in implementation, the functions (and the flags you can pass to the lock function) behave in the same way across platforms. Such cross-platform packaging of differently implemented but equivalent functionality is what lets you write cross-platform applications, which is one of Python's strengths.

When you write a cross-platform program, it's nice if the functionality that your program uses is, in turn, encapsulated in a cross-platform way. For file locking in particular, this is helpful to Perl users, who are used to an essentially transparent lock system call across platforms. More generally, if os.name== just does not belong in application-level code. It should ideally always be in the standard library or an application-independent module, as it is here.

4.25.4 See Also

Documentation on the fcntl module in the Library Reference; documentation on the win32file module at http://ASPN.ActiveState.com/ASPN/Python/Reference/Products/ActivePython/PythonWin32Extensions/win32file.html; Jonathan Feinberg's web site (http://MrFeinberg.com).