Archive for the ‘feature’ Category

Dynamic Module Loading in Python

Tuesday, May 29th, 2007

If you have a need to import a module in python and, for whatever reason, you will not know the name of the module until run-time, python provides the built-in __import__ function.

This allows you to put the name of the module to be imported into a string and assign the returned module object to any variable. This is can useful if your plan on dynamically importing a module whether it is stored in a configuration file, database, or in another way.

Here is an example:

module_name = "string"
module = __import__(module_name)

If you wish to use the “from module import…” method you will have a bit more work to do. At a basic level you will build a string which represents your import command, and then pass it to ‘exec‘ as such:

import_string = "from string import ascii_letters"
exec import_string

Here is an example of a second method which I used while modifying a custom solution to replace the default UNIX crontab. This specific script uses a set of configuration files to build a static host specific configuration object. This object is then pickled and pushed out to each host using rdist (tunnled through SSH).

import os
import sys
import glob
from cPickle import dump as pkl_dump
from crons.cron_config import CRONS
 
PATH = '/home/kevin/hosts'
DIRLIST = glob.glob(PATH + '/hostname*')
 
for ob in DIRLIST:
    CRONTAB = []
    HOSTNAME = ""
    if os.path.isdir(ob):
        HOSTNAME = ob.split('/')[1]
 
        for app in CRONS:
            exec 'from crons.' + app + '_crontab import APP_CRON'
            for entry in APP_CRON:
                if 'hostname' in entry:
                    if entry['hostname'] == HOSTNAME:
                        CRONTAB.append(entry)
                else:
                    continue
            del APP_CRON
 
        fo = open("%s" % ob + "/cron.pkl","wb")
        pkl_dump(CRONTAB,fo,1)
        fo.close()
        del fo
        del CRONTAB
        del HOSTNAME

There is also the ‘imp‘ module which exposes an interface to the mechanisms used to implement the import statement. This allows you to make a completely custom import method. I’ve included the example from the Python docs for completeness sake. This example emulates the built-in import statement.

import imp
import sys
 
def __import__(name, globals=None, locals=None, fromlist=None):
    # Fast path: see if the module has already been imported.
    try:
        return sys.modules[name]
    except KeyError:
        pass
 
    # If any of the following calls raises an exception,
    # there's a problem we can't handle -- let the caller handle it.
 
    fp, pathname, description = imp.find_module(name)
 
    try:
        return imp.load_module(name, fp, pathname, description)
    finally:
        # Since we may exit via an exception, close fp explicitly.
        if fp:
            fp.close()

As you can see there are many options when it comes to dynamically loading a module in Python. If any of you know of additional examples, or better ways to implement my examples, please leave a comment below.

Tags: ,
Posted in feature | 18 Comments »