Dynamic Module Loading in Python

This is a static copy of a post I originally made on 2007-05-29. I am currently working on an update to this post.

If you have a need to import a module in python and, for whatever reason, you will not know the name of the module until run-time, python provides the built-in __import__ function.

This allows you to put the name of the module to be imported into a string and assign the returned module object to any variable. This is can useful if your plan on dynamically importing a module whether it is stored in a configuration file, database, or in another way.

Here is an example:

module_name = "string"
module = __import__(module_name)

If you wish to use the "from module import ..." method you will have a bit more work to do. At a basic level you will build a string which represents your import command, and then pass it to 'exec' as such:

import_string = "from string import ascii_letters"
exec import_string

Here is an example of a second method which I used while modifying a custom solution to replace the default UNIX crontab. This specific script uses a set of configuration files to build a static host specific configuration object. This object is then pickled and pushed out to each host using rdist (tunnled through SSH).

import os
import sys

import glob
from cPickle import dump as pkl_dump
from crons.cron_config import CRONS

PATH = '/home/kevin/hosts'
DIRLIST = glob.glob(PATH + '/hostname*')

for ob in DIRLIST:
    CRONTAB = []

    HOSTNAME = ""
    if os.path.isdir(ob):
        HOSTNAME = ob.split('/')[1]

        for app in CRONS:
            exec 'from crons.' + app + '_crontab import APP_CRON'
            for entry in APP_CRON:
                if 'hostname' in entry:
                    if entry['hostname'] == HOSTNAME:
                        CRONTAB.append(entry)

                else:
                    continue
            del APP_CRON

        fo = open("%s" % ob + "/cron.pkl","wb")

        pkl_dump(CRONTAB,fo,1)
        fo.close()
        del fo
        del CRONTAB
        del HOSTNAME

There is also the 'imp' module which exposes an interface to the mechanisms used to implement the import statement. This allows you to make a completely custom import method. I've included the example from the Python docs for completeness sake. This example emulates the built-in import statement.

import imp
import sys

def __import__(name, globals=None, locals=None, fromlist=None):
    # Fast path: see if the module has already been imported.
    try:
        return sys.modules[name]

    except KeyError:
        pass

    # If any of the following calls raises an exception,
    # there's a problem we can't handle -- let the caller handle it.

    fp, pathname, description = imp.find_module(name)

    try:
        return imp.load_module(name, fp, pathname, description)
    finally:
        # Since we may exit via an exception, close fp explicitly.
        if fp:
            fp.close()

As you can see there are many options when it comes to dynamically loading a module in Python. If any of you know of additional examples, or better ways to implement my examples, please leave a comment below.

import imp, sys # create a module modname = 'mypackage.mymod' if sys.modules.has_key(modname): mod = sys.modules[modname] else: # create a module mod = imp.new_module(modname) # store it in sys.modules sys.modules[modname] = mod # get the package module packagemod = __import__( 'mypackage, globals(), locals(), ['mypackage'] ) # add module there too setattr(packagemod, 'mymod', mod) # our namespace is the module dictionary namespace = mod.__dict__ # test whether this has been done already if not hasattr(mod, 'myClass'): # compile and exec dynamic code in the module exec compile(pythoncode, '', 'exec') in namespace # get the extension extension = namespace.get('myClass') # instanciate the object myOb = extension()

#!/usr/bin/env python import imp import sys def import_module(path,name,unique_name, globals=None, locals=None, fromlist=None): # Fast path: see if unique_name has already been imported. try: return sys.modules[unique_name] except KeyError: pass sys.path.append(path) fp, pathname, description = imp.find_module(name) sys.path.pop(-1) try: return imp.load_module(unique_name,fp, pathname, description) finally: # Since we may exit via an exception, close fp explicitly. if fp: fp.close() mod1=import_module("mod1","module","module1") mod2=import_module("mod2","module","module2") mod2.test() mod1.test()

Andrei Says:
May 31st, 2007 at 09:46:09
Hmmm, I see your visitors are lazy to leave any comments - like mine?

We do the hard work, and they consume without feedback. Not nice...

Well, at least we enjoy doing what we do.

Thank you for this interesting part on Python
kevin Says:
May 31st, 2007 at 13:32:19
Thanks for the note, hopefully more people will start leaving feedback so I can make even better posts.

Kevin
Lars Says:
May 31st, 2007 at 22:30:27
:$ ... you're right Andrei... this is good and useful info - I just bookmarked it ;-) - and we SHOULD all be better at commenting

thanx Kevin

We're currently making a system (compiled to standalone with pyInstaller) for Mac, Lin & Win ... do you have any experience in multi-platform standalone "*.exe" support for this? ... can a module be "reloaded", "unloaded", etc. :-)

thanx (again)

- Lars
kevin Says:
June 1st, 2007 at 08:42:21
Unfortunately, I have not had a chance to do much multi-platform work, other than some basic utility scripting.

There is a way to reload a previously imported module, the built-in function reload() (http://docs.python.org/lib/built-in-funcs.html), though I do not know how well it will function with a standalone .exe

I can't find anything about specificly "unloading" a previously loaded module, but you can use "del" to remove the reference to it. Not sure if that helps or not.

Thanks for the comments.

Kevin
Joseph Says:
June 2nd, 2007 at 01:22:05
Wow thanks for the tip :)
kevin Says:
June 2nd, 2007 at 09:14:32
Glad it was helpful.

Kevin
Brennan Says:
September 9th, 2007 at 16:49:33
Just what I wanted, thanks. Part of an my experiment in moving modules about (name and location on a local network), to protect it from theft, while maintaining access at run time. An intermediate script queries a web server to get current name and location.
kwalo Says:
October 25th, 2007 at 05:49:00
Thanks for the tip, I was looking for a way to store some configuration in python script and didn't know about exec keyword, but config files are usually stored outside the python search path, so the fastest way to read them is to simply call:
```
f = file('/etc/myconfig.cfg', 'r')
exec f.read()
f.close()
```
All variables can be accessed from the current namespace, so it's pretty easy to use, but can cause some problems.
kevin Says:
December 5th, 2007 at 14:44:31
You can do the following:
```
import sys
sys.path.insert(0,"/path/to/module")
```
To add a config file to your python path at run time.

Kevin
Paul Says:
December 16th, 2007 at 14:52:43
I love this post! I went about the task of teaching myself basic python in my high school years, but never got far enough to figure this out. Kind of stopped and said "this was where someone who knew what they were doing would be really helpful..." Thanks for the flashback - and great blog! Just opened a new one of my own - and commenting is always hard. But you seem to be doing fine!
kevin Says:
December 17th, 2007 at 09:12:06
Thanks :-)

Hopefully I'll get some more posts up soon.

Kevin
Abhijeet Says:
December 25th, 2007 at 09:42:36
Cool stuff. Was looking for this and getting to your site wasn't difficult. Thanx!
kevin Says:
December 25th, 2007 at 16:47:14
I'm glad it's still a help. I've gotta get around to getting more content up here now!

Kevin
Radioactiveroach Says:
January 6th, 2008 at 15:41:41
Thanks I was looking for something exactly like this :)
daya Says:
September 3rd, 2008 at 21:20:44
Hi,
This is what I am looking for. I was breaking my head on how to load modules dynamically, Good work

Regards
Daya Sankar

Peter Says:
November 18th, 2008 at 09:37:54

Thanks for the hints, but that is not really dynamic. I was looking for a way to load modules from scratch without files and populate them with dynamically loaded extensions. Since I didn't find anything on the web, here my solution.
- mypackage is a real existing package, but is not necessary
- pythoncode is a variable containing the definition of 'myClass' as string (loaded from a file)

Tim Stoop Says:
March 11th, 2009 at 13:39:10
Hey thanks for this! Exactly what I was looking for.
Tim Stoop Says:
March 11th, 2009 at 13:48:33
Found this in the manual (that you linked to), seems a little bit cleaner than the exec:

On the other hand, the statement 'from spam.ham import eggs, sausage as saus' results in

_temp = __import__('spam.ham', globals(), locals(), ['eggs', 'sausage'], -1)
eggs = _temp.eggs
saus = _temp.sausage

Daniel Fiala Says:
March 24th, 2009 at 13:37:01

Thanks a lot for the nice exapmles, They helpend me a lot in setting up a dynamic loading with identical names...

Here the code that I derived:

As example env two directories are present that both include a __init__.py and a module.py providing different implementations of the functions test...

Nick Says:
May 7th, 2009 at 04:34:56
Thanks for this tip.

It is useful for overcoming the JVM memory limits of Jython? Why use Jython not python? It's what you're stuck with when using The Grinder

I'm load testing a website as thousands of users, with the user data held in one large array. Python can cope with this, but Jython can't, so I'll use your tip to break the array up into smaller arrays, one per Grinder agent process, and each agent will import the users it is going to simulate based on it's own name.