Dynamic Module Loading in Python

If you have a need to import a module in python and, for whatever reason, you will not know the name of the module until run-time, python provides the built-in __import__ function.

This allows you to put the name of the module to be imported into a string and assign the returned module object to any variable. This is can useful if your plan on dynamically importing a module whether it is stored in a configuration file, database, or in another way.

Here is an example:

module_name = "string"
module = __import__(module_name)

If you wish to use the “from module import…” method you will have a bit more work to do. At a basic level you will build a string which represents your import command, and then pass it to ‘exec‘ as such:

import_string = "from string import ascii_letters"
exec import_string

Here is an example of a second method which I used while modifying a custom solution to replace the default UNIX crontab. This specific script uses a set of configuration files to build a static host specific configuration object. This object is then pickled and pushed out to each host using rdist (tunnled through SSH).

import os
import sys
import glob
from cPickle import dump as pkl_dump
from crons.cron_config import CRONS
 
PATH = '/home/kevin/hosts'
DIRLIST = glob.glob(PATH + '/hostname*')
 
for ob in DIRLIST:
    CRONTAB = []
    HOSTNAME = ""
    if os.path.isdir(ob):
        HOSTNAME = ob.split('/')[1]
 
        for app in CRONS:
            exec 'from crons.' + app + '_crontab import APP_CRON'
            for entry in APP_CRON:
                if 'hostname' in entry:
                    if entry['hostname'] == HOSTNAME:
                        CRONTAB.append(entry)
                else:
                    continue
            del APP_CRON
 
        fo = open("%s" % ob + "/cron.pkl","wb")
        pkl_dump(CRONTAB,fo,1)
        fo.close()
        del fo
        del CRONTAB
        del HOSTNAME

There is also the ‘imp‘ module which exposes an interface to the mechanisms used to implement the import statement. This allows you to make a completely custom import method. I’ve included the example from the Python docs for completeness sake. This example emulates the built-in import statement.

import imp
import sys
 
def __import__(name, globals=None, locals=None, fromlist=None):
    # Fast path: see if the module has already been imported.
    try:
        return sys.modules[name]
    except KeyError:
        pass
 
    # If any of the following calls raises an exception,
    # there's a problem we can't handle -- let the caller handle it.
 
    fp, pathname, description = imp.find_module(name)
 
    try:
        return imp.load_module(name, fp, pathname, description)
    finally:
        # Since we may exit via an exception, close fp explicitly.
        if fp:
            fp.close()

As you can see there are many options when it comes to dynamically loading a module in Python. If any of you know of additional examples, or better ways to implement my examples, please leave a comment below.

Tags: ,

This entry was posted on Tuesday, May 29th, 2007 at 07:34:30 and is filed under feature. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

18 Responses to “Dynamic Module Loading in Python”

  1. Andrei Says:
    May 31st, 2007 at 09:46:09

    Hmmm, I see your visitors are lazy to leave any comments - like mine?

    We do the hard work, and they consume without feedback. Not nice…

    Well, at least we enjoy doing what we do.

    Thank you for this interesting part on Python

  2. kevin Says:
    May 31st, 2007 at 13:32:19

    Thanks for the note, hopefully more people will start leaving feedback so I can make even better posts.

    Kevin

  3. Lars Says:
    May 31st, 2007 at 22:30:27

    :$ … you’re right Andrei… this is good and useful info - I just bookmarked it ;-) - and we SHOULD all be better at commenting

    thanx Kevin

    We’re currently making a system (compiled to standalone with pyInstaller) for Mac, Lin & Win … do you have any experience in multi-platform standalone “*.exe” support for this? … can a module be “reloaded”, “unloaded”, etc. :-)

    thanx (again)

    - Lars

  4. kevin Says:
    June 1st, 2007 at 08:42:21

    Unfortunately, I have not had a chance to do much multi-platform work, other than some basic utility scripting.

    There is a way to reload a previously imported module, the built-in function reload() (http://docs.python.org/lib/built-in-funcs.html), though I do not know how well it will function with a standalone .exe

    I can’t find anything about specificly “unloading” a previously loaded module, but you can use “del” to remove the reference to it. Not sure if that helps or not.

    Thanks for the comments.

    Kevin

  5. Joseph Says:
    June 2nd, 2007 at 01:22:05

    Wow thanks for the tip :)

  6. kevin Says:
    June 2nd, 2007 at 09:14:32

    Glad it was helpful.

    Kevin

  7. Brennan Says:
    September 9th, 2007 at 16:49:33

    Just what I wanted, thanks. Part of an my experiment in moving modules about (name and location on a local network), to protect it from theft, while maintaining access at run time. An intermediate script queries a web server to get current name and location.

  8. kwalo Says:
    October 25th, 2007 at 05:49:00

    Thanks for the tip, I was looking for a way to store some configuration in python script and didn’t know about exec keyword, but config files are usually stored outside the python search path, so the fastest way to read them is to simply call:

    f = file(’/etc/myconfig.cfg’, ‘r’)
    exec f.read()
    f.close()

    All variables can be accessed from the current namespace, so it’s pretty easy to use, but can cause some problems.

  9. kevin Says:
    December 5th, 2007 at 14:44:31

    You can do the following:

    import sys
sys.path.insert(0,"/path/to/module")

    To add a config file to your python path at run time.

    Kevin

  10. Paul Says:
    December 16th, 2007 at 14:52:43

    I love this post! I went about the task of teaching myself basic python in my high school years, but never got far enough to figure this out. Kind of stopped and said “this was where someone who knew what they were doing would be really helpful…” Thanks for the flashback - and great blog! Just opened a new one of my own - and commenting is always hard. But you seem to be doing fine!

  11. kevin Says:
    December 17th, 2007 at 09:12:06

    Thanks :-)

    Hopefully I’ll get some more posts up soon.

    Kevin

  12. Abhijeet Says:
    December 25th, 2007 at 09:42:36

    Cool stuff. Was looking for this and getting to your site wasn’t difficult. Thanx!

  13. kevin Says:
    December 25th, 2007 at 16:47:14

    I’m glad it’s still a help. I’ve gotta get around to getting more content up here now!

    Kevin

  14. Radioactiveroach Says:
    January 6th, 2008 at 15:41:41

    Thanks I was looking for something exactly like this :)

  15. Alex Says:
    February 17th, 2008 at 10:47:40

    It might be not much relevant but there are so many sites like this one that have exactly the same problem. They are hard to read because of fixed width specified in pixels. These days most people has high-resolution wide screens. I personally find it difficult to read content that use pixels for font sizes and other things that don’t need that. I have to increase font size using browser’s feature but that doesn’t change content width that is in pixes causing content to be very unreadable. Two words per line at the best and only beginning of line for content elements that don’t use line wrapping. Not to mention 2/3 of screen width is wasted for no good reason.

    I like to read blogs and respect people who dedicate time and effort to share their knowledge and thoughts with others.

    Alex

  16. kevin Says:
    February 17th, 2008 at 18:46:45

    Thanks for the feedback Alex.

    Replacing the default WP theme is a top priority for me, as I have many of the same issues with it that you do.

    Kevin

  17. jones Says:
    July 14th, 2008 at 23:57:28

    nobody is ever going to want to post if you “require” an email.
    sorry if you get spam, but it’s the way things go!
    i liked some of your article, thanks!

  18. kevin Says:
    July 15th, 2008 at 13:53:00

    That’s pretty much standard for every Wordpress blog on the innerweb…

    I am glad that you enjoyed the post.

Leave a Reply