Loading Javascript Modules

Loading Javascript Modules

Sooner or later, every Javascript developer encounters the Module Pattern. It isn’t always used for modules in the conventional sense, but even when it is used for modules, the files in which those modules reside are still often loaded by carefully hand-listing script-elements, in dependency order, in a HTML page. I have just been through the process of moving one of my own projects from a naive file-as-module project structure (with implicit dependencies and explicit loading) to a structure with explicit modules, explicit import/export dependencies and automated loading in dependency order.

This is an explanatory post, outlining the process. Both a possible sequence of refactoring steps and a possible implementation of a module loader library are presented. I found it fun to figure out the module loader code myself, and the code provided should work well enough (tested Opera, Firefox, Safari; IE), but it is intended for reading, not production use (refer to existing module loader libraries instead). The intent of this post is threefold:

  • to make more readers aware of module patterns and module-loader libraries,
  • to look under the hood to see how such a libary might come into being,
  • and to illustrate the refactoring process: how use of such a module loader might be introduced into an existing multi-module project.

What is the Module Pattern, anyway?

The name is somewhat overloaded, so let me be specific about which of the many uses I refer to (the names used here are non-standard, descriptive):

Scoped Block with Sideeffects

  (function() {

   // local namespace for private
   // variables and functions
   var localVar;
   function localFun() {};

   // use/manipulate non-local things;
   // can use local vars/funs here
   globalVar = ..;

  })();

Scoped Block with Return

  var someThing = (function() {

     // local namespace for private
     // variables and functions
     var localVar;
     function localFun() {};

     // can use local vars/funs here
     return someThing;

    })();

Parameterized Scoped Block with Sideeffects

  (function(window, ..) {
   // local namespace for private
   // variables and functions

   var localVar;
   function localFun() {};

   // use/manipulate non-local things,
   // as provided via parameters;
   // can use local vars/funs here
   window.globalVar = ..;

  })(window, ..);

Parameterized Scoped Block with Return

  var someThing = (function(window, ..) {

     // local namespace for private
     // variables and functions

     var localVar;
     function localFun() {};

     // can use local vars/funs here,
     // as well as non-local parameters
     return someThing;

    })(window, ..);

These are all variations on the same theme: block scoping. Javascript (pre 1.7, where let can be used for block scoped definitions) does not have it – local scopes are delimited by functions, not by blocks (combined with hoisting of declarations, this can be confusing to those of us with experience in other programming languages). So the idea is to use an anonymous function to provide a block-like scope: the function is applied immediately – all we use it for are its scope delimiting abilities.

These variations can be mixed, the someThing returned can be arbitrarily complex, and the purposes of using such simulated block-scopes varies widely. It is somewhat unfortunate that the intended uses (blocks, singletons, namespaces, modules) have been mixed up with the technique used to achieve those aims.

For the purposes of this post, modules will come closest to the following pattern, though we will have to break it up a little, to facilitate separate loading:

Modules as Scoped Parameterized Blocks with Returns

  var module = (function(import1, import2, ..) {

     // local namespace for private variables and functions;
     // can use local vars/funs here, as well as non-local parameters

     var localVar;
     function localFun(..) {};

     function exportFun1(..) {};
     function exportFun2(..) {};

     return { export1: exportFun1
            , export2: exportFun2
            };

    })(import1, import2, ..);

We use a Parameterized Scoped Block with Return, where parameters will usually be other modules (dependencies/imports), and the thing to return will be an object literal referencing (and thereby exporting) module-local definitions.

Introducing modules, step 0 – separate Files are not enough

Now that we have an idea of what modules with explicit imports and exports might look like, we can start introducing such modules into our projects. Even fairly complex Javascript applications often start out as some script added to a (X)HTML page. The code grows, moves from inline to external script elements, and from a single script to several scripts working together. After organizing the code into modules (aka files, at this stage;-), that often leaves a main page looking somewhat like this:

  <html>
  <head>

  <script src="utils.js"></script>
  <script src="debug.js"></script>
  <script src="module1.js"></script>
  <script src="module2.js"></script>
  <script src="main.js"></script>

  </head>

  <body>
  </body>
  </html>

All modules are loaded explicitly, carefully sorted in dependency order (which is left implicit, probably accompanied by comments like “load this first” or “load utils.js before this”), and as to the effect of these modules on the global name space, who knows? If you want to reuse this Javascript application, you have to repeat the whole sequence of script elements, if I want to restructure the application, all clients have to change their invocation sequences, and if clients want to use my application in combinaton with other such applications, as part of something bigger, conflicts are not far away.

The task at hand is to refactor a multi-source project like this, with manual loading, implicit dependencies, and global namespacing into a multi-module project with explicit dependencies, automatic loading, and minimal impact on the global namespace.

When I started this refactoring, I already had a good idea on what my files were doing to the global namespace: the application introduces some objects with a short prototype chain, and kept all local variables out of the way, so all that remained where the classes themselves (with methods and properties attached to their prototypes, so not polluting the global namespace), some debug and utility definitions (logging to an in-page console, constructing namespaced elements from object literals, that sort of thing). Not bad, I thought, but after the refactoring, none of these need to end up in the global namespace.

If you feel confident about knowing the way your project scripts use the global namespace, you might be tempted to introduce modules in a single step. That might even work, but if things go wrong, it helps to limit the kind of things that can go wrong in each step, which is what the suggested sequence of steps is trying to achieve. If you are trying to introduce modules to a code-base you do not feel quite as familiar with, it is even more important that you can proceed in small steps. If you already use a module pattern, you should be able to skip over most of the initial steps, to module loading and dependency sorting.

Before you continue, if you are trying to do something similar to one of your projects, you should set up a test suite. We are going to change around definitions and file loading order, and while Javascript gives us the means to define our own module system, it will not even alert us of any accidental breakage in the scope chains until we actually run the code, let alone unintended lapses in functionality! Since one goal is to reduce global namespace pollution, you might also want to add a test that measures this (at minimum, loop over window before and after module loading, and report any additions; tools like JSLint also try to record global variables, but they usually do not execute the code to find bindings).

In my own refactoring, I first moved script loading from HTML to Javascript, and then figured out the rest. That worked well for the versions of Opera and Firefox I am using, but there are differences in the way program-generated external script elements are executed between browsers. It seems that the standard does not guarantee much about the order of execution of such scripts, and browsers are increasingly making use of this (see also the dynamic script execution order summary and discussion on the whatwg wiki). For such browsers, it is safer to leave the dynamic module loading to a later refactoring step, when we already have modules in place, and some shared code to control their execution.

Introducing modules, step 1 – wrap scripts in module pattern

If we want to avoid exposing exports globally, we need to make exports explicit and get a handle on where exactly which exports will be needed, so that we can link up exports and imports directly, without polluting the global namespace. To begin with, we need some structure that makes our scripts less dependent on the global name space. This is where we start encountering module patterns for real. However, there are a few small twists:

  1. we do not want the module’s code to run when the script is loaded, because
  2. we need to be able to discover and load any dependencies first, so
  3. we need to build a record of modules we have loaded
  4. and run each module’s code when all dependencies have been loaded

Point 1 is easily taken care of: we still use a function to wrap our module, but we only apply it after we have got hold of all its dependencies (point 4). Point 2 means that we need to list the dependendencies for a module outside the module function. To cover point 3, we will simply use the script file name as the module name, and build an object mapping module names to modules and module information. We will build a small library of module handling code to organize all this, and introduce features gradually:

// modules.js
(function () {

  var modules = {}; // private record of module data

  // modules are functions with additional information
  function module(name,imports,mod) {

    // record module information
    window.console.log('loading module '+name);
    modules[name] = {name:name, imports: imports, mod: mod};

    // just execute module code right away, no imports/exports yet
    mod();
  }

  // export module wrapper
  window.module = module;

})()

We have a module-construct with three parameters (module/file name,list of dependencies, module function). So far, this is nothing more than a named version of our module pattern, without support for imports or exports. Each module is recorded and logged, but then directly executed. A Scoped Block with Sideeffects is used to provide a local namespace for the modules library itself. The modules library is loaded before our project modules

  <html>
  <head>

  <script src="modules.js"></script>

  <script src="utils.js"></script>
  <script src="debug.js"></script>
  <script src="module1.js"></script>
  <script src="module2.js"></script>
  <script src="main.js"></script>

  </head>

  <body>
  </body>
  </html>

and we can start wrapping our scripts, using the module construct (which is just a call to the module function). An example module using this construct looks like this, where the original script remains largely unchanged in the body of the function, and the script file name is used as the module name:

// module2.js
module('module2.js',[],function() {

  // module2 definitions, split into local and global/exported definitions

  // local definitions

  var localVar = ..;
  function localFun ..

  // exports
  window.someFunction = ..;
  window.SomeObject = ..;

});

At this stage, we still use the global namespace to provide definitions and to access dependencies, so we do not have explicit import parameters nor do we return an export object, but we do have a module-local namespace at hand, so we can start to keep temporary/local definitions from leaking out. In fact, if we declare all our variables, we need to take explicit steps to add any definitions to the global namespace, such as assigning them to window. This also applies to any functions we define.

Keeping track of which definitions need to be global, and which can become local, is the main difficulty of this step. Our tests are important here, to show us if we accidentally make something local that has to be global.

Introducing modules, step 2 – make exports explicit

Handling module exports by directly assigning them to the global window object does not make the exports explicit (they only differ from other assignments in the destination). More importantly, it leaves assignments to global objects in modules (we want to factor that out, so that modules have no global assignments left, and the assignment code can be shared and modified without having to modify all modules), and gives no place to get the exports from when we switch away from poluting the global namespace (we need explicit export records to pass on to dependent modules).

Taking all these together, our modules lose their global assignments and acquire explicit export records:

// module2.js
module('module2.js',[],function() {

  // module2 definitions, split into local and exports

  // local definitions

  var localVar = ..;
  function localFun ..

  function someFunction ..;
  function SomeObject ..;

  // exports
  return { someFunction: someFunction
         , SomeObject:   SomeObject
         };

});

and the assignments move out to our modules library:

// modules.js
(function () {

  var modules = {}; // private record of module data

  // modules are functions with additional information
  function module(name,imports,mod) {

    // record module information
    window.console.log('loading module '+name);
    modules[name] = {name:name, imports: imports, mod: mod};

    // execute module code, record exports
    var exports = mod();

    // copy exports to global namespace
    for (var exp in exports)
      window[exp] = exports[exp];
  }

  // export module wrapper
  window.module = module;

})()

Introducing modules, step 3 – make import dependencies explicit

Now it is time to make the imports explicit, too, so that we may drop those assignments to the global namespace:

// module2.js
module('module2.js',['utils.js','module1.js']
                   ,function(utils,module1) {

  // module2 definitions, split into local and exports

  // local definitions, with explicitly qualified import references

  var localVar = ..;
  function localFun ..

  function someFunction ..;
  function SomeObject ..;

  // exports
  return { someFunction: someFunction
         , SomeObject:   SomeObject
         };

});

Finding all implicit references to items from imports, and explicitly qualifying them with their module of origin (eg, Utility becomes utils.Utility, AnotherObject becomes module1.AnotherObject) requires careful inspection. We can leave the global assignments in place while we are still adding import dependencies. Once we start removing the global assignments in our module loading code and pass imports explicitly instead, overlooked implicit import references should fail (if they refer to items previously exported to the global namespace), but we might need our testsuite to spot those failures for us!

// modules.js
(function () {

  var modules = {}; // private record of module data

  // modules are functions with additional information
  function module(name,imports,mod) {

    // record module information
    window.console.log('loading module '+name);
    modules[name] = {name:name, imports: imports, mod: mod};

    // collect import dependencies
    var deps = []; 
    for (var i in imports)
      deps.push(modules[imports[i]].linked);

    // execute module code, pass imports, record exports
    modules[name].linked = mod.apply(null,deps);

    // stop copying exports to global namespace
    // var exports = modules[name].linked;
    // for (var exp in exports)
    //   window[exp] = exports[exp];
  }

  // export module wrapper
  window.module = module;

})()

One anticipated consequence of dropping the global assignments of exports is that the global namespace is no longer polluted by these exports (only module itself needs to be global). We can still access these exports just fine, though: we just define an inline module that has the modules we want as import dependencies:

  <html>
  <head>

  <script src="modules.js"></script>

  <script src="utils.js"></script>
  <script src="debug.js"></script>
  <script src="module1.js"></script>
  <script src="module2.js"></script>
  <script src="main.js"></script>
  <script>
  module('client',['main.js','debug.js']
                 ,function(main,debug) {

    var result = main.doSomething();
    debug.log(result);

  });
  </script>
  </head>

  <body>
  </body>
  </html>

If we would like to expose some module-based functionality for use outside modules, we could also assign to the global namespace from within such an inline module. If we do not need to do so, only the module function is added to the global namespace (as our namespace test code should confirm). If the code is to manipulate the DOM, the inline module might need to move into the document body, or register an onload listener.

Up to here, we have added a module pattern to our project scripts, making imports and exports explicit. It has taken us several steps to get here, to keep the scope of upheaval (and the nature of accidental breakage) limited in each step. If your project already uses a module pattern in a similar way, you can go to this stage in a single step, by factoring the module pattern aspects into two parts, one to remain in the module (module names, listing of import dependencies, local namespace, listing of exports), the other one to move into a modules library (when and how to run the module code, fetching and providing import dependencies, storing module exports).

We have already seen some advantages to this way of factoring the module pattern – it puts the control over module execution and linking in a single place. This will help us as we proceed to take control of module loading as well. We can now make changes to the way our modules are loaded and linked without further modifications to the module sources. Only the modules library and the HTML client will be affected in the remaining refactoring steps.

Introducing modules, step 4 – move module loading to Javascript

We do not really want to expose our internal project file structure to every client HTML page. Those pages should only need to load one script for our project, so that clients are not affected every time we reorganize our internal structure.

  <html>
  <head>

  <script src="modules.js"></script>

  <!-- project script loading has moved to modules.js -->

  </head>

  <body>
  </body>
  </html>

Let us assume, for the moment, that modules.js not only defines our modules library code, but also triggers loading of all project files, by adding the script elements dynamically (be careful here: loadModule only triggers the script loading, which happens asynchronously). Naively, we might expect the code to look like this:

// modules.js
(function () {

  ..

  // trigger module loading by adding script element
  function loadModule(mod) {
    var element = document.createElement('script');
    element.setAttribute('type','text/javascript');
    element.setAttribute('src',mod);
    document.getElementsByTagName('head')[0].appendChild(element);
  }

  // trigger module loading
  // WARNING: this naive approach does not work cross-browser,
  //          as load order not guaranteed! 
  loadModule("utils.js");
  loadModule("debug.js");
  loadModule("module1.js");
  loadModule("module2.js");
  loadModule("main.js");

})()

In some browsers, this simple approach can actually work, because external scripts are executed in the order in which their script elements are added to the document. In other browsers, however, such external scripts will be executed in the order in which they become available, and that appears to follow the current specification (note: for script elements that are present in the original HTML document, there are two attributes – defer and async – that give further control; what these attributes should mean for script-inserted script elements is still being worked out, however). Unfortunately, those latter browsers will also not block document processing while waiting for the external scripts, so we cannot simply add a single onload listener to gather and link all modules when everything is ready (that listener could be triggered before all external scripts have been processed). In general, one might have to add event listeners to each of the script elements, and have those event listeners coordinate to figure out when all scripts have been loaded (this problem affects only script-inserted external scripts, not static script elements, unless the latter are explicitly marked async or defer).

Fortunately, we already have all the ingredients in place for taking control of module linking even without load-order guarantees:

  • We move creation of script elements to Javascript (new function loadModule). Script-inserted external scripts are loaded and executed asynchronously, but we record the order of calls to loadModule.
  • We do not have to link and run module code when we first encounter the module, but can leave that until all dependencies have become available (some code moves from module to the new linkModules).
  • We keep a record of modules scheduled for loading, and we keep track of modules becoming available (in the new loadModule and loadedModule).
  • If the current module was the last pending load in a given group of dependencies, we trigger a separate module linking and execution phase (module calls loadedModule, which may call linkModules, which links modules in the order in which they were loaded).

Here is the refined modules library code:

// modules.js
(function () {

  var modules = {}, // private record of module data
      order   = []; // private record of module load order

  // modules are functions with additional information
  function module(name,imports,mod) {

    // record module information
    window.console.log('found module '+name);
    modules[name] = {name:name, imports: imports, mod: mod};

    // allow for inline modules, which do not come via loadModule
    if (!(name in order)) order.push(name);

    // check whether this was the last module to be loaded
    // in a given dependency group
    loadedModule(name);
  }

  // trigger module loading by adding script element
  function loadModule(mod) {

    if (modules[mod])
      return;             // don't load the same module twice
    else
      modules[mod] = {};  // mark module as currently loading

    // add a script element to document head, with module as src
    var element = document.createElement('script');
    element.setAttribute('type','text/javascript');
    element.setAttribute('src',mod);
    document.getElementsByTagName('head')[0].appendChild(element);

    // keep record of loading order
    order.push(mod);
  }

  // check whether this was the last module to be loaded
  // in a given dependency group;
  // if yes, start linking and running modules
  function loadedModule(mod) {
    window.console.log('finished loading: '+mod);

    // collect modules marked as currently loading
    var pending=[];
    for (var m in modules)
      if (!modules[m].name) pending.push(m);

    // if no more modules need to be loaded, we can start 
    // linking the modules together
    if (pending.length===0) {
      window.console.log('all done loading');
      linkModules();
    } else {
      window.console.log('loads pending: '+pending.join(', '));
    }
  }

  // link and run loaded modules, keep record of results
  function linkModules() {
    window.console.log('linking modules');

    // link modules in loading order
    for (var nextName in order) {
      var name    = order[nextName];
      var module  = modules[name];
      var imports = module.imports;

      if (module.linked) {
        window.console.log('already linked '+name);
        continue;
      } 
      window.console.log('linking module '+name);

      // collect import dependencies
      var deps = []; 
      for (var i in imports)
        deps.push(modules[imports[i]].linked);

      // execute module code, pass imports, record exports
      modules[name].linked = module.mod.apply(null,deps);
    }
  }

  // export module wrapper
  window.module = module;

  // trigger module loading for our project
  // NOTE: we assume that all these loadModule calls are executed
  //       before any of the script elements they create; otherwise,
  //       we have a possible race condition (linkModules could be 
  //       called early, because not all dependencies are marked as
  //       currently loading yet)
  loadModule("utils.js");
  loadModule("debug.js");
  loadModule("module1.js");
  loadModule("module2.js");
  loadModule("main.js");

  // just calling linkModules here would not work, as we have only 
  // added the script elements, the scripts could still be loading;

  // calling linkModules in document onload would not work
  // in browsers which do not stop parsing while script-inserted
  // external scripts are loading;

  // therefore, we call linkModules when all modules in a dependency
  // group have been loaded, as checked by loadedModule;

})()

That is quite a bit of new code – since it is important to see how the pieces play together, I have used extensive inline comments instead of mixing code and blog fragments here. Most of the added complexity comes from splitting up tasks over several phases/functions, the way these functions interact, and the record keeping.

Introducing modules, step 5 – sort imports in dependency order

Looking at modules.js, the sequence of loadModule calls sticks out, for two reasons: it is the only part of that file that is specific to our project, and it carefully hand-codes a dependency-sorted loading order (which our modules library dutifully records and preserves when linking the modules together). Since we now have explicit import/export information, we can do both dependency discovery and dependency sorting in our module loader library, removing the explicit listing of dependencies as a potential source of coding errors.

Dependency discovery is easy – we just trigger loading of import dependencies for each module we encounter, in function module. When we come to processing the dependencies, they will recursively trigger loading of their own dependencies.

// modules.js
(function () {

  var modules = {}; // private record of module data

  // modules are functions with additional information
  function module(name,imports,mod) {

    // record module information
    window.console.log('found module '+name);
    modules[name] = {name:name, imports: imports, mod: mod};

    // trigger loading of import dependencies
    for (var imp in imports) loadModule(imports[imp]);

    // check whether this was the last module to be loaded
    // in a given dependency group
    loadedModule(name);
  }

In loadModule, we just drop the variable order and the tracking of loading order (note that we are already protected against repeated loads), loadedModule remains entirely unchanged.


  // trigger module loading by adding script element
  function loadModule(mod) {

    if (modules[mod])
      return;             // don't load the same module twice
    else
      modules[mod] = {};  // mark module as currently loading

    // add a script element to document head, with module as src
    var element = document.createElement('script');
    element.setAttribute('type','text/javascript');
    element.setAttribute('src',mod);
    document.getElementsByTagName('head')[0].appendChild(element);

    // no longer keep record of loading order
    // order.push(mod);
  }

  // check whether this was the last module to be loaded
  // in a given dependency group;
  // if yes, start linking and running modules
  function loadedModule(mod) {
    window.console.log('finished loading: '+mod);

    // collect modules marked as currently loading
    var pending=[];
    for (var m in modules)
      if (!modules[m].name) pending.push(m);

    // if no more modules need to be loaded, we can start 
    // linking the modules together
    if (pending.length===0) {
      window.console.log('all done loading');
      linkModules();
    } else {
      window.console.log('loads pending: '+pending.join(', '));
    }
  }

The next important changes are in linkModules, which calls a new function dependency_sort. This sorts modules in dependency order: we keep selecting modules for which all dependencies have already been sorted, pushing back modules for which dependencies are still pending. Sorting terminates if either all modules have been sorted or a dependency cycle has been discovered.

  // Sort modules by dependencies (dependents last),
  // returning sorted list of module names.
  function dependency_sort(modules) {

    var pending = [],   // modules remaining to be sorted
        sorted = [],    // modules already sorted
        been_here = {}; // remember length of pending list for each module 
                        // (if we revisit a module without pending
                        //  getting any shorter, we are stuck in a loop)

    // preparation: linked modules do not need to be sorted,
    //              all others go into pending
    for (var name in modules)
      if (modules[name].linked)
        sorted.push(name);  // allready linked by a previous run
      else
        pending.push(name); // sort for linking (after its dependencies)

    // has mod been sorted already?
    function issorted(mod){
      var result = false;
      for (var s in sorted) result = result || (sorted[s]===mod);
      return result;
    }

    // have all dependencies deps been sorted already?
    function aresorted(deps){
      var result = true;
      for (var d in deps) result = result && (issorted(deps[d]));
      return result;
    }

    // repeat while there are modules pending
    while (pending.length>0) {

      // consider the next pending module
      var m = pending.shift();

      // if we've been here and have not made any progress, we are looping
      // (no support for cyclic module dependencies)
      if (been_here[m] && been_here[m]<=pending.length)
        throw("can't sort dependencies: "+sorted+" < "+m+" < "+pending);
      else
        been_here[m] = pending.length;

      // consider the current module's import dependencies
      var deps = modules[m].imports;
      if (aresorted(deps))
        sorted.push(m);  // dependencies done; module done
      else
        pending.push(m); // some dependencies still pending;
                         // revisit module later
    }

    return sorted;
  }

The changes in linkModules are minor

  // link and run loaded modules, keep record of results
  function linkModules() {
    window.console.log('linking modules');

    // sort modules in dependency order
    var sortedNames = dependency_sort(modules);

    // link modules in dependency order
    for (var nextName in sortedNames) {
      var name    = sortedNames[nextName];
      var module  = modules[name];
      var imports = module.imports;

      if (module.linked) {
        window.console.log('already linked '+name);
        continue;
      } 
      window.console.log('linking module '+name);

      // collect import dependencies
      var deps = []; 
      for (var i in imports)
        deps.push(modules[imports[i]].linked);

      // execute module code, pass imports, record exports
      modules[name].linked = module.mod.apply(null,deps);
    }
  }

  // export module wrapper
  window.module = module;

  // trigger module loading for our project
  // NOTE: we assume that loadModule calls are executed before the 
  //       script elements they create; otherwise, we have a possible 
  //       race condition (linkModules could be called early, because 
  //       not all dependencies are marked as currently loading yet)
  // loadModule("utils.js");
  // loadModule("debug.js");
  // loadModule("module1.js");
  // loadModule("module2.js");
  loadModule("main.js");

  // just calling linkModules here would not work, as we have only 
  // added the script elements, the scripts could still be loading;

  // calling linkModules in document onload would not work
  // in browsers which do not stop parsing while script-inserted
  // external scripts are loading;

  // therefore, we call linkModules when all modules in a dependency
  // group have been loaded, as checked by loadedModule;

})()

Note how the import dependencies of main.js no longer need to be loaded explicitly – the modules library handles that now, loading what is needed and linking dependencies in the right order.

Introducing modules, step 6 – reusable module loading and inline modules

At this stage, we can drop the remaining project-specific line from modules.js – we have extracted all the module loading and linking code in a reusable library! To trigger project loading in our HTML client page, all we need to do is to write an inline module, listing the dependencies we would like to use, and the code we would like to call, and our modules library will do the rest:

  <html>
  <head>

  <script src="modules.js"></script>

  </head>

  <body>

  <script>
  module('client',['main.js','debug.js']
                 ,function(main,debug) {

    var result = main.doSomething();
    debug.log(result);

  });
  </script>

  </body>
  </html>

Note that the inline module is defined in a script element in the document body, assuming that our project will want to manipulate the DOM (defining an onload listener might be an alternative). If we would like to expose some module-based functionality for use outside modules, we could also assign to the global namespace from within such an inline module. If we do not need to do so, only the module function is added to the global namespace, so our module loader library enables us to keep tight control over namespace pollution (an improvement, even compared to a disciplined use of prototype methods and local module patterns).

References

I hope you have enjoyed this little tour of Javascript modules, refactoring Javascript projects to use modules, and looking under the hood of a simple module loader library. If this tour has made you want to try these ideas in your project, you do not have to start with my sample code. The ideas are not new, and once you know what to look for, you can find module loader support or discussion in several of the popular Javascript frameworks:

  • RequireJS – a framework-independent module loader library (can also be used with jQuery, which can load individual scripts, but has no module loader). Also has quite a bit of discussion of API rationale and implementation options, as well as comparison with node and commonjs modules, and dojo and YUI module loaders, not to mention a blog with further discussion (the RequireJS developer is very active on the topic of cross-toolkit Javascript module loading). Go there for further details and options!-)
  • dojo toolkit – has a package system with provide and require.
  • YUI 3 – has the YUI Loader in the YUI Global Object, to load its own component dependencies. The YUI theater also has an interesting video on Scalable JavaScript Application Architecture, discussing extensive modularisation of Javascript applications.
  • CommonJS seems to tend towards synchronously loaded modules in server-side Javascript, but has a proposal for asynchronously loaded modules. The CommonJS mailing list archive has lots of discussion of modules and packages.
  • LABjs seems to try and support parallel script loading with ordered script execution, for arbitrary Javascript code (without requiring module patterns with explicit dependencies or separated module loading and execution), leading to problems as browsers are moving to follow the HTML5 spec for script-inserted external scripts. Proposals and discussion are under way (HTML5 ticket).
  • A ScriptJunkie article in which the LABjs and RequireJS developers introduce their libraries.

Bonus steps

With the core modules library in place, there are various interesting extensions, including

  • replace the file-name-as-module-name-tags with an indirection layer, allowing configuration of the mapping from abstract module names to concrete module locations; this can also prepare for loading non-Javascript modules/resources;
  • loading CSS files or other resources as dependencies, so we can bundle them with our Javascript files (clients still need to be able to override our default styles, but our Javascript code can rely on those defaults being available);
  • using the module loader to load non-module Javascript sources, providing external scaffolding to enforce script execution order constraints without modifying the scripts to be loaded; one problem here is when non-module Javascript is executed right after loading – to enforce execution order, we have to control loading order; doing so is straightforward (we use the module functions as continuations/callbacks), but if we want to re-cover the early (and parallel) pre-loading aspect, we need additional tricks (which is also why LABjs is more affected by browser updates than RequireJS);
  • integrating with onload event (if browsers will not wait with the onload event until all script-inserted scripts have been executed, we can treat the main document as just another module dependency: create a pseudo-module that becomes available when onload is triggered; any modules depending on this pseudo-module can be sure that the document is ready when they execute; the similarities between module export/import and custom event trigger/listen might be interesting to explore in general);
  • packaging modules from multiple script files into a single project file (with modules sorted in dependency order), ready for minification and compression; this is roughly the same code as linkModules, just that we concatenate code fragments instead of applying module functions;
About these ads
This entry was posted in Uncategorized and tagged , , , , . Bookmark the permalink.

7 Responses to Loading Javascript Modules

  1. Pingback: JavaScript Magazine Blog for JSMag » Blog Archive » News roundup: Crankshaft, WebSockets disabled, 3d Christmas tree

  2. Pingback: the rasx() context » Blog Archive » “JavaScriptMVC 3.0: Good To Go!” and other links…

  3. Spencer Carney says:

    Found this link via jsmentors and I must say, good job. I’m still wet behind the ears with intermediate JS such as this but you do a really good of making it understandable and easy. Thanks!

  4. Francisco Fernández González says:

    It’s a wonderful article. Thank you!

  5. I also experienced this need for a progressive transition from modules declared synchronously in the global scope to module declared asynchronously by wrapping the declarations in a definition function.
    With this in mind, I developed a library called scopeornot:

    https://github.com/eric-brechemier/scopeornot

    It proves a useful tool, which helped me to migrate several projects to a more modular architecture.

    • Yes, using function wrappers as parameters to a function with varying definitions to give control over the transition was a core idea in the transition I outlined in this post. But even back then, my definitions were temporary/project-local – the time for advertising “yet another module loader” is long past. AMD-based loaders were a hard enough sell, and they seem to cover the ground well (including adapters for other module systems, without another level of abstraction on top).

      Over the next year, I expect the focus to shift towards ES6 modules, partly supported natively, partly transpiled into AMD or CommonJS (as in TypeScript, for instance), for not-yet-supporting engines. Time to work out the remaining issues in that spec, so that we can put the module system variations behind us.

      • ES6 modules will be a much needed addition to the language, but after the completion of the specification, several more years will pass until browsers that do not support ES6 fall into oblivion.
        In the meantime, I would like to have a simple way to declare modules to ensure that I won’t need to modify every single script to take advantage of ES6 Harmony for example.
        I find that the transition from namespaced modules to AMD is too steep, with a long tunnel effect caused by the inversion of control.
        I am fine with a temporary solution, local to a single project if need be. I am definitely not interested in implementing yet another loader, but I am not looking forward to rewriting a whole code base to adapt to yet another module system.
        What I am looking for is something partly similar to the Loader API of ES6:

        http://wiki.ecmascript.org/doku.php?id=harmony:module_loaders

        but simpler to implement, and adapted to today’s module definitions.

Comments are closed.