Introducing ESTR (ES traversals and transformations)

estr is a Javascript library for processing Javascript source code, building on the Esprima parser and AST. It comes with a commandline interface, based on nodejs, and an online demo, based on CodeMirror, showing how to use estr directly as a library, from the browser. A sample binding for using estr from Vim is also included.

What does this mean in practice: If your Javascript editor can be extended by calling out to Javascript code or commandline tools, you can now add scoped variable navigation (to binding, to next/previous occurrence) and variable renaming to your editor’s feature set.


In its current state, estr can:

  • generate (scope-aware) tags files for JS projects (for use with scoped_tags Vim mode); this replaces my earlier, doctorjs-based scoped tags generator
  • augment Esprima ASTs with scope information
  • locate identifier (with line/column info) in augmented AST, find its binding occurrence, scope, and other occurrences
  • transform JS source code to implement variable renaming, while preserving layout and comments (no pretty-printing)

The online demo shows how to use this functionality to enhance a standard CodeMirror JS editor. CodeMirror provides editing, its Javascript mode provides basic syntax highlighting, Esprima adds parsing and parse errors. On top of that, estr enables us to:

  • highlight unbound variables
  • highlight other occurrences of identifier under cursor (scope-aware)
  • navigate to next/previous identifier occurrence, or to the binding occurrence (scope-aware)
  • rename variables (scope-aware, with checks to prevent errors introduced by naive renaming)

The demo code is in the gh-pages branch of the estr repo.

If you find estr useful, please tell others;
if you find bugs in estr, please tell me!-)

If you have access to another JS renaming tool, you might want to try the estr renaming tests on it. If you want to integrate estr in your tool suite, please get in touch.


Posted in Uncategorized | Tagged , , , , , , | 3 Comments

Scoped tags support for Javascript, in Vim

Tags files

How do you navigate a Javascript project in Vim? Well, you generate a tags file, and use Vim’s extensive support for tags-based commands. A good Javascript-aware tags file generator is DoctorJS’s jsctags (itself written in Javascript).

Problem solved? Not quite. Javascript programming patterns tend to use nested declarations (module pattern, immediately applied function definitions for local namespaces, or just plain old local variable declarations and function parameters). Tags files were designed for languages with mostly global definitions, so if these local definitions show up at all in a tags file, there will be no scope-awareness (no navigation to the definition currently in scope).

Ok, so we need to see about extending the tags file information. The tags file format used in Vim is extensible, so we can add fields. Such added fields are already in use, for instance, to indicate the class or namespace of properties in object-oriented languages. DoctorJS, in earlier versions, used to generate such namespace information for Javascript tags (that feature seems to be broken in recent versions).

Scoped tags files

The difference here is that we want to support lexical scoping, not namespacing. Scoping differs quite widely between programming languages, even between language versions (Javascript’s function scoping is going to be accompanied by other scoping constructs in ES6). To avoid hard-coding language-specific scoping conventions into Vim, we use a language-independent form of indicating variable scope in a tags file, via a tag field


It is then up to Vim scripts to figure out the matching tag for an identifier under the cursor — a tag with the smallest scope range wrapping the cursor position (smallest to account for shadowing of bindings), and the work of translating the language scoping rules to scope ranges is left to language-specific tags file generators.

Continue reading

Posted in Uncategorized | 9 Comments

Loading Javascript Modules

Loading Javascript Modules

Sooner or later, every Javascript developer encounters the Module Pattern. It isn’t always used for modules in the conventional sense, but even when it is used for modules, the files in which those modules reside are still often loaded by carefully hand-listing script-elements, in dependency order, in a HTML page. I have just been through the process of moving one of my own projects from a naive file-as-module project structure (with implicit dependencies and explicit loading) to a structure with explicit modules, explicit import/export dependencies and automated loading in dependency order.

This is an explanatory post, outlining the process. Both a possible sequence of refactoring steps and a possible implementation of a module loader library are presented. I found it fun to figure out the module loader code myself, and the code provided should work well enough (tested Opera, Firefox, Safari; IE), but it is intended for reading, not production use (refer to existing module loader libraries instead). The intent of this post is threefold:

  • to make more readers aware of module patterns and module-loader libraries,
  • to look under the hood to see how such a libary might come into being,
  • and to illustrate the refactoring process: how use of such a module loader might be introduced into an existing multi-module project.

Continue reading

Posted in Uncategorized | Tagged , , , , | 7 Comments

Updating Vim haskellmode for GHC-7.0.1

New version of haskellmode plugins for Vim: haskellmode-20101118.vba .

Several features of the haskellmode plugins for Vim depend on building an index of locally installed Haddock documentation. If you are using Vim and Haskell, and haven’t used these plugins yet, the screencasts will give you an overview of the main features.

Apart from the obvious, looking up docs for the identifier under the cursor, I’ve found it useful to be able to refer to identifiers that have not yet been imported. There is a variant of identifier completion wrt documented identifiers, we can add import statements for the identifier under the cursor, and we can get suggestions on valid qualifiers for unqualified identifiers. All together, this supports a style of programming where you first decide the function you want, then have haskellmode fill in the details of imports and qualifiers (of course, that is in addition to traditional completion wrt identifiers already in scope).

Building such a Haddock index has been done by scraping identifiers and URLs from Haddock’s HTML indices, and the format of those has changed with the Haddock included with the new GHC-7.0.1. This update adapts the scraping code to these changes.

Scraping index information from HTML has never been pretty, and the format is no longer as stable as it used to be, either. Fortunately, Haddock has recently been growing an API, so I hope to extract the index directly from Haddock’s interface files in future. That code is almost ready, but since it will mean installing Haskell code together with haskellmode, and the matching version of Haddock has not hit hackage yet, I thought I’d update the old-fashioned way first, for users who want to start using the new GHC before the next Haskell Platform release comes out.

Posted in Uncategorized | Tagged , , , ,

GUI libs? No thanks, I’m just browsing

A recent thread on the haskell-cafe reminded me of an old experiment of mine, and that it could still be useful for current Haskellers.

The idea was to use a web-browser as a GUI for Haskell programs running locally, via a rudimentary HTTP-server on the Haskell-side and some AJAX on the browser-side. Even in 2007, that was not a new idea, but with web development often being somewhat separate from other development, and non-web developers being quite attached to their GUI lib of choice, it still is not as widely known as one might think.

The advantages are quite compelling, and getting more so as rich internet applications are becoming commonplace, web standards take on the best of browser and library experimental features, and all active browser implementors are finally working on implementing those standards. To mention just a few:

  • unlike GUI libraries, browsers are not linked to your code – you do not have to rebuild the browser when you update your GHC
  • if you stick to implemented standards, you and your users can choose which browser they prefer (if they know web development, they might be able to tune and style your GUI)
  • not only are your GUIs portable accross platforms, there are browsers that stick to whatever look and feel is native to that platform
  • with web-based platforms making strong inroads against locally installed software, some users will find their browser’s look more native than any single OS’s

Some of the (potential) disadvantages:

  • you’ve got a network connection between your application and its GUI (you need to secure that, there might be performance limitations)
  • in spite of progress, not all modern standards are yet supported equally well in all browsers

Unlike real web developers, you won’t have to support older, non-standards-compliant browsers – just ask your users to install a modern browser of their choice (they are smaller and easier to install than most GUI libraries). You might of course get requests to support your application on a mobile browser where Haskell hasn’t yet been ported to the underlying platform, but then you could at least run your Haskell application on a server and connect a mobile GUI to it!-).

[screenshot of JustBrowsing in Opera][screenshot of JustBrowsing in Firefox][screenshot of JustBrowsing in Safari]

To make things concrete, here is a silly little Haskell code example, not pretty (and Canvas isn’t new anymore, either;-), but demonstrating many of the ingredients. The code needed only minor updates (exception handling has changed, and some modern browsers seem to prefer [::1] instead of localhost). It doesn’t do anything useful, instead it provides simple examples of the common issues:

  • integrate a network server into your application (since we only need the most basic features of an HTTP server, I just coded them explicitly, on top of the network package). We have a server loop, which parses HTTP GET requests, dispatches to Haskell functions based on URL, and delivers HTTP replies based on the function results.
  • the simplest GUI is just an HTML page served at some known URL (for our purposes, a named server entry point).
  • GUI elements, such as buttons, can use XHR/AJAX to call into other URLs (the get/callback pair of Javascript functions).
  • Haskell code can generate Javascript code that will be evaluated browser-side, in order to manipulate the HTML page. As an example, Haskell function draw recursively computes Javascript instructions that draw on a HTML canvas (in real applications, one would avoid the general eval, writing a custom interpreter loop specifically for graphic commands instead).
  • While the draw example delivers a whole bunch of changes at once, addLabel gives an example of continued interaction (each browser-side click gets another label text from the Haskell server and adds it to the HTML page).
  • One typical scenario that is not demonstrated are server-initiated events (say the server was computing a feed of fibonacci numbers in the background, pushing them browser-side whenever ready). This has typically been handled by having the browser making an asynchronous server request that only returns when the server has a message to send, and is reestablished immediately (so there is always a browser callback waiting when the server has something to send).

To make this example self-contained in a single file, I chose to hack up my own HERE-file support (function commentData), so that I could include the GUI HTML and Javascript to be served in the Haskell source file, as a comment at the end. The additional thisFile hack enables the code to figure out its own file name, when run via runghc — if you hardcode the source file name instead, the rest of the code is portable Haskell (which wouldn’t have been the case if I had used Hugs’ HERE-doc support, or GHC’s quasiquoting equivalent).

These days, one should package the whole application as cabal package, managing both the dependency on the network package and any HTML pages or Javascript files one might want to include as data, without the HERE-file hack.

To try just run main in Canvas.hs (for the single-file version; needs package network installed), or unpack the tarball and cabal install the package, followed by running JustBrowsing (for the cabal-package version). Then point your browser at http://%5B::1]:8000/start.

Other things to consider in modernizing: HTTP packages on hackage, websockets instead of HTTP, standard server-sent events.

Posted in Uncategorized | Tagged , ,

Yet another blog

Ok, I’ve finally given in, and have started blogging. For several years now, it has become clear that mailing list posts alone were no longer working. Just as with Usenet posts before, the readership has split up (into specialist sublists, IRC channels, blogs, etc.).

What has tipped the balance at last were two things: archiving and interests spanning more than a single language community. In the past, it was worth writing longer answers to mailing list threads, because the search engines would find them in the archives.

For Haskell-related lists, that has been working less and less (some years ago, a mis-edited robots.txt file prevented web spiders from indexing the main list archives, and these days, there are several archives for each list, dividing any Google rankings between them; hint to Google: the main list archive is listed in each list posting’s header).

And though Haskellers tend to be interested in other languages, posting only to Haskell lists misses the other language’s community, and cross-posting isn’t all that nice, either.

So, here we are. Expect to see posts about my experiences with Haskell, Javascript, and Petri nets, among other things (if you only want to see parts of the contents, please use the tags). I might also try to reuse/rescue some of my mailing list posts from the archives, or mention updates to software I maintain. Or I might do things I haven’t thought of yet, being new to blogging!-)

Posted in Uncategorized | Tagged , ,