From Require.js to Webpack - Part 1 (the reasons)

Wherein I layout all of the reasons why we decided to move off of require.js and on to webpack.

About 2 months ago we completed a migration from require.js to webpack. Require.js had been the primary UI module system at PayPal and was baked into the default template kraken server we use to build our applications. Everyone was familiar with it and it worked pretty well. Despite our success with it, I identified a few features that were provided by webpack that ultimately led my team to migrate off of require.js. This is an overview of the reasons why we chose webpack.

I hope this proves helpful to you as you seek to improve your own module/build systems!

CommonJS By Default

Working in node all of the time, we sure like our common js modules. Despite a handful of CJS-like structures supported by require.js (including the "lite" CJS wrapper and partial support for the syntax within r.js), it really does not work as expected. There are a number of key differences.

1. Confusion about returning vs. module.exporting
2. Module resolution path (it will default to ".", which is basically like adding path.resolve() to everything.)
3. NPM support (including the package.json "main" field, etc.)

Client-side Unit Testing

One of the immediate benefits we experienced was being able to rip out this excessively annoying code we had to put in place to support client-side unit testing. There was a great deal of things we had to do to get mocha to support the files in the require.js. All of this code we were literally able to rip out and throw away once a file was moved to CJS, because node could just read the module and as long as it didn't rely on the DOM it could be tested like any other node module. Up until this point our client-side coverage had been terrible. With this change it was much easier to ratchet up the testing much further.

Re-use and modularity

Moving to CJS also opened up a flood of module reusability. This reusability was always available with AMD modules. Why the change? The reasons were 2 fold:

1. Syntax parity with our node.js code meant we started applying the same practices we were already applying to our node code to our client-side code. It turns out that we associated certain patterns with AMD code (monolothic) and certain types of patterns with our CJS code (modular). This meant that even though the shift was primarily mental, we quickly adopted a more modular approach to our UI code-bases when took that mental shift.

2. Actual compatibility with our server-side code meant that a number of important modules (date/time formatting, various utilities, etc.) could be directly shared and reused immediately.

Re-sharpening of our focus on modularity and sharing was one of the main benefits of the move to webpack.

NPM Support

While the CJS syntax itself is quite a bit nicer than the AMD syntax and provides a bunch of benefits as listed above. The most important reason for switching to webpack is full NPM support. Meaning I can install any module with NPM and without any extra work require() that into my library. Technically, this was possible with require.js, but it was really difficult and limited in scope:

NPM with require.js:

1. Expose your node_modules folder as a publicly accessible route in your app.
2. Add an alias from your module to the convoluted new module path in your config.js.
3. Make sure the module you want has 0 dependencies (because they won't work at all.)
3. define('your-module',  function(yourModule) { });

NPM with webpack:

1. require() the module.

Mature ES6/Plugin Support

Webpack Loaders are awesome. Require.js plugins on the other hand are not well supported and syntax can be confusing define(['css!styles/main'], function() { /* .. */}). The loader config in webpack is a breeze and the plugin ecosystem is much stronger.

With requirejs I tried to add ES6 support using the legacy es6 require.js plugin which is based on Traceur. I just couldn't get it to work, so we moved to 6to5 (babel) using a two-step process. First, we'd run babel to convert our files to JS and then we'd suck them up with require.js. Later we'd need to run some kind of "clean" command to remove the extra files it left hanging around our file system. 

Using babel-loader had proved to be a lot simpler and nicer and has cut both one step and additional time out of our build process. Source maps of course work out of the box as well.

With all of the great plugins including built-in support for uglify and other thing we barely need grunt at all for building our app. Almost all of it is done by webpack directly.


Concerns

When I showed the team the changes and talked with various people about them I'm frequently bombarded with the same two questions. People are generally under the assumption that it's no longer possible to do an asynchronous require() and others wonder why we didn't just go with browserify. There are good stories for both, so hear me out.

What about asynchronous/dynamic requires?

Asynchronous require's are often helpful in large projects. In our case we used the AMD require([]) at the router level to do something like this:

routeChange(path) {

    require(['/views/' + path], function(PageView) {

        app.setView(new PageView());

   }

}

This allowed us to avoid defer loading hundreds of kb of code until it was absolutely necessary and avoid loading code for different parts of the app that weren't being accessed at this time. It's a pretty neat trick and one we didn't really want to give up.

Turns out, we didn't have to give it up. Webpack supports this with only minor changes to the code above. In the end our various bundles were the same size or smaller than the ones created with r.js (not to mention the build time was cut down from 50s to around 10s).

I'll go more into the exact code we used to get this to work in the follow up, but if you're interested you can check out Pete Hunt's popular Webpack How-To for those getting started which has a nice section on async loading.

Why not browserify?

Browserify is pretty legit and indeed my first 2 attempts to migrate to away from require.js were both to browserify. Browserify has been innovating in this space for ages and was the obvious choice when I started this project. Where did it fall down?

1. Spotty support for AMD modules (even with the deAMDify plugin)
2. No built-in way to asynchronously require().

AMD Support

Because migrating all of our UI code from CommonJS to AMD meant changing nearly 1000 files, clearly we wanted to automate that in some way. A number of tools support this in one capacity or another, but none of them worked very well. As with most folks migrating away from AMD land, the easiest way with browserify was to use the deAMDify transform. This almost worked! Despite it being sold as the solution to our problems it fell down pretty hard in two areas:

1. The path resolution was still common.js based. It didn't know about our public/js folder. It didn't look for "module" in a relative path by default like AMD does, it looked for it in NPM. It ran into a lot of problems. Eventually I had to fork the transform in order to add support for this functionality, which was a pretty big pain in the neck.

2. It didn't support dependencies that weren't assigned to variables.

define('dependency', function() { }); was not translated to require('dependency') as expected. It just broke the system.

While this wasn't too hard to fix, the module itself had been abandoned by the maintainer and despite multiple tries to help, at that time we weren't able to get any of this stuff fixed. (Thankfully Tim Branyen has taken over the the deamdify module now and is very responsive to changes!) 

While the AMD support was frustrating, ultimately we were able to fix it, but we ran into an even bigger hurdle, which ultimately ended our test with browserify.

Async require()

I already covered how webpack supports AMD-style require() statements above. It turns out that there's nothing like that built in to browserify. The suggestion is to just basically load everything up front. This is all well and good for most apps. For an app the size of PayPal.com it turns out it's not such a good idea and leads to megabytes of extra code being loaded.

Another alternative to async require() with browserify is to manually bundle things. This will actually work. You need to add two <script> tags to your page. One for the main bundle and another for the specific "sub-app" that you're working with. This is pretty great if you start your app this way, but going from async require() to a lt;script&gt tag approach for each sub-app was extremely hard. I didn't have time to completely re-factor our routing system and test everything, so eventually I gave up.

Webpack allowed me to get the benefits of browserify without having to re-factor major parts of our codebase.

Conclusion

Webpack has been a huge help to our client-side code base and developer experience in general. It's allowed greater parity and reuse between our client-side and node code, it's made testing our code much easier and it's allowed us to cut way down on the config and extra support code needed to maintain two different module systems in the same code base. The most important thing it has provided though is access to the NPM ecosystem in the browser. Coding will never be the same again :)

Look for a follow-up post detailing the technical aspects of the migration in the next couple of weeks.

* In case you wanted to know most of the UI code at the time of the migration was written in Backbone. Unmentioned in the article was that we're starting to move a lot of stuff to React and NPM support was crucial for us to do that well.