It’s not our code
In many ways this represents a triumph for Open Source: developers are able to build on the communal value of shared code and collaborate on generalized solutions to their problems in a public forum.
“the dependencies we install from npm are stuck in 2014”
node_modules being ECMAScript 5. In some very rare cases, developers use bolted-on solutions to detect non-ES5 modules and preprocess them down to their desired output target (here’s a hacky approach you shouldn’t use). As a community, the backwards compatibility of each new ECMAScript version has allowed us to largely ignore the effect this has had on our applications, despite an ever-widening gap between the syntax we write and the syntax found in most of our favorite npm dependencies.
This has led to a general acceptance that npm modules should be transpiled before they are published to the registry. The publishing process for authors generally involves bundling source modules to multiple formats: JS Modules, CommonJS and UMD. Module authors sometimes denote these different bundles using a set of unofficial fields in a module’s package.json, where
"module" points to an
"unpkg" points to the UMD bundle, and
"main" is still left to reference a CommonJS file.
module field. Unfortunately, this approach is incompatible with today’s tooling - more specifically, it’s incompatible with the way we’ve all configured our tooling. These configurations are different for every project, which makes this a massive undertaking since the tools themselves are not what needs to be changed. Instead, the changes would need to be made in each and every application’s build configuration.
node_modules should be processed. These tools can be easily configured to treat
node_modules the same as authored code, but their documentation consistently recommends developers disable Babel transpilation for
node_modules. This recommendation is generally given citing build performance improvements, even though the slower build produces better results for end users. This makes any in-place changes to the semantics of importing code from
node_modules exceptionally difficult to propagate through the ecosystem, since the tools don’t actually control what gets transpiled and how. This control rests in the hands of application developers, which means the problem is decentralized.
The module author’s perspective
- We know app developers aren’t transpiling
node_modulesto match their support targets.
- We can’t rely on app developers to set up sufficient minification and optimization.
- Library size must be measured in bundled+minified+gzipped bytes to be realistic.
- There is still a widespread expectation that npm modules are delivered as ECMAScript 5.
- Increasing a module’s JS version requirement means the code is unavailable to some users.
Module authoring tools hurt, too
Just like with application bundlers being configurable without an implied default behaviour for
node_modules, changing the module authoring landscape is an unfortunately distributed problem. Since most module authors tend to roll their own build tooling as requirements vary from project to project, there isn’t really a set of canonical tools to which changes could be made. Microbundle has been gaining traction as a shared solution, and @pika/pack recently launched with similar goals to optimize the format in which modules are published to npm. Unfortunately, these tools still have a long way to go before being considered widespread.
Assuming a group of solutions like Microbundle, Pika and Angular’s library bundler could be influenced, it may be possible to shift the ecosystem using popular modules as an example. An effort on this scale would be likely to encounter some resistance from module consumers, since many are not yet aware of the limitations their bundling strategies impose. However, these upended expectations are the very shift our community needs.
It’s not all doom and gloom. While Webpack and Rollup encourage unprocessed npm module usage only through their documentation, Browserify actually disables all transforms within
node_modules by default. That means Browserify could be modified to produce modern/legacy bundles automatically, without requiring every single application developer to change their build configuration. Similarly, opinionated tools built atop Webpack and Rollup provide a few centralized places where we could make changes that bring modern JS to
node_modules. If we made these changes within Next.js, Create React App, Angular CLI, Vue CLI and Preact CLI, the resulting build configurations would eventually make their way out to a decent fraction of applications using those tools.
node_modules are left unprocessed. Babel announced some new features last year that allow selective transpiling of
node_modules, and Create React App recently started transpiling
The last piece
Let’s assume we could build automation and guidance into our tools, and that doing so would eventually move the thousands (millions?) of applications using those tools over to configurations that allow modern syntax to be used within
node_modules. In order for this to have any effect, we need to come up with a consistent way for package authors to specify the location of their modern JS source, and also get consensus on what “modern” means in that context. For a package published 3 years ago, “modern” could have meant ES2015. For a package published today, would “modern” include class fields, BigInt or Dynamic Import? It’s hard to say, since browser support and specification stage vary.
The problem is that, if we assume “modern” to mean “anything newer than ES5”, it becomes impossible to determine what syntax a package contains that needs to be transpiled in order to meet a given browser support target. We can address this problem by establishing a way for packages to express the specific set of syntax features they rely on, however this still requires maintaining many variant configurations to handle each set of input→output syntax pairs:
|Example “Downleveling” Transformations
|ES5 / nomodule
|ES5 / nomodule
|classes & tagged templates
|ES5 / nomodule
|async/await, classes & tagged templates
|ES5 / nomodule
|rest/spread, for-await, async/await, classes & tagged templates
|rest/spread & for-await
What would you do?