Monorepo Reflections

Translated from Simplified Chinese by AI.Read the original

I used to organize my personal projects with a Monorepo all the time, but whenever I started coding, there always seemed to be a vague question nagging at me: is my Monorepo configured correctly?

Recently I was getting ready to start a new project, so I took the opportunity to search through a lot of material and look at how some open-source projects organize theirs, trying to figure out the so-called "correct" way to organize a Monorepo.

Disclaimer

I'm not a Monorepo expert, so please take the following with a grain of salt. There may be factual errors; feel free to point them out, because as I said at the beginning, I'm not a Monorepo expert （。＾▽＾）

Build output or source code

I think this is the first thing that needs to be clear. In a typical Monorepo project, our project structure roughly looks like this:

.
├── apps
│   └── web
└── packages
    └── utils

As you can see, we've defined utils and want it to be usable by multiple Apps. It's very important to understand utils correctly—is it designed as a package that needs to be bundled, published to a Registry, and referenced by other projects? Or is it just a package that exists as source code for internal use?

Nx's approach

In Nx's Monorepo practice, it clearly distinguishes between Workspace Libraries and Publishable Libraries / Buildable Libraries.

Workspace Libraries are "packages that are not made to be published or built"; their TypeScript source code is imported directly by Apps and bundled together with the App by the App's bundler. This is the latter case above: "a package that exists only as source code for internal use."

In an Nx Integrated Repo, all Project dependencies are defined in the package.json at the Repo root. Workspace Libraries don't have their own package.json; Apps reference them by adding their directories to paths in the tsconfig.json at the Repo root, so there's no need to explicitly pnpm add ... in the App. This practice is also very common in Monorepos based on pnpm Workspace.

Aside: In Polyrepo projects, we also often set an alias like @components/ for the components folder in tsconfig.json. Could that also be seen as a kind of Monorepo? 23333（逃

The most obvious benefit of setting it up this way is that under dev you only need to start the App's Dev Server, because what we import is source code rather than dist, and transpilation and other operations happen in the App's toolchain. Any changes to the package can be detected by the App's Dev Server and hot-reloaded.

For packages that need to be published to a Registry / need to be built, Nx Integrated Repo creates a package.json for them, with the main and exports fields pointing to the build output, ensuring external projects can import the package correctly. When other Apps inside the Repo import this package, they still import source code via tsconfig.json rather than importing the build output, continuing to use the App's toolchain for bundling, which allows hot reloading.

Separating the build output used for publishing from the code used for internal imports is indeed very convenient, but there's an obvious problem: when importing code directly, the package's own bundling configuration stops applying.

Imagine a package has a function getVersion(), and in this package's vite.config.ts we use a text replacement plugin that replaces __VERSION__ in the code with the current version number. In the published output, because it's bundled using the package's own Vite config, the output can successfully contain the current version number. But when an App imports this library via source code, it uses the App's toolchain for bundling, and if the App hasn't configured this text replacement plugin, the App will only get the __VERSION__ placeholder.

If the user doesn't understand this, it can be misleading. In the article Monorepo Evolution - Are You Really Using Shared Packages?, you can see that this situation is actually quite common. A comment by Zhang Lili under it inspired me a lot; here's the original text:

The root cause is still whether you treat it as "a package that doesn't need publishing" or "a source directory in a project." If you treat it as a package, none of the problems are problems:

package.json issue: Can a package reference content inside the package? Of course it can, as long as it's published. But if it doesn't publish certain files, they can't be referenced. In a monorepo the "publishing" step is missing, but you still need the concept of "publishing." Usually before publishing you build and bundle; once all that is done, what ultimately gets referenced should be an entry, and specifying that entry in package.json is equally valid.

tsconfig.json issue: Would a package send out its tsconfig.json? Would sending it out have any effect? The answer is obviously clear. If you use it as a package, you naturally won't have tsconfig questions. More precisely, ts should not be referenced directly (no package publishes ts code for others to use).

phantom dependency issue: Dependencies inside the package naturally follow the package, and workspace tools will handle them. Dependencies whose versions should theoretically be controlled by the consumer should be designed as peerDependencies. (There's a case here where a monorepo has problems when peer+dev coexist; that really has no solution.)

common in a monorepo should be "a package that doesn't go through the publishing process," not "a source directory." Stick to this concept and some problems aren't problems in the first place.

Essentially, it's still not treating it as a "package" (directly referencing source, letting the referencing side handle build) while wanting it to have "package" behavior (package.json and tsconfig.json taking effect). That's very contradictory.

But I don't think this is an unacceptable practice, because hot reloading this way is really convenient (/▽＼). If we were to treat it as "a package that doesn't go through the publishing process"—that is, Apps import its build output—then when modifying the package we'd need another daemon to watch file changes in real time and rebuild, in order to achieve hot reloading. And you can expect that there might be some dependency optimization / caching issues on the App side that make hot reloading fail.

pnpm Workspace's approach

Unlike Nx Integrated Repo, every subproject in a pnpm Workspace should have its own package.json, and the subproject's dependencies are installed there too, not installed globally for the Monorepo (this is similar to Nx Package-Based Monorepo).

We can continue to set aliases in tsconfig.json and the bundler to import shared packages within the project (personally I find this approach a bit tricky), or we can choose to have the App add the package as a dependency and link it via the workspace: protocol. At this point we again have two choices, as mentioned above: importing build output or importing code.

Importing build output

In this case, we can simply point main and exports in the package's package.json to the dist directory, and then the App imports the build output. After the package code is modified it needs to be rebuilt, so we can define these scripts in the Repo root package.json:

{
  "scripts": {
    "dev": "pnpm -r --filter=./packages/** --parallel run dev"
    // ...
  }
  // ...
}

Running pnpm dev in the Repo root will run all packages' dev scripts in parallel, recompiling the output when changes are detected. This is what Slidev does.

Importing code

Similarly, we can also point main and exports to the code.

If the package can be bundled and published separately, we can override main and exports in publishConfig to point to dist.

{
  "main": "src/index.ts",
  "publishConfig": {
    "main": "dist/index.js",
    "types": "dist/index.d.ts"
  }
  // ...
}

This is also what BlockSuit does.

Single or multiple package.json

Nx Integrated Repo has only one package.json by default (unless you create Publishable Libraries).

In the article Your Monorepo Dependencies Are Asking for Trouble, the author explains in detail the version inconsistency problems caused when a Monorepo uses multiple package.json files. Using Nx Integrated Repo naturally handles this easily, but after a brief trial I found that it might not be a good fit for me right now.

One obvious problem with a single package.json is that the dependencies of various projects are mixed together and hard to sort out—I tried deleting a project with nx g rm <App Name>, but the dependencies in package.json weren't cleaned up. From this small example I can imagine the bigger picture: once the project grows large, how do you safely remove a dependency?

Also, many frameworks' scaffolding CLIs are currently designed for multiple package.json files, so dependencies for a newly created project won't be installed into the Repo root package.json. You can only create or migrate through plugins provided by Nx. Of course, Nx Integrated Repo also provides a guide for adding a Package-Based Project, but I probably just don't like the feeling of being somewhat vendor lock-in, so for now I'm not likely to use Nx Integrated Repo.

When using multiple package.json files, pnpm Workspace provides the catalog: protocol as a version variable for the whole Repo, which can be set in pnpm-workspace.yaml. This is also a way to solve version inconsistencies.

Project Reference

Imagine a project like this:

.
├── apps
│   └── web
│       └── tsconfig.json
├── packages
│   └── utils
│       └── tsconfig.json
└── tsconfig.json

In a Monorepo, a common practice is to put a tsconfig.json at the Repo root, have subprojects' TypeScript configs extend from the Repo root tsconfig.json, and then override on top of that.

If noUnusedLocals is enabled in the root tsconfig.json, but overridden to false in the utils package, what happens when the Web App imports the Utils package and then runs tsc to compile the project?

.
├── apps
│   └── web
│       └── tsconfig.json <- "extends": "../../tsconfig.json"
├── packages
│   └── utils
│       └── tsconfig.json <- "extends": "../../tsconfig.json"
│                            "noUnusedLocals": false (Override)
│
└── tsconfig.json         <- "noUnusedLocals": true

In the case of importing build output, because what we import is already JavaScript compiled from TypeScript and d.ts type definition files, the Web App doesn't get involved in compiling the Utils package.

In the case of importing source code, as described in the first section, because what we import is TypeScript source code, the compilation work is effectively handed over to the Web App, using the Web App's TypeScript config, so it errors.

../../packages/utils/src/index.ts:2:9 - error TS6133: 'unused' is declared but its value is never read.

2   const unused = 1
          ~~~~~~


Found 1 error in ../../packages/utils/src/index.ts:2

In this case, we can use TypeScript's Project References to solve this problem. In the utils package's TypeScript config, enable composite, declaration, declarationMap, turn off noEmit, and set outDir.

Then add this to the web TypeScript config:

{
  "references": [
    {
      "path": "../../packages/utils"
    }
  ] // ...
}

Finally, run tsc -b under web, and you can see the project compiles successfully, with compiled files appearing under utils.

The tsc -b command builds each referenced project in turn. If you use Nx to manage tasks and configure it properly, Nx will automatically run the build tasks of dependent projects, so using tsc at that point has the same effect.

But it's worth noting that in a Monorepo, TypeScript's project boundary checking stops working; if you're not careful, you might forget to add references. Maskbook's tsconfig.json explains this situation. The developers chose to create aliases for packages in paths to solve it, with the drawback that subpackages can be imported without being installed as dependencies.

A Turbo blog post You might not need TypeScript project references points out that sometimes Project References are unnecessary and advocates handing TypeScript code to the end user (the Web App) for compilation. I somewhat agree with this view and lean toward maintaining a self-consistent tsconfig.json at the Repo root, because most of the time the project's TypeScript config is fairly uniform (。・ω・。).

An attempt at importing build output

The previous sections focused more on importing code, but I personally prefer importing build output—treating subpackages as packages that don't go through the publishing process—because it feels more "proper" (￣▽￣).

The key to doing this is to compile dependent subpackages one by one according to the dependency graph; Nx's dependsOn can help us do this easily. But when it comes to dev / serve use cases that need hot reloading, Nx seems a bit powerless (if you set dependsOn: ["^dev"], when the dependency package's task runs in watch mode, it can't finish and will block the App's dev task).

This Discussion mentions this problem, but from when it was raised in 2021 until now there hasn't been a fairly complete solution. Nx's run-many can work around it, but you need to specify subpackage names one by one. Fortunately, Turborepo, another Monorepo management tool, provides filter syntax in its run command (using ... to mean the package and its dependencies), which can run the App and its dependencies' dev tasks at the same time:

{
  "scripts": {
    "dev:web": "turbo run dev -F=@a-monorepo/web..."
  } // ...
}

But there's still a gap between this and the ideal result: before the App's dev runs, the dependency packages' dev should run in advance and complete at least one round of compilation; otherwise the App's bundler will report errors. I really haven't found a better solution to this problem, and Turborepo's docs use it this way too.

When using Vite in this situation, changes in dependency packages cause Vite to redo Dep Optimization and then refresh the page, causing page state to be lost and making HMR impossible. Maybe there's a way to solve it, but I haven't tried (/▽＼).

A document from Microsoft's Rush Monorepo management tool mentions this need, but it's still experimental, and starting the App's dev at the end requires several commands, which is a bit tedious, like this:

# 构建所有依赖于 D 的项目（但不包括 D 本身），并在无限循环中重复这个操作
$ rush build:watch --to-except D

# 在项目 D 的目录下开启 Webpack 的开发服务器
# （这是示例中的 web 应用）
$ cd apps/D
$ heft start # 或者用自己的 "npm run start"

(Turborepo's watch, by combining filter syntax, could in theory also be operated in the way described in the Rush doc above.)

That's my simple attempt at importing build output. The conclusion is that the DX of this approach is currently worse than importing source code. I also looked at Turborepo's Examples, and they all use importing source code too—that is, pointing exports to src.

No silver bullet

I spent several days investigating and trying various Monorepo practices, and finally put them together in this article. Well, my efficiency really hasn't been high these past few weeks （；´д｀）ゞ

At the beginning I deliberately put quotes around "the 'correct' way to organize things," because although open-source projects organize their Monorepos differently, that doesn't stop them from being excellent open-source projects—there's no correct one, only a suitable one.

Whether it's Polyrepo or Monorepo, no matter how the Monorepo is organized, as long as it can adapt to the current developers' needs and handle the current software development environment, it's a good Repo <(￣︶￣)>（逃