pnpm
pnpm is the recommended package manager of choice for TypeScript projects at SEEK.
This topic details how to use pnpm with skuba.
Background
skuba serves as a wrapper for numerous developer tools such as TypeScript, Jest, Prettier & ESLint, abstracting the dependency management of those packages across SEEK projects. When you are using skuba, you do not need to declare these packages as direct devDependencies
. In our previously-recommended package manager, Yarn, these packages and others are automatically hoisted to create a flattened dependency tree.
{
"devDependencies": {
"skuba": "7.2.0"
}
}
node_modules
βββ jest
βββ prettier
βββ skuba
βββ other-skuba-deps
However, this behaviour can lead to some silly bugs when updating packages.
pnpm in skuba
pnpm addresses the hoisting issue with a symlinked structure. Each package is guaranteed to resolve compatible versions of its dependencies, rather than whichever versions were incidentally hoisted.
This behaviour is a double-edged sword for a toolkit like skuba. Dependencies like Prettier and ESLint end up nested in a node_modules/skuba/node_modules
subdirectory, where most editor and developer tooling integrations will not know to look.
node_modules
βββ skuba -> ./.pnpm/skuba@7.2.0
βββ .pnpm
βββ skuba@7.2.0
β βββ node_modules
β βββ prettier -> ../../prettier@3.0.0
βββ prettier@3.0.0
βββ node_modules
βββ other-dep -> <store>/other-dep
.npmrc
pnpm allows us to specify dependencies to hoist via command line or .npmrc
. The number of package patterns we need to hoist may fluctuate over time, so specifying hoist patterns via command line would be difficult to maintain.
The skuba-maintained .npmrc
currently instructs pnpm to hoist the following dependencies:
# managed by skuba
package-manager-strict-version=true
public-hoist-pattern[]="@types*"
public-hoist-pattern[]="*eslint*"
public-hoist-pattern[]="*prettier*"
public-hoist-pattern[]="esbuild"
public-hoist-pattern[]="jest"
public-hoist-pattern[]="tsconfig-seek"
# end managed by skuba
From the previous example, this will produce the following node_modules
layout, allowing external integrations to find prettier
in node_modules/prettier
as before.
node_modules
βββ prettier -> ./.pnpm/prettier@3.0.0
βββ skuba -> ./.pnpm/skuba@7.2.0
βββ .pnpm
βββ skuba@7.2.0
β βββ node_modules
β βββ prettier -> ../../prettier@3.0.0
βββ prettier@3.0.0
βββ node_modules
βββ other-dep -> <store>/other-dep
Committing pnpm configuration in .npmrc
can conflict with build pipelines that synthesise an ephemeral .npmrc
to access private SEEK packages on the npm registry. A solution to this problem is detailed in the migration guide below.
Migrating from Yarn 1.x to pnpm
This migration guide assumes that your project was scaffolded with a skuba template.
-
Install skuba 7.4.0 or greater
-
Add a
packageManager
key topackage.json
"packageManager": "pnpm@9.14.2",
-
Install pnpm
corepack enable && corepack install
(Check the install guide for alternate methods)
-
Create
pnpm-workspace.yaml
Skip this step if your project does not use Yarn workspaces.
packages: # all packages in direct subdirectories of packages/ - 'packages/*'
(Optional) If your sub-package
package.json
s reference one another using the syntaxfoo: *
, you can replace these references with the workspace protocol using the syntaxfoo: workspace:*
. -
Run
pnpm import && rm yarn.lock
This converts
yarn.lock
topnpm-lock.yaml
. -
Run
pnpm skuba format
-
Remove the
.npmrc
ignore entry from.gitignore
and.dockerignore
Heed the warning and ensure that a safe
.npmrc
is included in the same commit.yarn-error.log # end managed by skuba - - # Ignore .npmrc. This is no longer managed by skuba as pnpm projects use a managed .npmrc. - # IMPORTANT: if migrating to pnpm, remove this line and add an .npmrc IN THE SAME COMMIT. - # You can use `skuba format` to generate the file or otherwise commit an empty file. - # Doing so will conflict with a local .npmrc and make it more difficult to unintentionally commit auth secrets. - .npmrc
A safe
.npmrc
will be synthesised for you in the next step. -
Run
pnpm skuba format
This will synthesise managed hoist patterns into
.npmrc
. -
Include additional hoisting settings in
.npmrc
Skip this step if your project does not use Serverless.
# managed by skuba package-manager-strict-version=true public-hoist-pattern[]="@types*" public-hoist-pattern[]="*eslint*" public-hoist-pattern[]="*prettier*" public-hoist-pattern[]="esbuild" public-hoist-pattern[]="jest" public-hoist-pattern[]="tsconfig-seek" # end managed by skuba + + # Required for Serverless packaging + node-linker=hoisted + shamefully-hoist=true
-
Run
rm -rf node_modules && pnpm install
This will ensure your local workspace will not have any lingering hoisted dependencies from
yarn
.If you have a monorepo, delete all sub-package
node_modules
directories. -
Run
pnpm skuba lint
After running
pnpm install
, you may notice that some module imports no longer work. This is intended behaviour as these packages are no longer hoisted by default. Explicitly declare these asdependencies
ordevDependencies
inpackage.json
.For example:
Cannot find module 'foo'. Did you mean to set the 'moduleResolution' option to 'nodenext', or to add aliases to the 'paths' option? ts(2792)
Run
pnpm install foo
to resolve this error. -
Modify
Dockerfile
orDockerfile.dev-deps
Your build pipeline may have previously mounted an ephemeral
.npmrc
with an auth token at/workdir
. This needs to be mounted elsewhere to avoid overwriting the new pnpm configuration stored in.npmrc
.FROM --platform=arm64 node:20-alpine AS dev-deps + RUN --mount=type=bind,source=package.json,target=package.json \ + corepack enable pnpm && corepack install + RUN --mount=type=bind,source=package.json,target=package.json \ + pnpm config set store-dir /root/.pnpm-store WORKDIR /workdir - COPY package.json yarn.lock ./ - COPY packages/foo/package.json packages/foo/ - RUN --mount=type=secret,id=npm,dst=/workdir/.npmrc \ - yarn install --frozen-lockfile --ignore-optional --non-interactive + RUN --mount=type=bind,source=.npmrc,target=.npmrc \ + --mount=type=bind,source=package.json,target=package.json \ + --mount=type=bind,source=pnpm-lock.yaml,target=pnpm-lock.yaml \ + --mount=type=secret,id=npm,dst=/root/.npmrc,required=true \ + pnpm fetch
Move the
dst
of the ephemeral.npmrc
from/workdir/.npmrc
to/root/.npmrc
, and use a bind mount in place ofCOPY
to mountpnpm-lock.yaml
.pnpm fetch
does not requirepackage.json
to be copied to resolve packages; trivial updates topackage.json
like a change inscripts
will no longer result in a cache miss.pnpm fetch
is also optimised for monorepos and does away with the need to copy nestedpackage.json
s. However, this command only serves to populate a local package store and stops short of installing the packages, the implications of which are covered in the next step.Review
Dockerfile.dev-deps
from the newkoa-rest-api
template as a reference point. -
Replace
yarn
withpnpm
inDockerfile
As
pnpm fetch
does not actually install packages, run a subsequentpnpm install --offline
before any command which may reference a dependency. Swap outyarn
commands forpnpm
commands, and drop the unnecessaryAS deps
stage.- FROM ${BASE_IMAGE} AS deps - - RUN yarn install --ignore-optional --ignore-scripts --non-interactive --offline --production - - ### - FROM ${BASE_IMAGE} AS build COPY . . - RUN yarn build + RUN pnpm install --offline + RUN pnpm build + RUN pnpm install --offline --prod ### FROM --platform=arm64 gcr.io/distroless/nodejs20-debian12 AS runtime WORKDIR /workdir COPY --from=build /workdir/lib lib - COPY --from=deps /workdir/node_modules node_modules + COPY --from=build /workdir/node_modules node_modules ENV NODE_ENV=production
-
Modify plugins in
.buildkite/pipeline.yml
Your build pipeline may have previously output an ephemeral
.npmrc
with an auth token on the build agent. This needs to be output elsewhere to avoid overwriting the new pnpm configuration stored in.npmrc
.Swap out caching on
yarn.lock
for.npmrc
andpnpm-lock.yaml
at the same time.We are also using an updated caching syntax on
package.json
which caches only on thepackageManager
key. This requires the seek-oss/docker-ecr-cache plugin version to be >= 2.2.0.seek-oss/private-npm#v1.3.0: env: NPM_READ_TOKEN + output-path: /tmp/
-Β seek-oss/docker-ecr-cache#v2.1.0: + seek-oss/docker-ecr-cache#v2.2.1: cache-on: - - package.json - - yarn.lock + - .npmrc + - package.json#.packageManager + - pnpm-lock.yaml dockerfile: Dockerfile.dev-deps - secrets: id=npm,src=.npmrc + secrets: id=npm,src=/tmp/.npmrc
-
Run
pnpm install --offline
and replaceyarn
withpnpm
in.buildkite/pipeline.yml
- label: π§ͺ Test & Lint commands: + - echo '--- pnpm install --offline' + - pnpm install --offline - - echo '+++ yarn test:ci' - - yarn test:ci - - echo '--- yarn lint' - - yarn lint + - echo '+++ pnpm test:ci' + - pnpm test:ci + - echo '--- pnpm lint' + - pnpm lint
-
Search for other references to
yarn
in your project. Replace these withpnpm
where necessary.
FAQ
Q: Iβm running into ERR_PNPM_CANNOT_DEPLOYβ A deploy is only possible from inside a workspace
A: pnpm deploy
is a reserved command. Use pnpm run deploy
instead.
Q: Iβm seeing ERR_PNPM_RECURSIVE_EXEC_FIRST_FAILβ Command "<NAME>" not found
in my pipeline
A: Ensure pnpm install --offline
is referenced earlier within pipeline step as shown in step 14.
Q: Iβm seeing ERR_PNPM_RECURSIVE_EXEC_FIRST_FAILβ Command "workspace" not found
in my pipeline
A: pnpm workspace <PACKAGE_NAME>
does not work. Replace it with the --filter
flag.
Contributing
This guide is not comprehensive just yet, and it may not account for certain intricacies of your project.
If you run into an issue that is not documented here, please start a discussion or contribute a change so others can benefit from your findings. This page may be edited on GitHub.