Worker
lambda-sqs-worker
An asynchronous worker built on AWS Lambda and deployed with Serverless.
Modelled after the “enricher” pattern where an event is received from a source SNS topic and a corresponding enrichment is published to a destination SNS topic. For fault tolerance, a message queue is employed between the source topic and the Lambda function, and unprocessed events are sent to a dead-letter queue for manual triage.
Added in 3.4.1
Changelog
-
9.0.0: Point Docker base images to AWS ECR Public and remove constant
--platform
arguments (#1684) -
8.2.1: Remove JSON schema definitions from Buildkite pipeline files (#1624)
This reverts #1611 due to incompatibility with pipeline signing.
-
8.2.0: Add JSON schema definitions to Buildkite pipeline files (#1611)
-
8.1.0: Add extension recommendations to
.vscode/extensions.json
(#1556) -
8.1.0: Make all configuration values explicit (#1560)
Previously,
src/config.ts
included optional configuration values and inheritance between environments in the spirit of DRY. While the templated file was wired up in a “safe” way—the production environment never inherited from other environments and explicitly specified all its configuration values—its pattern was misappropriated elsewhere and led to local configuration values affecting production environments.Instead, we now list all configuration values explicitly against each environment.
-
8.1.0: Remove deprecated
docker-compose.yml
version (#1570)Docker has ignored this for a while, and now generates a warning on every build: https://github.com/compose-spec/compose-spec/blob/master/04-version-and-name.md
-
8.0.1: Install specific pnpm version via Corepack (#1515)
Previously, our Dockerfiles ran
corepack enable pnpm
without installing a specific version. This does not guarantee installation of the pnpm version specified inpackage.json
, which could cause a subsequentpnpm install --offline
to run Corepack online or otherwise hang on stdin:FROM --platform=arm64 node:20-alpine RUN corepack enable pnpm
{ "packageManager": "pnpm@8.15.4", "engines": { "node": ">=20" } }
Corepack is about to download https://registry.npmjs.org/pnpm/-/pnpm-8.15.4.tgz. Do you want to continue? [Y/n]
To avoid this issue, modify (1) Buildkite pipelines to cache on the
packageManager
property inpackage.json
, and (2) Dockerfiles to mountpackage.json
and runcorepack install
:- seek-oss/docker-ecr-cache#v2.1.0: + seek-oss/docker-ecr-cache#v2.2.0: cache-on: - .npmrc + - package.json#.packageManager - pnpm-lock.yaml
FROM --platform=arm64 node:20-alpine - RUN corepack enable pnpm + RUN --mount=type=bind,source=package.json,target=package.json \ + corepack enable pnpm && corepack install
-
8.0.0: Remove
BUILDPLATFORM
from Dockerfiles (#1350)Previously, the built-in templates made use of
BUILDPLATFORM
and a fallback value:FROM --platform=${BUILDPLATFORM:-arm64} gcr.io/distroless/nodejs20-debian11
- Choose the platform of the host machine running the Docker build. An AWS Graviton Buildkite agent or Apple Silicon laptop will build under
arm64
, while an Intel laptop will build underamd64
. - Fall back to
arm64
if the build platform is not available. This maintains compatibility with toolchains like Gantry that lack support for theBUILDPLATFORM
argument.
This approach allowed you to quickly build images and run containers in a local environment without emulation. For example, you could
docker build
anarm64
image on an Apple Silicon laptop for local troubleshooting, while your CI/CD solution employedamd64
hardware across its build and runtime environments. The catch is that your localarm64
image may exhibit different behaviour, and is unsuitable for use in youramd64
runtime environment without cross-compilation.The built-in templates now hardcode
--platform
as we have largely converged onarm64
across local, build and runtime environments:FROM --platform=arm64 gcr.io/distroless/nodejs20-debian11
This approach is more explicit and predictable, reducing surprises when working across different environments and toolchains. Building an image on a different platform will be slower and rely on emulation.
- Choose the platform of the host machine running the Docker build. An AWS Graviton Buildkite agent or Apple Silicon laptop will build under
-
8.0.0: Remove account-level tags from resources (#1494)
This partially reverts #1459 and #1461 to avoid unnecessary duplication of account-level tags in our templates.
-
7.4.0: Use
propagate-environment
for Docker Compose Buildkite plugin (#1392)This simplifies the Docker Compose environment variable configuration required for Buildkite and GitHub integrations.
In your
docker-compose.yml
:services: app: - environment: - # Enable Buildkite + GitHub integrations. - - BUILDKITE - - BUILDKITE_AGENT_ACCESS_TOKEN - - BUILDKITE_BRANCH - - BUILDKITE_BUILD_NUMBER - - BUILDKITE_JOB_ID - - BUILDKITE_PIPELINE_DEFAULT_BRANCH - - BUILDKITE_STEP_ID - - GITHUB_API_TOKEN image: ${BUILDKITE_PLUGIN_DOCKER_IMAGE:-''} init: true volumes: - ./:/workdir # Mount agent for Buildkite annotations. - /usr/bin/buildkite-agent:/usr/bin/buildkite-agent # Mount cached dependencies. - /workdir/node_modules
In your
.buildkite/pipeline.yml
:steps: - commands: - pnpm lint - pnpm test env: # At SEEK, this instructs the build agent to populate the GITHUB_API_TOKEN environment variable for this step. GET_GITHUB_TOKEN: 'please' plugins: - *aws-sm - *private-npm - *docker-ecr-cache - docker-compose#v4.16.0: + environment: + - GITHUB_API_TOKEN + propagate-environment: true run: app
-
7.4.0: Set maximum concurrency (#1412)
This prevents messages from going directly to the DLQ when the function reaches its reserved concurrency limit.
-
7.3.1: Update to Node 20 (#1317)
Consider upgrading the Node.js version for your project across:
.nvmrc
package.json#/engines/node
serverless.yml
@types/node
package version- CI/CD configuration (
.buildkite/pipeline.yml
,Dockerfile
, etc.)
If you are updating your AWS Lambda runtime to
nodejs20.x
, consider reading the release announcement as there are some breaking changes with this upgrade. -
7.3.0: seek-oss/docker-ecr-cache 2.1 (#1266)
This update brings a new
skip-pull-from-cache
option which is useful onWarm
/Build Cache
steps.At SEEK, our build agents no longer persist their Docker build cache from previous steps. This option allows a preparatory step to proceed on a cache hit without pulling the image from ECR, which can save on average ~1 minute per build for a 2GB Docker image.
-
7.3.0: Mount npm build secret to a separate directory (#1278)
Our templated Buildkite pipelines currently retrieve a temporary
.npmrc
. This file contains an npm read token that allows us to fetch private@seek
-scoped packages.New projects now write this file to
/tmp/
on the Buildkite agent and mount it as a secret to/root/
in Docker. This separation allows you to commit a non-sensitive.npmrc
to your GitHub repository while avoiding accidental exposure of the npm read token. This is especially important if you are migrating a project to pnpm, which houses some of its configuration options in.npmrc
.Existing projects are generally advised to wait until we’ve paved a cleaner migration path for pnpm.
-
7.0.0: Change some info logs to debug (#1178)
The “Function succeeded” log message was changed from
info
todebug
to reduce the amount of unnecessary logs in production. The message will still be logged in dev environments but at adebug
level. -
7.0.0: Bump aws-sdk-client-mock to 3.0.0 (#1197)
AWS SDK v3.363.0 shipped with breaking type changes.
-
6.2.0: Include manifest files in CODEOWNERS (#1162)
Our templates previously excluded
package.json
andyarn.lock
from CODEOWNERS. This was intended to support advanced workflows such as auto-merging PRs and augmenting GitHub push notifications with custom tooling. However, we are reverting this configuration as it is more common for SEEKers to prefer a simpler CODEOWNERS-based workflow.This will not affect existing projects. If you create a new project and wish to restore the previous behaviour, you can manually extend
.github/CODEOWNERS
:* @<%- ownerName %> + # Configured by Renovate + package.json + yarn.lock
-
5.1.0: Declare
dd-trace
dependency (#1051)This resolves a
Runtime.ImportModuleError
that occurs if this transitive dependency is not installed:Runtime.ImportModuleError Error: Cannot find module 'dd-trace'
Alternatively, you can configure the Datadog Serverless plugin to bundle these dependencies via Lambda layers:
serverless.yml custom: datadog: - addLayers: false + addLayers: true
package.json { "dependencies": { - "datadog-lambda-js: "x.y.z", - "dd-trace: "x.y.z" }, "devDependencies": { + "datadog-lambda-js: "x.y.z", + "dd-trace: "x.y.z" } }
-
5.1.0: Bump Node.js version to 18 (#1049)
This release contains some breaking changes to the Lambda runtime such as the removal of AWS SDK V2 in favour of AWS SDK V3. See the AWS Lambda Node.js 18.x runtime announcement for more information.
-
5.1.0: Use single hyphen in
renovate-
branch name prefix (#1050) -
5.0.0: Bump greeter and API templates to Node.js 18 (#1011)
Node.js 18 is now in active LTS. The Lambda templates are stuck on Node.js 16 until the new AWS Lambda runtime is released.
-
5.0.0: Replace Runtypes with Zod as default schema validator (#984)
-
5.0.0: Adjust Buildkite pipelines for new
renovate--
branch name prefix (#1022)See the pull request that aligns our Renovate presets for more information.
-
5.0.0: Support AMD64 Docker builds via
BUILDPLATFORM
(#1021)See the Docker documentation for more information. Note that this does not allow you to build on AMD64 hardware then deploy to ARM64 hardware and vice versa. It is provided for convenience if you need to revert to an AMD64 workflow and/or build and run an image on local AMD64 hardware.
-
4.4.1: Switch to modern Datadog integration (#965)
Datadog’s CloudWatch integration and the associated
createCloudWatchClient
function fromseek-datadog-custom-metrics
have been deprecated. We recommend Datadog’s Serverless Framework Plugin along with their first-party datadog-lambda-js and dd-trace npm packages. -
4.3.0: Remove
.me
files (#902)SEEK is moving away from Codex to off-the-shelf software powered by Backstage
catalog-info.yaml
files.At the moment we’re only asking teams to document their systems, which typically span across multiple repositories. We may add
catalog-info.yaml
files back to the templates if there’s a need for teams to document their components at a repository level. -
4.3.0: Use ARM64 architecture (#873)
We now recommend building and running projects on ARM64 hardware for greater cost efficiency. This requires a Graviton-based Buildkite cluster; see our ARM64 guide for more information.
-
4.2.2: Avoid mutation of logger context (#879)
We now perform a shallow copy when retrieving the logger context from
AsyncLocalStorage
.- mixin: () => loggerContext.getStore() ?? {}, + mixin: () => ({ ...loggerContext.getStore() }),
-
4.2.1: Time out Buildkite test steps after 10 minutes (#842)
Successful testing and linting should complete within this window. This timeout prevents commands from hanging and indefinitely preoccupying your Buildkite agents.
steps: - label: 🧪 Test & Lint + timeout_in_minutes: 10
-
4.2.1: Use AsyncLocalStorage to track logger context (#871)
We now employ this Node.js API to thread logging context through the handler of your Lambda function. This enables use of a singleton
logger
instance instead of manually propagating Lambda context and jugglingrootLogger
s andcontextLogger
s, and is equivalent to #864.Before:
import createLogger from '@seek/logger'; import { Context } from 'aws-lambda'; const rootLogger = createLogger(); const contextLogger = ({ awsRequestId }: Context) => rootLogger.child({ awsRequestId }); const handler = async (_event: unknown, ctx: Context) => { rootLogger.info('Has no context'); contextLogger(ctx).info('Has context'); };
After:
import { AsyncLocalStorage } from 'async_hooks'; import createLogger from '@seek/logger'; import { Context } from 'aws-lambda'; const loggerContext = new AsyncLocalStorage<{ awsRequestId: string }>(); const logger = createLogger({ mixin: () => ({ ...loggerContext.getStore() }), }); const handler = (_event: unknown, { awsRequestId }: Context) => loggerContext.run({ awsRequestId }, async () => { logger.info('Has context'); });
-
4.2.0: Propagate Buildkite environment variables for lint autofixing (#800)
-
4.2.0: Exclude DOM type definitions by default (#822)
TypeScript will now raise compiler errors when DOM globals like
document
andwindow
are referenced in new projects. This catches unsafe usage of Web APIs that will throw exceptions in a Node.js context.If you are developing a new npm package for browser use or require specific Node.js-compatible Web APIs like the Encoding API, you can opt in to DOM type definitions in your
tsconfig.json
:{ "compilerOptions": { - "lib": ["ES2020"] + "lib": ["DOM", "ES2020"] } }
If you have an existing backend project, you can opt out of DOM type definitions in your
tsconfig.json
.For Node.js 14:
{ "compilerOptions": { + "lib": ["ES2020"], "target": "ES2020" } }
For Node.js 16:
{ "compilerOptions": { + "lib": ["ES2021"], "target": "ES2021" } }
-
4.1.1: Disable type checking in tests (#787)
Newly initialised projects will skip TypeScript type checking on
skuba test
as it’s already covered byskuba lint
. You can now iterate on your tests without running into annoying compilation errors like TS6133 (unused declarations).This will be defaulted for existing projects in a future major version. You can opt in early by setting the
globals
configuration option in yourjest.config.ts
:export default Jest.mergePreset({ globals: { 'ts-jest': { // seek-oss/skuba#626 isolatedModules: true, }, }, // Rest of config });
-
4.1.0: Remove
variablesResolutionMode
(#768)This resolves the following deprecation warning in Serverless Framework v3:
Starting with v3.0, the "variablesResolutionMode" option is now useless. You can safely remove it from the configuration More info: https://serverless.com/framework/docs/deprecations/#VARIABLES_RESOLUTION_MODE
-
4.1.0: Add
NODE_ENV=production
to environment variables (#763) -
4.1.0: Move environment variables to
provider.environment
to reduce repetition (#767) -
4.0.0: Use
--enable-source-maps
(#761)Stable source map support has landed in Node.js 14.18+ via the built-in
--enable-source-maps
option.We recommend migrating off of custom source map implementations in favour of this option. Upgrading to skuba-dive v2 will remove
source-map-support
from theskuba-dive/register
hook.For a containerised application, update your Dockerfile:
- FROM gcr.io/distroless/nodejs:12 AS runtime + FROM gcr.io/distroless/nodejs:16 AS runtime + # https://nodejs.org/api/cli.html#cli_node_options_options + ENV NODE_OPTIONS=--enable-source-maps
For a Serverless Lambda application, update your
serverless.yml
:provider: - runtime: nodejs12.x + runtime: nodejs14.x functions: Worker: environment: + # https://nodejs.org/api/cli.html#cli_node_options_options + NODE_OPTIONS: --enable-source-maps
For a CDK Lambda application, update your stack:
new aws_lambda.Function(this, 'worker', { - runtime: aws_lambda.Runtime.NODEJS_12_X, + runtime: aws_lambda.Runtime.NODEJS_14_X, environment: { + // https://nodejs.org/api/cli.html#cli_node_options_options + NODE_OPTIONS: '--enable-source-maps', }, });
-
4.0.0: Disable
tty
on deploy step (#753)Serverless Framework v3 renders progress spinners on interactive terminals. We recommend disabling tty in CI/CD for cleaner log output.
-
4.0.0: Replace
custom.env
configuration withparams
(#752)You can now define environment specific variables using the new Serverless parameters feature. See https://www.serverless.com/framework/docs/guides/parameters for more details.
-
4.0.0: Remove
provider.lambdaHashingVersion
(#751)This resolves the following deprecation warning in Serverless Framework v3:
Setting "20201221" for "provider.lambdaHashingVersion" is no longer effective as new hashing algorithm is now used by default. You can safely remove this property from your configuration.
-
3.17.2: Remove qualifier from smoke test invocation (#743)
Previously, this template’s smoke test hook specified a
$LATEST
qualifier in itsLambda.Invoke
API call. AWS authorised the call based on the unqualified Lambda ARN in ourserverless.yml
IAM policy, but will stop doing so after April 2022.To avoid deployment failures, remove the qualifier in
src/hooks.ts
. An unqualified call is equivalent to targeting$LATEST
.- Qualifier: '$LATEST', + Qualifier: undefined,
-
3.16.0: Use correct
environment
key indocker-compose.yml
(#654) -
3.16.0: Switch to ARM64 architecture (#640)
These are a bit cheaper and a bit faster than x86 Lambdas: https://aws.amazon.com/blogs/aws/aws-lambda-functions-powered-by-aws-graviton2-processor-run-your-functions-on-arm-and-get-up-to-34-better-price-performance/
The underlying Lambda architecture should be invisible to typical TypeScript Lambdas.
-
3.16.0: Bump non-Lambda templates to Node.js 16 (#633)
Node.js 16 is now in active LTS. The Lambda templates are stuck on Node.js 14 until the new AWS Lambda runtime is released.
-
3.16.0: Ignore
.gantry
YAML paths via.prettierignore
(#636)Gantry resource and value files often live in the
.gantry
subdirectory and may use non-standard template syntax. -
3.16.0: Propagate environment variables for GitHub annotations (#642)
This enables GitHub annotations for newly-initialised projects with the appropriate Buildkite configuration.
-
3.15.2: Convert Serverless
isProduction
config value to boolean (#602)This avoids potentially surprising behaviour if you try to make use of this config value in a context that tests for truthiness. The boolean is still correctly applied as a string
seek:env:production
tag value. -
3.15.2: Opt in to new Serverless variables resolver (#601)
-
3.15.2: Remove README tables of contents (#596)
GitHub’s Markdown renderer now generates its own table of contents.
-
3.15.2: pino-pretty ^6.0.0 (#594)
pino-pretty@7 requires pino@7, which has not been released on its stable channel yet.
-
3.15.1: Remove
unknown
specifier in catch clauses (#580)Strict TypeScript 4.4 now defaults to typing catch clause variables as
unknown
. -
3.15.0: Configure environment variables and volume mounts for Buildkite annotations (#558)
-
3.15.0: Prime dev ECR cache in Buildkite pipeline (#503)
This should result in faster “Deploy Dev” times as the ECR cache will already be warm.
-
3.15.0: Remove
@types/node
resolution override (#498)Jest 27.1 is compatible with newer versions of
@types/node
. -
3.15.0: Remove Yarn cache from worker Docker images (#499)
This shrinks the cached Docker images that our worker templates generate.
-
3.14.3: Force
@jest/types
resolution to fix clean installs (#468) -
3.14.3: Use Docker Build secrets (#476)
-
3.14.3: Group Buildkite pipeline anchors (#474)
This provides a bit more structure to our
pipeline.yml
s and allows anchored plugins to be recognised by Renovate. -
3.14.3: Default Docker Compose image to empty string (#469)
This suppresses Docker Compose CLI warnings and errors when running outside of Buildkite.
-
3.14.3: Use BUILDKITE_PIPELINE_DEFAULT_BRANCH in
pipeline.yml
(#475) -
3.14.2: Set
memorySize
for smoke test hook to 128 MiB (#457) -
3.14.0: Prune
devDependencies
instead of installing twice in Docker (#435)The template-bundled Dockerfiles would previously run
yarn install
twice to build a separate stage for productiondependencies
only. These have been updated to correctly share the Yarn cache across stages and to useyarn install --production
to perform offline pruning. -
3.13.0: Upgrade to Node 14 (#347)
Node.js 14 is now supported on AWS Lambda. This lets us upgrade the Node.js requirement for skuba’s templates.
This should only impact newly created projects. You can use the template changes in this PR as an example of how to upgrade an existing project. A future version of skuba may include a fixup command to automatically upgrade your project to the most recent LTS release.
-
3.13.0: Use new
serverless.yml#/provider/iam
grouping (#357)The
provider.iamRoleStatements
property will be removed in Serverless v3. -
3.13.0: serverless-plugin-canary-deployments ^0.5.0 (#394)
The plugin now patches in CodeDeploy permissions to your
iamRoleStatements
, so you can clean yourserverless.yml
:- - Action: codedeploy:PutLifecycleEventHookExecutionStatus - Effect: Allow - Resource: !Sub arn:aws:codedeploy:${AWS::Region}:${AWS::AccountId}:deploymentgroup:*/${WorkerLambdaFunctionDeploymentGroup}
-
3.13.0: Remove custom Serverless variable syntax (#350)
serverless@2.3.0
bundled native support for CloudFormation pseudo parameters. This even works with arbitrary logical IDs like!Sub ${WorkerLambdaFunctionDeploymentGroup}
. -
3.13.0: Use new
serverless.yml#/package/patterns
property (#415)The
package.exclude
andpackage.include
properties will be removed in Serverless v3. -
3.13.0: Add GitHub repository settings and Renovate to init checklist (#388)
-
This brings back versioned functions along with
serverless-prune-plugin
to control Lambda storage consumption. By default we configureserverless-plugin-canary-deployments
for an instantaneous switch once the smoke test has passed, but this can be customised as necessary. -
3.12.0: Enable retry of successful deployment steps (#311)
This should be used with caution, but may be necessary if you need to rapidly roll back a broken deployment.
-
3.12.0: Lock Serverless
lambdaHashingVersion
(#329)This gets rid of the following warning when deploying:
Deprecation warning: Starting with next major version, default value of provider.lambdaHashingVersion will be equal to "20201221" More Info: https://www.serverless.com/framework/docs/deprecations/#LAMBDA_HASHING_VERSION_V2
-
3.10.2: Default to unversioned Lambdas (#245)
Our baseline template does not do canary deployments, and this makes it less likely to hit code storage limits down the road.
-
3.10.0: Remove region from subscription example snippet (#223)
-
3.10.0: Recommend
@seek/logger
(#225)This provides logging structure, trimming and redaction over plain Pino.
-
3.7.7: Remove explicitly set NPM_READ_TOKEN from Dockerfile commands (#168)
lambda-sqs-worker-cdk
An asynchronous worker built on AWS Lambda and deployed with AWS CDK.
SNS -> SQS (with a dead-letter queue) -> Lambda
Comes with configuration validation and infrastructure snapshot testing.
Added in 3.13.0
Changelog
-
9.1.0: Align template with Serverless template (#1577)
This adds the same boilerplate code available in
lambda-sqs-worker
along with Datadog integration. -
9.0.0: Replace custom hooks with
@seek/aws-codedeploy-infra
(#1644) -
9.0.0: Point Docker base images to AWS ECR Public and remove constant
--platform
arguments (#1684) -
8.2.1: Remove JSON schema definitions from Buildkite pipeline files (#1624)
This reverts #1611 due to incompatibility with pipeline signing.
-
8.2.0: Add JSON schema definitions to Buildkite pipeline files (#1611)
-
8.1.0: Add extension recommendations to
.vscode/extensions.json
(#1556) -
8.1.0: Make all configuration values explicit (#1560)
Previously,
src/config.ts
included optional configuration values and inheritance between environments in the spirit of DRY. While the templated file was wired up in a “safe” way—the production environment never inherited from other environments and explicitly specified all its configuration values—its pattern was misappropriated elsewhere and led to local configuration values affecting production environments.Instead, we now list all configuration values explicitly against each environment.
-
8.1.0: Remove deprecated
docker-compose.yml
version (#1570)Docker has ignored this for a while, and now generates a warning on every build: https://github.com/compose-spec/compose-spec/blob/master/04-version-and-name.md
-
8.0.1: Install specific pnpm version via Corepack (#1515)
Previously, our Dockerfiles ran
corepack enable pnpm
without installing a specific version. This does not guarantee installation of the pnpm version specified inpackage.json
, which could cause a subsequentpnpm install --offline
to run Corepack online or otherwise hang on stdin:FROM --platform=arm64 node:20-alpine RUN corepack enable pnpm
{ "packageManager": "pnpm@8.15.4", "engines": { "node": ">=20" } }
Corepack is about to download https://registry.npmjs.org/pnpm/-/pnpm-8.15.4.tgz. Do you want to continue? [Y/n]
To avoid this issue, modify (1) Buildkite pipelines to cache on the
packageManager
property inpackage.json
, and (2) Dockerfiles to mountpackage.json
and runcorepack install
:- seek-oss/docker-ecr-cache#v2.1.0: + seek-oss/docker-ecr-cache#v2.2.0: cache-on: - .npmrc + - package.json#.packageManager - pnpm-lock.yaml
FROM --platform=arm64 node:20-alpine - RUN corepack enable pnpm + RUN --mount=type=bind,source=package.json,target=package.json \ + corepack enable pnpm && corepack install
-
8.0.1: Align dead letter queue naming with Serverless template (#1542)
-
8.0.1: Replace CDK context based config with TypeScript config (#1541)
-
8.0.0: Remove
BUILDPLATFORM
from Dockerfiles (#1350)Previously, the built-in templates made use of
BUILDPLATFORM
and a fallback value:FROM --platform=${BUILDPLATFORM:-arm64} gcr.io/distroless/nodejs20-debian11
- Choose the platform of the host machine running the Docker build. An AWS Graviton Buildkite agent or Apple Silicon laptop will build under
arm64
, while an Intel laptop will build underamd64
. - Fall back to
arm64
if the build platform is not available. This maintains compatibility with toolchains like Gantry that lack support for theBUILDPLATFORM
argument.
This approach allowed you to quickly build images and run containers in a local environment without emulation. For example, you could
docker build
anarm64
image on an Apple Silicon laptop for local troubleshooting, while your CI/CD solution employedamd64
hardware across its build and runtime environments. The catch is that your localarm64
image may exhibit different behaviour, and is unsuitable for use in youramd64
runtime environment without cross-compilation.The built-in templates now hardcode
--platform
as we have largely converged onarm64
across local, build and runtime environments:FROM --platform=arm64 gcr.io/distroless/nodejs20-debian11
This approach is more explicit and predictable, reducing surprises when working across different environments and toolchains. Building an image on a different platform will be slower and rely on emulation.
- Choose the platform of the host machine running the Docker build. An AWS Graviton Buildkite agent or Apple Silicon laptop will build under
-
8.0.0: Remove account-level tags from resources (#1494)
This partially reverts #1459 and #1461 to avoid unnecessary duplication of account-level tags in our templates.
-
7.5.0: Update tests to use a stable identifier for the
AWS::Lambda::Version
logical IDs in snapshots. This avoid snapshot changes on unrelated source code changes. (#1450) -
7.4.0: Use
propagate-environment
for Docker Compose Buildkite plugin (#1392)This simplifies the Docker Compose environment variable configuration required for Buildkite and GitHub integrations.
In your
docker-compose.yml
:services: app: - environment: - # Enable Buildkite + GitHub integrations. - - BUILDKITE - - BUILDKITE_AGENT_ACCESS_TOKEN - - BUILDKITE_BRANCH - - BUILDKITE_BUILD_NUMBER - - BUILDKITE_JOB_ID - - BUILDKITE_PIPELINE_DEFAULT_BRANCH - - BUILDKITE_STEP_ID - - GITHUB_API_TOKEN image: ${BUILDKITE_PLUGIN_DOCKER_IMAGE:-''} init: true volumes: - ./:/workdir # Mount agent for Buildkite annotations. - /usr/bin/buildkite-agent:/usr/bin/buildkite-agent # Mount cached dependencies. - /workdir/node_modules
In your
.buildkite/pipeline.yml
:steps: - commands: - pnpm lint - pnpm test env: # At SEEK, this instructs the build agent to populate the GITHUB_API_TOKEN environment variable for this step. GET_GITHUB_TOKEN: 'please' plugins: - *aws-sm - *private-npm - *docker-ecr-cache - docker-compose#v4.16.0: + environment: + - GITHUB_API_TOKEN + propagate-environment: true run: app
-
7.4.0: Add blue-green deployment, smoke test and version pruning functionality (#1327)
-
7.4.0: Set maximum concurrency (#1412)
This prevents messages from going directly to the DLQ when the function reaches its reserved concurrency limit.
-
7.4.0: Introduce bundling with esbuild,
--hotswap
and--watch
(#1321)This template now uses the
aws_lambda_nodejs.NodejsFunction
construct which uses esbuild to bundle the Lambda function. This reduces cold start time and time to build on CI.The
--hotswap
and--watch
options allow you to rapidly deploy your code changes to AWS, enhancing the developer feedback loop. This change introducesdeploy:hotswap
anddeploy:watch
scripts to thepackage.json
manifest and aDeploy Dev (Hotswap)
step to the Buildkite pipeline. Read more about watch and hotswap on the AWS Developer Tools Blog. -
7.3.1: Update to Node 20 (#1317)
Consider upgrading the Node.js version for your project across:
.nvmrc
package.json#/engines/node
serverless.yml
@types/node
package version- CI/CD configuration (
.buildkite/pipeline.yml
,Dockerfile
, etc.)
If you are updating your AWS Lambda runtime to
nodejs20.x
, consider reading the release announcement as there are some breaking changes with this upgrade. -
7.3.0: seek-oss/docker-ecr-cache 2.1 (#1266)
This update brings a new
skip-pull-from-cache
option which is useful onWarm
/Build Cache
steps.At SEEK, our build agents no longer persist their Docker build cache from previous steps. This option allows a preparatory step to proceed on a cache hit without pulling the image from ECR, which can save on average ~1 minute per build for a 2GB Docker image.
-
7.3.0: Mount npm build secret to a separate directory (#1278)
Our templated Buildkite pipelines currently retrieve a temporary
.npmrc
. This file contains an npm read token that allows us to fetch private@seek
-scoped packages.New projects now write this file to
/tmp/
on the Buildkite agent and mount it as a secret to/root/
in Docker. This separation allows you to commit a non-sensitive.npmrc
to your GitHub repository while avoiding accidental exposure of the npm read token. This is especially important if you are migrating a project to pnpm, which houses some of its configuration options in.npmrc
.Existing projects are generally advised to wait until we’ve paved a cleaner migration path for pnpm.
-
7.0.0: Change some info logs to debug (#1178)
The “Function succeeded” log message was changed from
info
todebug
to reduce the amount of unnecessary logs in production. The message will still be logged in dev environments but at adebug
level. -
7.0.0: Bump aws-sdk-client-mock to 3.0.0 (#1197)
AWS SDK v3.363.0 shipped with breaking type changes.
-
6.2.0: Include manifest files in CODEOWNERS (#1162)
Our templates previously excluded
package.json
andyarn.lock
from CODEOWNERS. This was intended to support advanced workflows such as auto-merging PRs and augmenting GitHub push notifications with custom tooling. However, we are reverting this configuration as it is more common for SEEKers to prefer a simpler CODEOWNERS-based workflow.This will not affect existing projects. If you create a new project and wish to restore the previous behaviour, you can manually extend
.github/CODEOWNERS
:* @<%- ownerName %> + # Configured by Renovate + package.json + yarn.lock
-
5.1.0: Declare
dd-trace
dependency (#1051)This resolves a
Runtime.ImportModuleError
that occurs if this transitive dependency is not installed:Runtime.ImportModuleError Error: Cannot find module 'dd-trace'
Alternatively, you can configure the Datadog Serverless plugin to bundle these dependencies via Lambda layers:
serverless.yml custom: datadog: - addLayers: false + addLayers: true
package.json { "dependencies": { - "datadog-lambda-js: "x.y.z", - "dd-trace: "x.y.z" }, "devDependencies": { + "datadog-lambda-js: "x.y.z", + "dd-trace: "x.y.z" } }
-
5.1.0: Bump Node.js version to 18 (#1049)
This release contains some breaking changes to the Lambda runtime such as the removal of AWS SDK V2 in favour of AWS SDK V3. See the AWS Lambda Node.js 18.x runtime announcement for more information.
-
5.1.0: Use single hyphen in
renovate-
branch name prefix (#1050) -
5.0.0: Bump greeter and API templates to Node.js 18 (#1011)
Node.js 18 is now in active LTS. The Lambda templates are stuck on Node.js 16 until the new AWS Lambda runtime is released.
-
5.0.0: Replace Runtypes with Zod as default schema validator (#984)
-
5.0.0: Replace Runtypes with Zod as default schema validator (#984)
-
5.0.0: Adjust Buildkite pipelines for new
renovate--
branch name prefix (#1022)See the pull request that aligns our Renovate presets for more information.
-
5.0.0: Support AMD64 Docker builds via
BUILDPLATFORM
(#1021)See the Docker documentation for more information. Note that this does not allow you to build on AMD64 hardware then deploy to ARM64 hardware and vice versa. It is provided for convenience if you need to revert to an AMD64 workflow and/or build and run an image on local AMD64 hardware.
-
4.4.1: Switch to modern Datadog integration (#965)
Datadog’s CloudWatch integration and the associated
createCloudWatchClient
function fromseek-datadog-custom-metrics
have been deprecated. We recommend Datadog’s Serverless Framework Plugin along with their first-party datadog-lambda-js and dd-trace npm packages. -
4.3.0: Remove
.me
files (#902)SEEK is moving away from Codex to off-the-shelf software powered by Backstage
catalog-info.yaml
files.At the moment we’re only asking teams to document their systems, which typically span across multiple repositories. We may add
catalog-info.yaml
files back to the templates if there’s a need for teams to document their components at a repository level. -
4.3.0: Use ARM64 architecture (#873)
We now recommend building and running projects on ARM64 hardware for greater cost efficiency. This requires a Graviton-based Buildkite cluster; see our ARM64 guide for more information.
-
4.2.2: Avoid mutation of logger context (#879)
We now perform a shallow copy when retrieving the logger context from
AsyncLocalStorage
.- mixin: () => loggerContext.getStore() ?? {}, + mixin: () => ({ ...loggerContext.getStore() }),
-
4.2.1: Time out Buildkite test steps after 10 minutes (#842)
Successful testing and linting should complete within this window. This timeout prevents commands from hanging and indefinitely preoccupying your Buildkite agents.
steps: - label: 🧪 Test & Lint + timeout_in_minutes: 10
-
4.2.1: Use AsyncLocalStorage to track logger context (#871)
We now employ this Node.js API to thread logging context through the handler of your Lambda function. This enables use of a singleton
logger
instance instead of manually propagating Lambda context and jugglingrootLogger
s andcontextLogger
s, and is equivalent to #864.Before:
import createLogger from '@seek/logger'; import { Context } from 'aws-lambda'; const rootLogger = createLogger(); const contextLogger = ({ awsRequestId }: Context) => rootLogger.child({ awsRequestId }); const handler = async (_event: unknown, ctx: Context) => { rootLogger.info('Has no context'); contextLogger(ctx).info('Has context'); };
After:
import { AsyncLocalStorage } from 'async_hooks'; import createLogger from '@seek/logger'; import { Context } from 'aws-lambda'; const loggerContext = new AsyncLocalStorage<{ awsRequestId: string }>(); const logger = createLogger({ mixin: () => ({ ...loggerContext.getStore() }), }); const handler = (_event: unknown, { awsRequestId }: Context) => loggerContext.run({ awsRequestId }, async () => { logger.info('Has context'); });
-
4.2.0: Propagate Buildkite environment variables for lint autofixing (#800)
-
4.2.0: Exclude DOM type definitions by default (#822)
TypeScript will now raise compiler errors when DOM globals like
document
andwindow
are referenced in new projects. This catches unsafe usage of Web APIs that will throw exceptions in a Node.js context.If you are developing a new npm package for browser use or require specific Node.js-compatible Web APIs like the Encoding API, you can opt in to DOM type definitions in your
tsconfig.json
:{ "compilerOptions": { - "lib": ["ES2020"] + "lib": ["DOM", "ES2020"] } }
If you have an existing backend project, you can opt out of DOM type definitions in your
tsconfig.json
.For Node.js 14:
{ "compilerOptions": { + "lib": ["ES2020"], "target": "ES2020" } }
For Node.js 16:
{ "compilerOptions": { + "lib": ["ES2021"], "target": "ES2021" } }
-
4.1.1: Disable type checking in tests (#787)
Newly initialised projects will skip TypeScript type checking on
skuba test
as it’s already covered byskuba lint
. You can now iterate on your tests without running into annoying compilation errors like TS6133 (unused declarations).This will be defaulted for existing projects in a future major version. You can opt in early by setting the
globals
configuration option in yourjest.config.ts
:export default Jest.mergePreset({ globals: { 'ts-jest': { // seek-oss/skuba#626 isolatedModules: true, }, }, // Rest of config });
-
4.1.0: Remove
variablesResolutionMode
(#768)This resolves the following deprecation warning in Serverless Framework v3:
Starting with v3.0, the "variablesResolutionMode" option is now useless. You can safely remove it from the configuration More info: https://serverless.com/framework/docs/deprecations/#VARIABLES_RESOLUTION_MODE
-
4.1.0: Add
NODE_ENV=production
to environment variables (#763) -
4.1.0: Add
NODE_ENV=production
to environment variables (#763) -
4.1.0: Move environment variables to
provider.environment
to reduce repetition (#767) -
4.0.0: Use
--enable-source-maps
(#761)Stable source map support has landed in Node.js 14.18+ via the built-in
--enable-source-maps
option.We recommend migrating off of custom source map implementations in favour of this option. Upgrading to skuba-dive v2 will remove
source-map-support
from theskuba-dive/register
hook.For a containerised application, update your Dockerfile:
- FROM gcr.io/distroless/nodejs:12 AS runtime + FROM gcr.io/distroless/nodejs:16 AS runtime + # https://nodejs.org/api/cli.html#cli_node_options_options + ENV NODE_OPTIONS=--enable-source-maps
For a Serverless Lambda application, update your
serverless.yml
:provider: - runtime: nodejs12.x + runtime: nodejs14.x functions: Worker: environment: + # https://nodejs.org/api/cli.html#cli_node_options_options + NODE_OPTIONS: --enable-source-maps
For a CDK Lambda application, update your stack:
new aws_lambda.Function(this, 'worker', { - runtime: aws_lambda.Runtime.NODEJS_12_X, + runtime: aws_lambda.Runtime.NODEJS_14_X, environment: { + // https://nodejs.org/api/cli.html#cli_node_options_options + NODE_OPTIONS: '--enable-source-maps', }, });
-
4.0.0: Disable
tty
on deploy step (#753)Serverless Framework v3 renders progress spinners on interactive terminals. We recommend disabling tty in CI/CD for cleaner log output.
-
4.0.0: Replace
custom.env
configuration withparams
(#752)You can now define environment specific variables using the new Serverless parameters feature. See https://www.serverless.com/framework/docs/guides/parameters for more details.
-
4.0.0: Remove
provider.lambdaHashingVersion
(#751)This resolves the following deprecation warning in Serverless Framework v3:
Setting "20201221" for "provider.lambdaHashingVersion" is no longer effective as new hashing algorithm is now used by default. You can safely remove this property from your configuration.
-
3.17.2: Remove qualifier from smoke test invocation (#743)
Previously, this template’s smoke test hook specified a
$LATEST
qualifier in itsLambda.Invoke
API call. AWS authorised the call based on the unqualified Lambda ARN in ourserverless.yml
IAM policy, but will stop doing so after April 2022.To avoid deployment failures, remove the qualifier in
src/hooks.ts
. An unqualified call is equivalent to targeting$LATEST
.- Qualifier: '$LATEST', + Qualifier: undefined,
-
3.17.0: Fix docker-compose volume mount and deploy output (#695)
-
3.16.0: Use correct
environment
key indocker-compose.yml
(#654) -
3.16.0: Switch to ARM64 architecture (#640)
These are a bit cheaper and a bit faster than x86 Lambdas: https://aws.amazon.com/blogs/aws/aws-lambda-functions-powered-by-aws-graviton2-processor-run-your-functions-on-arm-and-get-up-to-34-better-price-performance/
The underlying Lambda architecture should be invisible to typical TypeScript Lambdas.
-
3.16.0: Bump non-Lambda templates to Node.js 16 (#633)
Node.js 16 is now in active LTS. The Lambda templates are stuck on Node.js 14 until the new AWS Lambda runtime is released.
-
3.16.0: Ignore
.gantry
YAML paths via.prettierignore
(#636)Gantry resource and value files often live in the
.gantry
subdirectory and may use non-standard template syntax. -
3.16.0: Propagate environment variables for GitHub annotations (#642)
This enables GitHub annotations for newly-initialised projects with the appropriate Buildkite configuration.
-
3.15.2: Convert Serverless
isProduction
config value to boolean (#602)This avoids potentially surprising behaviour if you try to make use of this config value in a context that tests for truthiness. The boolean is still correctly applied as a string
seek:env:production
tag value. -
3.15.2: Opt in to new Serverless variables resolver (#601)
-
3.15.2: Remove README tables of contents (#596)
GitHub’s Markdown renderer now generates its own table of contents.
-
3.15.2: pino-pretty ^6.0.0 (#594)
pino-pretty@7 requires pino@7, which has not been released on its stable channel yet.
-
3.15.1: Remove
unknown
specifier in catch clauses (#580)Strict TypeScript 4.4 now defaults to typing catch clause variables as
unknown
. -
3.15.0: Configure environment variables and volume mounts for Buildkite annotations (#558)
-
3.15.0: Prime dev ECR cache in Buildkite pipeline (#503)
This should result in faster “Deploy Dev” times as the ECR cache will already be warm.
-
3.15.0: Remove
@types/node
resolution override (#498)Jest 27.1 is compatible with newer versions of
@types/node
. -
3.15.0: Run “Test, Lint & Build” step in prod (#503)
This reduces our dependence on a dev environment to successfully deploy to prod.
-
3.15.0: Remove Yarn cache from worker Docker images (#499)
This shrinks the cached Docker images that our worker templates generate.
-
3.14.3: Force
@jest/types
resolution to fix clean installs (#468) -
3.14.3: Use Docker Build secrets (#476)
-
3.14.3: Use Docker Build secrets (#476)
-
3.14.3: Group Buildkite pipeline anchors (#474)
This provides a bit more structure to our
pipeline.yml
s and allows anchored plugins to be recognised by Renovate. -
3.14.3: Default Docker Compose image to empty string (#469)
This suppresses Docker Compose CLI warnings and errors when running outside of Buildkite.
-
3.14.3: Use BUILDKITE_PIPELINE_DEFAULT_BRANCH in
pipeline.yml
(#475) -
3.14.3: Build once upfront (#477)
This employs Buildkite artifacts to share compiled code with each subsequent deployment step.
-
3.14.2: Set
memorySize
for smoke test hook to 128 MiB (#457) -
3.14.0: Always build before deploy (#428)
This prevents stale compiled code from being cached and deployed from ECR.
-
3.14.0: Prune
devDependencies
instead of installing twice in Docker (#435)The template-bundled Dockerfiles would previously run
yarn install
twice to build a separate stage for productiondependencies
only. These have been updated to correctly share the Yarn cache across stages and to useyarn install --production
to perform offline pruning.