March 28, 2021

Understanding Vercel lambda size errors

Stumbling across the error

Last night I was working on adding the ability to dynamically generate og:images for this site. Following some similar implementations, I decided to use puppeteer-core and chrome-aws-lambda to take a snapshot of a page that I built that acts as a preview of the og:image. Once I got it set up, I ran across a weird error trying to deploy Vercel.

⚠️
Error: The Serverless Function "api/feed/atom" is 50.38mb which exceeds the maximum size limit of 50mb. Learn More: https://vercel.link/serverless-function-size

This error was weird to me because I know my atom feed API route (and its dependencies) are nowhere near 50 mb. The learn more link to the error description essentially says what's already apparent: You can't have lambdas that big. This lambda isn't that big though, right?

The only possible culprit for this would have to be puppeteer. None of my other dependencies are remotely that big. The weird part is that the feed API doesn't import puppeteer. On top of that Vercel's output for the atom feed API in the build summary said it was only supposed to be ~70 kb. While trying to reason through all this, I noticed the following line above the error message in the logs.

Traced Next.js server files in: 15500.049ms

I took two things from this

  1. Vercel is analyzing the all the lambdas' dependencies to see what should be included for server execution.
  1. Given that it took so long and the bundle was reported to be so large, they may actually be bundling all the server dependencies together and shipping that for each lambda. Maybe via a lambda layer?

While the exact mechanics weren't clear to me, I had enough to go on to start digging through GitHub for answers. My first question was simple: how do they trace the server dependencies?

Tracing the clues

After searching around in their GitHub, I stumbled across vercel/nft (Node File Trace). What's the first thing in the readme?

This package is used in @vercel/node and @vercel/next to determine exactly which files (including node_modules) are necessary for the application runtime.

bingo.

The API for this library is incredibly simple. Give it a list of files to analyze and it'll return a list of all those files dependencies.

const { nodeFileTrace } = require('@vercel/nft');
const files = ['./src/main.js', './src/second.js'];
const { fileList } = await nodeFileTrace(files);

All I needed then was the list of files to trace. The tracing step in the logs is listed right after Next's build step, so the build output directory seemed like a good place to look.

Next generates a .next directory anytime you run next build. My hope was to find something to key me off in there.

Next's build output
Next's build output

Turns out that hope was well founded. Inside of .next there's a server directory containing all the resources for running things server side. There's a pages-manifest.json file that looked like exactly what I was looking for. Also note that there's a pages directory which contains all the server modules compiled and ready to go.

{
  "/_app": "pages/_app.js",
  "/_document": "pages/_document.js",
  "/_layout": "pages/_layout.html",
  "/about": "pages/about.js",
  "/api/feed/atom": "pages/api/feed/atom.js",
  "/api/feed/json": "pages/api/feed/json.js",
  "/api/feed/rss": "pages/api/feed/rss.js",
  "/api/post/[slug]": "pages/api/post/[slug].js",
  "/": "pages/index.js",
  "/og/posts/[slug]": "pages/og/posts/[slug].js",
  "/posts/[post]": "pages/posts/[post].js",
  "/posts/_layout": "pages/posts/_layout.html",
  "/tips": "pages/tips.js",
  "/_error": "pages/_error.js",
  "/404": "pages/404.html"
}

The first thing I wanted to know here was if somehow puppeteer was accidently getting bundled up into the atom feed lambda. I wrote a out a little script just to test that.

const { nodeFileTrace } = require('@vercel/nft')

nodeFileTrace(['.next/server/pages/api/feed/atom.js']).then(({ fileList }) =>
  fileList.forEach((file) => console.log(file))
)

Running this confirmed what I had already expected: No puppeteer dependency. At this point, I'm relatively confident that my earlier assumption that everything is bundled together is true. I wanted to go a little further though and have a detailed log of all the dependencies, their sizes, and the total size of the output. To achieve that goal I wrote this trace script. It passes the values of the server manifest file to node file trace, maps through the results and uses fs.stat to get the file size for each, does some formatting to make it readable, and dumps it to a file called trace-debug.log. The results were... revealing.

The fruits of our labor

I won't dump the whole 1200 lines of files being bundled together here, but the top of the log says enough.

Total size: 55.8 MB

44.0 MB node_modules/chrome-aws-lambda/bin/chromium.br
3.6 MB .next/server/pages/tips.js
1.5 MB node_modules/chrome-aws-lambda/bin/aws.tar.br
991.4 KB node_modules/chrome-aws-lambda/bin/swiftshader.tar.br
529.8 KB node_modules/lodash/lodash.js
267.7 KB node_modules/framer-motion/dist/framer-motion.cjs.js
132.7 KB .next/server/pages/posts/[post].js
129.3 KB .next/server/pages/index.js
124.0 KB .next/server/pages/about.js

The 55.8 MB is a bit bigger than the 50.38 MB that Vercel showed in the error, but not by much. There's likely something in this list that's getting stripped out. Still... that's a lot. Unsurprisingly chromium is the behemoth. That means that if chrome-aws-lambda (even without puppeteer) is included in your build you've already lost 88% of the code capacity of your server bundle. The entirety of the remainder of your site and all its dependencies needs to fit in 6 MB. If you do something crazy like me and ship prettier in one of your lambdas to format code snippets then you're just SOL.

Now what?

I plan to release a more generic version of the script at some point, but it should work well enough if you just plop it in your project now. This at least helps reveal what's been hard to debug up to this point.

The obvious next question is probably this... how do we fix it? I just thought I'd need to host the puppeteer rendering somewhere else. Björn Rave showed me otherwise.

So, your tip of the day. While Next.js bundles all it's server dependencies together, any API endpoints living at the root in api/ will be bundled separately. These functions are Vercel's serverless functions which don't have all the niceties of Next's functions, but are individually bundled. I've got that working on my site now if you want to check it out. If folks are interested I could write a guide for that.

Takeaways for the Vercel crew

If there's any of the Vercel crew reading this, I do have a few takeaways that I think would make this experience better.

  1. While I understand that how lambda dependencies are bundled is an implementation detail that most folks don't necessarily need to know about, it's also a leaky abstraction. It may be worth putting some details in the docs about how Next bundles its server dependencies on Vercel and linking to that instead of just the generic lambda size error docs if the lambda size for Next is too big.
  1. There should absolutely be warnings in the build logs when a build is getting close to running over the 50mb limit. I think it'd also be useful just to list out the top 5 or 10 files by size to inform the user what's contributing to the bloat. Likewise if it errors out... list the top files and their sizes as possible causes.