Understanding nodejs-yarn And Caching

[brockwood]

Greetings,

First off, thank you for this buildpack. We're currently exploring deploys that can target both Heroku and a Docker platform and this seems like a good fit.

One area that we are having an issue with is image size. When running pack build, the resulting Docker image is gigabytes bigger than trying to build this directly with a Dockerfile. While exec'ing into the final image, we've discovered that /layers/heroku_nodejs-yarn/deps/cache contains the cached Nodejs modules from the yarn install which, for our app, is over 1.5 gigabytes in size. Most likely due to my limited understanding, I've tried to get around this by using the flag --publish along with --cache-image but the final app image still contains all the saved zip file data. Reading docs for the build process, it is not obvious if there's a workaround for this behavior. Is it possible to limit the heroku/nodejs-yarn:deps layer to the cache image? Or, if this isn't the right place to ask, do you have a recommendation where I should direct this question? Thank you for your time.

[colincasey]

@brockwood This looks like an unintended side-effect of supporting Yarn's Plug'n'Play feature which, due to the buildpack configuration, requires the /layers/heroku_nodejs-yarn/deps/cache layer to be present at launch to work. This is not ideal since, as you noted, the result is a much larger Docker image than necessary for Yarn installs that don't use the pnp feature.

I believe this will need to be addressed by adding code to detect when pnp mode is required and only making the deps layer available at runtime in that case. I would have to look into this more though to see what other options are available.

Keep in mind, these Cloud Native Buildpacks are currently in preview so issues are to be expected. Your feedback here is definitely appreciated!

[brockwood]

@colincasey Thank you for the detailed reply. One final question since this is fresh in your mind. I have found that if I perform a build and specify --volume /path/to/cache:/layers/heroku_nodejs-yarn:rw, the image avoids storing the zipped modules and it starts as expected. Do you foresee any issues using this approach for the time being?

[colincasey]

That's an interesting workaround @brockwood 😄

Where that approach might run into trouble:

• Running an application configured to use the Yarn pnp feature since those dependencies will not be present at /layers/heroku_nodejs-yarn/deps
• Running an application configured to use Yarn 1.x or a vendored Yarn release (i.e.; yarnPath since the global binary is installed at /path/to/cache:/layers/heroku_nodejs-yarn/dist. This should be avoidable by installing Yarn via corepack.

Another option would be to modify the image during the build phase using an inline buildpack to strip out the contents of the Yarn cache without messing with other layers. For example, if you added a project.toml file to your application with the following contents:

[_]
schema-version = "0.2"

[[io.buildpacks.group]]
id = "heroku/nodejs"
version = "3.2.13"

[[io.buildpacks.group]]
id = "yarn/delete-cache"
[io.buildpacks.group.script]
api = "0.10"
inline = """
echo "===> Deleting yarn cache"
rm -rf /layers/heroku_nodejs-yarn/deps/cache
"""

You could also combine this technique with yours to keep build caching of modules run a couple scripts that:

• move modules a provided volume into /layers/heroku_nodejs-yarn/deps/cache before heroku/nodejs
• move the files from /layers/heroku_nodejs-yarn/deps/cache into the provided volume after heroku/nodejs