I was debugging a problem that only existed when the server was running in the cluster. That meant that I changed the code, waited a whooping 10-15 minutes to see if the change worked or not or to see the extra logging I had added. What a waste of time. This is how I managed to get that build time to only about 5 minutes.
Using yarn instead of npm
This might be obvious to some but as nodejs is not my favorite environment and not what I (want to) work with full-time, I was not aware of the differencies of npm and yarn. The major difference in this case was that yarn is multithreaded while npm is not. This makes a huge difference when the tool downloads dependencies when building the container.
So the old build pipeline was using many npm install commands in this manner:
cd /some/sub/project1 && npm install --production && \
cd /some/sub/project2 && npm install --production && \
cd /some/sub/project3 && npm install --production && \
cd /some/sub/project4 && npm install --production && \
cd /some/directory/ && npm install --production
The total time this took for the full backend build was around 30-40 minutes including admin front-end which I am not addressing here.
- Game Worker - 2-4 minutes build time
- Game API - 2-4 minutes build time
- Admin Backend - 3-5 minutes build time
That totals up to around 13 minutes of build time. Sometimes it took even more than that, up to 15 minutes or so.
The change to yarn is quite simple. Just remove the package-lock.json and run yarn to generate the yarn lock file. Then in the dockerfile you need to replace npm with yarn.
cd /some/sub/project1 && yarn install --production && \
cd /some/sub/project2 && yarn install --production && \
cd /some/sub/project3 && yarn install --production && \
cd /some/sub/project4 && yarn install --production && \
cd /some/directory/ && yarn install --production
After Changing npm to yarn the build times decreased drastically
- Game Worker - 1-2 minutes build time
- Game API - 1-2 minutes build time
- Admin Backend - 1-2 minutes build time
That totals up to 6 minutes build time. That's a good improvement in my book.
Multiple build stages
Then I figured that a lot of the build process is about the common libraries used by all of the services. I wanted to try out if it helps build times to combine the common library in to a single docker image that is then being used as a base for each of the services.
So I added a Dockerfile.common that has the build steps for the common libraries and removed those from the main docker containers.
cd /some/sub/project1 && yarn install --production && \
cd /some/sub/project2 && yarn install --production && \
cd /some/sub/project3 && yarn install --production && \
cd /some/sub/project4 && yarn install --production && \
cd /some/directory/ && yarn install --production
This mess became Dockerfile.common with
cd /some/sub/project1 && yarn install --production && \
cd /some/sub/project2 && yarn install --production && \
cd /some/sub/project3 && yarn install --production && \
cd /some/sub/project4 && yarn install --production
and the main service Dockerfiles then had just this
cd /some/directory/ && yarn install --production
This further reduced the build times as the common docker container had to be built only once, which takes around 1-1.5 minutes
- Common libraries - 1.5 minutes build time
- Game Worker - <1 minutes build time
- Game API - <1 minutes build time
- Admin Backend - <1 minutes build time
And that's how the whole project builds now in less than five minutes. I'm happy about this result.
Further improvements
I have not tried it yet as I have not felt it being necessary but I think the build process could be further optimized by running the common library build commands in parallel with GNU Parallel or something similar.