I recently finished a really big refactoring from NextJS 13 to NextJS 14, mostly prompted by the server actions stuff, allowing me to entirely drop a publicly exposed GraphQL service, and do everything in backend Postgres.
That all worked wonderfully. Very pleased with that – any time I can delete stuff and yet maintain the exact same (or better) functionality, I am thrilled. Less stuff, less bugs. I’d say less grey hairs, but now that I’m competing against Prince William for most shocking male hairline, I don’t really have that as a concern.
You can see the site I’m talking about at BikeClimbs.com:
And that actually highlights two things.
One is that I royally screwed up the localisation of over 2.8 million climbs.
The other is that for every country in the world, I have a sub page which contains all the bike climbs I am aware of, broken down by state, county, and suburb.
That’s a hell of a lot of pages. It’s 2.8m + however many unique countries, states, counties, and suburbs there are in the world. Which is a lot.
What Problem Does Pre-Rendering Solve?
Landing on the climb list, you are first presented with a big list of unique countries.
Under each country is a count of all the climbs in that country.
I ended up having to do a materialised view to get this data, but even with that basically pre-queried data it’s still way slower than I would like. Slow enough to make fetching and rendering that data seem super sluggish. I know NextJS is pretty good (read: aggressive af) at caching this sort of stuff, but even so, it’s a crap user experience for the poor sucker user who hits that page for the first time.
So I then use NextJS static page functionality to build the pages at compile time, and so essentially what the end user sees is a static copy of the page – no database involved. It goes from taking about 20 seconds to load the page down to about 200ms.
Can’t be bad.
Now, I hit on some issues with this. Issues with basically having so many pages to pre-render that NextJS would, for want of a better word, crap out during the build.
I can’t remember the exact error from NextJS 13, and I’m not about to switch branches back and re-set all that back up, just to get it, but the gist of it was that it would cause the database to die. Don’t worry, if you like database errors, I got some more coming shortly even with the NextJS 14 approach – but thankfully, with a ‘fix’.
So here’s what I had:
// www/pages/climb/[country]/index.tsx
export async function getStaticProps({ params }) {
const data = await fetchAPI(combinedQuery(params.country));
const stateEdges = data.findStatesForCountry.edges;
const states = stateEdges.map((edge) => {
const { state, count } = edge.node;
return {
state,
count,
};
});
const allSegments = data.allSegments;
return {
props: {
states,
top100Climbs: top100Climbs(allSegments),
totalClimbCount: allSegments.totalCount,
},
};
}
export async function getStaticPaths() {
const uniqueCountriesData = await allCountries();
const uniqueCountryParams = uniqueCountriesData.map(({ country }) =>
slugIt(["climb", country])
);
return {
paths: uniqueCountryParams,
fallback: false, // See the "fallback" section below
};
}Code language: TypeScript (typescript)This worked fine enough for countries. There are about 96 countries or something, so throwing 96 queries at the database, give or take, didn’t cause too many problems. At least, not on the PC I build on.
But throw in all the states under a country, and I did hit some issues.
This bumps the count of pages needing pre-rendering up to 2,200, and for each page we’re doing several queries… so the DB just crapped out.
For what it’s worth, I think I ended up doing those calls via the API simply because I had to expose the GraphQL API to the front end for a part of how the site worked. And as I already had all these queries, I simply re-used them, rather than then going direct to the DB just for this part. It was probably me trying to decouple things… sounds like some logic I’d have had at some point.
Here’s what I ended up with:
// www/pages/climb/[country]/[state]/index.tsx
export async function getStaticProps({ params }) {
const data = await fetchAPI(combinedQuery(params.country, params.state));
const countyEdges = data.findCountiesForState.edges;
const counties = countyEdges.map((edge) => {
const { county, count } = edge.node;
return {
county,
count,
};
});
const allSegments = data.allSegments;
return {
props: {
counties,
top100Climbs: top100Climbs(allSegments),
totalClimbCount: allSegments.totalCount,
},
};
}
export async function getStaticPaths() {
const states = await throttledPromises(
async ({ country, countryNormalised }) => {
const data = await fetchAPI(`{ ${findStatesForCountryQuery(country)} }`);
const stateEdges = await data.findStatesForCountry.edges;
return stateEdges.map((edge) => {
return {
params: {
country: countryNormalised,
state: edge.node.stateNormalised,
},
};
});
},
await allCountries(),
450,
10
);
return {
paths: states.flat(),
fallback: false, // See the "fallback" section below
};
}
Code language: TypeScript (typescript)And this actually worked pretty well.
Throttling the generation of the static paths seemed to stall the NextJS page generator sufficiently.
Here’s the actual implementation of that function:
async function asyncForEach(array, callback) {
for (let index = 0; index < array.length; index++) {
await callback(array[index], index, array);
}
}
function split(arr, n) {
var res = [];
while (arr.length) {
res.push(arr.splice(0, n));
}
return res;
}
const delayMS = (t = 200) => {
return new Promise((resolve) => {
setTimeout(() => {
resolve(t);
}, t);
});
};
export const throttledPromises = (
asyncFunction,
items = [],
batchSize = 1,
delay = 0
) => {
return new Promise(async (resolve, reject) => {
const output = [];
const batches = split(items, batchSize);
await asyncForEach(batches, async (batch) => {
const promises = batch.map(asyncFunction).map((p) => p.catch(reject));
const results = await Promise.all(promises);
output.push(...results);
await delayMS(delay);
});
resolve(output);
});
};Code language: JavaScript (javascript)It’s not typescript.
And whilst it looks exactly like the sort of thing I would now palm off to ChatGPT, this one actually pre-dates me having access to a $20 genius, so I can only assume I did the other honourable thing, which is to shamelessly steal it from Stack Overflow.
The point is, it works.
The first argument is a function that gets the output from the second argument, which is the thing we want to throttle. Then the third argument is the amount of results to return per batch, and the fourth argument is how long to ‘pause’ between returning those batches.
That all works / worked fine for NextJS 13.
But What About generateStaticParams?
In NextJS 14 the concepts of getStaticProps and getStaticPaths are not available if you are using the App router.
This presented me with a problem.
I had been able to trim down the country page to this:
// /src/app/(display root)/climb/[country]/page.tsx
export async function generateStaticParams() {
return await getUniqueCountries();
}Code language: TypeScript (typescript)Don’t get me started on that (display root) dir. I think NextJS are going a bit too far with this stuff. But
Anyway, yeah, I cut down from that, what, 25 line setup for getStaticProps and getStaticPaths down to a one liner.
And for the other page:
// /src/app/(display root)/climb/[country]/[state]/page.tsx
export async function generateStaticParams() {
const staticParams = await getDistinctCountriesAndStates();
return staticParams.map(({ country_normalised, state_normalised }) => ({
country: country_normalised,
state: state_normalised,
}));
}Code language: JavaScript (javascript)So again, a big improvement.
And I’d say a lot of NextJS 14 code is like this, when compared to 13 and earlier.
The cost of that is a ton more is now happening under the hood, and that’s all fine until you need to do something a little “off grid”, and it becomes a bit of a mission.
Anyway, during dev this was all working absolutely fine.
But then I went to build:
Creating an optimized production build ...
✓ Compiled successfully
Skipping validation of types
✓ Linting
✓ Collecting page data
Generating static pages (15/2252) [=== ]
Generating static pages (22/2252) [ ===]
error: sorry, too many clients already
at /home/chris/Development/bikeclimbs.com/node_modules/pg-pool/index.js:45:11
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async d (/home/chris/Development/bikeclimbs.com/.next/server/app/(display root)/climb/[country]/page.js:4:320)
at async Promise.all (index 0)
at async Module.x (/home/chris/Development/bikeclimbs.com/.next/server/app/(display root)/climb/[country]/page.js:16:250)
at async tR (/home/chris/Development/bikeclimbs.com/node_modules/next/dist/compiled/next-server/app-page.runtime.prod.js:12:227547)
at async tP (/home/chris/Development/bikeclimbs.com/node_modules/next/dist/compiled/next-server/app-page.runtime.prod.js:12:227788)
at async tT (/home/chris/Development/bikeclimbs.com/node_modules/next/dist/compiled/next-server/app-page.runtime.prod.js:13:2220)
at async /home/chris/Development/bikeclimbs.com/node_modules/next/dist/compiled/next-server/app-page.runtime.prod.js:13:2632 {
length: 85,
severity: 'FATAL',
code: '53300',
detail: undefined,
hint: undefined,
position: undefined,
internalPosition: undefined,
internalQuery: undefined,
where: undefined,
schema: undefined,
table: undefined,
column: undefined,
dataType: undefined,
constraint: undefined,
file: 'proc.c',
line: '351',
routine: 'InitProcess'
}
error: sorry, too many clients already
at /home/chris/Development/bikeclimbs.com/node_modules/pg-pool/index.js:45:11
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async a (/home/chris/Development/bikeclimbs.com/.next/server/app/(display root)/climb/[country]/page.js:1:3834)
at async h (/home/chris/Development/bikeclimbs.com/.next/server/app/(display root)/climb/[country]/page.js:16:795)
at async Promise.all (index 1)
at async v (/home/chris/Development/bikeclimbs.com/.next/server/app/(display root)/climb/[country]/page.js:16:900) {
length: 85,
severity: 'FATAL',
code: '53300',
detail: undefined,
hint: undefined,
position: undefined,
internalPosition: undefined,
internalQuery: undefined,
where: undefined,
schema: undefined,
table: undefined,
column: undefined,
dataType: undefined,
constraint: undefined,
file: 'proc.c',
line: '351',
routine: 'InitProcess'
}
Error occurred prerendering page "/climb/bolivia". Read more: https://nextjs.org/docs/messages/prerender-error
Code language: JavaScript (javascript)So it managed about 20 pages before blowing up.
This is using the node library pg and from there I have a little helper function:
import { Pool } from "pg";
import config from "../../config";
const getDbPool = new Pool({
connectionString: config.db.climbs.connectionString,
max: 32,
});
export default getDbPool;Code language: TypeScript (typescript)Very quickly the 32 pool connections were being exhausted.
I think by default the Postgres container runs 100 max concurrent connections, with 3 ‘reserved’ for superuser access. Even if I had set max 100, I very much doubt this would have made it past 100 statically rendered pages before blowing up in a similar fashion.
The problem here was that the same solution from NextJS 13 doesn’t work.
I naively tried this:
export async function generateStaticParams() {
return await throttledPromises(
async (data) => {
console.log(`data`,data );
return data.map(({ country_normalised, state_normalised }) => ({
country: country_normalised,
state: state_normalised,
}));
},
await getDistinctCountriesAndStates(),
5,
10,
);
}Code language: TypeScript (typescript)This slows down how quickly the pages are made available to NextJS, but it doesn’t slow down how quickly it queries them. In fact it causes a different problem. And again I don’t have the error to hand any longer as that terminal session is very much dead, but the gist was that it timed out returning the pages to NextJS, not timed out trying to fetch them.
It’s a bit like me having a stack of A4 sheets of paper, and me handing them to you 1 by 1, pausing for a second between each sheet. You never start reading them till I pass you them all, so after me taking 60 seconds of your life, you say Chris, stop being an ass and let’s try this another way, shall we?
And to follow on from that, once you finally had them all, you read them at lightning speed, creating the same problem we already had.
Now, try as I might, I could not find a way to slow this down…
The ‘Fix’: Experimental Settings? Oh Yes
God knows I tried a lot of things to fix this, but you know it’s bad times when Googles throw up stuff from 2018 or whatever – not much use given NextJS14 is pretty new, and all the blogs and Stack Overflows and what have you on the topic of Next are assuming NextJS13 or less still.
I also combed the Github issues, and found nothing.
It’s weird, I would expect there to be a lot of people doing big pre-renders.
Anyway, so the fix:
// next.config.js
module.exports = {
// other stuff here as necessary
experimental: {
cpus: 2,
},
};Code language: JavaScript (javascript)Yeah.
My PC has one processor but 16 cores. At a guess, CPU here means core.
By default, it seems to be set to 4.
Yeah, I ended up prodding around the NextJS core code to try and find some sort of hook or way to throttle this.
In the end, I just lucked out. Not gunna lie.
Setting this to setting to 2 obviously slows down the build.
But so what? It completes and I can deploy the site. And that’s all that matters.
This doesn’t solve the root cause. It would be nice if we could throttle the build using a setting, and hey, maybe we can. But I couldn’t find it.