How I solved my over-fetching Notion problem

Posted on: 18 June 2026

A few years ago I wrote about bringing my Notion database to life with Eleventy. This has formed the bedrock of the data on my site. However every single build re-queried my entire journal database. Here's how I switched to an incremental cache that, in theory, never has to fetch everything again.

The problem with fetching everything

My weekly feed is powered by a Notion database. Every day I log entries — TV, films, podcasts, books, todos — and on a weekly basis, I rebuild my site. This instructs Eleventy to pull it all in at build time to generate my weekly stats feed.

The original approach was simple: query the whole database, paginating through every entry, on every build. That worked fine for a while, but it's fundamentally wasteful. The vast majority of my journal never changes — last year's entries are relatively set in stone. Yet there I was, paging through years of history just to pick up the handful of rows I'd touched that week.

A while back I wrote about how if you're not moving forward, you're regressing — and the example in that post was this very bottleneck. Builds had crept up to requesting 13+ pages of journal data, taking five minutes or more to complete, if they completed at all. I diagnosed the problem there and promised myself a fresh solution that leaned less heavily on Notion's free API. This is that solution.

I already cached the result to avoid hammering Notion during development, but a cache that expires still means re-fetching the entire database the moment it goes stale. And it was going stale, by design, every week. I wanted something smarter: fetch everything once, then only ever ask Notion for what's actually changed.

The idea: cache the lot, then top it up

The plan came together in three parts:

  1. Query everything once and cache it for a long time — 12 weeks.
  2. When serving from that cache, fire a second, tiny query asking Notion only for rows edited since the cache was last written.
  3. Merge any changes back into the main cache and reset its expiry, so it stays warm indefinitely.

In theory, after that first full fetch, the expensive query never runs again. Each build does a cheap "what's new since last time?" request and stitches the results in.

Asking Notion for only what changed

I added a simple Last edited time column to my journal database. It was easy enough to use this field in my incremental query to filter just those records:

// Only fetch entries edited since the last time we saved the cache
const recentFilter = (from) => {
  return {
    property: 'Last edited time',
    date: {
      on_or_after: from.toISOString(),
    }
  };
};

The key is knowing when the cache was last written (the from attribute). I reworked my notion.queryDatabase method so that instead of just returning the data, it tells me whether the data came from cache and when it was stored:

if (asset.isCacheValid(duration)) {
  return {
    fromCache: true,
    cachedAt: new Date(asset.cachedObject.cachedAt),
    data: await asset.getCachedValue(),
  };
}

If the cache was cold, we do a full query for all journal data, and tell the caller the data was not from cache:

  const fresh = await callback();

  asset.save(fresh, 'json');

  return { fromCache: false, cachedAt: new Date, data: fresh };

Now the data file can make a decision. If we're working from cache, fire the incremental query using the cache's own timestamp as the from date:

module.exports = async function() {
  const notion = new NotionApi('<DATABASE ID>');

  let { data, fromCache, cachedAt } = await notion.queryDatabase('journal-data', '12w');

  if (fromCache) {
    const recent = recentFilter(cachedAt);
    const freshResults = await notion.queryDatabase('updated-journal-data', '5m', recent);

    // If we get fresh results, not from cache, update the original entry in the main cache
    if (!freshResults.fromCache && freshResults.data.length > 0) {
      notion.updateCache('journal-data', mergeFreshData(freshResults.data, data));
    }
  }

  // ...map the rows into my feed shape
};

The full dataset is cached under journal-data for 12 weeks. The incremental "what changed?" query is cached separately under updated-journal-data for just 5 minutes — enough to dedupe builds that happen in quick succession, without going stale. This is a small timesaver that prevents me needing to re-query Notion's API when I'm rebuilding multiple times in development. It does mean I have to wait 5 minutes after making a change in Notion for it to be reflected, but I decided this was a worthwhile trade-off.

Merging the deltas

When the incremental query does return something, I need to fold those rows into the cached dataset. Updated rows should replace their existing counterparts in place; genuinely new rows should land at the top of the feed:

const mergeFreshData = (fresh, existing) => {
  const freshData = Object.fromEntries(fresh.map((row) => [row.id, row]));

  const newData = existing.map(row => {
    const updatedRow = freshData[row.id] || row;
    delete freshData[row.id];
    return updatedRow;
  });

  Object.values(freshData).reverse().forEach(row => newData.unshift(row));

  return newData;
};

I walk the existing cache, swapping in any fresh version of a row I find (and removing it from the freshData map as I go). Whatever's left in freshData afterwards must be brand new, so it gets prepended. That merged array is written straight back to the journal-data cache, resetting its 12-week clock.

Caching the current week too

There was one snag I came across towards the end of making this change. Previously I asked Notion to exclude the in-progress current week, so I always cached (and displayed in my feed) complete weeks. But that meant edits to entries in the current week weren't being picked up when I came round to rebuilding my site.

So I flipped it: cache everything, including the current week, and filter the cutoff out in my own code instead. Now any edit gets pulled into the cache, and the "only show complete weeks" rule is applied at render time rather than at the Notion query.

Was it worth it?

Absolutely. The development experience of updating the feed each week is significantly improved now I don't need to page throguh my entire journal history to rebuild my site. My load on the Notion API is significantly less, which is a win for me and Notion. And ultimately I'm doing less pointless processing. It was a nice little win and the kind of small, self-contained problem that makes tinkering with your own site so enjoyable.

Tags: coding notion eleventy javascript