A general guide on how to handle common pagination patterns.
Pagination is one of the most common patterns for fetching from a large pool of items. When we have too many items to fetch and display on a single page, we limit the amount of returned items per fetch. We can fetch and filter additional items as needed.
Because we are fetching from a large pool of items, performance and speed become a top priority when implementing and designing pagination.
Optimizing Fetches
Fetching paginated items becomes slow when we are fetching a higher number page. Let's say that we have a total of 100 items, and each page shows 10 items. That means there are 100 items / 10 items / page = 10 pages total. On the initial page load, we always fetch the first page, and this page always fetches the fastest out of the others pages.
Let's say that we jump from the 1st page to the 3rd page. Here is what happens behind the scenes in the case of unoptimized pagination (slow). This is the order of events that will take place:
The most important concept to know in pagination is the following:
Skip is a very slow operation
In general, you always want to limit the number of items skipped to be as small as possible. This is because the skip operation has to physically go through all those documents in the cursor to skip it.
Side note: In the above example, we can see why the first page always loads the fastest. Because we are fetching the first 10 items, we can skip step 3 because we have no items to skip.
Now, lets' see how we can optimize our pagination fetch from page 1 to page 3:
Using the last/first element
We can take advantage of the fact that we already fetched 10 items and have the data of these items displayed on the current page. We can use the data from these items to optimize our fetch. Our new fetch would look something like this:
We are adding an additional filter to return fewer items for our initial search. We can use the last known date (this can be a different variable depending on what you are sorting by) to prevent the first 10 elements to be returned from our initial filter. Because the current 10 items on the page are not being returned, we only need to skip 10 elements rather than 20! Note that the skip amount is limited to the relative distance from the current page. That means that the farther away page the user navigates to, the more items we need to skip in our query, and the slower the query will be. Keep this in mind when deciding how many pages to display at any one time for your paging method.
Going backwards (from a larger page to a smaller page) will look slightly different. Instead of filtering for items greater than the last element, we filter for elements less than the first element. This will then return all items before the first current first element. Because our cursor location is now flipped, we need to negate the original sort and reverse the returned array of items.
Design Choices
Paging
Paging involves associating page numbers to the order of items that are returned from pagination. Think of any job board that allows you to toggle the number page you are on. You can either make these pages an entirely new route or utilize query variables to trigger re-fetches within the same page. My personal preference is using query variables since the page stays the same outside of the fetched items. But this might differ for your application!
Infinite Scroll
Infinite scroll is when you can scroll for what seems like an infinite amount of time and items continuously populate the screen. Whenever we fetch additional items, it stems from the user doing something in your application. With paging, this involves the user navigating to a new page number. With infinite scroll, it involves the user reaching a certain point within the list of items to trigger a fetch. So when the user gets close to the bottom of the page (maybe 3 elements from the last index), we can fetch more items. Infinite scroll also guaranteed that we won't need to skip any items because we are always fetching in sequential order.
Router vs. State
You will need to store filters and fetch variables in a local component state that triggers a re-fetch every time the search criteria are updated. This is the main mechanism used to handle filtering paginated items. However, it is up to you whether you want to also push these variables onto the URL. The advantage of doing this would be that whenever the user refreshes or shares that URL, the items will match the search criteria in the URL. The downside is that the URL will look ugly. So up to you if you decide to implement router pagination!
Filters
When fetching paginated items, there most likely will be some kinds of filters placed on these items. If you are on a job board, you might want to first filter the jobs by location. Make sure to apply these filters prior to using the pagination patterns described above.