Sometimes I run into issues with NextJS's Fast Refresh. This pages documents some fixes to them.
Performing full page refresh
Naming page components the same as another page can cause a full reload. Make sure the page name is unique.
Higher order components can cause issues with fast refresh for this reason as well. Honestly I've found it best to avoid them.