The main performance pitfall with server-side rendered code occurs when it's time to hydrate the components with React.
When you receive HTML from a server, React has no way to interact with it. Hydration is the phase where your server-side rendered React components become interactive.
When you perform a Lighthouse test, hydration is taken into account in the Time-to-Interact indicator: the longer it takes, the more it negatively impacts your score.
As a rule of thumb: the bigger the bundle, the longer it will take. Decreasing your bundle size is often arduous though because performance improvement often results in functional sacrifices.
Lazy hydration remains much easier to implement and brings immediate returns. It consists in dividing the hydration process to prioritize elements above the fold: if the component can't be seen, and thus interacted with, we don't need to hydrate it just yet so we delay it for later.
The impact is proportional to the size of your Document Object Model, so it won't be as meaningful if the webpage is short.
In React, the simplest way to implement lazy hydration is to use the react-lazy-hydration package. It's downloaded 15,000 times per week on average, so you can't go wrong. It also gives you the ability to skip hydration entirely for the static parts of your components, which is a non-negligible performance boost as well.