Yeah I struggle too. For me the problem happens because I have a limited amount of time to play, and if I have any large gaps between playtime I'm likely to start the game from the beginning to start fresh.
It's also why I really appreciate recaps on load screens, like with dragon quest XI.
To be fair it almost feels like a homage to early bioware. I remember a few bioware games that had this issue. I remember the forums being full of complaints about surprise romances in mass effect or dragon age.
I do think the issue is more prevalent I'm BG3 though.