Even if they’re not on your friend list you can still be on theirs. The friend list works more like Twitter where you can follow someone without you following back.
You should set the crew to invite only once in game.
Also experiencing several issues on Xbox one X since last weeks patch. There was a few posts about it not too long after the patch dropped.
I’m experiencing low quality textures, textures failing to load etc. Also seems as though the resolution has taken a hit alongside the inconsistent frames.
I’m sure it’s a bug rather than an intentional downgrade so I’d recommend submitted a support ticket detailing your experience.
There was a noticeable downgrade post launch which was upsetting at the time that affected the LODs and severely increased the pop in. I still experience rocks and animals pop in ten feet from me at times. Sometimes players disappear out of sight if I’m in the crows nest overlooking an island beside me etc.
Hopefully these new issues get sorted in a hot fix or at least in the next patch.