I have mostly the same experience as O.P., and I play pretty similarly as well (I never fire unless fired at first), and will only become an "aggressor" when dealing with PvP greifers, or if needing to re-acquire stolen loot.
It's rather laughable to be sailing to an island to see that ship in the distance turn from their current heading on sight, even when you hardly have anything, or nothing at all. I highly doubt this was the intended behavior of the game and it really puts a nail in the coffin for some of these ideas that Rare threw up on the roadmap of "needing crews to work together". One of my friends put it in a perfect fashion, everyone can work to finish the AI off, but then they'll just be right back at it once it goes down, looking to just swipe the rewards from the people who helped.
The "It's a pirate game" isn't a valid argument either (Looking at @MolanAbram particularly here), open world games like these are built to garnish interaction of players, and having cooperative experiences lend a hand much further towards community building versus competitive experiences, the results speak for themselves across numerous titles. Sure having some PvP is good, don't get me wrong, but from a design perspective, you're much more likely to lose out on players when multiple hours of work can be erased in a few minutes just because someone wants to be a jerk about it.
I'll continue to state that a philosophy change is needed with this game, and it needs to happen sooner rather than later. At the moment, Rare has effectively created a large-scale Dark Zone from the Division. I don't think I need to remind people what happened to that game when they tried to market it as the "endgame", because I can tell you that a very similar playerbase cut off is very likely to occur if this continues.
Either way, I'll continue playing with friends, the experience can be enjoyable, but I'll continue to advocate for change in the game's philosophy, as I'd much rather see a game with a flourishing playerbase than one that drops off harshly because of one bad design choice.