The same could be said for Rum Bottles as well. I see this as part of the intended experience. A fair bit of what Merchants are meant to transport are fragile things in some way (Animals can die, Plants can wither, Cloth can be soaked, Bottles can break, Gunpowder can be detonated). So to me it just feels in theme with the Faction design.
As for it being more risky in a Sloop, that is also working as designed I feel like. The Sloop is essentially always in a more risky position than any other size ship. Less crew, less space, less storage options by a full barrel.
All in all I don't think this needs adjustment, this is exactly how it is supposed to work. If you're not a fan of the stuff that carries that Fragile Goods facet of things you'll just need to be more picky about which Merchant Voyages you decide to actually do. You have the ability to cancel a Voyage and propose a new one without actually doing it, and that is a viable strategy if you don't feel the Voyage you got by RNG aligns with your goals or feels less profitable than it is worth.
But the Voyages cost Gold you might say, and not everyone has loads of Gold to toss around. Well, that is where Sunken Loot, Message in a Bottle Voyages, and Emergent Loot come in to play. You always have the ability to get more Gold for Voyages, and they are so cheap it basically just requires turning in a single piece of just about any Loot to have enough Gold to grab one. So cancellation of Voyages is wildly viable in this game thankfully.
So, my advice is to either accept that some Goods you'll be transporting will be Fragile, or simply start skipping Voyages (wholesale or in part, you can always do the things on a Voyage you're willing to do and then cancel as well).
Alternately, if you don't like Merchant Voyages at all, you're in luck since getting to Pirate Legend only requires you to have Rank 50 in 3 Factions, which Factions those are is up to you. So you still have Gold Hoarders, Order of Souls, Hunters Call, Reapers Bones, and Sea Dogs as viable options to fill out your card. You could just skip the Merchants entirely, and level them up via delivering Gems which can be taken by any Faction other than Athena's Fortune (and Sea Dogs, since they don't really exist within Adventure in reality).
Now, if you're a Completionist and hate the Merchants, all I can say is that as a Completionist you must be used to doing tasks in games you're not a fan of to get that job done. This is just gonna be another one of those things I fear, in that particular case.
So, in summation, you have a lot of options here, so I don't think this actually needs to be changed or balanced further. This advice is equally here to help out anyone else who pops in and may be feeling the same, but isn't aware of all the details (the OP very well might be aware of all this, in which case please don't see it as talking down to you - this is just as much for others).
Good luck out there, and may you find the Adventures you are looking for!