I think the brig is a terrible idea since inmates (assuming that they are in there for legitimate reasons) take up a required crew position, and still gain from this. However, if you intend to keep it, then you should also restrict their ability to interact with objects.
There is a way to break out of the brig and once this happens you still have full functionality to empty barrels, fire cannons, pick up and drop treasures, drop anchor. The crew can always vote to release you and then lock you in again but that assumes they've noticed.
Removing the ability to interact whilst locked in the brig means that breaking out has no benefit.
