From agility point of view, I did some testing with all three ships..
For timing purposes, I used a basic wallclock, so accuracy might not be what you'd like :D All numbers are in seconds and all actions have been done with one person only
Turning wheel from far left to far right
Sloop: 6, Brig: 10, Galleon: 15
Raising any sail from full to none
Sloop: 7, Brig: 10, Galleon: 12
Turning the sail from far left to far right
Sloop: 7, Brig: 12, Galleon: 17
Raising the anchor
Sloop: 8, Brig: 18, Galleon: 30
Doing a 360 turn on spot
Sloop: 40, Brig: 55, Galleon: 80
Edit: my opinion about this is that a crew of 3 on brig can outmaneuver a solo sloop if everyone is in equal skill level, as a solo slooper can only perform one action at the time. Galleon's operations being so much slower, even 4 man crew would have trouble doing the tasks fast enough compared to a solo sloop.