Build the ship around the weapons. Add the minimum crew. You will end up with a smaller ship.
We do: It's called a frigate, and it has a minimal crew.
Also, not all ships need to carry all weapons so long as the task force is capable of managing all threats.
Well, none of your dinky toys 100 tons ship (260 tons actually, if the Bras D'or is your measuring stick) can carry some of the stuff we need to manage some of the threats.
BTW: concerning HMCS Bras D'or, she was never, ever, at any time, considered for anything else but ASW duties.
The Bras D'or was an experiment at a time when the West had to decide how to deal with a new threat to the resupply of Europe: nuclear submarines that could run faster underwater than the escort ships of the day. The idea was that she would sprint, dip a small sonar (not a VDS - too big for her - but something akin to a helicopter dipping sonar), sprint again to get closer, then dip, then move, etc. until she could drop a light torpedo on the submarine. The experiment was cancelled for two reasons. First, she went on a Bermuda ex deployment, with her full crew of twenty-five
(so small, in fact that she did not carry any cooks - the meals were all frozen "TV diner" type and the crew prepared their own meals using, get this, some of the first micro-wave ovens in the world , pre commercialization of the process). On that ex, she carried no weapons system nor any dipping sonar, but merely simulated how she would operate. It was a disaster as the crew became too fatigued to operate properly after two weeks of operation. Second, she had short legs when she sprinted a lot, which would obviously happen when engaged in escort duties, and, as demonstrated in basin tests, could not refuel alongside (because, like a small fish, she got "reeled-in" whenever the tension was put on the high wire). That meant refuelling astern only - not the best of choices.
She was finally done in for by a combination of world crisis: the first oil crisis of the early 70's, which meant she was too expansive to run, and technological developments in the ASW fight: the foreseeable entry in service of towed away sonars and the general use of ASW helicopters onboard escort ships.