Smoothbore weapons deliver rounds at much higher velocity, which would seriously affect the "squash" part of the HESH round.
In principle, there is no reason a HESH round could not be made for a smooth bore cannon, but it would be fairly large and complex for the size of the warhead (needing pop out stabilizing fins), have a very low powered driving charge, and the ballistic computer and sights might have to be reprogrammed and modified to take into account the very slow flight time and "lobbing" trajectory.
Going the other way, the French actually made a HEAT round for use in their rifled cannons (the 105mm for the AMX-10RC armoured car and AMX-30 MBT), but this involved suspending the round on ball bearings and the driving band acting as the bearing race as the round went down the barrel. These rounds were not compatible with the 105mm L7 cannon either.
For the most part, HESH has been supplemented by such things as HEAT-MP rounds which have larger fragmentation effects and more behind armour effect, or enhanced HE like 84mm HEDP rounds for the Infantry. I also think (personal opinion) that the increasing use of compound armours negates much of the effect of the shockwave travelling through the armour plate and reduces the spalling effect inside the target, making HESH less effective these days. It is much harder to stop a KE penetrator or the hyper velocity "jet" of a HEAT round.
(Just as an aside, the jet of a HEAT round is moving at @ Mach 25, which is similar to the speed of a deorbiting projectile proposed in the THOR project. When you consider the damage a few ounces of metal moving at that speed can do, then scale it up to dozens or hundreds of kilograms [tactical THOR penetrators are supposed to be small, the size of broom handles, while the strategic ones that can penetrate missile silos and deep bunkers are indeed the size of telephone poles] the effects will be dramatic, to say the least).