There is a tonne of abandoned rail-line in Eastern Canada, like a metric tonne. Moreover, the dedicated trackage VIA uses is all mostly double track in the Corridor. The issue isn't the track itself or really even the freight trains. There is more than enough room to facilitate meets and additional crossovers and/or sidings could be installed to facilitate additional passenger rail service, a cost to the Govt of course.
It's the 1000s of unprotected at grade crossings that exist on the corridor and really, what amounts to poor municipal/urban planning, that gets in the way. It gets worse the closer you get to major urban centres. All those folks that wanted waterfront property which just happened to be next to former industrial sites and rail operations.