We'll start with terminology: there are no "side of international waters" involved here. International waters are just that: international. Thus no one has jurisdiction over anyone in international waters other than over their own flagged ships.
What you do have between Canada and the USA in the great lakes, part of the St Lawrence River and on some other smaller waterways (such as Champlain Lake) are waters that are either internal or territorial to Canada or the US, on either side of agreed lines that make up the international border.
The general rule is that if you are transiting territorial waters (i. the Great Lakes - other than Michigan which is internal to the US - or the St Lawrence river) to get from a point in Canada to another point in Canada or a point in the US to another point in the US without stopping, then you need not report to customs/border services of either country. However, and regardless, at any given time you are either in Canadian or in American waters, in which case you are subject to all the laws and rules of that country and are answerable to the proper law enforcement authorities of the country you are in.
The US Coast Guard does law enforcement in US waters - generally, that does not include border protection/customs work, which is usually done by border patrol. In the case at hand, the allegation was that the USCG involvement arose from an infraction to fishing laws and regulations, which would appropriately fall under their jurisdiction.