That happens because of exhaust velocity. Notice how a header is all bent funny? Wouldn't straight tubing be simpler?
Yes.
But, the tubes are made from exact lengths for a reason... when each exhaust pulse hits the collector at the end of the header, the pulses are timed in such a way that it creates a small vacuum (ie: Scavenging) that increases power by reducing pumping losses from the exhaust stroke, and helping to suck in the new charge of fresh air during valve overlap.
When your exhaust is TOO large, then the speed at which the pulses hit the collector goes down, reducing the scavenging effect. This is why some old farts say when they tell you that "Backpressure is NEEDED by the engine to make sure you valves work properly, blah, blah, blah". Truth is, most of them just don't understand the purpose of an exhaust manifold (header).
Turbocharged (but not supercharged) engines don't count, as the location of the turbo prevents any sort of scavenging from taking place. However, super/turbocharged engines are designed with more valve overlap, so that the pressurized charge of fresh air blows out the exhaust during the cycle, having a similar effect.
And, of course, a smaller exhaust is undesirable because of restriction (much like an intake). The bigger the straw, the more milkshake you get (intake), and the bigger bubbles you can blow in your soda (exhaust).