Flying equals 'getting high', so to speak.
In the Europe of the Middle Ages and into the Renaissance, bread was made, in large part, with rye. And rye and rye-like plants can host fungus—ergot—that can be a powerful hallucinogen. Records from the 14th to the 17th century mention Europeans' affliction with "dancing mania," which found groups of people dancing through streets—often speaking nonsense and foaming at the mouth as they did so—until they collapsed from exhaustion.
Ergot has been connected to unusual behaviors in Salem, Massachusetts, and may have led to many a lady being burned at the stake for BBAW (Behavior Befitting A Witch).
As for the connection with brooms? You'll need to investigate that yourself. You will never look at a broom the same way again, however. We guarantee.