Indeed, if you only have one mount, you're not really a cavalryman. You're just one mishap away from being a (bad) infantryman.
In many ancient steppe cultures (such as the Skythians) you weren't considered a rider (ie a warrior) unless you had at least four mounts. As an aside, I had the following house rule on remounts:
"Remounts - if the party has two mounts per character (including henchmen/hirelings), they may move at double the long-distance movement rates shown. This is double the speed of the slowest mount in the group. If the party has three mounts per character, they may move at 2.5 times long-distance rates. If the party has four or more mounts per character, they may move at triple the long-distance rates."
This is to account for the much faster overland speeds of steppe armies. If you have lots of horses/ponies (and all the logistical complications they entail) you can move a lot faster.