I think a number of our discussions right now about starting domains early in play come down to the question of what it means to be a follower.
Beedo’s blog http://dreamsinthelichhouse.blogspot.com/2011/08/under-new-management-castle-ravenloft.html says:
It takes a garrison to secure a domain, so the PCs will have to secure the services of 512gp value in mercenaries or men-at-arms. No free followers appear until 9th level! 512gp would buy around 40 heavy infantry or crossbowman - typical chainmail wearing mooks.
It seems to me that part of what makes someone a follower is that they are not a NPC any more. There’s a chance that those hired mooks will go on strike, or secretly be working to your enemy.
That’s the stick. But a good rule rewards you for following it, not just punishes you for not. If the only benefit of followers is that the Judge can’t turn them against you, it puts a big burden on the Judge to screw you with hired mercenaries in a way that could easily become un-fun. (Although, if this is a table or dice result that is known to the players and comes up randomly until you have followers - like you have to roll morale checks for hirelings but not followers, and a roll of snake-eyes makes bad things happen - I think that’s more fun and less a GM screwjob).
So I think the carrot is that when you get followers you get XP for their activities, at some reduced rate. These XP wouldn’t be enough for Conqueror-level characters to advance at the usual pace, but it would supplement their advancement. One reason not to have your higher-level PCs do all the adventuring is that you want to keep them alive, so running an adventure with disposable Adventurers becomes more attractive if it also helps level your higher-level guys without risking their skins.
Put together, I think this creates an awesome situation where at lower levels, you luck into a situation where you might wind up ruling a domain bigger than you’d be able to build, or you start building a wooden palisade fort or a seagoing vessel and hiring untrustworthy mercenaries to man it. At these levels, you have to do all the work; you always need to be worrying about the loyalty of your mercenaries, especially if you’re housing them in a vast castle you can’t afford to repair and it’s been months since you could afford to pay them. And you generally need to deal with threats yourself - you need your mercenaries to staff whatever structure you have, so if there’s trouble on the borders you have to deal with it.
At higher levels, maybe you’ve had your fill of these kind of adventures. If so, you go chase some other game, and leave your followers to run things. Or maybe you really like this kind of play, so now you switch to playing your followers and leave your original characters to level up slowly based on the XP you pass on. Or maybe you’re not all that interested in domain play; you gain followers and tell them “go build my domain offstage while I continue to kick ass in person.”