My group plays it as visible in both cases. A declaration of defensive movement shows a combatant positioning himself for safe withdrawl whereas a declared casting involves starting the complex process of spell-casting (involving chanting and gesturing).
I would say that casting with the Quiet Magic proficiency would be must harder to detect, requiring a hear noise check (possibly not required in a one-on-one or similarly intimate engagement). Likewise, the skirmishing proficiency, as it allows defensive movement without declaration, signifies a skill in disengaging quickly without giving that “tell” to the opponent.
The rules are not explicit on this point. From a Talmudic reading of the manual, I have ruled:
1. A declaration of spellcasting is apparent. How do we know? Because the spellcasting can be interrupted. Since the spellcasting can be interrupted, we can conclude that whatever is involved in spellcasting is happening; and since spellcasting requires vigorous speaking and gestures, the declaration must be visible. Note that although the declaration of spellcasting is apparent, the exact spell or target is not. A character with Collegiate Wizardry might be able to identify the spell being cast from the motions, words, and spell signature.
2. A declaration of full retreat is apparent. How do we know? Because the character suffers a penalty to AC that round. The AC penalty kicks in when the retreat is declared, and not when the retreat occurs. Therefore, some retreat-like activity must have begun at the moment of declaration.
3. A declaration of fighting withdrawal is not apparent. How do we know? Well, we don't, but what we do know is that the fighting withdrawal can't be interrupted and doesn't impose any penalty to the retreater. We also know that if an enemy *follows* a character who withdraws, the character who withdraws gets to attack, but no such ruling enables a withdrawing character to strike at an enemy who hits him before he withdraws. This suggests that the withdrawal doesn't actually occur when declared, but instead takes place on the character's initiative.
I think all of the above are also the correct interpretations in terms of fun gameplay. Having spellcasting be apparent means that spellcasters have a period of vulnerability where they can be interrupted. Having full retreat be apparent allows you to press the attack or allow them to escape, depending on alignment, goals, etc. Having withdrawal not be apparent means that a character can sometimes manage to fall back safely, with a pursuer unwlling to risk the free attack if he follows.