On the note of removing penalities, is that more the issue with some stats? For some stats there are options to remove said penalities, others not so much. For example the only real ability to offset a dex penality and add to AC is via Swashbuckling, which is no good for medium or heavly armoured folk, thus going to the light armoured folk that would likely have good Dex anyway.
For example maybe some defensive inspired fighting styles/proficencies?
Medium Armored: the character has trained to make the most of medium armour (Ring Mail or Chain Mail). They ignore any dexterity modifiers (bonuses or penalities) to their armour class and gain an additional +1 AC to the armour worn.
Heavily Armoured: the character has trained to make the most of wearing Heavy Armour (Banded Plate or Plate Armour). They ignore any dexterity modifiers (bonuses or penalities) and reduce the weight of the armour by 1 (2?) Stone while worn.