Susan, that sounds like an awesome device, and exactly the sort of weapon that Dwarven Machinists ought to be making.
Attacks made with the “hand ballista” would use the attack throw of its operator.
Damage should be a maximum of 21 points of damage per round and an average of half that. I would recommend 2d10+1, 3d6+3, or 5d4 as reasonable damage codes for the hand ballista.
Below are some items that the dwarven machinist created in my Eastern Krysea mini-campaign:
Arcanometer (AC 0, HD1, hp5): The operator of this strange contraption can detect magic at will. The presence of magic is revealed by a gauge on the handle. Calibrating the device requires 1 turn each time it is used. (1 stone, 2,000gp)
Clockwork Griffin (AC 3, Move 120’ flying, HD6, hp24, #AT2 talons and 1 bite, Dmg 1d4+1/1d4+1/1d8, ML N/A): The clockwork griffin is a drone that follows simple instructions in the same manner as a golem. It cannot carry passengers due to its many complex gears. It weighs 36 stone (360lbs). (27,000gp)
2 Clockwork War Titans (AC 9, Move 60’, HD18, hp90, #AT 2 fists, Dmg 2d12+3/2d12+3, ML N/A): Clockwork war titans are drones that follow simple instructions in the same manner as a golem. They cannot carry passengers but can carry and operate the repeating steam ballistas (below). Each weighs 324 stone (3,240lbs). (46,000gp).
2 Repeating Steam Ballistas (AC3, HD4, hp20): The operator may fire a steam-powered spear once per round. Each spear does 2d6 damage to a maximum range of 240 yards. The device weighs 16 stone (160lbs) and cannot move on its own. (8,000gp)
Ornithopter Wing-Pack (AC 0, HD1, hp5): The operator of these mechanical wings can fly at an exploration rate of 60’/ turn. He cannot hover or carry other passengers. (1 stone, 7,000gp)