When ssh'd onto the external machine the backspace acts as a delete key.
There is some outdated advice here http://www.macosxhints.com/article.php?story=20060825071728278
I fixed the Problem by going to terminal.app
Terminal -> Prefferences settings tab, advanced sub tab then select "Delete sends Ctrl-H"
This fixed the problem for me.