Joe Duffy: ‘I was so demoralised with the onslaught. I saw my own staff crying with the phone calls we were getting’