Prince Harry source says country’s withdrawal from Invictus Games was a ‘consequence’ of King’s letter