Legal ‘loopholes’, fast-track plans and ghost town fears: These residents are going to war with the Big Build