Up to 38,000 staff from HM Revenue and Customs (HMRC) will be expected to move large distances as part of a reorganisation, a report has found.
Some may need to relocate by up to 174 miles if they want to keep their jobs, the National Audit Office (NAO) said.
The changes are part of long-term plans by HMRC to replace 170 offices across the UK with 13 large regional centres.
HMRC said the plan would eventually save the taxpayer £83m a year, and help it provide a better service.
Nevertheless the NAO said that the tax authority had already recognised that its original plans were unrealistic, and HMRC was considering changing them.
“The changes HMRC is considering will inevitably increase uncertainty among some of its employees, and have the potential to harm its relationship with them,” the NAO report said.
The cost of the new buildings involved had also escalated by £600m since the plan was first proposed in 2015, the report said.
In the most extreme case, staff from an office in Redruth, Cornwall, will be invited to move 174 miles to a new office in Bristol.
Workers from Aberdeen and Dundee will be relocated to Edinburgh, a distance of up to 127 miles.
And a big new office in Croydon, Surrey will see staff being moved from as far away as Norwich, more than 130 miles away.
However the average move will be just 18 miles, the NAO said.
As a result of the disruption, HMRC expects that around 5,000 staff who are unwilling to relocate will leave their jobs.
The NAO said that could at first affect the ability of staff to do their jobs properly.
“Staff that will be recruited at the new regional centres may have lower initial performance, and may take time to reach the productivity levels of more experienced staff,” the report said.
The Public and Commercial Services (PCS) trade union, which represents the affected staff, has already said that it is worried that the cuts could harm HMRC’s ability to collect tax.
“With costs rising, and the cracks beginning to show, it is now imperative that HMRC halts these plans and allows MPs and the public to have their say,” said Mark Serwotka, the PCS general secretary.
But HMRC said the changes would deliver better value.
“Our 13 new regional centres are an essential part of our work to modernise HMRC and provide an even better service for our customers,” an HMRC spokesperson said.
“This will deliver annual savings to the taxpayer of £83 million from 2025-26 and will mean modern and cost-effective buildings with the latest technology and more training opportunities for staff.”