Monthly figures from the Office of National Statistics showed construction output was down 3.6% in March, compared to February, with the decline relating to both new work and repair and maintenance.
Output also fell in the quarter from January to March this year, down 1.1% on Q4 2015 and down 1.9% on Q1 2015.
The one area of growth was in housebuilding, which was up 4.8% in Q1 2016 compared to the previous quarter.
Former Royal Institute of Chartered Surveyors (RICS) chairman and north London estate agent Jeremy Leaf said the housebuilding figure was encouraging but expects things to decline in this area before long.
“This is potentially a last hurrah as confidence is likely to dip in response to weakening activity,” said Leaf.
“Those housebuilders who are building now made the decision to do so six months ago when demand was higher so the next quarter’s figures may be okay but the quarter after that might see a dip to reflect the current lack of activity.”
He said that people are nervous, not only about the potential of a Brexit, but about the strength of the economy in general.
“Brexit is a bit of a smokescreen – the strength of the economy is a bigger issue. While in the suburbs and outside the centre of London the housing industry is still confident, in the centre of London where there is oversupply and a lot of cranes, they are not.”
The ONS figures also found infrastructure to be a down, falling for a third consecutive quarter. It decreased 5.6% in Q1 2016 compared to the previous quarter.