Mortgage brokers have said they would be unwilling to use a lender that withdrew products with little notice in the future, a survey from a lender has found.