Diversification of funding is vital for specialist lending success, and specialist lenders should not primarily rely on one source of funding, experts have said.