Two of the biggest specialist lenders are in advanced talks about a possible merger which would create a business completing around £4bn of mortgage lending a year.