Little is known about his life outside of the basics. Fermor was the wool staple at Calais, and made large sums supplying the king's army with armor and munitions for the campaign at Tournai. As a reward, he was given wool export licenses, and his losses due to piracy or political strife were settled by the king's ambassadors.
In 1524, he traveled to Florence to assist Wolsey's agent in trying to secure Wolsey's election as the pope, a campaign that ultimately failed, but its certain that Fermor's financial assistant to Wolsey's agent was appreciated by the king. At the time of Wolsey's death, he owed Fermor 125£, so the financial investment must have been significant. Fermor became a very wealthy man, and one of some import.
His wife was Anne Browne, daughter of Sir William Browne, Lord Mayor of London. They had ten children, only five of whom lived to adulthood.
Will Somers is said to have been either Fermor's fool, or discovered by him. It's unknown how Will actually came to the king's service. It could have been during one of the king's visits to Calais, or perhaps Will was taken to court as a gift for the king, as he is in Under These Restless Skies.
Fools were essentially owned by their employer, and could be traded or sold to another household, whether or not the fool wanted to go. Patch is recorded to have fought so violently against being sent to the king that it took six men to tie him to the horse.
Will Somers seems to have had some regard for Fermor. In 1540, Fermor was attained by the king and put into prison. He had concealed a Catholic priest in his home, and after the priest had been arrested, Fermor visited him in prison to bring him two shirts and a bit of money. Prison was horrific for a destitute man, since a prisoner had to pay for the basic comforts, such as bedding, and had to surrender the clothes they wore as soon as they entered. The priest would have been freezing in a bare cell.
The old story has it that Will Somers pleaded with the dying king to show Fermor mercy, and Henry agreed to allow Fermor's release, and, eventually to restore some of his property. He was never pardoned, though, until the reign of Henry's son, Edward IV. That pardon is also attributed to Will's efforts on Fermor's behalf.
Fermor seems to have been a deeply religious man. In Collins' Peerage of England, it is recorded:
It is further remarkable, that having some foreknowledge of his own death, he invited on that very day many of his friends and neighbours, and taking leave of them, retired to his devotions, and was found dead in that posture.Fermor left behind an estate of 200 marks - a respectable sum -and property in Oxfordshire for his children. His tomb is in Easton Neston church.