The Lennox sisters were daughters of Charles Lennox, 2nd Duke of Richmond, an illegitimate descendant of King Charles II of England. In 1744, Lady Caroline eloped with Henry Fox, 1st Baron Holland, a politician who was eighteen years her senior. The marriage was disapproved of by her parents, but proved a happy one. The couple had several sons, including the great Whig politician, Charles James Fox. Their home, Holland House in Kensington, London, became famous as a social and political focus.
Two of her sisters, Emily and Louisa, married and went to live in Ireland. When both of the girls' parents died within a year of one another, Caroline was passed over for the care of her two youngest sisters, Sarah and Cecilia, and this slight was the starting-point for a quarrel between Caroline and Emily. Political matters aggravated the quarrel, which was only made shortly before Caroline's death. However, when her youngest sister, Lady Sarah Lennox, returned to England, Caroline brought her to court, where she enjoyed a brief flirtation with the future King George III of the United Kingdom.
Caroline's spoiled sons, especially Stephen, the eldest, caused her much embarrassment and distress by gambling and falling into debt. She was already suffering from a mysterious and painful illness when her beloved husband, who had suffered a stroke, died. She followed within a short time.