The nucleus of the estate was a small farm of 100 acres, called Cartleyhole, nicknamed Clarty (i.e., muddy) Hole, and bought by Scott on the lapse of his lease (1811) of the neighbouring house of Ashestiel. He at first built a small villa (now the western end of the house) and named it Abbotsford, creating the name from a ford near by where previously abbots of Melrose Abbey used to cross the river. Scott then built additions to the house and made it into a mansion, building into the walls many sculptured stones from ruined castles and abbeys of Scotland. In it he gathered a large library, and a collection of ancient furniture, arms and armor, and other relics and curiosities, especially connected with Scottish history.
The last and principal acquisition being that of Toftfield (afterwards named Huntlyburn), purchased in 1817. The new house was then begun and completed in 1824.
The general ground-plan is a parallelogram, with irregular outlines, one side overlooking the Tweed; and the style is mainly the Scottish Baronial. Into various parts of the fabric were built relics and curiosities from historical structures, such as the doorway of the old Tolbooth in Edinburgh.
Scott had only enjoyed his residence one year when (1825) he met with that reverse of fortune which involved the estate in debt. In 1830 the library and museum were presented to him as a free gift by the creditors. The property was wholly disencumbered in 1847 by Robert Cadell, the publisher, who cancelled the bond upon it in exchange for the family's share in the copyright of Sir Walter's works.
Scott's only son Walter did not live to enjoy the property, having died on his way from India in 1847. Among subsequent possessors were Scott's son-in-law, J. G. Lockhart, J. R. Hope Scott, Q.C., and his daughter (Scott's great-granddaughter), the Hon. Mrs Maxwell Scott.
Abbotsford gave its name to the "Abbotsford Club," a successor of the Bannatyne and Maitland clubs, founded by W. B. D. D. Turnbull in 1834 in Scott's honour, for printing and publishing historical works connected with his writings. Its publications extended from 1835 to 1864.