Annan was a maritime town whose ship building yards built many clippers and other boats. A cairn on the jetty commemorates Robert Burns who worked as an exciseman here in the 1790s.
The writer Thomas Carlyle was educated at Annan Academy, a Georgian building now known as Bridge House. The Town Hall is Victorian, built from local sandstone in 1878. There is also a Historic Resources Centre.
In Port Street, some of the windows were (and are still) blocked up to avoid paying Window Tax of 1701.
Hoddom Castle nearby was built ca. 1552-1565 by John Maxwell, 4th Lord Herries.
Annan Castle was the original home of the 'de Brus' family, later to be known as Bruce, whose most famous son was Robert The Bruce.