Sir Anthony was born in 1925, as a son of Sir James Berry, 1st Viscount Kemsley, and his wife Mary (née Holmes). His aristocratic connections led to his becoming a treasurer of the royal household.
He married firstly, in 1954, the Hon. Mary Cynthia Burke Roche (born 1934). They had four children, named Alexandra (born 1955), Antonia (born 1957), Joanna Cynthia (born 1957, twin of Antonia), and Edward (born 1960). They divorced in 1966. It is perhaps worth noting that Sir Anthony's first wife was the elder sister of the Hon. Frances Ruth Burke Roche (born 1936), who was to become Countess Spencer and mother of Diana, Princess of Wales.
Since Magee's release, Sir Anthony's daughter Jo Tuffnell (a mother of three living in North Wales) has become renowned for her series of controversial meetings with the Brighton bomber, as part of her quest to come to terms with the bombing and, in her own words, "to bring something positive out of it". Some of their discussions were filmed for an Everyman programme, shown on BBC2 on Thursday 13th December 2001. Although she has received some criticism for these meetings, she is regarded by many as a courageous campaigner for peace.