Lyttelton attended Eton College, where his father was a housemaster. At Eton he developed his love for jazz, forming a quartet there in 1936 after teaching himself the trumpet. After leaving school, he served in the Grenadier Guards. Following demob after World War II, he attended Camberwell Art College.
In the 1950s Lyttelton led the British "trad jazz" movement, a revival of traditional Dixieland and older jazz forms, from New Orleans to early Duke Ellington. In 1956, he had his only hit, Bad Penny Blues, which was in the UK charts for 6 weeks.
Lyttelton has presented The Best of Jazz on BBC Radio 2 since 1967.
In 2001, Lyttleton and his band added trad jazz elements to a free-form, experimental Radiohead song "Life In A Glass House" on the Amnesiac album.