The band was founded in 1954, and its first long-term director was Skitch Henderson (1954-57). Henderson was replaced by Jose Melis in 1957, when Jack Paar took over as host from Steve Allen, and in turn replaced Melis in 1962 when Johnny Carson took over from Paar. While working for Carson he created a major jazz big band by hiring some of the best jazz musicians in New York City, including Clark Terry, Bobby Rosengarden, Doc Severinsen, Urbie Green, and Ernie Royal, and commissioning charts from top jazz arrangers.
In 1966 Henderson left the show and was replaced by Milton DeLugg, who in 1967 was replaced by Severinsen. Severinsen continued to add important musicians to the band, including Louis Bellson and Snooky Young.
In 1992, when Jay Leno replaced Carson, Branford Marsalis took over as musical director, and the number of band members was reduced. Marsalis was succeeded in 1995 by Kevin Eubanks.