While in high school in Philadelphia, Pennsylvania he played with several other promising young musicians, including John Coltrane, Red Garland, Jimmy Heath, Percy Heath, Philly Joe Jones, and Red Rodney. After graduating from Howard University he joined Bull Moose Jackson's rhythm and blues band; Tadd Dameron, whom Golson came to consider the most important influence on his writing, was Jackson's pianist at the time. From 1953 to 1959 he played with Dameron's band and then with the bands of Lionel Hampton, Johnny Hodges, Earl Bostic, Dizzy Gillespie, and Art Blakey (the Jazz Messengers). During this period Golson composed the jazz standards "I Remember Clifford," "Stable Mates," "Killer Joe," "Whisper Not," and "Along Came Betty."
From 1959 to 1962 he co-led the Jazztet with Art Farmer. He then left jazz to concentrate on studio and orchestral work for 12 years, after which he returned to jazz recording. In 1983 he re-organized the Jazztet.
In 1995 he received the Jazz Masters Award of the National Endowment for the Arts.