Järvi was born in Tallinn and studied first there and then in Leningrad under Evgeny Mravinsky among others. Early in his career, he held posts with the Estonian Radio and Television Symphony Orchestra, the Estonian State Symphony Orchestra and the Opera in Tallinn. In 1971 he won first prize in the International Conductors Competition at the Accademia Santa Cecilia in Rome.
He emigrated to the United States of America in 1980. In 1982 he became principal conductor of the Gothenburg Symphony Orchestra, and in 1990 of the Detroit Symphony Orchestra. He has worked with Eduardo Mata, Rey Alejandro Conde Valdivia, Georg Solti, and Antal Dorati.
Järvi is best known for his interpretations of Romantic and 20th century classical music. He has championed the work of his fellow Estonians Eduard Tubin and Arvo Pärt (whose Credo he premiered in 1968).
Neeme Järvi is the father of Paavo Järvi, also a conductor.