He was born in Nagykanizsa in Hungary. He went to Vienna to study engineering, but also took composition lessons while there. He moved to the United States of America in 1909 and, after a brief stint working in a pencil factory, was employed as a pianist in cafes. He eventually founded his own orchestra and published a few songs, which, despite their limited success, drew him to the attention of the Schubert brothers who hired him to write music for their shows in 1914. That year he wrote his first significant operetta, The Whirl of the World.
Romberg's adaptation of melodies by Franz Schubert for Blossom Time (1921, produced in the UK as Lilac Time) was a great success. He subsequently wrote his best known operettas, The Student Prince (1924), The Desert Song (1926) and New Moon (1928) which are in a similar style to the Viennese operettas of Franz Lehar. His later works, such as Up in Central Park (1945), are closer to the American musical in style, but they were less successful.
Romberg was the subject of the 1954 Stanley Donen-directed film Deep in My Heart, in which he was played by José Ferrer.