He was born in Wutai, Shanxi province, China. Xu was admitted to Whampoa Academy in 1924. He held various officer ranks in the National Revolutionary Army between 1925 and 1927. Xu joined Chinese Communist Party in 1927 and later became a commander in the 4th Army of Red Army.
During the Second Chinese-Japanese War, Xu fought against the Japanese invaders. After the communist takeover of China in 1949, Xu served as the General Chief of Staff of the People's Liberation Army and was awarded the rank of marshal in 1955.