With the arrival of the Mormon pioneers in 1847, disputes arose between Bridger and the new settlers. By 1853, a militia of Mormons was sent to arrest Bridger for selling alcohol and firearms to the Native Americans. Bridger escaped capture, temporarily returning to the East. In 1855, Bridger sold the fort to the Mormons for $8,000.
On the night of October 7, 1857, William "Bill" Hickman set fire to the fort to keep it from falling into the hands of the approaching United States Army. At the end of hostilities, the United States Congress rejected Brigham Young's claim to the fort, nor did it recognize Jim Bridger's continuing claims to the fort. Instead, the fort was profitably run by William Alexander Carter. However, by 1869 the Union Pacific Railroad made the fort obsolete.
In 1928, Fort Bridger was sold to the Wyoming Historic Landmark Commission for preservation as a historic monument.