The son of Orlando Harriman, an Episcopal clergyman, and Cornelia (Neilson) Harriman. His great-grandfather, William Harriman, emigrated from England in 1795 and engaged successfully in trading and commercial pursuits.
In 1879 he married Mary Williamson Averell, the daughter of William J. Averell, a banker of Ogdensburg, New York, who was president of the Ogdensburg & Lake Champlain Railroad Company. This relationship aroused his interest in up-state transportation and two years later his career as a rebuilder of bankrupt railroads began with a small broken-down railroad called the Lake Ontario Southern which he renamed the Sodus Bay & Southern, reorganized, and sold with considerable profit to the Pennsylvania.
Harriman was nearly fifty years old when in 1897 he became a director of the Union Pacific Railroad. By May 1898 he was chairman of the executive committee, and from that time until his death his word was law on the Union Pacific system. In 1903 he assumed the office of president of the company.