Born in County Wexford, Ireland, O'Herlihy first appeared in Carol Reed's Odd Man Out in 1947. His first American film was Macbeth in the following year, appearing with Orson Welles. In 1952 he starred in Luis Buñuel's The Adventures of Robinson Crusoe, which garnered him an Oscar nomination for Best Actor.
O'Herlihy later starred in John Huston's The Dead in 1987, in RoboCop, in the cult television program Twin Peaks in 1991, and in RoboCop 2.
He is the brother of director Michael O'Herlihy and the father of actor Gavan O'Herlihy.