He was born at Lyon, the son of Jehan de l'Orme, who practised the same art and brought his son up to it. At an early age Philibert was sent to Italy to study (1533-1536) and was employed there by Pope Paul III. Returning to France he was patronized by Cardinal du Bellay at Lyon, and was sent by him about 1540 to Paris, where he began the Chateau de St Maur, and enjoyed royal favour; in 1545 he was made architect to Francis I of France and given the charge of works in Brittany.
In 1548 Henry II gave him the supervision of Fontainebleau, Saint-Germain-en-Laye and the other royal buildings; but on his death (1559) Philibert fell into disgrace. Under Charles IX, however, he returned to favour, and was employed to construct the Tuileries, in collaboration with Jean Brillant. He died in Paris.
Much of his work has disappeared, but his fame remains. An ardent humanist and student of the antique, he yet vindicated resolutely the French tradition in opposition to Italian tendencies; he was a man of independent mind and a vigorous originality. His masterpiece was the Château d'Anet (1552-1559), built for Diane de Poitiers, the plans of which are preserved in Du Cerceau's Plus excellens bastimens de France, though part of the building alone remains; and his designs for the Tuileries (also given by Du Cerceau), begun by Catherine de’ Medici in 1565, were magnificent. His work is also seen at Chenonceaux and other famous ''châteaux; and his tomb of Francis I. at Saint Denis Basilica remains a perfect specimen of his art. He wrote two books on architecture (1561 and 1567).