For example, given a vector space V we can take G to be GL(V), the group of invertible linear transformations of V, and X to be the set of all (ordered) bases of V. Then G acts on X in the way that it acts on vectors of V; and it acts transitively since any basis can be transformed via G to any other. What is more, a linear transformation fixing each vector of a basis will fix all v in V: so that X is indeed a principal homogeneous space. One way to follow basis-dependence in a linear algebra argument is to track variables x in X.
Another linear algebra example is the affine space concept: the idea of the affine space A underlying V can be said succinctly by saying that A means a principal homogeneous space for V as additive group.
The principal homogeneous space concept is a special case of that of principal bundle: it means a principal bundle with base a single point. In other words the local theory of principal bundles is that of a family of principal homogeneous spaces depending on some parameters in the base. The 'origin' can be supplied by a section of the bundle - such sections are usually assumed to exist locally on the base - the bundle being locally trivial, so that the local structure is that of a cartesian product. But sections will often not exist globally. For example a differential manifold M has a principal bundle of frames associated to its tangent bundle. A global section will exist, tautologically, only when M is parallelizable; which implies strong topological restrictions.
In number theory there is a (superficially different) reason to consider principal homogeneous spaces, for elliptic curves E defined over a field K (and more general abelian varieties. Once this was understood various other examples were collected under the heading, for other algebraic groups: quadratic forms for orthogonal groups, and Severi-Brauer varieties for projective linear groups being two. Also the shorter term torsor gained currency for principal homogeneous space.
The reason of the interest for Diophantine equations, in the elliptic curve case, is that K may not be algebraically closed. There can exist curves C that have no point defined over K, and which become isomorphic over a larger field to E, which by definition has a point over K to serve as identity element for its addition law. That is, for this case we should distinguish C that have genus 1, from elliptic curves E that have a K-point (or, in other words, provide a Diophantine equation that has a solution in K). The curves C turn out to be torsors over E, and form a set carrying a rich structure in the case that K is a number field (the theory of the Selmer group)). In fact a typical plane cubic curve C over Q has no particular reason to have a rational point; the standard Weierstrass model always does, namely the point at infinity, but you need a point over K to put C into that form over K.
This theory has been developed with great attention to local analysis, leading to the definition of the Tate-Shafarevich group. In general the approach of taking the torsor theory, easy over an algebraically closed field, and trying to get back 'down' to a smaller field is an aspect of descent. It leads at once to questions of Galois cohomology, since the torsors represent classes in group cohomology H1.