In greater detail, given a finite-dimensional vector space V we can consider the symmetric algebra S(V), and the action on it of GL(V). It is actually more accurate to consider the projective representation of GL(V), if we are going to speak of invariants: that's because a scalar multiple of the identity will act on a tensor of rank r in S(V) through the r-th power 'weight' of the scalar. The point is then to define the subalgebra of invariants I(V) for the (projective) action. We are, in classical language, looking at n-ary r-ics, where n is the dimension of V.
These days it might be more natural to look to decompose the degree r part of S(V) into irreducible representations of GL(V): the formulation just given is the same as saying we are concerned only with the occurrence of one-dimensional representations. The representation theory required came later, though, with Schur.
It is customary to say that the work of David Hilbert, proving abstractly that I(V) was finitely presented, put an end to classical invariant theory. That is far from being true: the classical epoch in the subject may have continued to the final publications of Alfred Young, more than 50 years later. Explicit calculations for particular purposes have been known in modern times (for example Shioda, with the binary octavics).
The modern formulation of geometric invariant theory is due to David Mumford, and emphasizes the construction of a quotient by the group action that should capture invariant information through its coordinate ring. It is a subtle theory, in that success is obtained by excluding some 'bad' orbits and identifying others with 'good' orbits. In a separate development the symbolic method of invariant theory, an apparently heuristic combinatorial notation, has been rehabilitated.
For the invariant theory of finite groups, see Molien series.