Invariant theory is a branch of abstract algebra dealing with actions of groups on algebraic varieties, such as vector spaces, from the point of view of their effect on functions. Classically, the theory dealt with the question of explicit description of polynomial functions that do not change, or are invariant, under the transformations from a given linear group. For example, if we consider the action of the special linear group SLn on the space of n by n matrices by left multiplication, then the determinant is an invariant of this action because the determinant of A X equals the determinant of X, when A is in SLn.
Invariant theory of infinite groups is inextricably linked with the development of linear algebra, especially, the theories of quadratic forms and determinants. Another subject with strong mutual influence was projective geometry, where invariant theory was expected to play a major role in organizing the material. One of the highlights of this relationship is the symbolic method. Representation theory of semisimple Lie groups has its roots in invariant theory.
David Hilbert's work on the question of the finite generation of the algebra of invariants (1890) resulted in the creation of a new mathematical discipline, abstract algebra. A later paper of Hilbert (1893) dealt with the same questions in more constructive and geometric ways, but remained virtually unknown until David Mumford brought these ideas back to life in the 1960s, in a considerably more general and modern form, in his geometric invariant theory. In large measure due to the influence of Mumford, the subject of invariant theory is seen to encompass the theory of actions of linear algebraic groups on affine and projective varieties. A distinct strand of invariant theory, going back to the classical constructive and combinatorial methods of the nineteenth century, has been developed by Gian-Carlo Rota and his school. A prominent example of this circle of ideas is given by the theory of standard monomials.
Cayley first established invariant theory in his "On the Theory of Linear Transformations (1845)." In the opening of his paper, Cayley credits an 1841 paper of George Boole, "investigations were suggested to me by a very elegant paper on the same subject... by Mr Boole." (Boole's paper was Exposition of a General Theory of Linear Transformations, Cambridge Mathematical Journal.)
Classically, the term "invariant theory" refers to the study of invariant algebraic forms (equivalently, symmetric tensors) for the action of linear transformations. This was a major field of study in the latter part of the nineteenth century. Current theories relating to the symmetric group and symmetric functions, commutative algebra, moduli spaces and the representations of Lie groups are rooted in this area.
In greater detail, given a finite-dimensional vector space V of dimension n we can consider the symmetric algebra S(Sr(V)) of the polynomials of degree r over V, and the action on it of GL(V). It is actually more accurate to consider the relative invariants of GL(V), or representations of SL(V), if we are going to speak of invariants: that is because a scalar multiple of the identity will act on a tensor of rank r in S(V) through the r-th power 'weight' of the scalar. The point is then to define the subalgebra of invariants I(Sr(V)) for the action. We are, in classical language, looking at invariants of n-ary r-ics, where n is the dimension of V. (This is not the same as finding invariants of GL(V) on S(V); this is an uninteresting problem as the only such invariants are constants.) The case that was most studied was invariants of binary forms where n = 2.
The work of David Hilbert, proving that I(V) was finitely presented in many cases, almost put an end to classical invariant theory for several decades, though the classical epoch in the subject continued to the final publications of Alfred Young, more than 50 years later. Explicit calculations for particular purposes have been known in modern times (for example Shioda, with the binary octavics).
Hilbert (1890) proved that if V is a finite-dimensional representation of the complex algebraic group G = SLn(C) then the ring of invariants of G acting on the ring of polynomials R = S(V) is finitely generated. His proof used the Reynolds operator ρ from R to RG with the properties
Hilbert constructed the Reynolds operator explicitly using Cayley's omega process Ω, though now it is more common to construct ρ indirectly as follows: for compact groups G, the Reynolds operator is given by taking the average over G, and non-compact reductive groups can be reduced to the case of compact groups using Weyl's unitarian trick.
Given the Reynolds operator, Hilbert's theorem is proved as follows. The ring R is a polynomial ring so is graded by degrees, and the ideal I is defined to be the ideal generated by the homogeneous invariants of positive degrees. By Hilbert's basis theorem the ideal I is finitely generated (as an ideal). Hence, I is finitely generated by finitely many invariants of G (because if we are given any – possibly infinite – subset S that generates a finitely generated ideal I, then I is already generated by some finite subset of S). Let i1,...,in be a finite set of invariants of G generating I (as an ideal). The key idea is to show that these generate the ring RG of invariants. Suppose that x is some homogeneous invariant of degree d > 0. Then
for some aj in the ring R because x is in the ideal I. We can assume that aj is homogeneous of degree d − deg ij for every j (otherwise, we replace aj by its homogeneous component of degree d − deg ij; if we do this for every j, the equation x = a1i1 + ... + anin will remain valid). Now, applying the Reynolds operator to x = a1i1 + ... + anin gives
We are now going to show that x lies in the R-algebra generated by i1,...,in.
First, let us do this in the case when the elements ρ(ak) all have degree less than d. In this case, they are all in the R-algebra generated by i1,...,in (by our induction assumption). Therefore, x is also in this R-algebra (since x = ρ(a1)i1 + ... + ρ(an)in).
In the general case, we cannot be sure that the elements ρ(ak) all have degree less than d. But we can replace each ρ(ak) by its homogeneous component of degree d − deg ij. As a result, these modified ρ(ak) are still G-invariants (because every homogeneous component of a G-invariant is a G-invariant) and have degree less than d (since deg ik > 0). The equation x = ρ(a1)i1 + ... + ρ(an)in still holds for our modified ρ(ak), so we can again conclude that x lies in the R-algebra generated by i1,...,in.
Hence, by induction on the degree, all elements of RG are in the R-algebra generated by i1,...,in.
The modern formulation of geometric invariant theory is due to David Mumford, and emphasizes the construction of a quotient by the group action that should capture invariant information through its coordinate ring. It is a subtle theory, in that success is obtained by excluding some 'bad' orbits and identifying others with 'good' orbits. In a separate development the symbolic method of invariant theory, an apparently heuristic combinatorial notation, has been rehabilitated.
One motivation was to construct moduli spaces in algebraic geometry as quotients of schemes parametrizing marked objects. In the 1970s and 1980s the theory developed interactions with symplectic geometry and equivariant topology, and was used to construct moduli spaces of objects in differential geometry, such as instantons and monopoles.