We describe the generalized Sato-Tate group attached to an abelian variety and introduce an approach to characterize it through the character theory of compact Lie groups. We illustrate the method with examples of generic curves of low genus, with Sato-Tate group $\mathrm{USp}(2g)$; special curves which yield proper subgroups, and a family of Katz giving rise to Galois representations in $\mathrm{SO}(2g+1)$.