Following Zagier and Beukers, we show that the sequences used by Apery in his proofs of the irrationality of zeta(2) and zeta(3) are special cases of more general sequences having surprisingly only integer values, and that many of these sequences can be parametrized by modular forms. Following Almkwist and Zudilin, we also explain that the degree three sequences used for zeta(3) and generalizations can be automatically obtained via a Clausen type hypergeometric identity from the degree two sequences used for zeta(2) and generalizations.