Let $E$ and $f$ be an Eisenstein series and a cusp form, respectively, of the same weight $k\geq 2$ and of the same level $N$, both eigenfunctions of the Hecke operators, and both normalized so that $a_1 = 1$. The main result we prove is that when $E$ and $f$ are congruent mod a prime $\mathfrak{p}$ (which we take to be a prime of $\overline{Q}$ lying over a rational prime $p >2$), the algebraic parts of the special values $L(E,\chi ,j)$ and $L(f,\chi ,j)$ satisfy congruences mod the same prime. More explicitly, we prove that, under certain conditions, \[ \frac{\tau (\bar{\chi })L(f,\chi ,j)}{(2 \pi i)^{j-1}\Omega _f^{\text{sgn}(E)}} \equiv \frac{\tau (\bar{\chi })L(E,\chi ,j)}{(2 \pi i)^{j}\Omega _E} \pmod{\mathfrak{p}} \] where the sign of $E$ is $\pm 1$ depending on $E$, and $\Omega _f^{\text{sgn}(E)}$ is the corresponding canonical period for $f$. Also, $\chi$ is a primitive Dirichlet character of conductor $m$, $\tau (\bar{\chi })$ is a Gauss sum, and $j$ is an integer with $0< j< k$ such that $(-1)^{j-1}\cdot \chi(-1) = \text{sgn}(E)$. Finally, $\Omega _E$ is a $\mathfrak{p}$-adic unit which is independent of $\chi$ and $j$. This is a generalization of earlier results of Stevens and Vatsal for weight $k=2$. The main point is the construction of a modular symbol associated to an Eisenstein series.