Maps between elliptic curves, also called isogenies, are fixed once the image on sufficiently many points is known. Last year, a method was discovered that computationally recovers isogenies just by knowing information about their image points, leading to the break of the key-exchange scheme SIDH. Isogeny-based key-exchange proposals relying on class group actions, such as CSIDH, remain unaffected, because such image information is not directly available. Using the theory of pairings on elliptic curves, we show that sometimes one may recover such information anyway, and classify when this approach results in a key-recovery attack.