This paper investigates absolute and relative pose estimation under refraction, which are essential problems for refractive structure from motion. We first present an absolute pose estimation algorithm by leveraging an efficient iterative refinement. Then, we derive a novel refractive epipolar constraint for relative pose estimation. The epipolar constraint is established based on the virtual camera transformation, making it in a succinct form and can be efficiently optimized. Evaluations of the proposed algorithms on synthetic data show superior accuracy and computational efficiency to state-of-the-art methods. For further validation, we demonstrate the performance on real data and show the application in 3D reconstruction of objects under refraction.