Ghost imaging (GI) is a second-order correlation imaging technique based on optical fields, with advantages like wide spectral range, high sensitivity and strong anti-disturbance ability, applied in X-ray imaging, spectral imaging, etc. However, its imaging speed is affected by spatial light modulators such as digital micromirror devices (DMD) in practical use, and balancing imaging quality and efficiency is a key challenge. Inspired by human eyes acquiring region of interest (ROI), this paper proposes a deep learning-based foveated pattern affine transformation method, Retina Affine Transformer (RAT) network, with low parameters and FLOPs. It improves GI efficiency by accelerating target localization and pattern generation in ROI via affine matrix parameter prediction, enhances ROI imaging quality with foveated patterns while keeping field of view, suitable for single/multi-ROI. Simulations and experiments verify its superiority over traditional methods in PSNR, SSIM, pattern generation time and computational efficiency.