We present a study of the terahertz-frequency magnetoelectric effect in ferrimagnetic pyroelectric CaBaCo4O7 and its Ni-doped variants. The terahertz absorption spectrum of these materials consists of spin excitations and low-frequency infrared-active phonons. We studied the magnetic-field-induced changes in the terahertz refractive index and absorption in magnetic fields up to 17 T. We find that the magnetic field modulates the strength of infrared-active optical phonons near 1.2 and 1.6 THz. We use the Lorentz model of the dielectric function to analyze the measured magnetic-field dependence of the refractive index and absorption. We propose that most of the magnetoelectric effect is contributed by the optical phonons near 1.6 THz and higher frequency resonances. Our experimental results can be used to construct and validate more detailed theoretical descriptions of magnetoelectricity in CaBaCo4-xNixO7. © 2017 American Physical Society.