Compact stars with strong magnetic fields (magnetars) have been observationally determined to have surface magnetic fields of order of 1014-1015 G, the implied internal field strength being several orders larger. We study the equation of state and composition of dense hypernuclear matter in strong magnetic fields in a range expected in the interiors of magnetars. Within the non-linear Boguta-Bodmer-Walecka model we find that the magnetic field has sizable influence on the properties of matter for central magnetic field B ≥ 1017 G, in particular the matter properties become anisotropic. Moreover, for the central fields B ≥ 1018 G, the magnetized hypernuclear matter shows instability, which is signalled by the negative sign of the derivative of the pressure parallel to the field with respect to the density, and leads to vanishing parallel pressure at the critical value Bcr ≃ 1019 G. This limits the range of admissible homogeneously distributed fields in magnetars to fields below the critical value Bcr. © 2012 Elsevier B.V..