Compact stars having strong magnetic fields (magnetars) have been observationally determined to have surface magnetic fields of order of 10 14-1015 G, the implied internal field strength being several orders larger. We study the equation of state and composition of hypernuclear matter and quark matter - two forms of dense matter in strong magnetic fields. We find that the magnetic field has substantial influence on the properties of hypernuclear matter and quark matter for magnetic field B ≥ 1017 G and B ≥ 1018 G respectively. In particular the matter properties become anisotropic. Moreover, above a critical field B cr, both hypernuclear and quark matter show instability, although the values of Bcr are different for two kinds of matter. © Published under licence by IOP Publishing Ltd.