Single-photon emission from the nitrogen-vacancy defect in diamond constitutes one of its many proposed applications. Owing to its doubly degenerate 3 E electronic excited state, photons from this defect can be emitted by two optical transitions with perpendicular polarization. Previous measurements have indicated that orbital-selective photoexcitation does not, however, yield photoluminescence with well-defined polarizations, thus hinting at orbital-averaging dynamics even at cryogenic temperatures. Here we employ femtosecond polarization anisotropy spectroscopy to investigate the ultrafast electronic dynamics of the 3 E state. We observe subpicosecond electronic dephasing dynamics even at cryogenic temperatures, up to five orders of magnitude faster than dephasing rates suggested by previous frequency- and time-domain measurements. Ab initio molecular dynamics simulations assign the ultrafast depolarization dynamics to nonadiabatic transitions and phonon-induced electronic dephasing between the two components of the 3 E state. Our results provide an explanation for the ultrafast orbital averaging that exists even at cryogenic temperatures.