The two-temperature model (TTM) has been widely employed in describing ultrafast relaxation dynamics, providing a simple yet powerful framework to study energy relaxation in photoexcited systems. Recently, the time-dependent Boltzmann equation (TDBE) has revealed the limitations of TTM. However, current implementations of the TDBE assume instantaneous electronic thermalization. In this work, we employ first-principles Boltzmann transport simulations to explicitly examine the impact of non-thermal electronic distributions on relaxation processes. By comparing gold, silver, and aluminum as representative cases, we show that while phonons can indeed remain far from equilibrium, the neglect of non-thermal electrons is far more consequential. For gold, the absence of strongly coupled scattering channels makes the influence of non-thermal electrons negligible, rendering the TTM valid. For silver, deviations from the TTM stem mainly from non-thermal electronic effects, and for aluminum, both non-thermal electrons and phonons lead to substantial discrepancies. These findings demonstrate that non-thermalized electrons play a decisive role in out-of-equilibrium ultrafast energy transfer between electrons and phonons, offering a new perspective on the limitations of the TTM.