Traditional cell analysis approaches based on quantitative phase imaging (QPI) necessitate a reconstruction stage, which utilizes digital holography. However, phase retrieval processing can be complicated and time-consuming since it needs numerical reconstruction and then phase unwrapping. For analysis of cardiomyocyte (CM) dynamics, it was reported that by estimating the spatial variance of the optical path difference from QPI, the spatial displacement of CMs can be quantified, thereby enabling monitoring of the excitation-contraction activity of CMs. Also, it was reported that the Farnebäck optical flow method could be combined with the holographic imaging information from QPI to characterize the contractile motion of single CMs, enabling monitoring of the mechanical beating activity of CMs for cardiotoxicity screening. However, no studies have analyzed the contractile dynamics of CMs based on raw holograms. In this paper, we present a fast, label-free, and high throughput method for contractile dynamic analysis of human-induced pluripotent stem cell-derived CMs using raw holograms or the filtered holograms, which are obtained by filtering only The proposed approach obviates the need for time-consuming numerical reconstruction and phase unwrapping for CM's dynamic analysis while still having performance comparable to that of the previous methods. Accordingly, we developed a computational algorithm to characterize the CM's functional behaviors from contractile motion waveform obtained from raw or filtered holograms, which allows the calculation of various temporal metrics related to beating activity from contraction-relaxation motion-speed profile. To the best of our knowledge, this approach is the first to analyze drug-treated CM's dynamics from raw or filtered holograms without the need for numerical phase image reconstruction. For one hologram, the reconstruction process itself in the existing methods takes at least three times longer than the process of tracking the contraction-relaxation motion-speed profile using optical flow in the proposed method. Furthermore, our proposed methodology was validated in the toxicity screening of two drugs (E-4031 and isoprenaline) with various concentrations. The findings provide information on CM contractile motion and kinetics for cardiotoxicity screening.