The integration of paper-based microfluidics with deep learning represents a pivotal trend in enhancing diagnostic capabilities. This paper introduces a new approach to improve the performance of a paper-based microfluidic enzyme-linked immunosorbent assay (ELISA) by training the temporal sequence colorimetric data rather than static data conventionally, using deep learning. Traditional deep learning-assisted ELISA analysis methods usually rely on a single snapshot of the reaction at its end, which limits the further improvement of sensitivity and specificity (or accuracy for combined evaluation), as it misses dynamic changes in the reaction over time. In this work, we developed a temporal sequence-enhanced paper analytical device (TimePAD) that captures continuous video data of the ELISA reaction, which contains the dynamic colorimetric changes. With the YOLOv8 deep learning alogrithm and the Rabbit IgG as the model for ELISA assay, we can use the initial 20 min signal instead of waiting for 30 min for full reaction, achieving a 33% reduction in the turnaround time. Moreover, the overall accuracy at 20 min is 94.1%, which is slightly improvement to the 93.5% using a traditional single snapshot method at 30 min. This method not only accelerates result interpretation but also enhances the overall efficiency of diagnostics, making it particularly valuable for time-sensitive point-of-care testing applications. Lastly, to demonstrate its real-world use, we expanded to the disease biomarker cTnI detection and obtained accuracy of 98.1% within only 10 min, compared to 25 min with 97.8% accuracy in traditional methods.