Modern viral kinetic modeling and its application to therapeutics is a field that attracted the attention of the medical, pharmaceutical and modeling communities during the early days of the AIDS epidemic. Its successes lead to applications of modeling methods not only to HIV but a plethora of other viruses, such as hepatitis C virus (HCV), hepatitis B virus (HBV) and cytomegalovirus (CMV), which along with HIV cause chronic diseases, and viruses such as influenza, respiratory syncytial virus (RSV), West Nile virus (WNV), Zika virus, and SAR-CoV-2, which generally cause acute infections. Here we first review the historical development of mathematical models to understand HIV and HCV infections and the effects of treatment by fitting the models to clinical data. Furthermore, we then focus on recent efforts and contributions of applying these models towards understanding SARS-CoV-2 infection and highlight outstanding questions where modeling can provide crucial insights and help to optimize non-pharmaceutical and pharmaceutical interventions of the COVID-19 pandemic. The review is written from our personal perspective emphasizing the power of simple target cell limited models that provided important insights and then their evolution into more complex models that captured more of the virology and immunology. To quote Albert Einstein ?Everything should be made as simple as possible, but not simpler?, and this idea underlies the modeling we describe below.