The significant death toll from major viral outbreaks during the past century highlights the importance of understanding the factors that determine whether zoonotic spillovers escalate into full-blown human epidemics. Hundreds of thousands of humans are made sick by animal-derived viruses such as Lassa, Nipah, Marburg, and Ebola every year. This includes thousands of fatalities. Moreover, many viruses, including the four pathogens mentioned above, exhibit human-to-human spread. Given this information, an explanation is needed for why human viral pandemics caused by animal viruses are relatively rare. This review proposes that variations in the fidelity of viral replication play an important role and that low-fidelity replication creates a barrier to extended human-to-human transmission. After a zoonotic transfer to humans, a virus will be subject to unusually strong selection, and full adaptation to the new host would typically require multiple mutations. A rapid route for accumulating such mutations involves minor alterations to the virus's polymerase that increase its mutation frequency, allowing more rapid adaptation. I refer to this route as the mutation-induced enhanced mutability pathway (MIEMP). However, the cost of this route may be the loss of the virus’s long-term viability when harmful mutations accumulate. Recombination with a strain with a high-fidelity polymerase could, however, restore fidelity and longevity. This would typically require another virus jump from the original animal reservoir to an infected human. We might expect the MIEMP pathway to be active after zoonosis and also when a virus is subjected to strong selective pressures. There is some evidence from the COVID-19 pandemic to support this scenario: firstly, COVID-19 epidemics in several countries showed rapid rises followed by sudden collapses. Some lineages appeared to lose viability throughout large geographical areas very suddenly, which may have been a result of low fidelity. This is compatible with a MIEMP origin. Secondly, several SARS-CoV-2 variants, including the Omicron variants, appeared with a jump in the number of mutations compared to previous lineages, which is compatible with MIEMP followed by recombination. The same variants also had many mutations in the Spike gene but fewer in the rest of the genome. Moreover, an anomalously low proportion of the mutations in the Spike of Omicron (and other variants) were C-to-T nucleotide "transitions." Many coronaviruses have an excess of C-to-T transitions (often caused by host modifications of viral RNA). In contrast, low-fidelity polymerases are expected to generate all nucleotide exchanges randomly. This mutational pattern, which is otherwise difficult to explain, is therefore consistent with recombination events, where much of the right-hand-end "structural protein" sections of the genomes of these variants, including the Spike genes, came from error-prone partners, while much of the non-structural protein sections, including the polymerases, came from high-fidelity partners. These conclusions lead to recommendations for avoiding conditions that might allow dangerous recombination between well-adapted strains and high-fidelity spillovers.