Stochastic differential equations (SDEs) are one of the most commonly studied probabilistic dynamical systems, and widely used to model complex biological processes. Building upon the previously introduced idea of performing inference of dynamical systems by parametrising their coefficient functions via neural networks, we propose a novel formulation for an optimisation objective that combines simulation-based penalties with pseudo-likelihoods. This greatly improves prediction performance compared to the state-of-the-art, and makes it possible to learn a wide variety of dynamics without any prior assumptions on analytical structure.