The ability to solve differential equations represents a key step in the modeling and understanding of complex systems. There exist several analytical and numerical methods for solving differential equations, each with their own advantages and limitations. Physics-informed neural networks (PINNs) offer an alternative perspective. Although PINNs deliver promising results, many stones remain unturned about this method. In this paper, we introduce a method that improves the efficiency of PINNs in solving differential equations. Our method is related to the formulation of the problem: Instead of training a network to solve an nth order differential equation, we propose transforming the problem into the equivalent system of n first-order equations in phase space. The target of the network is to solve all equations of the system simultaneously, effectively introducing a multitask optimization problem. We compare both approaches empirically on various problems, ranging from secondorder differential equations with constant coefficients to higher-order and nonlinear problems. We also show that our approach is suited for solving partial differential equations. Our results show that the system approach performs equal or better in most experiments performed. We analyze the learning process for the few runs that did not perform well and show that the problem stems from conflicting gradients during training, effectively obstructing multitask learning. The result of this paper is a straightforward heuristic that can be incorporated into any subsequent research that builds on PINNs solving differential equations. Moreover, it also shows how to make PINNs even more efficient by implementing techniques from multitask learning literature.