Paperlane

no. 7 · Tuesday 3 November 2026

Derivative Computation in PINNs: Automatic Differentiation, Finite Differences and Beyond

Maciej J. Mikulski and Tadeusz Uhl

math.NA · v2 · 22 pp

Abstract

We systematically investigate finite-difference (FD) derivative computation in Physics-Informed Neural Networks (PINNs) as an alternative to automatic differentiation (AD). On three benchmark PDEs we show that, with a properly calibrated step size, FD matches AD in accuracy on every problem while running faster across the full tested batch-size range and using substantially less GPU memory, and that a stochastic variant we propose outperforms AD on a stationary problem. We further show that for neural architectures with inter-sample dependencies (e.g. BatchNorm, self-attention) the standard PyTorch autograd idiom is silently incorrect; the correct per-sample alternative is computationally infeasible at PINN-relevant batch sizes, while FD provides a forward-only approximation that is empirically an order of magnitude closer to the true per-sample derivative.

comments
v2: journal version under review; 22 pages, 12 figures
fields
Numerical Analysis
pdf ↗ page arXiv:2608.11020 ↗