GPS is tricky! You’re getting very weak signals from very far away that depend on precise timing and a really good view of the sky and horizon.
Make sure your GPS device is laying flat on its back, with its antenna pointed up at the sky, or use an external antenna. Cold starts of 3-5 minutes are normal, use a coin cell battery for warm or hot starts to keep the GPS memory fresh.
And make sure you check the fix quality and number of satellites you’re getting back from the GPS. It has a good idea of how accurate the result is that it’s giving you, so you can use that to avoid accidentally warping hundreds of miles away.