At the core, this is a problem rooted in your device's inability to accurately gather your location.
If you have this problem, you will notice that opening any map app on your phone is similarly misrepresenting your location, not just Detour.
Do a quick check to make sure your WiFi is turned on, or turn it off then on again, because we use the help of WiFi to help improve your location detection.
You will also want to make sure to check your phone settings to ensure your location permissions all look good.
It may also help to put your phone into airplane mode (this turns your cell signal off) then turning it on again to force a reset.
Hopefully this does the trick.
It can also be informative to tap the "Map" icon in the Detour Tour Player and it will tell you where it "thinks" you are, in relation to the start point.