I was rebuilding my UDP socket after a reconnect and this was causing problems with my ability to reconnect, I put a short timer after my reconnect to delay before restarting the socket and it has solved any problem I've been having.
May not affect you but drawing your attention to it in case!