Yes, you have to press RESET first, then MODE right away.
No, not unless you change the timing to if(millis() - startTime > 10000) Another way to do it would be to comment out this line BUTTON_ResetDebouncedState(BUTTON1); so that the background code can still see the button being held for 10 seconds. Actually I like this latter idea a lot. The only problem is the interrupt code that monitors the button only resets debounce time when you press the button again. If it reset the debounce time when you let go, we could use the mode button very easily in our code without interfering with the SmartConfig stuff. I'll experiment with this and see if it works, then if so submit a pull request.
Not at this time, but you can certainly roll your own debounce routine on any input. Exposing the MODE function on any input pin is an item on the backlog though, so in the future you should be able to do this 
Not for the Mode button stuff. They are pretty self explanatory though, one just holds the current number of milliseconds that it has been pressed for. The other resets that time.