When you order by number, typing a number jumps to that numbered row. This feature has been added on user demand some years ago. Adding a modifier key probably restores the system behavior as you discovered by yourself.
The real solution would be adding an option to disable the number search feature.
In the meanwhile you could change the conflicting control modifier menu shortcuts (maybe adding the option key):