All you have to do is use the Ctrl+Z keyboard shortcut, or use Edit \ Undo on the menu.
This is especially handy when you are trying to select a bunch of files, and accidentally move the mouse and copy them into the same folder instead:
Once you do that, just immediately use the Ctrl+Z shortcut and the files will be removed, although you will get the regular delete confirmation dialog:
This can be confusing, but since the accidentally copied file is now going to be removed permanently, it makes sense that you’ll have to accept the deletion.
Also note that this works in any version of Windows.