Only apply patches if we have any, and make sure to cleanup every file we patched at the end to leave the tree clean