uloop: remove file descriptors if neither read nor write notification is requested