A web-search on the error message yielded this Technipages answer (but read my comments below before trying this):

When trying to delete folder, you may get the error message: “Cannot Delete [foldername]: The directory is not empty”. Follow these steps to solve this problem.

Go to START–> RUN. In the OPEN: line type CHKDSK /F

Most likely you will get a message like the following:

"Chkdsk cannot run because the volume is in use by another process. Would you like to schedule this volume to be checked the next time the system restarts? (Y/N)"

Type Y for yes and press ENTER on your keyboard. Restart your computer, and the scan will run. Afterward you should be able to delete the folder you wanted to delete because CHKDSK has fixed the errors preventing it.

This is a good start, but is not the complete solution for a plug-in USB stick-- as it turns out, the command above is indeed running a CHKDSK, but on your C: drive. Doesn't hurt but totally doesn't help this particular case either.

The complete set of CHKDSK command-suffixes is given by Wikipedia.

As it turns out, the command you actually want to run for a USB stick is:

chkdsk f: /f
      (assuming your external drive reads as "F:" in your Windows Explorer.
      Substitute the appropriate drive letter as needed.)

Following the Technipages steps above, but using the command chkdsk f: /f at the OPEN prompt,
CHKDSK started up and I got the following message:

"The type of the file system is FAT.
Volume Serial Number is 0432-29ED
Windows is verifying files and folders...
Unrecoverable error in folder \Stuck Undeletable Houses.
Convert folder to file (Y/N)?"

I said 'Y'.

CHKDSK continued:

"File and folder verification is complete.
Convert lost chains to files (Y/N)?"

I said 'Y'.

CHKDSK finished and the DOS window closed.

Now looking at the root of the USB stick in Windows Explorer, there were two new items:
     - A 32KB file called "Stuck Undeletable Houses" (same name as the former undeletable folder).
     - An item looking like a faded-out folder named "FOUND.000".

I was then able to delete both the 32KB file and the faded folder without apparent incident.

Hopefully this will do it for you too.