debug and release dirs are still generated when "debug_and_release_target" is removed

In my project, I manually define where intermediate build files are placed for both debug and release mode and I use "CONFIG += debug_and_release" in my project file.

In Qt4, this worked exactly as expected.

In Qt5+, it appears that empty "debug" and "release" directories are created no matter what. I've tried adding "CONFIG -= debug_and_release_target" to my project file but it doesn't appear to have any affect; the empty directories are still created.