Source/header switching not working if header, source not in same directory

I have the following directory structure:

foo/include/foo/bar.hpp
foo/src/bar.cpp

Opening bar.cpp and pressing f4 does not switch to bar.hpp, or the other way around (if the files are not opened as part of a project). This is true regardless of having

#include <foo/bar.hpp>
#include <bar.hpp>
#include "../include/foo/bar.hpp"

in bar.cpp. At least in the last case, switching from the cpp file to the hpp file could work.