Here's a suggestion on how it could be implemented on X11 and Wayland.

X11:
1. Check whether _NET_SUPPORTED has _GTK_FRAME_EXTENTS (implemented by lots of WMs including KWin)
2. If it has, draw a client-side shadow for the resize handles and use _GTK_FRAME_EXTENTS to communicate the extents of the client-side to the WM. Here are some implementations:
https://gitlab.gnome.org/GNOME/gtk/-/blob/ab7d969700164e4306bc6c5e52c39501a67d70ac/gdk/x11/gdksurface-x11.c#L210-244
https://invent.kde.org/frameworks/kwindowsystem/-/blob/08353b02e890a2ea6e2929ac02133f1ad5ff2166/src/platforms/xcb/netwm.cpp#L3406-3417
(please make sure to unset _GTK_FRAME_EXTENTS when not doing client-side shadows rather than just setting it to 0)
3. If it doesn't have, draw borders for the resize handles.

Wayland:
The QPA already has some infrastructure for client-side decorations including communicating shadow margins but all of them currently draw borders rather than shadows. Just teach the decos to draw shadows and then draw only shadows/borders.