QML ListView unable to properly show dynamic delegates of different heights

As the title say, when a delegate of a ListView loads different components based on different model roles, QML ListView is having a hard time estimating its contentHeight. This is evident in my block editor (get-notes.com), in 2 ways:

1. The scrollbar isn't being calculated properly (see how the scrollbar height keep changing in the attached video).
2. Very tall delegates causes scrolling to be not smooth and worsen user-experience. Also sometimes scrolling fast can cause stuttering/jumping. I don't think it's related to large loading/creation time but rather from inability to calculate the delegate height. 
3. It actually causes many other bugs (for example, sometimes when a user tries to scroll while selecting text it suddenlt stops scrolling since my "pixelToTravel" calculation is probably inaacurate due to the wrong contentHeight and contentY).

pixelsToTravel = blockEditorView.contentHeight - blockEditorView.contentY - selectionArea.height; 

I know this must be a difficult problem, but I've seen native macOS apps doing fine, so I assume there must be a solution. My idea for a solution will be a way to for the model to tell the view (for example, on C++ side), the estimatedHeight of the item/delegate. Maybe there are better ways tho. Are there any other tricks I'm unaware of?

One way I'm mitigating this is by increasing ListView's cacheBuffer to height*2. But this is both inefficient, and only helps when the ListView is very small so it can calculate the height better. I'm curious how this can be resolved.

Sidenote: Not sure why but I think it might be only a problem when scrolling upward (maybe just in my case)?