You can open the FontDialog and press tab to go to the font list; I wonder if it would be better to have it focused by default though.

Then there is the bug that TableView jumps two rows for each press of the arrow key, but that's a bug in TableView.

After you tab to the table of font sizes, arrowing through the list should maybe change the selected font size in the spinbox above.

Finally when you press enter, it doesn't switch to the font that you selected, but just dismisses the dialog.