A sample .qml file attached. It draws 3 Shape's. The first one has no antialiasing techniques applied. The second has:

antialiasing: true

and the 3rd has:

layer.enabled: true
layer.samples: 4

The result is that "antialiasing: true" seems to do nothing noticeable. The edges (especially the sloped edge on left side) is still quite coarse. MSAA seems to work well though, even without "antialiasing" being turned on.