Standard shader element for QML

Overview

Several QML projects are using GLSL/ES 2.0 shaders to provide user-defined effects:

• Qt/3D's ShaderProgram element - http://doc.qt.nokia.com/qt3d-snapshot/qml-shaderprogram.html
• Scenegraph's ShaderEffectItem element - http://qt.gitorious.org/qt-labs/scene-graph/blobs/master/src/effects/shadereffectitem.cpp

The common features of such integrations are:

• GLSL/ES 2.0 language support (with Qt's QGLShaderProgram providing translation to GLSL on desktop systems).
• "vertexShader" and "fragmentShader" properties to define the shader source.
• Connection of user-defined QML properties to GLSL/ES uniform variables for shader-specific textures and parameters.

The purpose of this task is to unify the various projects and devise a common understanding of how QML and GLSL/ES should interoperate:

• Standard shader variable names for hooking into the surrounding environment (attributes, uniforms, and varyings).
• Best practice for hooking user-defined properties up to uniform variables (supported types).
• Specifying shader code inline in QML vs out of line in a separate file?
• Do fragment/vertex shaders default to a reasonable default, or must they always be specified?
• Common variables (qgl_Vertex) vs domain-specific (qgl_Light).
• What tooling does Creator need to support our various use cases? e.g. "Scenegraph Effect" as a shader wizard option in Creator's "New" dialog.

With common variable naming and meaning of user-defined variables, it is more likely that a user's knowledge about how to write a QML-compatible shader will carry across different projects. It also simplifies the tooling problem for Creator.

Current Examples

Qt/3D

ShaderProgram {
    id: program
    texture: "textures/basket.jpg"
    property variant texture2 : "textures/qtlogo.png"
    property real interpolationFactor : 1.0

    SequentialAnimation on interpolationFactor {
        running: true
        loops: Animation.Infinite
        NumberAnimation { to : 1.0; duration: 750; }
        PauseAnimation { duration: 550 }
        NumberAnimation { to : 0.0; duration: 750; }
        PauseAnimation { duration: 550 }
    }

    vertexShader: "
        attribute highp vec4 qgl_Vertex;
        attribute highp vec4 qgl_TexCoord0;
        uniform mediump mat4 qgl_ModelViewProjectionMatrix;
        varying highp vec4 texCoord;

        void main(void)
        {
            gl_Position = qgl_ModelViewProjectionMatrix * qgl_Vertex;
            texCoord = qgl_TexCoord0;
        }
    "
    fragmentShader: "
        varying highp vec4 texCoord;
        uniform sampler2D qgl_Texture0;
        uniform sampler2D texture2;
        uniform mediump float interpolationFactor;

        void main(void)
        {
            mediump vec4 col1 = texture2D(qgl_Texture0, texCoord.st);
            mediump vec4 col2 = texture2D(texture2, texCoord.st);
            gl_FragColor = mix(col1, col2, interpolationFactor);
        }
    "
}

Qt/3D's ShaderProgram element has built-in properties for "color" (bound to qgl_Color), "texture" (bound to qgl_Texture0), and "material" (bound to qgl_Material). Additional colors and textures are provided via user-defined uniform variables. This probably needs to be improved so that all properties are consistent with nothing "special". The "vertexShader" and "fragmentShader" variables must be supplied - it is an error for either to be omitted.

Scenegraph

Specifying a shader (DropShadow.qml):

ShaderEffectItem
{
    fragmentShader:
            "varying highp vec2 qt_TexCoord;                                        \n" +
            "varying highp vec2 my_TexCoord1;                                       \n" +
            "varying highp vec2 my_TexCoord2;                                       \n" +
            "varying highp vec2 my_TexCoord3;                                       \n" +
            "varying highp vec2 my_TexCoord4;                                       \n" +
            "uniform sampler2D source;                                              \n" +
            "void main() {                                                          \n" +
            "    lowp vec4 pix = texture2D(source, qt_TexCoord);                    \n" +
            "    lowp float shadow = (texture2D(source, my_TexCoord1).w             \n" +
            "                        + texture2D(source, my_TexCoord2).w            \n" +
            "                        + texture2D(source, my_TexCoord3).w            \n" +
            "                        + texture2D(source, my_TexCoord4).w) * 0.1;    \n" +
            "    gl_FragColor = mix(vec4(0, 0, 0, shadow), pix, pix.w);             \n" +
            "}"

    vertexShader:
            "attribute highp vec4 qt_Vertex;                                        \n" +
            "attribute highp vec2 qt_MultiTexCoord0;                                \n" +
            "uniform highp mat4 qt_Matrix;                                          \n" +
            "uniform highp float xOffset;                                           \n" +
            "uniform highp float xDisplacement;                                     \n" +
            "uniform highp float yOffset;                                           \n" +
            "uniform highp float yDisplacement;                                     \n" +
            "varying highp vec2 qt_TexCoord;                                        \n" +
            "varying highp vec2 my_TexCoord1;                                       \n" +
            "varying highp vec2 my_TexCoord2;                                       \n" +
            "varying highp vec2 my_TexCoord3;                                       \n" +
            "varying highp vec2 my_TexCoord4;                                       \n" +
            "void main() {                                                          \n" +
            "    highp vec2 s1 = vec2(xOffset, yOffset);                            \n" +
            "    highp vec2 s2 = vec2(-xOffset, yOffset);                           \n" +
            "    qt_TexCoord = qt_MultiTexCoord0;                                   \n" +
            "    vec2 shadowPos = qt_MultiTexCoord0 + vec2(-xDisplacement, yDisplacement); \n" +
            "    my_TexCoord1 = shadowPos + s1;                                     \n" +
            "    my_TexCoord2 = shadowPos - s1;                                     \n" +
            "    my_TexCoord3 = shadowPos + s2;                                     \n" +
            "    my_TexCoord4 = shadowPos - s2;                                     \n" +
            "    gl_Position = qt_Matrix * qt_Vertex;                               \n" +
            "}"

    property real xOffset: 0.66 / width;
    property real yOffset: 0.66 / height;
    property real xDisplacement: 10 / width;
    property real yDisplacement: 12 / height;
    property variant source
    smooth: true
}

Specifying the source image for a shader:

DropShadow
{
    anchors.fill: effectRoot
    source: ShaderEffectSource {
        sourceItem: effectRoot
        filtering: ShaderEffectSource.Linear
    }
    blending: true
}

ShaderEffectItem defines the uniforms qt_Matrix and qt_Opacity, and attributes qt_Vertex and qt_MultiTexCoord0. qt_Matrix and qt_Opacity are scenegraph specific.

ShaderEffectItem currently supports qreal, QColor, QTransform, int, QSize(F), QPoint(F), QRect(F), QVector3D and ShaderEffectSource. A QML property must exist for each uniform (except qt_Matrix and qt_Opacity), and the uniform and property must have the exact same name. sampler2Ds are bound to ShaderEffectSources. The ShaderEffectSource can either load a texture from file or render a QML item into an FBO, but it is not a visual item by itself. It has properties for wrap, filtering and mipmapping.

The ShaderEffectItem has default implementations for vertex and fragment shaders that implement simple texturing. In most of the use cases, the default vertex shader has been sufficient, so it is very convenient not having to specify it everywhere. The default vertex shader passes a varying qt_TexCoord to the fragment shader. The default fragment shader assumes that there is a ShaderEffectSource property called "source".

Variable Naming

As can be seen in the above examples, there are some differences in variable naming between the systems.

Update: It has been agreed by random toss of a die to use "qt_" as the standard prefix.

Update 2: Variable name changes have been completed in both Qt/3D and Scenegraph. Issue now closed.

Attributes

Uniforms

Varyings

Varyings don't need specific names if the user supplies both the vertex and fragment shaders. However, if we allow shaders to be optional (as ShaderEffectItem does), then some varying names will need standardization.

Specifying shader source code

There are several possibilities for specifying shader source in QML. First, directly as a string as shown above:

vertexShader: "
   void main(void) ...
"

The QML syntax can handle embedded newlines, but Creator needs some modification to support multi-line strings properly. Second, force shaders out into a separate file:

vertexShader: "blur.vert"
fragmentShader: "blur.frag"

This has the advantage that Creator doesn't need to recognize GLSL strings specially for GLSL syntax highlighting. But in actual usage to date, users prefer the inline version because most shaders are short. It may be useful to support both strings and external files. Third, add new syntax to QML:

vertexShader: ShaderScript {
   void main(void) ...
}

The QML parser can count matching braces to skip over the shader code, otherwise treating it as a multi-line literal string.

Creator tooling

• Syntax highlighting for GLSL and GLSL/ES language variants.
• Semantic support for code completion, builtin variable/function names, etc.
• Recognizing inline shaders in QML files to apply highlighting.
• Crossing the language boundary: auto-completing corresponding QML property and GLSL uniform variable names.
• Wizards under File-New for creating standard shader types, including the QML shell for animatable properties.

Other issues

• Binding attributes to QML properties that contain user-defined geometry? May be useful for morphing mesh animations in QML/3D.
• Are colors always premultiplied, or is it up to the shader program or surface?