Uploaded image for project: 'Qt'
  1. Qt
  2. QTBUG-99485

Qt WebEngine AutomationId issue

    XMLWordPrintable

Details

    • Windows
    • d95f6a6b23 (qt/qtwebengine/dev) bcb7ab3412 (qt/qtwebengine/6.3) bcb7ab3412 (qt/qtwebengine/6.3.0)

    Description

      User reported following issue:

      "

      Brief issue description:
      QWebEngineView does not provide Id/AutomationId value for html elements.
      It is not possible to get Id/AutomationId by means of Inspect/Spy tools.
      Also there is no way to pass Id/AutomationId from html to Qt through Chromium.
      The only Id value for html elements we can get with Inspect/Spy is RuntimeId.
      But this value is always different for the same html element from one run to another.
      We need Id value, which will be the same for each re-run of application to build automation tests in a proper way.
      Or we need ability to pass our own custom property (Id/AutomationId/CustomAutomationId/whatever) from html element to Qt through Chromium to allow Inspect/Spy to detect it.

      Details:
      Chrome browser, which is based on Chromium engine (like QWebEngineView does) returns AutomationId (Inspect/Spy can detect none-empty AutomationId).
      In case of Chrome browser its AutomationId value usually is equel to html element Id attribute.
      I.e. for element <input id="myId" .../> AutomdtionId value will be "myId".
      Browser based on QWebEngineView does not provide Id/AutomationId property at all (this is valid for standard SimpleBrowser example).

      For native Qt widgets AutomdtionId works in a proper way: Inspect/Spy can detect none-empty value.
      Also, there is a method to set AutomationId: QObject::setObjectName(const QString &name;).
      But there is no way to set AutomationId for QWebEngineView elements, since constructing of elements occures inside WebEngin internally.

      There is a method in WindowsPlatformPlugin, which highlights the source of the issue:
      https://code.woboq.org/qt6/qtbase/src/plugins/platforms/windows/uiautomation/qwindowsuiamainprovider.cpp.html#519

      // Generates an ID based on the name of the controls and their parents.
      QString QWindowsUiaMainProvider::automationIdForAccessible(const QAccessibleInterface *accessible)
      {
      QString result;
      if (accessible) {
      QObject *obj = accessible->object(); // The source of the issue: in case of HTML element QWebEngineView always returns NULL here
      while (obj) {
      QString name = obj->objectName();
      if (name.isEmpty())
      return result;
      if (!result.isEmpty())
      result.prepend(u'.');
      result.prepend(name);
      obj = obj->parent();
      }
      }
      return result;
      }

      We found few approaches, which allows to achive expected results, but they are not reliable for us (required Qt sources modification, complicates build/install process, complicates Qt version update in future):
      1) Modify WebEngine.
      For each html element create QObject which implements QAccessibleInterface in a proper way.
      Since QWebEngineView implements its own QAccessibleInterface for all its child html elements, it is required to modify Qt sources.
      Also it is required to re-compile our own WebEngine. This is complex procedure and it makes migrating to new QtWebEngine versions even more complicated.
      2) Modify Qt WindowsPlatformPlugin.
      content::BrowserAccessibilityQt *obj_qt = static_cast<content::BrowserAccessibilityQt *>(accessible);
      content::BrowserAccessibility* obj = static_cast<content::BrowserAccessibility *>(obj_qt);
      ui::AXNode *node = obj->node();
      const ui::AXNodeData &data; = node->data();
      base::StringPairs attributes = data.html_attributes;
      find attribute Id and assign it to IdAutomdtion
      return AutomdtionId
      It is required to re-compile WindowsPlatformPlugin. It is easear, then WebEngine re-compiling, but also it make it hard to build/install app and complicates Qt version updating in future.

      "

      Attachments

        Issue Links

          For Gerrit Dashboard: QTBUG-99485
          # Subject Branch Project Status CR V

          Activity

            People

              qt_webengine_team Qt WebEngine Team
              tero.pelkonen Tero Pelkonen
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes