Uploaded image for project: 'Coin'
  1. Coin
  2. COIN-284

Move module instructions to their own repositories

    XMLWordPrintable

Details

    • Epic
    • Resolution: Fixed
    • P2: Important
    • None
    • None
    • None
    • Module Instruction Abstraction

    Description

      History

      Since the beginning of Coin project, it was decided to move build and test instructions under CI source repository to speedup initial development. As the time passed, this has turned out to be expensive long-term solution that increased the CI maintenance burden. Not only because the module instructions would require CI service restart every time that the module instruction is changed but also the new instructions are not tested through a normal CI cycle.

       

      Solution

      Since commit ab9875436699c18f1c9133569d3d34032e47e762, Coin yaml -like configuration files that can express Build/Test Instructions that contain commands to be executed on the VM.

       

      How

      The functionality is enabled by adding a file module_config.yaml in coin/ sub-directory in the module repository. This has already been done for the CI self tests, which looks like this (old vs. new):

      qtqa/tqtc-coin-ci.git:src/modules_ci.py (legacy):

      class QtCiModule(Module):
          """Self tests for the Qt CI."""
      
          async def priority(self, repositoryState: storagestructs.RepositoryState) -> ModulePriority:
              return ModulePriority.ModuleMatch if repositoryState.project.endswith('coin-ci') else ModulePriority.NoMatch
      
          @staticmethod
          def setup_ssh(instructions: CommonInstructions, moduleConfig: messages.ModuleConfiguration) -> CommonInstructions:
              """ The ssh config is required for cloning private repositories from Gerrit. """
              if Feature.FAKESSHSETUP in moduleConfig.workItemConfiguration.features:
                  instructions.writeToFile("fakefile", "fake data")
                  return instructions
              path_to_secrets = os.path.join(os.path.dirname(os.path.realpath(__file__)), "../secrets")
              files_to_write = (("cibot_id_rsa", 0o0600),
                                ("cibot_id_rsa.pub", 0o0600),
                                ("config", 0o0644),
                                ("known_hosts", 0o0644))
              instructions.mkdir("/home/qt/.ssh")
              for filename, mode in files_to_write:
                  filepath = os.path.join(path_to_secrets, filename)
                  with open(filepath, "r") as file:
                      instructions.writeToFile("/home/qt/.ssh/%s" % filename, file.read(), fileMode=mode, hiddenContent=True)
              return instructions
      
          async def build_instructions(self, platformInstructions: CommonInstructions, moduleConfig: messages.ModuleConfiguration) -> messages.AgentInstructions:
              repositoryState = moduleConfig.repositoryState
              platformInstructions.installSourceArchive(repositoryState, windowsLineFeeds=False)
              platformInstructions.cd('{{.SourceDir}}')
              platformInstructions.make(['-t', 'submodules'])
              # FIXME Building thrift.protocol might throw warning
              #  PyErr_Format(PyExc_OverflowError, "size exceeded specified limit: %d", limit);
              #  that causes make citest fail unintentionally.
              platformInstructions.executeCommand(["make"],
                                                  onIntegrationFailureMessage % "Could not run make.",
                                                  maxTime=60 * 60,
                                                  ignoreExitCode=True)
              platformInstructions.make(["citest"])
              return platformInstructions.instructions()
      
          async def test_instructions(self, platformInstructions: CommonInstructions, moduleConfig: messages.ModuleConfiguration) -> messages.AgentInstructions:
              return platformInstructions.instructions()
      

      Note: ssh_setup() method is a hack that can be ignored by the module configuration.

      qtqa/tqtc-coin-ci.git:coin/module_config.yaml:

      (tqtc-coin-ci) 12:15:32~/tqtc-coin-ci (master) $ cat coin/module_config.yaml 
      version: 1
      accept_configuration:
        condition: and
        conditions:
            - condition: property  # No cross compilation
              property: host.os
              equals_property: target.os
            - condition: property  # Coin works only on linux
              property: host.os
              equals_value: Linux
      build_instructions:
        # We do not have real checkout so we need to convince make that we have up to date submodules
        - type: ExecuteCommand
          command: make --touch submodules
          userMessageOnFailure: >
            Could not update the timestamps on sources. It is probably a bug in the Coin build system.
            Please check if there is no changes to it.
        - type: ExecuteCommand
          maxTimeInSeconds: 3600
          maxTimeBetweenOutput: 60
          command: make
          userMessageOnFailure: >
            Could not build Coin project. Please investigate changes that could cause the failure.
        - type: ExecuteCommand
          command: make citest
          maxTimeInSeconds: 36000
          maxTimeBetweenOutput: 600
          userMessageOnFailure: >
            One or more tests failed. Please check logs to see which ones. There is a big likelihood
            that it is a regression caused by a tested change.
      

       

       

      Next

      A Jira task will be created for every custom module instructions and linked to this epic

      src/modules_3dstudio.py
      src/modules_automation.py
      src/modules_ci.py
      src/modules_custom.py
      src/modules_ifw.py
      src/modules_pyside.py
      src/modules_qt5.py
      src/modules_saferenderer.py
      src/modules_yocto.py
      

       

      Other

      Bugs related to the instructions or features that are needed can also be linked to this epic.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            tosaario Toni Saario
            aakeskimo Aapo Keskimölö
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes