Details
-
Epic
-
Resolution: Fixed
-
P2: Important
-
None
-
None
-
None
-
Module Instruction Abstraction
Description
History
Since the beginning of Coin project, it was decided to move build and test instructions under CI source repository to speedup initial development. As the time passed, this has turned out to be expensive long-term solution that increased the CI maintenance burden. Not only because the module instructions would require CI service restart every time that the module instruction is changed but also the new instructions are not tested through a normal CI cycle.
Solution
Since commit ab9875436699c18f1c9133569d3d34032e47e762, Coin yaml -like configuration files that can express Build/Test Instructions that contain commands to be executed on the VM.
How
The functionality is enabled by adding a file module_config.yaml in coin/ sub-directory in the module repository. This has already been done for the CI self tests, which looks like this (old vs. new):
qtqa/tqtc-coin-ci.git:src/modules_ci.py (legacy):
class QtCiModule(Module): """Self tests for the Qt CI.""" async def priority(self, repositoryState: storagestructs.RepositoryState) -> ModulePriority: return ModulePriority.ModuleMatch if repositoryState.project.endswith('coin-ci') else ModulePriority.NoMatch @staticmethod def setup_ssh(instructions: CommonInstructions, moduleConfig: messages.ModuleConfiguration) -> CommonInstructions: """ The ssh config is required for cloning private repositories from Gerrit. """ if Feature.FAKESSHSETUP in moduleConfig.workItemConfiguration.features: instructions.writeToFile("fakefile", "fake data") return instructions path_to_secrets = os.path.join(os.path.dirname(os.path.realpath(__file__)), "../secrets") files_to_write = (("cibot_id_rsa", 0o0600), ("cibot_id_rsa.pub", 0o0600), ("config", 0o0644), ("known_hosts", 0o0644)) instructions.mkdir("/home/qt/.ssh") for filename, mode in files_to_write: filepath = os.path.join(path_to_secrets, filename) with open(filepath, "r") as file: instructions.writeToFile("/home/qt/.ssh/%s" % filename, file.read(), fileMode=mode, hiddenContent=True) return instructions async def build_instructions(self, platformInstructions: CommonInstructions, moduleConfig: messages.ModuleConfiguration) -> messages.AgentInstructions: repositoryState = moduleConfig.repositoryState platformInstructions.installSourceArchive(repositoryState, windowsLineFeeds=False) platformInstructions.cd('{{.SourceDir}}') platformInstructions.make(['-t', 'submodules']) # FIXME Building thrift.protocol might throw warning # PyErr_Format(PyExc_OverflowError, "size exceeded specified limit: %d", limit); # that causes make citest fail unintentionally. platformInstructions.executeCommand(["make"], onIntegrationFailureMessage % "Could not run make.", maxTime=60 * 60, ignoreExitCode=True) platformInstructions.make(["citest"]) return platformInstructions.instructions() async def test_instructions(self, platformInstructions: CommonInstructions, moduleConfig: messages.ModuleConfiguration) -> messages.AgentInstructions: return platformInstructions.instructions()
Note: ssh_setup() method is a hack that can be ignored by the module configuration.
qtqa/tqtc-coin-ci.git:coin/module_config.yaml:
(tqtc-coin-ci) 12:15:32~/tqtc-coin-ci (master) $ cat coin/module_config.yaml version: 1 accept_configuration: condition: and conditions: - condition: property # No cross compilation property: host.os equals_property: target.os - condition: property # Coin works only on linux property: host.os equals_value: Linux build_instructions: # We do not have real checkout so we need to convince make that we have up to date submodules - type: ExecuteCommand command: make --touch submodules userMessageOnFailure: > Could not update the timestamps on sources. It is probably a bug in the Coin build system. Please check if there is no changes to it. - type: ExecuteCommand maxTimeInSeconds: 3600 maxTimeBetweenOutput: 60 command: make userMessageOnFailure: > Could not build Coin project. Please investigate changes that could cause the failure. - type: ExecuteCommand command: make citest maxTimeInSeconds: 36000 maxTimeBetweenOutput: 600 userMessageOnFailure: > One or more tests failed. Please check logs to see which ones. There is a big likelihood that it is a regression caused by a tested change.
Next
A Jira task will be created for every custom module instructions and linked to this epic
src/modules_3dstudio.py src/modules_automation.py src/modules_ci.py src/modules_custom.py src/modules_ifw.py src/modules_pyside.py src/modules_qt5.py src/modules_saferenderer.py src/modules_yocto.py
Other
Bugs related to the instructions or features that are needed can also be linked to this epic.