Uploaded image for project: 'Qt Quality Assurance Infrastructure'
  1. Qt Quality Assurance Infrastructure
  2. QTQAINFRA-6421

Archive and delete data older than 2 years for influxdb on testresults

    XMLWordPrintable

Details

    • Task
    • Resolution: Fixed
    • P1: Critical
    • None
    • None
    • None

    Description

      Right now we have too much of old (older than 2 years) data on influxdb on test results server. This causes slowdown and increases memory usage. We need to archive old data and remove it from influx
      affects coin database on influx.

      measurement: blacklisted_test, 2022 data start: 2022-01-01T00:32:24.593909Z
      measurement: crashed_test, 2022 data start: 2022-01-01T17:04:11.003882Z
      measurement: failed_test, 2022 data start: 2022-01-01T01:13:38.188538Z
      measurement: flaky_test, 2022 data start: 2022-01-01T00:32:24.593909Z
      measurement: integration, 2022 data start: 2022-01-01T14:32:05Z
      measurement: workitem, 2022 data start: 2022-01-01T00:12:43Z

      Right now we do not have retention policies:
      > show retention policies on coin
      name duration shardGroupDuration replicaN default
      ---- -------- ------------------ -------- -------
      autogen 0s 168h0m0s 1 true
      >

      see python notebook (displayed only in firefox at my machine):
      https://git.qt.io/qtqa/notebooks/-/blob/main/old_data/2022_old_data.ipynb?ref_type=heads

      Should we just drop older data? Afik nobody was ever checking the archive data we store.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            jimis Dimitrios Apostolou
            anwojcie Anna Wojciechowska
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes