Uploaded image for project: 'Qt Installer Framework'
  1. Qt Installer Framework
  2. QTIFW-2287

Can't update existing repository with repogen

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • P1: Critical
    • 4.2.0
    • 4.1.0, 4.0.0, 4.1.1
    • Tools
    • Windows 10, QT IFW 4.1.1.
    • Windows
    • Qt Installer Sprint 37-Summer

    Description

      Can't update existing repository

      Description

      Updating an existing repository with repogen produce a 7zip unknown exception.

      Could you please confirm if bug and solution found are correct.

      Reproduce issue

      Create repository with at least 2 components

      > C:\Qt\QtIFW-4.1.1\bin\repogen.exe -p packages1 repository

      Add updated or new components to repository

      > C:\Qt\QtIFW-4.1.1\bin\repogen.exe -p packages2 -update repository

      Caught 7zip exception: Unknown exception caught (void __cdecl Lib7z::createArchive(const class QString &,const class QStringList &,enum Lib7z::TmpFile,enum Lib7z::Compression,class Lib7z::UpdateCallback *))

      Debug

      Versions

      Bug not present in 3.2.2, but in 4.0 and 4.1.  
      Was introduced after unifyMetadata functionality was created.

      Going through call stack

      installer-framework\src\libs\ifwtools\repositorygen.cpp

      • QInstallerTools::createRepository 
      • QInstallerTools::compressMetaDirectories 
      • QInstallerTools::unifyMetadata  

      installer-framework\src\libs\installer\lib7z_facade.cpp

      • Lib7z::createArchive  

      installer-framework\src\libs\7zip\win\CPP\7zip\UI\Common\Update.cpp

      • UpdateArchive 
      • Compress  

      installer-framework\src\libs\7zip\win\CPP\7zip\UI\Common\UpdatePair.cpp

      • GetUpdatePairInfoList  
         -> line 123. ThrowError k_Duplicate_inDir_Message  

       

      So it seems that the 7Zip unknown error correspond to duplicate filenames passed as arguments.

      Source code

      repositorygen.cpp, QInstallerTools::unifyMetadata, line 807, foreach loop.

      Checking args passed through the functions, the bug seems to be on Qt ifw side.  
      When a unified zip containing metadata of previous and new packages is created.  

      For this test, we assume that package A, B and C are in temp dir, we cant to add packages D,E.

      Before loop

      A list containing the absolute paths of each component we want to add is created.

      QStringList absPaths;
      QDir dir(repoDir);
      const QStringList entryList = dir.entryList(QDir::Dirs | QDir::NoDotAndDotDot);

      foreach (const QString &i, entryList) {
          dir.cd;
          const QString absPath = dir.absolutePath();
          absPaths.append(absPath);
          dir.cdUp();
      }

      Abspaths content :

       ("C:/Users/USERNAME/AppData/Local/Temp/repogen-LDfpzY/com.vendor.packageD",
       "C:/Users/USERNAME/AppData/Local/Temp/repogen-LDfpzY/com.vendor.packageE")

      Then we check for packages already existing in repo, to add their path too.

      //Checking if a repo exist already
      QTemporaryDir existingRepoTempDir; 
      QString existingRepoTemp = existingRepoTempDir.path();

      if (!existingRepoDir.isEmpty()) {
          existingRepoTempDir.setAutoRemove(false);
          QFile archiveFile(existingRepoDir);
          QInstaller::openForRead(&archiveFile);
          Lib7z::extractArchive(&archiveFile, existingRepoTemp);
          QDir dir2(existingRepoTemp);

          //Entrylist contains names of packages, such as com.vendor.product1, com.vendor.product2
          QStringList existingRepoEntries = dir2.entryList(QDir::Dirs | QDir::NoDotAndDotDot);
          foreach (const QString existingRepoEntry, existingRepoEntries) {
              if (entryList.contains(existingRepoEntry)) {
                  continue;
              } else {
                  dir2.cd(existingRepoEntry);
                  const QString absPath = dir2.absolutePath();
                  absPaths.append(absPath);
              }
          }
      }

      After loop

      Abspaths now contains path of previous and new components.  
      But there are duplicates.

       ("C:/Users/USERNAME/AppData/Local/Temp/repogen-LDfpzY/com.vendor.packageD",
       "C:/Users/USERNAME/AppData/Local/Temp/repogen-LDfpzY/com.vendor.packageE",
       "C:/Users/USERNAME/AppData/Local/Temp/repogen-mtFrRf/com.vendor.packageA",
       "C:/Users/USERNAME/AppData/Local/Temp/repogen-mtFrRf/com.vendor.packageA",
       "C:/Users/USERNAME/AppData/Local/Temp/repogen-mtFrRf/com.vendor.packageA")

      Then the Lib7z::createArchive call that throw the exception because of duplicate filenames.

      // Compress all metadata from repository to one single 7z
      const QString metadataFilename = QDateTime::currentDateTime().
              toString(QLatin1String("yyyy-MM-dd-hhmm")) + QLatin1String("_meta.7z");
      const QString tmpTarget = repoDir  QDir::separator()  metadataFilename;

      Lib7z::createArchive(tmpTarget, absPaths, Lib7z::TmpFile::No);

      Solution

      The foreach > else part seems to be missing a dir.cdUp().  
      It cause the dir2.cd(...) to enter the first package directory, and search all the subsequent packages inside of it, and fail.

      Then dir2.absolutePath() always return the path of the first package.

       foreach (const QString existingRepoEntry, existingRepoEntries) {
              if (entryList.contains(existingRepoEntry)) {
                  continue;
              } else {
                  dir2.cd(existingRepoEntry);
                  const QString absPath = dir2.absolutePath();
                  absPaths.append(absPath);

                  dir2.cdUp(); // ADDED
              }
          }

      After fix, abspaths seems to be correct before entering createArchive function.  
      And 7Zip error doesn't appear again.

      ("C:/Users/USERNAME/AppData/Local/Temp/repogen-LDfpzY/com.vendor.packageD",
       "C:/Users/USERNAME/AppData/Local/Temp/repogen-LDfpzY/com.vendor.packageE",
       "C:/Users/USERNAME/AppData/Local/Temp/repogen-mtFrRf/com.vendor.packageA",
       "C:/Users/USERNAME/AppData/Local/Temp/repogen-mtFrRf/com.vendor.packageB",
       "C:/Users/USERNAME/AppData/Local/Temp/repogen-mtFrRf/com.vendor.packageC")

      Attachments

        1. test.zip
          19 kB
          Valentin Hulot
        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            arttu.tarkiainen Arttu Tarkiainen
            siriusval Valentin Hulot
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes