Back end, English

Best practices for keeping your Qlik framework under version control

I’ve worked with multiple Qlik environments where  Qlik Deployment Framework (QDF) is combined with a version control system during the past years. This hands on experience from multiple environments combined with several interesting discussions with colleagues, clients and Qlik have given me an insight on dos and do nots. The following post is a part of my best practice when working with version control in a QlikView environment.

Create a repository per container

A container is its own universe, you can copy it, replicate it or move it and it will still work without modifications. It is a good idea to create a repo per container.

In GIT you are limited to check in/out whole repositories, you are not able to select designated parts of if. If you choose to have a single repository for the whole environment you will be forced to check out your whole qlik environment everytime you check out a project. I always recommend to split an environment into multiple repositories, one per container.

Using Subversion you are not enforced to the full repository, it is pretty easy limit which part of the repository you want to check out. You may still consider the one-repository-per-container approach using Subversion. One scenario is where different developers are responsible for and should have access to different applications.  By configuring repository access according to the developer rights you may prevent unauthorized/accidential  commits by unauthorized developers.

ContainerscropThe QDF opens up for the architect or administrator to rearrange container structures without breaking script functionality. There could be several reasons for moving a container, but often it is related to business changes. The location/arrangement of containers does not change any Qlik functionally and should neither affect the repository, is a one-repo-per-container setup separation is maintained, but in a single repository environment a change in container structures will generate a lot of changes in the version control system.

Exclude your qvw and data files

Always exclude your binaries and data files. I see the version control as as system for tracking changes in your source code, it is not a back-up system.

The QlikView file (qvw) contains both the layout, script and data. It is stored binary and it is impossible for a version control system to determine which changes that have been done, it will basically commit your whole application into the repository database on every commit.  A large sized application will rapidly increase the size of your version tracking database, hence every time it runs the binary qvw file will be diffrent from the previous execution.

Always put all qvw and qvd files into the ignore list of the version control system.

It is sufficient  to track the prj folder instead of the qvw. The prj folder contains all the layout and data modeling information of the QlikView file. The files of the prj-folder is stored as readable xml files. With the solely the prj folder it is possible to build a new replica of the qvw file it represents empty for data transactions.

Data you load into applications will most likely change over time , data files should therfore never be committed into the version control system. This principle is valid both for data files you import into (excel, csv, etc.) and data files created by your application (qvd).

Keep the frame of the framework outside the version control.

Scaffolding
FreeImages.com/Bo de Visser

 

I consider the framework files not to be a part of my developed source code. I see the framework files as the frame in where I put my developed source code. The frame is not in the developers responsibilities and she should not be make any changes into it. If changes are to be made they should be global for all containers, therefore I consider it the Qlik architect or administrator’s responsibilty. If every developer where allowed to make small adjustments and tweaks to the framework files and then the organization desides to upgrade the framework to and later release and thereby overwrite the framework files tweaks.

By excluding the default qdf container files from version control it becomes very clear to the developer what is the project specific source code files and what is not.

 

Track your development, but stage for the rest

Only the development envrionment should be connected to the version control system. When  the development process is finished the developer should stage her changed files so that they may be copied throughout the DTAP street.

Keep the development in the development environment by cutting all ties to the version control system in test, acceptance and production environment

There is a QlikView server specific reason why you should avoid connecting  the test, acceptance and production environment to fetch from your version control system. I have earlier argued why qvw files should be kept out of the version control.  The QlikView desktop client will first consider the prj-folder information when opening a QlikView qvw-file. The server on the other hand will not consider the prj folder at all, it will only what is prestored into the qvw file. If all changes are pushed out to the prj-folder’s xml-files in the DTAP street the server will not recognize any changes until you have manually opened and saved the qvw file with the desktop client. To avoid this repetative task in all your environments you should always prepare your qvw files in the development and push them through the DTAP street.

There is also the principle that all changes should reside in the development environment, no changes should be committed after are put into any of the other envrionments. Cutting ties to the version control in test, acceptance and production will help prevent occational developing beeing done in these enviroments.

Limitation

There are limitations in version handling solely the prj-folder files of an application in QlikView. Always one selected value is one example where this property is lost when the qlikview file is stripped for data, certain action triggers are another feature that Qlik handles porly when it comes to version control. In cased where these are important features I make sure to stage my qvw from the same location and qvw-file every time, and I make sure the staged qvw-file is not empty.

I have been using these principles in QlikView Qlik Sense coexistence environments, where script mainly resides in external Qlik Script Files, but it does not cover how to track and handles Qlik Sense applications? If you have any inputs please share them in the comments field below or on your own blog.

Additional reads.

Qlik IDE Development environment – Qlik’s take on how to hand on combine QDF and version control ( Tortoise SVN)  using Sublime text editor.

Using Git with Qlikview to version control your projects – Hands on getting started blogpost from BI Experience

14 comments on “Best practices for keeping your Qlik framework under version control

  1. I am not using TortoiseGit at the moment. Haven’t used it for a while as a matter of fact and I have never used Git Extesion. It looks interesting. I will def. take a look at it!

    Juan

  2. Hi,

    Very nice blog! Thank you for this! I have a question on the DTAP process.
    We are versioning control -prj folder and all qvs files in Dev environment to source control system (i.e. GIT). I believe this is only for rollback/changes revert purpose on the DEV environment itself?

    On the DTAP street, we will still copy the latest qvw version from DEV to higher environment. Am i correct?

    How the defect fix/changes can be done in a typical development process i.e. defect found in Test environment, but take back to DEV to fix. However, the version of the qvw in DEV has already added some other additional changes, if we fix that and promote to Test again, it probably will introduce new feature/changes?

    1. Hi Jason,
      I’m glad you like my blog, it makes me happy.

      You are absoulutely correct about moving the qvw file and not the prj-folder up the DTAP street, hence QlikView Publisher/QlikView Server does not consider the prj content as the QlikView desktop client does.

      I tend to put all my development-iterations into branches (from the latest production commit). I.e. I would create an “Add_feature_sliced_milk” branch from the last production for the assignment off adding “sliced milk” into my application. Only when the sliced milk-feature is approved I merge it into my production branch. The additional changes to the new feature “liquid bread” done during the “sliced milk” testing phase are done in another branch “Add_feature_liquid_bread”.

      When the Sliced milk feature returns from test with a defect you may continue the development in your “Add_feature_sliced_milk” branch and the merge into prouction will only be done when testing is approved.

      Your other branches that are still in development/test need to be updated with any changed made to production branch, you need to merge your prod changes into the “Add_feature_liquid_bread” branch.

      /Vegar

  3. I still don’t understand how to keep the “container”/QDF out of reach of the developers and the server…

    It’s all connected.
    Lets say I:
    1) Use the QDF variable editor to add a global/shared variable (into the 99.Shared_Folders)
    2) I also add some stuff to the current containers folders (lets say changes to the 3.Include folder)
    3) I change my main qvw-document to use both the shared variable and some new fancy file in the “3.Include-folder”
    4) I checkin and tag the commit/revision as “31337 release, RC1”

    Well, if I -only- should take the qvw down the DTAP street, how the %&”#¤”/!&%¤ am I suppose to get the correct version/state of the framework files onto the testserver?

    My understanding is that it would be better to just have -all- the containers in the qlikview-server-root and just do a “git checkout tag ‘31337 release, RC1′”.
    I would then have to recreate the qvw’s from the -prj’s manually (this sucks, there should really be a command-line tool to do this, so it could be hooke’d into the checkout process).
    This would then get the entire framework into the correct state.

    How am I supposed to keep track of all the framework-file changes in the different environments (dev/test/prod) if I only were to copy the qvw’s`?

    1. And also, while I’m at it. I have no idea how to actually let 2 devs edit the same app in parallel, since that most likely leads to a scenario where one developer needs to do a manual merge of a 30 000, 50 000 or even hundred of thousands rows long XML file with a internal qlikview stucture…

      Or even have one developer have several development-branches, since this also most likely will lead to merge-problems later on.

      Lets say we have the main-branch, the bugfix branch which will merge into main, and several development-branches..
      When a dev-feature is done, I cant just do a “overwrite” when merging into main since main has changed with bugfixes….

      The problem is that the developers do gui-changes in the gui, not in XMl, so I have no idea how to actually in a secure/correct way do manual-merges of the XML.
      (I’m not even sure that automerge in TFS would be able to do a correct work…)

      1. QlikView development is in one aspect upside-down compared to “normal” development. In QlikView you are working with the binary file that you compile into plain text xml-files, normally it is the other way around you start with the source code that compiles into binary. It makes it a bit troublesome sometimes to track changes in QlikView.

        I don’t encourage 2 devs do to much edits in the same app paralell, but I also have diffrent dev branches active in the same time. The merge process can be a pain. However if the developers limit them self to separate objects/sheets I’ve found the merging to be doable most of the time. I have found my self checking out two diffrent versions and via the QlikView desktop merging the two develop paths into one.

        However there is no issues with letting one developer edit the script in an include file and another developer changing the layout in the qvw. As long the script developer is not checking in his working copy -prj folder then there will never be a conflict.

    2. I see your point Basse, but my main argument for not keeping the whole container root in one repository is that you always need to check out the whole repository when using GIT. For the small Qlik environment this is doable, but as the environment grows with more container that is not related to each other then it is not sustainable.

      I’m not arguing that you should only move the qvw through the DTAP street, all files that is not included in the default empty container should be moved through DEV-TEST-ACCEPTANCE-PROD. This means that the fancy include-file should be sent to the version control and move up together with the qvw file.

      When adding stuff into different containers during a development phase we make the developer keep track of which containers that is to be a part of the delivery. So if the developer make changes to 99.Shared_folders or an Extract container then it is the developers responisbility to include this info in a deployment document.

      Personally I tend to use 99.Shared_folders as little as possible because the more relations between containers the more complexity and the more complexity the more time I would have to put into testing. Any change into 99.Shared_folders could in theory affect all other containers in the QDF structure hence all containers fetching stuff from that container.

  4. Hi,
    Qlik Sense files are not easy to manage using classic source control like Git or Svn.
    We switched to a third party tool ebiexperts Version Manager for Qlik that manage integrated QlikView and Qlik Sense, source / version control, compare and deployment. They have a new feature that allow to automatically hide the scripts on deployment.
    Have a look on their web site http://www.ebiexperts.com
    Hayder

    1. Thanks for sharing Hayder.
      I know there are some specialised tools around such as wbiexperts version manager, but I have had the oppurtunity to trie any of them.
      /Vegar

  5. Thanks for sharing. Did you ever use git with the entire QDF source folder with all your containers and run into issues? I’m doing it this way and find it convenient but like you said, I have not had to move any containers.

    1. No I have not done so in GIT, but I’ve done it using SVN. Beside the drawbacks moving containers around GIT does not allow you to check out fractions of your repository as SVN does. If you check all containers into one too then all developers would need to check out the whole QDF structure for all development tasks. It could be convenient in a small controlled environment, but not in a large implementation. /Vegar

Leave a Reply

Your email address will not be published. Required fields are marked *