Structured Bindings with Qt SQL

Some time ago, I wrote a post about integrating Qt’s associative containers with the fancy new C++ features, range-based for loops with structured bindings.

That post inspired KDAB’s own Giuseppe D’Angelo to add the asKeyValueRange member function to both QHash and QMap. Now it’s possible to iterate over them with a simple range-based for loop, like so:

    for (auto [key, value] : map.asKeyValueRange()) {
        // ...
    }

The second part of my previous post demonstrates how we can iterate over Qt SQL results using a range-based for loop as if it were an ordinary collection. In addition, it announces this second part of the post that shows you how to add the support for structured bindings into the mix.

Structured Bindings, Revisited

Recall that structured bindings allow us to decompose structures such as std::pair and QPair, so that we can give more intuitive names to the fields in those structures than the first and second names provided by both pair types:

    auto [x, y] = mousePosition();

In the previous example, when the range-based for loop iterates over map.asKeyValueRange(), it takes each key-value pair from the map and assigns the name key to the first element of the pair and the name value to the second element.

Internally, the pair is stored in an invisible variable, and the names key and value just refer to the fields inside of that variable.

Behind the scenes of structured bindings

Out-of-the-box, structured bindings can also be used with arrays, tuple-like types, or, a bit less useful outside of generic programming, with ordinary user-defined structures:

    std::tuple<bool, QString, QString> employee;
    auto &[active, name, team_name] = employee;

    Employee employees[2];
    auto &[first_employee, second_employee] = employees;

    struct Employee {
        bool m_active;
        QString m_name;
        QString m_team;
    };
    Employee employee = ...;
    auto &[active, name, team_name] = employee;

Structured Bindings Support for Custom Types

It is also possible to make our own types decomposable with structured bindings, by making sure our types implement the so-called tuple protocol or, in other words, by making our types look like tuples.

Imagine we don’t want the m_active member variable to be seen when using structured bindings on the previously defined Employee type. Instead, we just want to be able to bind m_name and m_team:

    auto &[name, team_name] = employee;

In order to specify how our type should be decomposed with structured bindings, we need to define a few things:

  • into how many values an instance of our type can be decomposed;
  • the type of each of those values;
  • a getter for each of those values.

Tuple-like Employee

The first part of the tuple protocol is simple — we need to specialize the std::tuple_size template for our type Employee. Since we only want to bind m_name and m_team, the size of our tuple-like type will be 2.

    #include <utility>

    ...

    namespace std {
        template<>
        struct tuple_size<::Employee> {
            static constexpr std::size_t value = 2;
        };
    }

The next step is to specialize the std::tuple_element template:

    namespace std {
        template<>
        struct tuple_element<0, ::Employee> {
            // The type of m_name
            using type = QString;
        };

        template<>
        struct tuple_element<1, ::Employee> {
            // The type of m_team
            using type = QString;
        };
    }

The value we defined in the std::tuple_size specialization tells the compiler how many values it will get when it decomposes an instance of Employee, and the types we defined in the std::tuple_element specializations are the types of those values. In our case, both values are QStrings.

The last step is to create a get function template. It can be a member of Employee, but it can also be a free function (non-member) template.

     template <std::size_t Idx>
     auto& get(Employee& employee) {
         if constexpr (Idx == 0) return employee.m_name;
         if constexpr (Idx == 1) return employee.m_team;
     }

It’s worth noting that this implementation will not accept const objects and you’ll need to provide a get implementation that takes a reference to a const Employee, if you want to support those as well.

Decomposing QSqlRecord

Now that we know what we need to implement in order for our types to be usable with structured bindings, we can try to do it with QSqlResultIterator, which we implemented in part 1 of this blog post.

As a reminder, in the first part of the post, we implemented the QSqlResultIterator class that can be used to iterate over all results of a QSqlQuery. We also implemented operator[] on it, which allows us to access fields in a result.

    class QSqlResultIterator {
        // ...
        QVariant operator[] (int index) const
        {
            return m_query.value(index);
        }
    };

We can use this to base the get function template on. To demonstrate that get doesn’t need to be a free function, we will implement it as a member of QSqlResultIterator:

    class QSqlResultIterator {
        // ...
        template <std::size_t Idx>
        QVariant get() const
        {
            return m_query.value(index);
        }
    };

The remaining things that need to be implemented are the specializations of std::tuple_size and std::tuple_element.

Since all values in QSqlResult are QVariants, specializing std::tuple_element is trivial. For any index we’re given, we just need to set type = QVariant:

    namespace std {
        template<std::size_t Idx>
        struct tuple_element<Idx, QSqlResultIterator> {
            using type = QVariant;
        };
    }

The std::tuple_size, on the other hand, is tricky. SQL queries are a runtime thing, and we need to know the number of fields in a record at compile time. This means that we need to allow the user to explicitly define the number of fields in a record when creating the QSqlResultIterator. One way to do it is to make QSqlResultIterator a class template with one std::size_t parameter:

    template <std::size_t FieldCount>
    class QSqlResultIterator {
        // ...
    };

This will allow us to define everything we need to allow QSqlResultIterator to be used with structured bindings:

    template <std::size_t FieldCount>
    class QSqlResultIterator {
        template <std::size_t Idx>
        QVariant get() const
        {
            return m_query.value(index);
        }

        // ...
    };

    namespace std {
        template<std::size_t FieldCount>
        struct tuple_size<QSqlResultIterator<FieldCount>> {
            statuc constexpr std::size_t value = FieldCount;
        };

        template<std::size_t Idx, std::size_t FieldCount>
        struct tuple_element<Idx, QSqlResultIterator<FieldCount>> {
            using type = QVariant;
        };
    }

We could even add a few static_asserts that would check that Idx is less than FieldCount everywhere.

Broken Range-based For Loop Use-case

When we added the FieldCount template parameter to QSqlResultIterator, we broke the use-case we had in the part 1 of this post. We now require FieldCount to be specified explicitly when an instance of QSqlResultIterator is created, and we are not creating it anywhere explicitly.

As a reminder, the QSqlResultIterator was instantiated by the range-based for loop which called begin on the QSqlQuery instance we passed to it:

    for (auto result: query) {
        // ...
    }

It happened behind the scenes and we cannot control how begin is called by the range-based for loop to be able to pass in FieldCount somehow.

Or, can we?

We can write a simple wrapper similar to what we did for asKeyValueRange. Then, instead of begin being defined for QSqlQuery directly, it would be defined for that wrapper, and it would be able to create QSqlResultIterator with the proper FieldCount value.

It could look something like this:

    template <std::size_t FieldCount>
    class QSqlResultRange {
    public:
        QSqlResultRange(QSqlQuery query)
            : m_query(std::move(query))
        {}

        QSqlResultIterator<FieldCount> begin()
        {
            return { m_query };
        }

        QSqlResultSentinel end() const
        {
            return {};
        }

    private:
        QSqlQuery m_query;
    };

Then we can use it as follows:

    for (auto [active, name, team] : QSqlResultRange<3>(query)) {
        // ...
    }

Adding Types to the Mix

So far, we’ve implemented a range object that allows us to iterate over SQL results using a range-based for loop with structured bindings.

In the previous example, we get bindings active, name and team that all have QVariant type.

Can we improve the type-safety of this code, considering our almost always storing concrete types in a database and having to unwrap all QVariants, manually, is tedious and error prone?

Can we specify that the range object returns a bool and two QStrings in each iteration?

As usual in C++, the answer here is a resounding yes. The only thing that we need to do is replace all occurrences of the FieldCount parameter with a variadic pack of types, which will allow us to specify the exact types we expect to get in each resulting row of an SQL query.

In order avoid mixing these with the previously defined types, we’ll add Typed to the names of classes we’ve created so far.

    // We want to allow the user to specify the types
    // of the fields in an SQL row
    template <typename ...Types>
    class QSqlResultTypedIterator {
    public:
        // The constructor and all the basic functions
        // we had in QSqlResultIterator remain unchanged

        QSqlResultTypedIterator(QSqlQuery& query)
            : m_query(query)
        {
            m_query.next();
        }

        QSqlResultTypedIterator& operator++()
        {
            m_query.next();
            return *this;
        }

        bool operator!=(QSqlResultSentinel sentinel) const
        {
            Q_UNUSED(sentinel);
            return m_query.isValid();
        }

        QSqlResultTypedIterator& operator*()
        {
            return *this;
        }

        // The only one that differs is the tuple-compatible
        // get member function. It can return different types
        // depending on the provided index.
        template <size_t Idx>
        auto get() const
        {
            using ResultType = std::tuple_element_t<Idx, std::tuple<Types...>>;
            // We can assert that the type stored inside of QVariant
            // is the type that we expect to be in it.
            Q_ASSERT(m_query.value(Idx).canView<ResultType>());

            // .value returns a QVariant. Then we call .value
            // on said variant to convert it to the desired type.
            return m_query.value(Idx).value<ResultType>();
        }

    private:
        QSqlQuery& m_query;
    };

    namespace std {
        // The tuple_size for QSqlResultTypedIterator is the
        // number of types inside of Types... which we can
        // easily get with sizeof...(Types)
        template<typename... Types>
        struct tuple_size<QSqlResultTypedIterator<Types...>> : public integral_constant<size_t, sizeof...(Types)> {};

        // The simplest way to implement tuple_element on our type
        // is to just base it on the implementation of std::tuple itself.
        // When we are asked for tuple_element<Idx, QSqlResultTypedIterator<Types...>>,
        // we will just replace QSqlResultTypedIterator with std::tuple,
        // and return tuple_element<Idx, std::tuple<Types...>>
        template<std::size_t Idx, typename... Types>
        struct tuple_element<Idx, QSqlResultTypedIterator <Types...>>:
               tuple_element<Idx, std::tuple              <Types...>>
        {
        };
    }


    // The complex part was in the QSqlResultTypedIterator, and the
    // range object remains as simple as QSqlResultIterator was.
    // The only change is that FieldCount is replaced by Types... everywhere
    template <typename ...Types>
    class QSqlResultTypedRange {
    public:
        QSqlResultTypedRange(QSqlQuery query)
            : m_query(std::move(query))
        {
        }

        QSqlResultTypedIterator<Types...> begin()
        {
            return { m_query };
        }

        QSqlResultSentinel end() const
        {
            return {};
        }

    private:
        QSqlQuery m_query;
    };

Wrap-up

This was a lengthy post to follow, and it had a lot of non-trivial code for something as simple as being able to write:

    for (auto [active, name, team] : QSqlResultTypedRange<bool, QString, QString>(query)) {
        qDebug() << "active(bool):" << active;
        qDebug() << "name(QString):" << name;
        qDebug() << "team(QString):" << team;
    }

While this might not look like it is worth doing, remember that this is something you need to write only once and you will use it a lot, if you have an application that is SQL-heavy.

It will make your code more easy-to-read and as type-safe as possible when SQL is concerned, since the debug builds will assert that you are not trying to coerce values of one type into being something that they are not, when crossing the border between SQL and C++.

It is also easily extendible, to allow the user to skip the conversion from QVariant to a specific type by skipping the .value<ResultType>() part when the user-specified QVariant is the desired type, or to support using wrapper types such as std::optional when you have fields in an SQL table that can be NULL.

 

About KDAB

If you like this article and want to read similar material, consider subscribing via our RSS feed.

Subscribe to KDAB TV for similar informative short video content.

KDAB provides market leading software consulting and development services and training in Qt, C++ and 3D/OpenGL. Contact us.

The post Structured Bindings with Qt SQL appeared first on KDAB.

Call for Presentations for Qt World Summit 2022

Qt is looking for speakers, collaborators and industry thought leaders to share their expertise and thoughts with the community during the upcoming virtualQt World Summit on November 9, 2022. 

Qt for MCUs 2.1.1 Released

Qt for MCUs 2.1.1 has been released and is available for download. As a patch release, Qt for MCUs 2.1.1 provides bug fixes and other improvements, and maintains source compatibility with Qt for MCUs 2.1.0. It does not add any new functionality.

Extending the range of time

Until Qt 6.2, QDateTime's ability to take time-zone adjustments – both seasonal daylight-saving time and occasional changes (on the whims of politicians) to a zone's standard offset from UTC – into account was limited to the years 1970 through 2037, with some kludges in place to extrapolate beyond 2037. As the default assignee for time-related bugs, I'd long wanted to fix this, to use such information as we do have available. So – finally, once Qt 6 had been released – I made that change and discovered what it broke.

Packaging PySide6 applications into a macOS app with PyInstaller (updated for 2022)

There is not much fun in creating your own desktop applications if you can't share them with other people — whether than means publishing it commercially, sharing it online or just giving it to someone you know. Sharing your apps allows other people to benefit from your hard work!

The good news is there are tools available to help you do just that with your Python applications which work well with apps built using PySide6. In this tutorial we'll look at the most popular tool for packaging Python applications: PyInstaller.

This tutorial is broken down into a series of steps, using PyInstaller to build first simple, and then more complex PySide6 applications into distributable macOS app bundles. You can choose to follow it through completely, or skip to the parts that are most relevant to your own project.

We finish off by building a macOS Disk Image, the usual method for distributing applications on macOS.

You always need to compile your app on your target system. So, if you want to create a Mac .app you need to do this on a Mac, for an EXE you need to use Windows.

Example Disk Image for macOS Example Disk Image Installer for macOS

If you're impatient, you can download the Example Disk Image for macOS first.

Requirements

PyInstaller works out of the box with PySide6 and as of writing, current versions of PyInstaller are compatible with Python 3.6+. Whatever project you're working on, you should be able to package your apps.

You can install PyInstaller using pip.

bash
pip3 install PyInstaller

If you experience problems packaging your apps, your first step should always be to update your PyInstaller and hooks package the latest versions using

bash
pip3 install --upgrade PyInstaller pyinstaller-hooks-contrib

The hooks module contains package-specific packaging instructions for PyInstaller which is updated regularly.

Install in virtual environment (optional)

You can also opt to install PySide6 and PyInstaller in a virtual environment (or your applications virtual environment) to keep your environment clean.

bash
python3 -m venv packenv

Once created, activate the virtual environment by running from the command line —

bash
call packenv\scripts\activate.bat

Finally, install the required libraries. For PySide6 you would use —

python
pip3 install PySide6 PyInstaller

Getting Started

It's a good idea to start packaging your application from the very beginning so you can confirm that packaging is still working as you develop it. This is particularly important if you add additional dependencies. If you only think about packaging at the end, it can be difficult to debug exactly where the problems are.

For this example we're going to start with a simple skeleton app, which doesn't do anything interesting. Once we've got the basic packaging process working, we'll extend the application to include icons and data files. We'll confirm the build as we go along.

To start with, create a new folder for your application and then add the following skeleton app in a file named app.py. You can also download the source code and associated files

python
from PySide6 import QtWidgets

import sys

class MainWindow(QtWidgets.QMainWindow):

    def __init__(self):
        super().__init__()

        self.setWindowTitle("Hello World")
        l = QtWidgets.QLabel("My simple app.")
        l.setMargin(10)
        self.setCentralWidget(l)
        self.show()

if __name__ == '__main__':
    app = QtWidgets.QApplication(sys.argv)
    w = MainWindow()
    app.exec()

This is a basic bare-bones application which creates a custom QMainWindow and adds a simple widget QLabel to it. You can run this app as follows.

bash
python app.py

This should produce the following window (on macOS).

Simple skeleton app in PySide6 Simple skeleton app in PySide6

Building a basic app

Now we have our simple application skeleton in place, we can run our first build test to make sure everything is working.

Open your terminal (command prompt) and navigate to the folder containing your project. You can now run the following command to run the PyInstaller build.

python
pyinstaller --windowed app.py

The --windowed flag is necessary to tell PyInstaller to build a macOS .app bundle.

You'll see a number of messages output, giving debug information about what PyInstaller is doing. These are useful for debugging issues in your build, but can otherwise be ignored. The output that I get for running the command on my system is shown below.

bash
martin@MacBook-Pro pyside6 % pyinstaller --windowed app.py
74 INFO: PyInstaller: 4.8
74 INFO: Python: 3.9.9
83 INFO: Platform: macOS-10.15.7-x86_64-i386-64bit
84 INFO: wrote /Users/martin/app/pyside6/app.spec
87 INFO: UPX is not available.
88 INFO: Extending PYTHONPATH with paths
['/Users/martin/app/pyside6']
447 INFO: checking Analysis
451 INFO: Building because inputs changed
452 INFO: Initializing module dependency graph...
455 INFO: Caching module graph hooks...
463 INFO: Analyzing base_library.zip ...
3914 INFO: Processing pre-find module path hook distutils from '/usr/local/lib/python3.9/site-packages/PyInstaller/hooks/pre_find_module_path/hook-distutils.py'.
3917 INFO: distutils: retargeting to non-venv dir '/usr/local/Cellar/python@3.9/3.9.9/Frameworks/Python.framework/Versions/3.9/lib/python3.9'
6928 INFO: Caching module dependency graph...
7083 INFO: running Analysis Analysis-00.toc
7091 INFO: Analyzing /Users/martin/app/pyside6/app.py
7138 INFO: Processing module hooks...
7139 INFO: Loading module hook 'hook-PyQt6.QtWidgets.py' from '/usr/local/lib/python3.9/site-packages/PyInstaller/hooks'...
7336 INFO: Loading module hook 'hook-xml.etree.cElementTree.py' from '/usr/local/lib/python3.9/site-packages/PyInstaller/hooks'...
7337 INFO: Loading module hook 'hook-lib2to3.py' from '/usr/local/lib/python3.9/site-packages/PyInstaller/hooks'...
7360 INFO: Loading module hook 'hook-PyQt6.QtGui.py' from '/usr/local/lib/python3.9/site-packages/PyInstaller/hooks'...
7397 INFO: Loading module hook 'hook-PyQt6.QtCore.py' from '/usr/local/lib/python3.9/site-packages/PyInstaller/hooks'...
7422 INFO: Loading module hook 'hook-encodings.py' from '/usr/local/lib/python3.9/site-packages/PyInstaller/hooks'...
7510 INFO: Loading module hook 'hook-distutils.util.py' from '/usr/local/lib/python3.9/site-packages/PyInstaller/hooks'...
7513 INFO: Loading module hook 'hook-pickle.py' from '/usr/local/lib/python3.9/site-packages/PyInstaller/hooks'...
7515 INFO: Loading module hook 'hook-heapq.py' from '/usr/local/lib/python3.9/site-packages/PyInstaller/hooks'...
7517 INFO: Loading module hook 'hook-difflib.py' from '/usr/local/lib/python3.9/site-packages/PyInstaller/hooks'...
7519 INFO: Loading module hook 'hook-PyQt6.py' from '/usr/local/lib/python3.9/site-packages/PyInstaller/hooks'...
7564 INFO: Loading module hook 'hook-multiprocessing.util.py' from '/usr/local/lib/python3.9/site-packages/PyInstaller/hooks'...
7565 INFO: Loading module hook 'hook-sysconfig.py' from '/usr/local/lib/python3.9/site-packages/PyInstaller/hooks'...
7574 INFO: Loading module hook 'hook-xml.py' from '/usr/local/lib/python3.9/site-packages/PyInstaller/hooks'...
7677 INFO: Loading module hook 'hook-distutils.py' from '/usr/local/lib/python3.9/site-packages/PyInstaller/hooks'...
7694 INFO: Looking for ctypes DLLs
7712 INFO: Analyzing run-time hooks ...
7715 INFO: Including run-time hook '/usr/local/lib/python3.9/site-packages/PyInstaller/hooks/rthooks/pyi_rth_subprocess.py'
7719 INFO: Including run-time hook '/usr/local/lib/python3.9/site-packages/PyInstaller/hooks/rthooks/pyi_rth_pkgutil.py'
7722 INFO: Including run-time hook '/usr/local/lib/python3.9/site-packages/PyInstaller/hooks/rthooks/pyi_rth_multiprocessing.py'
7726 INFO: Including run-time hook '/usr/local/lib/python3.9/site-packages/PyInstaller/hooks/rthooks/pyi_rth_inspect.py'
7727 INFO: Including run-time hook '/usr/local/lib/python3.9/site-packages/PyInstaller/hooks/rthooks/pyi_rth_pyqt6.py'
7736 INFO: Looking for dynamic libraries
7977 INFO: Looking for eggs
7977 INFO: Using Python library /usr/local/Cellar/python@3.9/3.9.9/Frameworks/Python.framework/Versions/3.9/Python
7987 INFO: Warnings written to /Users/martin/app/pyside6/build/app/warn-app.txt
8019 INFO: Graph cross-reference written to /Users/martin/app/pyside6/build/app/xref-app.html
8032 INFO: checking PYZ
8035 INFO: Building because toc changed
8035 INFO: Building PYZ (ZlibArchive) /Users/martin/app/pyside6/build/app/PYZ-00.pyz
8390 INFO: Building PYZ (ZlibArchive) /Users/martin/app/pyside6/build/app/PYZ-00.pyz completed successfully.
8397 INFO: EXE target arch: x86_64
8397 INFO: Code signing identity: None
8398 INFO: checking PKG
8398 INFO: Building because /Users/martin/app/pyside6/build/app/PYZ-00.pyz changed
8398 INFO: Building PKG (CArchive) app.pkg
8415 INFO: Building PKG (CArchive) app.pkg completed successfully.
8417 INFO: Bootloader /usr/local/lib/python3.9/site-packages/PyInstaller/bootloader/Darwin-64bit/runw
8417 INFO: checking EXE
8418 INFO: Building because console changed
8418 INFO: Building EXE from EXE-00.toc
8418 INFO: Copying bootloader EXE to /Users/martin/app/pyside6/build/app/app
8421 INFO: Converting EXE to target arch (x86_64)
8449 INFO: Removing signature(s) from EXE
8484 INFO: Appending PKG archive to EXE
8486 INFO: Fixing EXE headers for code signing
8496 INFO: Rewriting the executable's macOS SDK version (11.1.0) to match the SDK version of the Python library (10.15.6) in order to avoid inconsistent behavior and potential UI issues in the frozen application.
8499 INFO: Re-signing the EXE
8547 INFO: Building EXE from EXE-00.toc completed successfully.
8549 INFO: checking COLLECT
WARNING: The output directory "/Users/martin/app/pyside6/dist/app" and ALL ITS CONTENTS will be REMOVED! Continue? (y/N)y
On your own risk, you can use the option `--noconfirm` to get rid of this question.
10820 INFO: Removing dir /Users/martin/app/pyside6/dist/app
10847 INFO: Building COLLECT COLLECT-00.toc
12460 INFO: Building COLLECT COLLECT-00.toc completed successfully.
12469 INFO: checking BUNDLE
12469 INFO: Building BUNDLE because BUNDLE-00.toc is non existent
12469 INFO: Building BUNDLE BUNDLE-00.toc
13848 INFO: Moving BUNDLE data files to Resource directory
13901 INFO: Signing the BUNDLE...
16049 INFO: Building BUNDLE BUNDLE-00.toc completed successfully.

If you look in your folder you'll notice you now have two new folders dist and build.

build & dist folders created by PyInstaller build & dist folders created by PyInstaller

Below is a truncated listing of the folder content, showing the build and dist folders.

bash
.
&boxvr&boxh&boxh app.py
&boxvr&boxh&boxh app.spec
&boxvr&boxh&boxh build
&boxv   &boxur&boxh&boxh app
&boxv       &boxvr&boxh&boxh Analysis-00.toc
&boxv       &boxvr&boxh&boxh COLLECT-00.toc
&boxv       &boxvr&boxh&boxh EXE-00.toc
&boxv       &boxvr&boxh&boxh PKG-00.pkg
&boxv       &boxvr&boxh&boxh PKG-00.toc
&boxv       &boxvr&boxh&boxh PYZ-00.pyz
&boxv       &boxvr&boxh&boxh PYZ-00.toc
&boxv       &boxvr&boxh&boxh app
&boxv       &boxvr&boxh&boxh app.pkg
&boxv       &boxvr&boxh&boxh base_library.zip
&boxv       &boxvr&boxh&boxh warn-app.txt
&boxv       &boxur&boxh&boxh xref-app.html
&boxur&boxh&boxh dist
    &boxvr&boxh&boxh app
    &boxv   &boxvr&boxh&boxh libcrypto.1.1.dylib
    &boxv   &boxvr&boxh&boxh PySide6
    &boxv   ...
    &boxv   &boxvr&boxh&boxh app
    &boxv   &boxur&boxh&boxh Qt5Core
    &boxur&boxh&boxh app.app

The build folder is used by PyInstaller to collect and prepare the files for bundling, it contains the results of analysis and some additional logs. For the most part, you can ignore the contents of this folder, unless you're trying to debug issues.

The dist (for "distribution") folder contains the files to be distributed. This includes your application, bundled as an executable file, together with any associated libraries (for example PySide6) and binary .so files.

Since we provided the --windowed flag above, PyInstaller has actually created two builds for us. The folder app is a simple folder containing everything you need to be able to run your app. PyInstaller also creates an app bundle app.app which is what you will usually distribute to users.

The app folder is a useful debugging tool, since you can easily see the libraries and other packaged data files.

You can try running your app yourself now, either by double-clicking on the app bundle, or by running the executable file, named app.exe from the dist folder. In either case, after a short delay you'll see the familiar window of your application pop up as shown below.

Simple app, running after being packaged Simple app, running after being packaged

In the same folder as your Python file, alongside the build and dist folders PyInstaller will have also created a .spec file. In the next section we'll take a look at this file, what it is and what it does.

The Spec file

The .spec file contains the build configuration and instructions that PyInstaller uses to package up your application. Every PyInstaller project has a .spec file, which is generated based on the command line options you pass when running pyinstaller.

When we ran pyinstaller with our script, we didn't pass in anything other than the name of our Python application file and the --windowed flag. This means our spec file currently contains only the default configuration. If you open it, you'll see something similar to what we have below.

python
# -*- mode: python ; coding: utf-8 -*-


block_cipher = None


a = Analysis(['app.py'],
             pathex=[],
             binaries=[],
             datas=[],
             hiddenimports=[],
             hookspath=[],
             hooksconfig={},
             runtime_hooks=[],
             excludes=[],
             win_no_prefer_redirects=False,
             win_private_assemblies=False,
             cipher=block_cipher,
             noarchive=False)
pyz = PYZ(a.pure, a.zipped_data,
             cipher=block_cipher)

exe = EXE(pyz,
          a.scripts,
          [],
          exclude_binaries=True,
          name='app',
          debug=False,
          bootloader_ignore_signals=False,
          strip=False,
          upx=True,
          console=False,
          disable_windowed_traceback=False,
          target_arch=None,
          codesign_identity=None,
          entitlements_file=None )
coll = COLLECT(exe,
               a.binaries,
               a.zipfiles,
               a.datas,
               strip=False,
               upx=True,
               upx_exclude=[],
               name='app')
app = BUNDLE(coll,
             name='app.app',
             icon=None,
             bundle_identifier=None)


The first thing to notice is that this is a Python file, meaning you can edit it and use Python code to calculate values for the settings. This is mostly useful for complex builds, for example when you are targeting different platforms and want to conditionally define additional libraries or dependencies to bundle.

Because we used the --windowed command line flag, the EXE(console=) attribute is set to False. If this is True a console window will be shown when your app is launched -- not what you usually want for a GUI application.

Once a .spec file has been generated, you can pass this to pyinstaller instead of your script to repeat the previous build process. Run this now to rebuild your executable.

bash
pyinstaller app.spec

The resulting build will be identical to the build used to generate the .spec file (assuming you have made no changes). For many PyInstaller configuration changes you have the option of passing command-line arguments, or modifying your existing .spec file. Which you choose is up to you.

Tweaking the build

So far we've created a simple first build of a very basic application. Now we'll look at a few of the most useful options that PyInstaller provides to tweak our build. Then we'll go on to look at building more complex applications.

Naming your app

One of the simplest changes you can make is to provide a proper "name" for your application. By default the app takes the name of your source file (minus the extension), for example main or app. This isn't usually what you want.

You can provide a nicer name for PyInstaller to use for the app (and dist folder) by editing the .spec file to add a name= under the EXE, COLLECT and BUNDLE blocks.

python
exe = EXE(pyz,
          a.scripts,
          [],
          exclude_binaries=True,
          name='Hello World',
          debug=False,
          bootloader_ignore_signals=False,
          strip=False,
          upx=True,
          console=False
         )
coll = COLLECT(exe,
               a.binaries,
               a.zipfiles,
               a.datas,
               strip=False,
               upx=True,
               upx_exclude=[],
               name='Hello World')
app = BUNDLE(coll,
             name='Hello World.app',
             icon=None,
             bundle_identifier=None)

The name under EXE is the name of the executable file, the name under BUNDLE is the name of the app bundle.

Alternatively, you can re-run the pyinstaller command and pass the -n or --name configuration flag along with your app.py script.

bash
pyinstaller -n "Hello World" --windowed app.py
# or
pyinstaller --name "Hello World" --windowed app.py

The resulting app file will be given the name Hello World.app and the unpacked build placed in the folder dist\Hello World\.

Application with custom name "Hello World" Application with custom name "Hello World"

The name of the .spec file is taken from the name passed in on the command line, so this will also create a new spec file for you, called Hello World.spec in your root folder.

Make sure you delete the old app.spec file to avoid getting confused editing the wrong one.

Application icon

By default PyInstaller app bundles come with the following icon in place.

Default PyInstaller application icon, on app bundle Default PyInstaller application icon, on app bundle

You will probably want to customize this to make your application more recognisable. This can be done easily by passing the --icon command line argument, or editing the icon= parameter of the BUNDLE section of your .spec file. For macOS app bundles you need to provide an .icns file.

python
app = BUNDLE(coll,
             name='Hello World.app',
             icon='Hello World.icns',
             bundle_identifier=None)

To create macOS icons from images you can use the image2icon tool.

If you now re-run the build (by using the command line arguments, or running with your modified .spec file) you'll see the specified icon file is now set on your application bundle.

Custom application icon (a hand) on the app bundle Custom application icon on the app bundle

On macOS application icons are taken from the application bundle. If you repackage your app and run the bundle you will see your app icon on the dock!

Custom application icon in the dock Custom application icon on the dock

Data files and Resources

So far our application consists of just a single Python file, with no dependencies. Most real-world applications a bit more complex, and typically ship with associated data files such as icons or UI design files. In this section we'll look at how we can accomplish this with PyInstaller, starting with a single file and then bundling complete folders of resources.

First let's update our app with some more buttons and add icons to each.

python
from PySide6.QtWidgets import QMainWindow, QApplication, QLabel, QVBoxLayout, QPushButton, QWidget
from PySide6.QtGui import QIcon

import sys

class MainWindow(QMainWindow):

    def __init__(self):
        super().__init__()

        self.setWindowTitle("Hello World")
        layout = QVBoxLayout()
        label = QLabel("My simple app.")
        label.setMargin(10)
        layout.addWidget(label)

        button1 = QPushButton("Hide")
        button1.setIcon(QIcon("icons/hand.png"))
        button1.pressed.connect(self.lower)
        layout.addWidget(button1)

        button2 = QPushButton("Close")
        button2.setIcon(QIcon("icons/lightning.png"))
        button2.pressed.connect(self.close)
        layout.addWidget(button2)

        container = QWidget()
        container.setLayout(layout)

        self.setCentralWidget(container)

        self.show()

if __name__ == '__main__':
    app = QApplication(sys.argv)
    w = MainWindow()
    app.exec_()

In the folder with this script, add a folder icons which contains two icons in PNG format, hand.png and lightning.png. You can create these yourself, or get them from the source code download for this tutorial.

Run the script now and you will see a window showing two buttons with icons.

Window with two icons Window with two buttons with icons.

Even if you don't see the icons, keep reading!

Dealing with relative paths

There is a gotcha here, which might not be immediately apparent. To demonstrate it, open up a shell and change to the folder where our script is located. Run it with

bash
python3 app.py

If the icons are in the correct location, you should see them. Now change to the parent folder, and try and run your script again (change <folder> to the name of the folder your script is in).

bash
cd ..
python3 <folder>/app.py

Window with two icons missing Window with two buttons with icons missing.

The icons don't appear. What's happening?

We're using relative paths to refer to our data files. These paths are relative to the current working directory -- not the folder your script is in. So if you run the script from elsewhere it won't be able to find the files.

One common reason for icons not to show up, is running examples in an IDE which uses the project root as the current working directory.

This is a minor issue before the app is packaged, but once it's installed it will be started with it's current working directory as the root / folder -- your app won't be able to find anything. We need to fix this before we go any further, which we can do by making our paths relative to our application folder.

In the updated code below, we define a new variable basedir, using os.path.dirname to get the containing folder of __file__ which holds the full path of the current Python file. We then use this to build the relative paths for icons using os.path.join().

Since our app.py file is in the root of our folder, all other paths are relative to that.

python
from PySide6.QtWidgets import QMainWindow, QApplication, QLabel, QVBoxLayout, QPushButton, QWidget
from PySide6.QtGui import QIcon

import sys, os

basedir = os.path.dirname(__file__)

class MainWindow(QMainWindow):

    def __init__(self):
        super().__init__()

        self.setWindowTitle("Hello World")
        layout = QVBoxLayout()
        label = QLabel("My simple app.")
        label.setMargin(10)
        layout.addWidget(label)

        button1 = QPushButton("Hide")
        button1.setIcon(QIcon(os.path.join(basedir, "icons", "hand.png")))
        button1.pressed.connect(self.lower)
        layout.addWidget(button1)

        button2 = QPushButton("Close")
        button2.setIcon(QIcon(os.path.join(basedir, "icons", "lightning.png")))
        button2.pressed.connect(self.close)
        layout.addWidget(button2)

        container = QWidget()
        container.setLayout(layout)

        self.setCentralWidget(container)

        self.show()

if __name__ == '__main__':
    app = QApplication(sys.argv)
    w = MainWindow()
    app.exec_()

Try and run your app again from the parent folder -- you'll find that the icons now appear as expected on the buttons, no matter where you launch the app from.

Packaging the icons

So now we have our application showing icons, and they work wherever the application is launched from. Package the application again with pyinstaller "Hello World.spec" and then try and run it again from the dist folder as before. You'll notice the icons are missing again.

Window with two icons missing Window with two buttons with icons missing.

The problem now is that the icons haven't been copied to the dist/Hello World folder -- take a look in it. Our script expects the icons to be a specific location relative to it, and if they are not, then nothing will be shown.

This same principle applies to any other data files you package with your application, including Qt Designer UI files, settings files or source data. You need to ensure that relative path structures are replicated after packaging.

Bundling data files with PyInstaller

For the application to continue working after packaging, the files it depends on need to be in the same relative locations.

To get data files into the dist folder we can instruct PyInstaller to copy them over. PyInstaller accepts a list of individual paths to copy, together with a folder path relative to the dist/<app name> folder where it should to copy them to. As with other options, this can be specified by command line arguments or in the .spec file.

Files specified on the command line are added using --add-data, passing the source file and destination folder separated by a colon :.

The path separator is platform-specific: Linux or Mac use :, on Windows use ;

bash
pyinstaller --windowed --name="Hello World" --icon="Hello World.icns" --add-data="icons/hand.png:icons" --add-data="icons/lightning.png:icons" app.py

Here we've specified the destination location as icons. The path is relative to the root of our application's folder in dist -- so dist/Hello World with our current app. The path icons means a folder named icons under this location, so dist/Hello World/icons. Putting our icons right where our application expects to find them!

You can also specify data files via the datas list in the Analysis section of the spec file, shown below.

python
a = Analysis(['app.py'],
             pathex=[],
             binaries=[],
             datas=[('icons/hand.png', 'icons'), ('icons/lightning.png', 'icons')],
             hiddenimports=[],
             hookspath=[],
             runtime_hooks=[],
             excludes=[],
             win_no_prefer_redirects=False,
             win_private_assemblies=False,
             cipher=block_cipher,
             noarchive=False)

Then rebuild from the .spec file with

bash
pyinstaller "Hello World.spec"

In both cases we are telling PyInstaller to copy the specified files to the location ./icons/ in the output folder, meaning dist/Hello World/icons. If you run the build, you should see your .png files are now in the in dist output folder, under a folder named icons.

The icon file copied to the dist folder The icon file copied to the dist folder

If you run your app from dist you should now see the icon icons in your window as expected!

Window with two icons Window with two buttons with icons, finally!

Bundling data folders

Usually you will have more than one data file you want to include with your packaged file. The latest PyInstaller versions let you bundle folders just like you would files, keeping the sub-folder structure.

Let's update our configuration to bundle our icons folder in one go, so it will continue to work even if we add more icons in future.

To copy the icons folder across to our build application, we just need to add the folder to our .spec file Analysis block. As for the single file, we add it as a tuple with the source path (from our project folder) and the destination folder under the resulting folder in dist.

python
# ...
a = Analysis(['app.py'],
             pathex=[],
             binaries=[],
             datas=[('icons', 'icons')],   # tuple is (source_folder, destination_folder)
             hiddenimports=[],
             hookspath=[],
             hooksconfig={},
             runtime_hooks=[],
             excludes=[],
             win_no_prefer_redirects=False,
             win_private_assemblies=False,
             cipher=block_cipher,
             noarchive=False)
# ...

If you run the build using this spec file you'll see the icons folder copied across to the dist\Hello World folder. If you run the application from the folder, the icons will display as expected -- the relative paths remain correct in the new location.

Alternatively, you can bundle your data files using Qt's QResource architecture. See our tutorial for more information.

Building the App bundle into a Disk Image

So far we've used PyInstaller to bundle the application into macOS app, along with the associated data files. The output of this bundling process is a folder and an macOS app bundle, named Hello World.app.

If you try and distribute this app bundle, you'll notice a problem: the app bundle is actually just a special folder. While macOS displays it as an application, if you try and share it, you'll actually be sharing hundreds of individual files. To distribute the app properly, we need some way to package it into a single file.

The easiest way to do this is to use a .zip file. You can zip the folder and give this to someone else to unzip on their own computer, giving them a complete app bundle they can copy to their Applications folder.

However, if you've install macOS applications before you'll know this isn't the usual way to do it. Usually you get a Disk Image .dmg file, which when opened shows the application bundle, and a link to your Applications folder. To install the app, you just drag it across to the target.

To make our app look as professional as possible, we should copy this expected behaviour. Next we'll look at how to take our app bundle and package it into a macOS Disk Image.

Making sure the build is ready.

If you've followed the tutorial so far, you'll already have your app ready in the /dist folder. If not, or yours isn't working you can also download the source code files for this tutorial which includes a sample .spec file. As above, you can run the same build using the provided Hello World.spec file.

bash
pyinstaller "Hello World.spec"

This packages everything up as an app bundle in the dist/ folder, with a custom icon. Run the app bundle to ensure everything is bundled correctly, and you should see the same window as before with the icons visible.

Two icons Window with two icons, and a button.

Creating an Disk Image

Now we've successfully bundled our application, we'll next look at how we can take our app bundle and use it to create a macOS Disk Image for distribution.

To create our Disk Image we'll be using the create-dmg tool. This is a command-line tool which provides a simple way to build disk images automatically. If you are using Homebrew, you can install create-dmg with the following command.

bash
brew install create-dmg

...otherwise, see the Github repository for instructions.

The create-dmg tool takes a lot of options, but below are the most useful.

bash
create-dmg --help
create-dmg 1.0.9

Creates a fancy DMG file.

Usage:  create-dmg [options] <output_name.dmg> <source_folder>

All contents of <source_folder> will be copied into the disk image.

Options:
  --volname <name>
      set volume name (displayed in the Finder sidebar and window title)
  --volicon <icon.icns>
      set volume icon
  --background <pic.png>
      set folder background image (provide png, gif, or jpg)
  --window-pos <x> <y>
      set position the folder window
  --window-size <width> <height>
      set size of the folder window
  --text-size <text_size>
      set window text size (10-16)
  --icon-size <icon_size>
      set window icons size (up to 128)
  --icon file_name <x> <y>
      set position of the file's icon
  --hide-extension <file_name>
      hide the extension of file
  --app-drop-link <x> <y>
      make a drop link to Applications, at location x,y
  --no-internet-enable
      disable automatic mount & copy
  --add-file <target_name> <file>|<folder> <x> <y>
      add additional file or folder (can be used multiple times)
  -h, --help
        display this help screen

The most important thing to notice is that the command requires a <source folder> and all contents of that folder will be copied to the Disk Image. So to build the image, we first need to put our app bundle in a folder by itself.

Rather than do this manually each time you want to build a Disk Image I recommend creating a shell script. This ensures the build is reproducible, and makes it easier to configure.

Below is a working script to create a Disk Image from our app. It creates a temporary folder dist/dmg where we'll put the things we want to go in the Disk Image -- in our case, this is just the app bundle, but you can add other files if you like. Then we make sure the folder is empty (in case it still contains files from a previous run). We copy our app bundle into the folder, and finally check to see if there is already a .dmg file in dist and if so, remove it too. Then we're ready to run the create-dmg tool.

bash
#!/bin/sh
# Create a folder (named dmg) to prepare our DMG in (if it doesn't already exist).
mkdir -p dist/dmg
# Empty the dmg folder.
rm -r dist/dmg/*
# Copy the app bundle to the dmg folder.
cp -r "dist/Hello World.app" dist/dmg
# If the DMG already exists, delete it.
test -f "dist/Hello World.dmg" && rm "dist/Hello World.dmg"
create-dmg \
  --volname "Hello World" \
  --volicon "Hello World.icns" \
  --window-pos 200 120 \
  --window-size 600 300 \
  --icon-size 100 \
  --icon "Hello World.app" 175 120 \
  --hide-extension "Hello World.app" \
  --app-drop-link 425 120 \
  "dist/Hello World.dmg" \
  "dist/dmg/"

The options we pass to create-dmg set the dimensions of the Disk Image window when it is opened, and positions of the icons in it.

Save this shell script in the root of your project, named e.g. builddmg.sh. To make it possible to run, you need to set the execute bit with.

bash
chmod +x builddmg.sh

With that, you can now build a Disk Image for your Hello World app with the command.

bash
./builddmg.sh

This will take a few seconds to run, producing quite a bit of output.

bash
 No such file or directory
Creating disk image...
...............................................................
created: /Users/martin/app/dist/rw.Hello World.dmg
Mounting disk image...
Mount directory: /Volumes/Hello World
Device name:     /dev/disk2
Making link to Applications dir...
/Volumes/Hello World
Copying volume icon file 'Hello World.icns'...
Running AppleScript to make Finder stuff pretty: /usr/bin/osascript "/var/folders/yf/1qvxtg4d0vz6h2y4czd69tf40000gn/T/createdmg.tmp.XXXXXXXXXX.RvPoqdr0" "Hello World"
waited 1 seconds for .DS_STORE to be created.
Done running the AppleScript...
Fixing permissions...
Done fixing permissions
Blessing started
Blessing finished
Deleting .fseventsd
Unmounting disk image...
hdiutil: couldn't unmount "disk2" - Resource busy
Wait a moment...
Unmounting disk image...
"disk2" ejected.
Compressing disk image...
Preparing imaging engine…
Reading Protective Master Boot Record (MBR : 0)…
   (CRC32 $38FC6E30: Protective Master Boot Record (MBR : 0))
Reading GPT Header (Primary GPT Header : 1)…
   (CRC32 $59C36109: GPT Header (Primary GPT Header : 1))
Reading GPT Partition Data (Primary GPT Table : 2)…
   (CRC32 $528491DC: GPT Partition Data (Primary GPT Table : 2))
Reading  (Apple_Free : 3)…
   (CRC32 $00000000:  (Apple_Free : 3))
Reading disk image (Apple_HFS : 4)…
...............................................................................
   (CRC32 $FCDC1017: disk image (Apple_HFS : 4))
Reading  (Apple_Free : 5)…
...............................................................................
   (CRC32 $00000000:  (Apple_Free : 5))
Reading GPT Partition Data (Backup GPT Table : 6)…
...............................................................................
   (CRC32 $528491DC: GPT Partition Data (Backup GPT Table : 6))
Reading GPT Header (Backup GPT Header : 7)…
...............................................................................
   (CRC32 $56306308: GPT Header (Backup GPT Header : 7))
Adding resources…
...............................................................................
Elapsed Time:  3.443s
File size: 23178950 bytes, Checksum: CRC32 $141F3DDC
Sectors processed: 184400, 131460 compressed
Speed: 18.6Mbytes/sec
Savings: 75.4%
created: /Users/martin/app/dist/Hello World.dmg
hdiutil does not support internet-enable. Note it was removed in macOS 10.15.
Disk image done

While it's building, the Disk Image will pop up. Don't get too excited yet, it's still building. Wait for the script to complete, and you will find the finished .dmg file in the dist/ folder.

The Disk Image in the dist folder The Disk Image created in the dist folder

Running the installer

Double-click the Disk Image to open it, and you'll see the usual macOS install view. Click and drag your app across the the Applications folder to install it.

The Disk Image containing your file The Disk Image contains the app bundle and a shortcut to the applications folder

If you open the Showcase view (press F4) you will see your app installed. If you have a lot of apps, you can search for it by typing "Hello"

The app is installed! The app installed on macOS

Repeating the build

Now you have everything set up, you can create a new app bundle & Disk Image of your application any time, by running the two commands from the command line.

bash
pyinstaller "Hello World.spec"
./builddmg.sh

It's that simple!

Wrapping up

In this tutorial we've covered how to build your PySide6 applications into a macOS app bundle using PyInstaller, including adding data files along with your code. Then we walked through the process of creating a Disk Image to distribute your app to others. Following these steps you should be able to package up your own applications and make them available to other people.

For a complete view of all PyInstaller bundling options take a look at the PyInstaller usage documentation.

For an in-depth guide to building GUIs with Python see my PySide6 book.

Training at Qt Developer Conference 2022

KDAB is delighted to offer, with our partners The Qt Company and Software Compliance Academy, 9 superb one day training classes to kick off our QtDevCon event in Berlin this summer. Each class is led by an expert in the field with content that is slap up-to-date with the latest changes and derived from comprehensive longer courses.

As well as a wide range of more obviously Qt-focused courses, subjects offered range from the oft-overlooked but vital FOSS Compliance that every developer should be aware of, to the nitty gritty on What’s New in C++20 – not for the faint-hearted.

Whether you’re a relative newcomer to your chosen subject, or just need to catch up with the latest tips and tricks, these courses are terrific value and not-to-be-missed*.

Training ticket sales close on May 23rd so Sign up and choose your training NOW while seats last.

Here’s a brief rundown of what’s on offer (click on the titles to get more info):

From KDAB

QML Application Architecture

André Somers will show where to put the boundaries between what you do in C++ and QML and demonstrate some of the tools available in QML that can help you achieve well-performing, well-structured, and well-maintainable QML applications.

Custom components and rendering deep-dive

James Turner will show how to create and design custom components, especially how to get data on-screen. He’ll talk about portability, performance, and custom event handling. He’ll also cover the RHI layer in Qt6 and how to work on different target platforms supporting Metal, Vulkan and D3D.

Introduction to CMake

Kevin Funk will explain how CMake’s broad functionality helps solve real world problems and advanced build requirements. These include cross-platform builds, feature detection, built-time configurable feature switches, custom build steps, code generators, third-party package retrieval and more.

What’s new in C++20?

Giuseppe D’Angelo will explain to attendees with solid experience with Modern C++ (11/14/17 (a prerequisite): Concepts, Comparison (3-way), Coroutines, the char8_t mess, immediate functions, lambda improvements, modules, multi-threading improvements, ranges, span, <bit> and more.

Profiling & Debugging for Linux

Milian Wolff, the main author of Massif-Visualizer, heaptrack, hotspot and ctf2ctf – tools now used widely to improve C++ applications performance, will introduce profiling with Linux perf, measuring memory consumption with heaptrack, and debugging with RR and the sanitizers.

Introduction to QML

Jesper K. Pedersen of youtube infamy and Shantanu Tushar will show UI developers interested in starting with QML and learning the basics, how to develop an application in QML and connect it to C++ back end code.

From Software Compliance Academy

FOSS Compliance

Dr Catharina Maracke will make clear why you should care about Intellectual Property and licensing, even if you choose ‘Free’ and Open software. She’ll cover the FOSS Licensing Model, warranties and liability, end-to-end compliance management and offer some examples and a reality check.

From The Qt Company and froglogic

Programming for Microcontrollers with Qt for MCUs (STMicroelectronics)

David Sliwa and Piotr Mucko will introduce the Qt Quick Ultralite rendering engine, running on bare metal or FreeRTOS, with Ultralite Controls and an optimized subset of QML, enabling you to create smartphone-like user experiences on microcontrollers with the Qt Creator IDE and other tools.

Qt GUI Automated Testing with Squish

Tomasz Pawlowski will show how to design and implement cross-platform automated tests using Squish GUI Tester for Qt, QML & QtQuick applications that continue to work as your product evolves, allowing you to achieve high quality applications during the testing process.

Choose your training and Sign up!

Qt Developer Conference, Berlin, June 13 – 15, with Training Day on June 13th.

Our sponsors

Main Sponsor:


Sponsor:


Contributors:

* This one day training offer is for attendees of Qt Developer Conference only.

The post Training at Qt Developer Conference 2022 appeared first on KDAB.

How to integrate Qml and C++? Expose object and register C++ class to Qml

This tutorial about how to expose object and register C++ class to Qml is the first post in the new post series on Scythe Studio blog. The series is going to be entitled „How to integrate Qml and C++”. With the new Qt 6 version it’s not that clear how to properly and easily implement this crucial mechanism in Qt software development. Especially as many people currently move to CMake from qmake. Therefore, we thought that it would be a nice opportunity to explain various ways of Qml and C++ integration in details.
Future posts will cover mechanisms like models and plugins, but for now let’s focus on the basics and let’s explain how to access C++ object from Qml and how to register C++ class to Qml.

JSONify All Things

The nlohmann/json library is everything a developer can expect from a modern library — easy to integrate and JSON objects are treated as first class citizens with a very intuitive API.

However, it has one problem that is widely mentioned across the internet, which I’ll tell you about below. Various solutions to the problem have been developed and shared, but none seem to be easy-to-use.

In this blog post, we will see how one can serialize and deserialize almost anything by extending the library a bit.

The Problem

I particularly like how you can easily define serialization/deserialization for your own type:

struct SimpleStruct {
    int id;
    std::string text;
};
NLOHMANN_DEFINE_TYPE_NON_INTRUSIVE(SimpleStruct, id, text);

But what about for a more complex data structure, using a std::optional or std::variant, like this:

struct ComplexStruct {
    std::variant<std::string, int> id;
    std::optional<std::string> text;
    std::optional<std::variant<std::vector<int>, bool>> values;
};

Unfortunately, it’s not supported out-of-the-box by nlohmann/json; see some of the issues here:

I’ll tell you what I want, what I really, really want: being able to write:

NLOHMANN_JSONIFY_ALL_THINGS(ComplexStruct, id, text, values)

Variants

nlohmann/json gives you everything you need to write down your own serialization code; see the documentation for arbitrary type conversions. In our case, this means writing an adl_serializer for std::variant.

namespace nlohmann {
template <typename... Ts>
struct adl_serializer<std::variant<Ts...>> {
    static void to_json(nlohmann::json &j, const std::variant<Ts...> &data) { /*TODO*/ }
    static void from_json(const nlohmann::json &j, std::variant<Ts...> &data) { /*TODO*/ }
};
}

We just need to fill the TODO. The issue I linked earlier gives one solution. Unfortunately, it requires storing the index of the type close to the value in the JSON file. This is only possible if you control the whole chain. If you need to integrate in an existing protocol, it may not be possible.

The solution below will focus on the case where you have only one value for the variant and no indication of the type.

Serialization

To serialize the to_json method, we just want to automatically set into j the type that is in the variant. Fortunetaly, std::visit comes to the rescue and it ends up being a one liner:

static void to_json(nlohmann::json &j, const std::variant &data) {
    // Will call j = v automatically for the right type
    std::visit([&j](const auto &v) { j = v; }, data);
}

Deserialization

To deserialize is a bit more complex as we don’t know the exact type. So, we need to try them all to find the right one. If you can use C++17, this can be done quickly with fold expressions:

// Try to set the value of type T into the variant data if it fails, do nothing
template <typename T, typename... Ts>
void variant_from_json(const nlohmann::json &j, std::variant<Ts...> &data) {
    try { 
        data = j.get<T>();
    } catch (...) {
    }
}   
static void from_json(const nlohmann::json &j, std::variant<Ts...> &data) {
    // Call variant_from_json for all types, only one will succeed
    (variant_from_json<Ts>(j, data), ...);
}

The fold expression (line 11) allows us to handle variadic templates without the need of writing recursive code. The ... means that it’s going to repeat what is on the left of the comma for all types.

Be careful; this solution has some issues:

  • The types need to be “exclusive,” meaning you shouldn’t be able to convert one from another. For example, std::variant<int, long long> won’t work as expected.
  • During deserialization, if your type variant has n types, you will raise (and catch) n-1 exceptions.

Optionals

Optionals are more complex, because we can’t have a to_json/from_json at the level of the property, as the property may not exist at all. So we need to go one level up. There are actually some detailed explanations on the issue I linked before, thanks to all the people who’ve shared their solutions.

First Step

The first thing is to write down code to serialize/deserialize an optional. This code will be called later on in the parent json value to_json/from_json:

template <class T>
void optional_to_json(nlohmann::json &j, const char *name, const std::optional<T> &value) {
    if (value)
        j[name] = *value;
}
template <class T>
void optional_from_json(const nlohmann::json &j, const char *name, std::optional<T> &value) {
    const auto it = j.find(name);
    if (it != j.end())
        value = it->get<T>();
    else
        value = std::nullopt;
}

But we still need to write down, explicitly, the code to serialize/deserialize the structure using the optionals. At this point, we have only done half of the work.

Second Step

To go further, we need to look at how the macro is implemented. Looking at the code, this is what is done for all properties you pass in the macro NLOHMANN_DEFINE_TYPE_NON_INTRUSIVE:

#define NLOHMANN_JSON_TO(v1) nlohmann_json_j[#v1] = nlohmann_json_t.v1;
#define NLOHMANN_JSON_FROM(v1) nlohmann_json_j.at(#v1).get_to(nlohmann_json_t.v1);

In the case of a std::optional, we want to call our own optional_[from|to]_json rather than the default. Again, with C++17 to the rescue, using if constexpr we can write:

template <typename>
constexpr bool is_optional = false;
template <typename T>
constexpr bool is_optional<std::optional<T>> = true;
 
template <typename T>
void extended_to_json(const char *key, nlohmann::json &j, const T &value) {
    if constexpr (is_optional<T>)
        optional_to_json(j, key, value);
    else
        j[key] = value;
}
template <typename T>
void extended_from_json(const char *key, const nlohmann::json &j, T &value) {
    if constexpr (is_optional<T>)
        optional_from_json(j, key, value);
    else
        j.at(key).get_to(value);
}

We then use our extended version to create our own macro, still copying what his done in the nlohmann/json library:

#define EXTEND_JSON_TO(v1) extended_to_json(#v1, nlohmann_json_j, nlohmann_json_t.v1);
#define EXTEND_JSON_FROM(v1) extended_from_json(#v1, nlohmann_json_j, nlohmann_json_t.v1);
 
#define NLOHMANN_JSONIFY_ALL_THINGS(Type, ...)                                          \
  inline void to_json(nlohmann::json &nlohmann_json_j, const Type &nlohmann_json_t) {   \ 
      NLOHMANN_JSON_EXPAND(NLOHMANN_JSON_PASTE(EXTEND_JSON_TO, __VA_ARGS__))            \
  }                                                                                     \
  inline void from_json(const nlohmann::json &nlohmann_json_j, Type &nlohmann_json_t) { \
      NLOHMANN_JSON_EXPAND(NLOHMANN_JSON_PASTE(EXTEND_JSON_FROM, __VA_ARGS__))          \
  }

Conclusion

That’s it! With this code, we are now able to write down:

struct ComplexStruct {
    std::variant<std::string, int> id;
    std::optional<std::string> text;
    std::optional<std::variant<std::vector<int>, bool>> values; 
};
NLOHMANN_JSONIFY_ALL_THINGS(ComplexStruct, id, text, values)

and it just works out of the box !

Now why is it not by default in the library? In my opinion, there are multiple reasons for that:

  • the current implementation for std::variant has a hard pre-requisite that all types must be exclusive;
  • the current implementation for std::variant is not the most performant one, as it requires multiple exceptions;
  • it would be better to store the index of the type’s variant in the json, if possible;
  • the current implementation for std::variant expects that an empty optional does not exist, while for some it may be on a null value in JSON.

Overall, the problem is complex and I don’t think one solution will fit them all, meaning the chances for inclusion in the library are probably very low.

A big thanks to Niels Lohmann for creating this amazing piece of the library! I love it and am using it when I can.

About KDAB

If you like this article and want to read similar material, consider subscribing via our RSS feed.

Subscribe to KDAB TV for similar informative short video content.

KDAB provides market leading software consulting and development services and training in Qt, C++ and 3D/OpenGL. Contact us.

Annex: full code all together

A big thank to Andrew for finding an issue in the code in the blog (now fixed). You’ll find the whole code all together below, if you want to just copy/paste it.

#include <nlohmann/json.hpp>

#include <optional>
#include <variant>

namespace nlohmann {

///////////////////////////////////////////////////////////////////////////////
// std::variant
///////////////////////////////////////////////////////////////////////////////
// Try to set the value of type T into the variant data if it fails, do nothing
template <typename T, typename... Ts>
void variant_from_json(const nlohmann::json &j, std::variant<Ts...> &data) {
    try {
        data = j.get<T>();
    } catch (...) {
    }
}

template <typename... Ts>
struct adl_serializer<std::variant<Ts...>>
{
    static void to_json(nlohmann::json &j, const std::variant<Ts...> &data) {
        // Will call j = v automatically for the right type
        std::visit([&j](const auto &v) { j = v; }, data);
    }

    static void from_json(const nlohmann::json &j, std::variant<Ts...> &data) {
        // Call variant_from_json for all types, only one will succeed
        (variant_from_json<Ts>(j, data), ...);
    }
};
///////////////////////////////////////////////////////////////////////////////
// std::optional
///////////////////////////////////////////////////////////////////////////////
template <class T>
void optional_to_json(nlohmann::json &j, const char *name, const std::optional<T> &value) {
    if (value)
        j[name] = *value;
}
template <class T>
void optional_from_json(const nlohmann::json &j, const char *name, std::optional<T> &value) {
    const auto it = j.find(name);
    if (it != j.end())
        value = it->get<T>();
    else
        value = std::nullopt;
}

///////////////////////////////////////////////////////////////////////////////
// all together
///////////////////////////////////////////////////////////////////////////////
template <typename>
constexpr bool is_optional = false;
template <typename T>
constexpr bool is_optional<std::optional<T>> = true;

template <typename T>
void extended_to_json(const char *key, nlohmann::json &j, const T &value) {
    if constexpr (is_optional<T>)
        nlohmann::optional_to_json(j, key, value);
    else
        j[key] = value;
}
template <typename T>
void extended_from_json(const char *key, const nlohmann::json &j, T &value) {
    if constexpr (is_optional<T>)
        nlohmann::optional_from_json(j, key, value);
    else
        j.at(key).get_to(value);
}

}

#define EXTEND_JSON_TO(v1) extended_to_json(#v1, nlohmann_json_j, nlohmann_json_t.v1);
#define EXTEND_JSON_FROM(v1) extended_from_json(#v1, nlohmann_json_j, nlohmann_json_t.v1);

#define NLOHMANN_JSONIFY_ALL_THINGS(Type, ...)                                          \
  inline void to_json(nlohmann::json &nlohmann_json_j, const Type &nlohmann_json_t) {   \ 
      NLOHMANN_JSON_EXPAND(NLOHMANN_JSON_PASTE(EXTEND_JSON_TO, __VA_ARGS__))            \
  }                                                                                     \
  inline void from_json(const nlohmann::json &nlohmann_json_j, Type &nlohmann_json_t) { \
      NLOHMANN_JSON_EXPAND(NLOHMANN_JSON_PASTE(EXTEND_JSON_FROM, __VA_ARGS__))          \
  }

The post JSONify All Things appeared first on KDAB.

Packaging PyQt5 applications into a Linux package with PyInstaller (updated for 2022)

In the previous tutorials, we've looked at packaging your PyQt5 applications for Windows and macOS -- turning them into EXE Installers and macOS bundles respectively. But to make your application truly cross-platform you should also provide installers for Linux. In this tutorial we'll look at how to do just that, first using PyInstaller to bundle our application into a executable app and then using a tool called fpm to convert that into a Linux package.

This tutorial is broken down into a series of steps, using PyInstaller to build first simple, and then more complex PyQt5 applications into Linux executables. You can choose to follow it through completely, or skip to the parts that are most relevant to your own project.

We finish off by building an Ubuntu .deb package, the usual method for distributing application on that systems. Thanks to the magic of fpm the instructions will also work for other Linux distributions, such as Redhat .rpm or Arch .pacman.

You always need to compile your app on your target system. So, if you want to create an Ubuntu package do this on Ubuntu.

Example Ubuntu Package Example Ubuntu Package

If you're impatient, you can download the Example Ubuntu Package first.

Requirements

PyInstaller works out of the box with PyQt5 and as of writing, current versions of PyInstaller are compatible with Python 3.6+. Whatever project you're working on, you should be able to package your apps. This tutorial assumes you have a working installation of Python with pip package management working.

You can install PyInstaller using pip.

bash
pip3 install PyInstaller

If you experience problems packaging your apps, your first step should always be to update your PyInstaller and hooks package the latest versions using

bash
pip3 install --upgrade PyInstaller pyinstaller-hooks-contrib

The hooks module contains package-specific packaging instructions for PyInstaller which is updated regularly.

Install in virtual environment (optional)

You can also opt to install PyQt5 and PyInstaller in a virtual environment (or your applications virtual environment) to keep your environment clean.

bash
python3 -m venv packenv

Once created, activate the virtual environment by running from the command line —

bash
call packenv\scripts\activate.bat

Finally, install the required libraries. For PyQt5 you would use —

python
pip3 install PyQt5 PyInstaller

Getting Started

It's a good idea to start packaging your application from the very beginning so you can confirm that packaging is still working as you develop it. This is particularly important if you add additional dependencies. If you only think about packaging at the end, it can be difficult to debug exactly where the problems are.

For this example we're going to start with a simple skeleton app, which doesn't do anything interesting. Once we've got the basic packaging process working, we'll extend the application to include icons and data files. We'll confirm the build as we go along.

To start with, create a new folder for your application and then add the following skeleton app in a file named app.py. You can also download the source code and associated files

python
from PyQt5 import QtWidgets

import sys

class MainWindow(QtWidgets.QMainWindow):

    def __init__(self):
        super().__init__()

        self.setWindowTitle("Hello World")
        l = QtWidgets.QLabel("My simple app.")
        l.setMargin(10)
        self.setCentralWidget(l)
        self.show()

if __name__ == '__main__':
    app = QtWidgets.QApplication(sys.argv)
    w = MainWindow()
    app.exec()

This is a basic bare-bones application which creates a custom QMainWindow and adds a simple widget QLabel to it. You can run this app as follows.

bash
python app.py

This should produce the following window (on Ubuntu).

Simple skeleton app in PyQt5 Simple skeleton app in PyQt5

Building a basic app

Now we have our simple application skeleton in place, we can run our first build test to make sure everything is working.

Open your terminal (shell) and navigate to the folder containing your project. You can now run the following command to run the PyInstaller build.

python
pyinstaller app.py

You'll see a number of messages output, giving debug information about what PyInstaller is doing. These are useful for debugging issues in your build, but can otherwise be ignored. The output that I get for running the command on my system is shown below.

bash
$ pyinstaller app.py
85 INFO: PyInstaller: 4.10
85 INFO: Python: 3.9.7
88 INFO: Platform: Linux-5.13.0-39-generic-x86_64-with-glibc2.34
89 INFO: wrote /home/martin/pyinstaller/linux2/no-datas/pyqt5/app.spec
91 INFO: UPX is not available.
91 INFO: Extending PYTHONPATH with paths
['/home/martin/pyinstaller/linux2/no-datas/pyqt5']
236 INFO: checking Analysis
240 INFO: Building because inputs changed
240 INFO: Initializing module dependency graph...
243 INFO: Caching module graph hooks...
255 INFO: Analyzing base_library.zip ...
2008 INFO: Processing pre-find module path hook distutils from '/home/martin/.local/lib/python3.9/site-packages/PyInstaller/hooks/pre_find_module_path/hook-distutils.py'.
2013 INFO: distutils: retargeting to non-venv dir '/usr/lib/python3.9'
4231 INFO: Caching module dependency graph...
4348 INFO: running Analysis Analysis-00.toc
4379 INFO: Analyzing /home/martin/pyinstaller/linux2/no-datas/pyqt5/app.py
4403 INFO: Processing module hooks...
4403 INFO: Loading module hook 'hook-PyQt5.py' from '/home/martin/.local/lib/python3.9/site-packages/PyInstaller/hooks'...
4559 WARNING: Hidden import "sip" not found!
4559 INFO: Loading module hook 'hook-xml.etree.cElementTree.py' from '/home/martin/.local/lib/python3.9/site-packages/PyInstaller/hooks'...
4560 INFO: Loading module hook 'hook-heapq.py' from '/home/martin/.local/lib/python3.9/site-packages/PyInstaller/hooks'...
4562 INFO: Loading module hook 'hook-distutils.py' from '/home/martin/.local/lib/python3.9/site-packages/PyInstaller/hooks'...
4568 INFO: Loading module hook 'hook-xml.py' from '/home/martin/.local/lib/python3.9/site-packages/PyInstaller/hooks'...
4627 INFO: Loading module hook 'hook-PyQt5.QtWidgets.py' from '/home/martin/.local/lib/python3.9/site-packages/PyInstaller/hooks'...
4709 INFO: Loading module hook 'hook-difflib.py' from '/home/martin/.local/lib/python3.9/site-packages/PyInstaller/hooks'...
4711 INFO: Loading module hook 'hook-multiprocessing.util.py' from '/home/martin/.local/lib/python3.9/site-packages/PyInstaller/hooks'...
4712 INFO: Loading module hook 'hook-sysconfig.py' from '/home/martin/.local/lib/python3.9/site-packages/PyInstaller/hooks'...
4713 INFO: Loading module hook 'hook-encodings.py' from '/home/martin/.local/lib/python3.9/site-packages/PyInstaller/hooks'...
4759 INFO: Loading module hook 'hook-PyQt5.QtGui.py' from '/home/martin/.local/lib/python3.9/site-packages/PyInstaller/hooks'...
4807 INFO: Loading module hook 'hook-lib2to3.py' from '/home/martin/.local/lib/python3.9/site-packages/PyInstaller/hooks'...
4820 INFO: Loading module hook 'hook-pickle.py' from '/home/martin/.local/lib/python3.9/site-packages/PyInstaller/hooks'...
4827 INFO: Loading module hook 'hook-PyQt5.QtCore.py' from '/home/martin/.local/lib/python3.9/site-packages/PyInstaller/hooks'...
4853 INFO: Loading module hook 'hook-distutils.util.py' from '/home/martin/.local/lib/python3.9/site-packages/PyInstaller/hooks'...
4862 INFO: Looking for ctypes DLLs
4897 INFO: Analyzing run-time hooks ...
4900 INFO: Including run-time hook '/home/martin/.local/lib/python3.9/site-packages/PyInstaller/hooks/rthooks/pyi_rth_subprocess.py'
4903 INFO: Including run-time hook '/home/martin/.local/lib/python3.9/site-packages/PyInstaller/hooks/rthooks/pyi_rth_pkgutil.py'
4905 INFO: Including run-time hook '/home/martin/.local/lib/python3.9/site-packages/PyInstaller/hooks/rthooks/pyi_rth_multiprocessing.py'
4910 INFO: Including run-time hook '/home/martin/.local/lib/python3.9/site-packages/PyInstaller/hooks/rthooks/pyi_rth_inspect.py'
4912 INFO: Including run-time hook '/home/martin/.local/lib/python3.9/site-packages/PyInstaller/hooks/rthooks/pyi_rth_pyqt5.py'
4916 INFO: Looking for dynamic libraries
6561 INFO: Looking for eggs
6561 INFO: Python library not in binary dependencies. Doing additional searching...
6596 INFO: Using Python library /lib/x86_64-linux-gnu/libpython3.9.so.1.0
6604 INFO: Warnings written to /home/martin/pyinstaller/linux2/no-datas/pyqt5/build/app/warn-app.txt
6625 INFO: Graph cross-reference written to /home/martin/pyinstaller/linux2/no-datas/pyqt5/build/app/xref-app.html
6643 INFO: checking PYZ
6645 INFO: Building because name changed
6645 INFO: Building PYZ (ZlibArchive) /home/martin/pyinstaller/linux2/no-datas/pyqt5/build/app/PYZ-00.pyz
6923 INFO: Building PYZ (ZlibArchive) /home/martin/pyinstaller/linux2/no-datas/pyqt5/build/app/PYZ-00.pyz completed successfully.
6926 INFO: checking PKG
6926 INFO: Building because name changed
6927 INFO: Building PKG (CArchive) app.pkg
6959 INFO: Building PKG (CArchive) app.pkg completed successfully.
6962 INFO: Bootloader /home/martin/.local/lib/python3.9/site-packages/PyInstaller/bootloader/Linux-64bit-intel/run
6963 INFO: checking EXE
6963 INFO: Building because name changed
6964 INFO: Building EXE from EXE-00.toc
6969 INFO: Copying bootloader EXE to /home/martin/pyinstaller/linux2/no-datas/pyqt5/build/app/app
6970 INFO: Appending PKG archive to custom ELF section in EXE
6979 INFO: Building EXE from EXE-00.toc completed successfully.
6981 INFO: checking COLLECT
6982 INFO: Building COLLECT COLLECT-00.toc
8674 INFO: Building COLLECT COLLECT-00.toc completed successfully.

If you look in your folder you'll notice you now have two new folders dist and build.

build & dist folders created by PyInstaller build & dist folders created by PyInstaller

Below is a truncated listing of the folder content, showing the build and dist folders.

bash
.
&boxvr&boxh&boxh app.py
&boxvr&boxh&boxh app.spec
&boxvr&boxh&boxh build
&boxv   &boxur&boxh&boxh app
&boxv       &boxvr&boxh&boxh localpycos
&boxv       &boxvr&boxh&boxh Analysis-00.toc
&boxv       &boxvr&boxh&boxh COLLECT-00.toc
&boxv       &boxvr&boxh&boxh EXE-00.toc
&boxv       &boxvr&boxh&boxh PKG-00.pkg
&boxv       &boxvr&boxh&boxh PKG-00.toc
&boxv       &boxvr&boxh&boxh PYZ-00.pyz
&boxv       &boxvr&boxh&boxh PYZ-00.toc
&boxv       &boxvr&boxh&boxh app
&boxv       &boxvr&boxh&boxh app.pkg
&boxv       &boxvr&boxh&boxh base_library.zip
&boxv       &boxvr&boxh&boxh warn-app.txt
&boxv       &boxur&boxh&boxh xref-app.html
&boxur&boxh&boxh dist
    &boxvr&boxh&boxh app
    &boxv   &boxvr&boxh&boxh lib-dynload
    &boxv   &boxvr&boxh&boxh PyQt5
    &boxv   ...
    &boxv   &boxvr&boxh&boxh app
    &boxv   &boxur&boxh&boxh libQt5Core.so.5
    &boxur&boxh&boxh app.app

The build folder is used by PyInstaller to collect and prepare the files for bundling, it contains the results of analysis and some additional logs. For the most part, you can ignore the contents of this folder, unless you're trying to debug issues.

The dist (for "distribution") folder contains the files to be distributed. This includes your application, bundled as an executable file, together with any associated libraries (for example PyQt5) and binary .so files.

Everything necessary to run your application will be in this folder, meaning you can take this folder and "distribute" it to someone else to run your app.

You can try running your app yourself now, by running the executable file, named app from the dist folder. After a short delay you'll see the familiar window of your application pop up as shown below.

Simple app, running after being packaged Simple app, running after being packaged

In the same folder as your Python file, alongside the build and dist folders PyInstaller will have also created a .spec file. In the next section we'll take a look at this file, what it is and what it does.

The Spec file

The .spec file contains the build configuration and instructions that PyInstaller uses to package up your application. Every PyInstaller project has a .spec file, which is generated based on the command line options you pass when running pyinstaller.

When we ran pyinstaller with our script, we didn't pass in anything other than the name of our Python application file. This means our spec file currently contains only the default configuration. If you open it, you'll see something similar to what we have below.

python
# -*- mode: python ; coding: utf-8 -*-

block_cipher = None

a = Analysis(['app.py'],
             pathex=[],
             binaries=[],
             datas=[],
             hiddenimports=[],
             hookspath=[],
             hooksconfig={},
             runtime_hooks=[],
             excludes=[],
             win_no_prefer_redirects=False,
             win_private_assemblies=False,
             cipher=block_cipher,
             noarchive=False)

pyz = PYZ(a.pure, a.zipped_data,
             cipher=block_cipher)

exe = EXE(pyz,
          a.scripts,
          [],
          exclude_binaries=True,
          name='app',
          debug=False,
          bootloader_ignore_signals=False,
          strip=False,
          upx=True,
          console=True,
          disable_windowed_traceback=False,
          target_arch=None,
          codesign_identity=None,
          entitlements_file=None )

coll = COLLECT(exe,
               a.binaries,
               a.zipfiles,
               a.datas,
               strip=False,
               upx=True,
               upx_exclude=[],
               name='app')

The first thing to notice is that this is a Python file, meaning you can edit it and use Python code to calculate values for the settings. This is mostly useful for complex builds, for example when you are targeting different platforms and want to conditionally define additional libraries or dependencies to bundle.

Once a .spec file has been generated, you can pass this to pyinstaller instead of your script to repeat the previous build process. Run this now to rebuild your executable.

bash
pyinstaller app.spec

The resulting build will be identical to the build used to generate the .spec file (assuming you have made no changes). For many PyInstaller configuration changes you have the option of passing command-line arguments, or modifying your existing .spec file. Which you choose is up to you.

Tweaking the build

So far we've created a simple first build of a very basic application. Now we'll look at a few things we can do to tweak our build.

Naming your app

One of the simplest changes you can make is to provide a proper "name" for your application. By default the app takes the name of your source file (minus the extension), for example main or app. This isn't usually what you want.

You can provide a nicer name for PyInstaller to use for your executable file (and dist folder) by editing the .spec file to add a name= under the EXE and COLLECT blocks. On Linux you will want to use a name with no spaces (use hyphens instead).

python
exe = EXE(pyz,
          a.scripts,
          [],
          exclude_binaries=True,
          name='hello-world',
          debug=False,
          bootloader_ignore_signals=False,
          strip=False,
          upx=True,
          console=False
         )
coll = COLLECT(exe,
               a.binaries,
               a.zipfiles,
               a.datas,
               strip=False,
               upx=True,
               upx_exclude=[],
               name='hello-world')

The name under EXE is the name of the executable file, the name under COLLECT is the name of the output folder. Usually you would want these to be the same.

Alternatively, you can re-run the pyinstaller command and pass the -n or --name configuration flag along with your app.py script.

bash
pyinstaller -n "hello-world" app.py
# or
pyinstaller --name "hello-world" app.py

The resulting executable file will be given the name hello-world and the unpacked build placed in the folder dist\hello-world\. The name of the .spec file is taken from the name passed in on the command line, so this will also create a new spec file for you, called hello-world.spec in your root folder.

If you've created a new .spec delete the old one to avoid getting confused!

Application with custom name "hello-world" Application with custom name "hello-world"

Application icon

One simple improvement we can make is to change the application icon which is shown while the application is running. We can set this icon in the code directly. To show an icon on our window we need to modify our simple application a little bit, to add a call to .setWindowIcon().

python
from PyQt5 import QtWidgets, QtGui
import sys

class MainWindow(QtWidgets.QMainWindow):

    def __init__(self):
        super().__init__()

        self.setWindowTitle("Hello World")
        l = QtWidgets.QLabel("My simple app.")
        l.setMargin(10)
        self.setCentralWidget(l)

        self.show()

if __name__ == '__main__':
    app = QtWidgets.QApplication(sys.argv)
    app.setWindowIcon(QtGui.QIcon('penguin.svg'))
    w = MainWindow()
    app.exec()

Here we've added the .setWindowIcon call to the app instance. This defines a default icon to be used for all windows of our application. You can override this on a per-window basis if you like, by calling .setWindowIcon on the window itself.

If you run the above application you should now see the icon appears on the dock.

Window showing the custom penguin icon Window showing the custom penguin icon

You can use a PNG file instead of SVG, but if using PNG make sure that the icon is large enough not to appear blurry due to scaling.

Even if you don't see the icon, keep reading!

Dealing with relative paths

There is a gotcha here, which might not be immediately apparent. To demonstrate it, open up a shell and change to the folder where our script is located. Run it with

bash
python3 app.py

If the icons are in the correct location, you should see them. Now change to the parent folder, and try and run your script again (change <folder> to the name of the folder your script is in).

bash
cd ..
python3 <folder>/app.py

Window with icon missing Window with icon missing.

The icons don't appear. What's happening?

We're using relative paths to refer to our data files. These paths are relative to the current working directory -- not the folder your script is in. So if you run the script from elsewhere it won't be able to find the files.

One common reason for icons not to show up, is running examples in an IDE which uses the project root as the current working directory.

This is a minor issue before the app is packaged, but once it's installed you don't know what the current working directory will be when it is run -- if it's wrong your app won't be able to find anything. We need to fix this before we go any further, which we can do by making our paths relative to our application folder.

In the updated code below, we define a new variable basedir, using os.path.dirname to get the containing folder of __file__ which holds the full path of the current Python file. We then use this to build the relative paths for icons using os.path.join().

Since our app.py file is in the root of our folder, all other paths are relative to that.

python
from PyQt5 import QtWidgets, QtGui
import sys, os

basedir = os.path.dirname(__file__)


class MainWindow(QtWidgets.QMainWindow):

    def __init__(self):
        super().__init__()

        self.setWindowTitle("Hello World")
        l = QtWidgets.QLabel("My simple app.")
        l.setMargin(10)
        self.setCentralWidget(l)
        self.show()

if __name__ == '__main__':
    app = QtWidgets.QApplication(sys.argv)
    app.setWindowIcon(QtGui.QIcon(os.path.join(basedir, 'penguin.svg')))
    w = MainWindow()
    app.exec_()

Try and run your app again from the parent folder -- you'll find that the icon now appear as expected, no matter where you launch the app from.

With this added to your script, running it should now show the icon on your window and taskbar. The final step is to ensure that this icon is correctly packaged with your application and continues to be shown when run from the dist folder.

Try it, it wont.

The issue is that our application now has a dependency on a external data file (the icon file) that's not part of our source. For our application to work, we now need to distribute this data file along with it. PyInstaller can do this for us, but we need to tell it what we want to include, and where to put it in the output.

In the next section we'll look at the options available to you for managing data files associated with your app.

Data files and Resources

So far we successfully built a simple app which had no external dependencies. However, once we needed to load an external file (in this case an icon) we hit upon a problem. The file wasn't copied into our dist folder and so could not be loaded.

In this section we'll look at the options we have to be able to bundle external resources, such as icons or Qt Designer .ui files, with our applications.

Bundling data files with PyInstaller

The simplest way to get these data files into the dist folder is to just tell PyInstaller to copy them over. PyInstaller accepts a list of individual file paths to copy over, together with a folder path relative to the dist/<app name> folder where it should to copy them to.

As with other options, this can be specified by command line arguments, --add-data

bash
pyinstaller --add-data "penguin.svg:." --name "hello-world" app.py

You can provide `--add-data` multiple times. Note that the path separator is platform-specific, on Linux or Mac use `:` while on Windows use `;`

Or via the datas list in the Analysis section of the spec file, shown below.

python
a = Analysis(['app.py'],
             pathex=[],
             binaries=[],
             datas=[('penguin.svg', '.')],
             hiddenimports=[],
             hookspath=[],
             runtime_hooks=[],
             excludes=[],
             win_no_prefer_redirects=False,
             win_private_assemblies=False,
             cipher=block_cipher,
             noarchive=False)

And then execute the .spec file with

bash
pyinstaller hello-world.spec

In both cases we are telling PyInstaller to copy the specified file penguin.svg to the location . which means the output folder dist. We could specify other locations here if we wanted. On the command line the source and destination are separated by the path separator :, whereas in the .spec file, the values are provided as a 2-tuple of strings.

If you run the build, you should see your .svg file now in the output folder dist ready to be distributed with your application.

The icon file copied to the dist folder The icon file copied to the dist folder

If you run your app from dist you should now see the icon on the window, and on the taskbar as expected.

The penguin icon showing on the dock The penguin icon showing on the dock

The file must be loaded in Qt using a relative path, and be in the same relative location to the EXE as it was to the .py file for this to work.

Bundling data folders

Usually you will have more than one data file you want to include with your packaged file. The latest PyInstaller versions let you bundle folders just like you would files, keeping the sub-folder structure. For example, lets extend our app to add some additional icons, and put them under a folder.

python
from PyQt5.QtWidgets import QMainWindow, QApplication, QLabel, QVBoxLayout, QPushButton, QWidget
from PyQt5.QtGui import QIcon
import sys, os

basedir = os.path.dirname(__file__)


class MainWindow(QMainWindow):

    def __init__(self):
        super().__init__()

        self.setWindowTitle("Hello World")
        layout = QVBoxLayout()
        label = QLabel("My simple app.")
        label.setMargin(10)
        layout.addWidget(label)

        button = QPushButton("Push")
        button.setIcon(QIcon(os.path.join(basedir, "icons", "lightning.svg")))
        button.pressed.connect(self.close)
        layout.addWidget(button)

        container = QWidget()
        container.setLayout(layout)

        self.setCentralWidget(container)

        self.show()

if __name__ == '__main__':
    app = QApplication(sys.argv)
    app.setWindowIcon(QIcon(os.path.join(basedir, "icons", "penguin.svg")))
    w = MainWindow()
    app.exec_()

The icons (both SVG files) are stored under a subfolder named 'icons'.

bash
.
&boxvr&boxh&boxh app.py
&boxur&boxh&boxh icons
    &boxur&boxh&boxh lightning.svg
    &boxur&boxh&boxh penguin.svg

If you run this you'll see the following window, with an icon on the button and an icon in the dock.

Two icons Window with two icons, and a button.

The paths are using the Unix forward-slash / convention, so they are cross-platform for macOS. If you're only developing for Windows, you can use \\

To copy the icons folder across to our build application, we just need to add the folder to our .spec file Analysis block. As for the single file, we add it as a tuple with the source path (from our project folder) and the destination folder under the resulting dist folder.

python
# -*- mode: python ; coding: utf-8 -*-


block_cipher = None


a = Analysis(['app.py'],
             pathex=[],
             binaries=[],
             datas=[('icons', 'icons')],   # tuple is (source_folder, destination_folder)
             hiddenimports=[],
             hookspath=[],
             hooksconfig={},
             runtime_hooks=[],
             excludes=[],
             win_no_prefer_redirects=False,
             win_private_assemblies=False,
             cipher=block_cipher,
             noarchive=False)
pyz = PYZ(a.pure, a.zipped_data,
             cipher=block_cipher)

exe = EXE(pyz,
          a.scripts,
          [],
          exclude_binaries=True,
          name='hello-world',
          debug=False,
          bootloader_ignore_signals=False,
          strip=False,
          upx=True,
          console=False,
          disable_windowed_traceback=False,
          target_arch=None,
          codesign_identity=None,
          entitlements_file=None )
coll = COLLECT(exe,
               a.binaries,
               a.zipfiles,
               a.datas,
               strip=False,
               upx=True,
               upx_exclude=[],
               name='hello-world')


If you run the build using this spec file you'll see the icons folder copied across to the dist folder. If you run the application from the folder, the icons will display as expected -- the relative paths remain correct in the new location.

Alternatively, you can bundle your data files using Qt's QResource architecture. See our tutorial for more information.

Creating a Linux Package (Ubuntu deb)

So far we've used PyInstaller to bundle the application into a Linux executable, along with the associated data files. The output of this bundling process is a folder. However, in order to share this application with other people and allow them to install it, we need to create a Linux package. Packages are distributable files which allow users to install software on their Linux system, as well as setting up things like application entries in the dock/menu.

On Ubuntu (and Debian) packages are named .deb files, on Redhat .rpm and on Arch Linux .pacman. These files are all different formats, but thankfully the process for building them is the same: using a tool named fpm.

In this tutorial we'll work through the steps for creating a Linux package, using an Ubuntu .deb file as an example. However, you will be able to use the same steps for your own system.

Installing fpm

The fpm tool is written in ruby and requires ruby to be installed to use it. Install ruby using your systems package manager, for example.

bash
$ sudo apt-get install ruby

Once ruby is installed, you can install fpm using the gem tool.

bash
$ gem install fpm --user-install

If you see a warning e.g. You don't have /home/martin/.local/share/gem/ruby/2.7.0/bin in your PATH you will need to add that to your path in your .bashrc file.

...and that's it. Once the installation is complete, you're ready to use fpm. You can check it is installed and working by running:

bash
$ fpm --version
1.14.2

Checking your build

In a terminal, change to the folder containing your application source files & run a PyInstaller build to generate the dist folder. Test that the generated build runs as expected (it works, and icons appear) by opening the dist folder in the file manager, and double-clicking on the application executable.

If everything works, you're ready to package the application -- if not, go back and double check everything.

It's always a good idea to test your built application before packaging it. Then if anything goes wrong, you know where the problem is!

Now let's package our folder using fpm.

Structuring your package

Linux files are used to install all sorts of applications, including system tools. Because of this they are set up to allow you to place files anywhere in the Linux filesystem -- and there are specific correct places to put different files. For a bundled package like ours, we can -- thankfully -- put our executable and associated data files all under the same folder (in /opt). However, to have our application show up in the menus/search we'll also need to install a .desktop file under /usr/share/applications.

The simplest way to ensure things end up in the correct location is to recreate the target file structure in a folder & then tell fpm to package using that folder as the root. This process is also easily automatable using a script (see later).

In your projects root folder, create a new folder called package and subfolders which map to the target filesystem -- /opt will hold our application folder hello-world, and /usr/share/applications will hold our .desktop file., while /usr/share/icons... will hold our application icon.

bash
$ mkdir -p package/opt
$ mkdir -p package/usr/share/applications
$ mkdir -p package/usr/share/icons/hicolor/scalable/apps

Next copy (recursively, with -r to include subfolders) the contents of dist/app to package/opt/hello-world -- the /opt/hello-world path is the destination of our application folder after installation.

bash
$ cp -r dist/hello-world package/opt/hello-world

We're copying the dist/hello-world folder. The name of this folder will depend on the application name configured in PyInstaller.

The icons

We've already set an icon for our application while it's running, using the penguin.svg file. However, we want our application to show it's icon in the dock/menus. To do this correctly, we need to copy our application icons into a specific location, under /usr/share/icons.

This folder contains all the icon themes installed on the system, but default icons for applications are always placed in the fallback hicolor theme, at /usr/share/icons/hicolor. Inside this folder, there are various folders for different sizes of icons.

bash
$ ls /usr/share/icons/hicolor/
128x128/          256x256/          64x64/            scalable/
16x16/            32x32/            72x72/            symbolic/
192x192/          36x36/            96x96/
22x22/            48x48/            icon-theme.cache
24x24/            512x512/          index.theme

We're using the scalable folder, since our icon is an SVG (Scalable Vector Graphics). If you're using a specifically sized PNG file, place it in the correct location -- and feel free to add multiple different sizes, to ensure your application icon looks good when scaled. Application icons go in the subfolder apps.

bash
$ cp icons/penguin.svg package/usr/share/icons/hicolor/scalable/apps/hello-world.svg

Name the destination filename of the icon after your application to avoid it clashing with any others! Here we're calling it hello-world.svg.

The .desktop file

The .desktop file is a text configuration file which tells the Linux desktop about a desktop application -- for example, where to fine the executable, the name and which icon to display. You should include a .desktop file for your apps to make them easy to use. An example .desktop file is shown below -- add this to the root folder of your project -- with the name hello-world.desktop, and make any changes you like.

ini
[Desktop Entry]

# The type of the thing this desktop file refers to (e.g. can be Link)
Type=Application

# The application name.
Name=Hello World

# Tooltip comment to show in menus.
Comment=A simple Hello World application.

# The path (folder) in which the executable is run
Path=/opt/hello-world

# The executable (can include arguments)
Exec=/opt/hello-world/hello-world

# The icon for the entry, using the name from `hicolor/scalable` without the extension.
# You can also use a full path to a file in /opt.
Icon=hello-world


For more information on creating .desktop files see this documentation.

Now the hello-world.desktop file is ready, we can copy it into our install package with.

bash
$ cp hello-world.desktop package/usr/share/applications

Permissions

Packages retain the permissions of installed files from when they were packaged, but will be installed by root. In order for ordinary users to be able to run the application, you need to change the permissions of the files created.

We can recursively apply the correct permissions 755 - owner can read/write/execute, group/others can read/execute. and 644, owner can read/write, group/others can red to the contents of our executable folder and icons/desktop files.

bash
$ find package/opt/hello-world -type f -exec chmod 755 -- {} +
$ find package/usr/share -type f -exec chmod 644 -- {} +

Building your package

Now everything is where it should be in our package "filesystem", we're ready to start building the package itself.

Enter the following into your shell.

bash
fpm -C package -s dir -t deb -n "hello-world" -v 0.1.0 -p hello-world.deb

The arguments in order are:

  • -C the folder to change to before searching for files: our package folder
  • -s the type of source(s) to package: in our case dir, a folder
  • -t the type of package to build: a deb Debian/Ubuntu package
  • -n the name of the application: "hello-world"
  • -v the version of the application: 0.1.0
  • -p the package name to output: hello-world-deb

For more command line arguments, see the fpm documentation.

You can create other package types (for other Linux distributions) by changing the -t argument.

After a few seconds, you should see a message to indicate that the package has been created.

bash
$ fpm -C package -s dir -t deb -n "hello-world" -v 0.1.0 -p hello-world.deb
Created package {:path=>"hello-world.deb"}

Installation

The package is ready! Let's install it.

bash
$ sudo dpkg -i hello-world.deb

You'll see some output as the install completes.

python
Selecting previously unselected package hello-world.
(Reading database ... 172208 files and directories currently installed.)
Preparing to unpack hello-world.deb ...
Unpacking hello-world (0.1.0) ...
Setting up hello-world (0.1.0) ...

Once installation has completed, you can check the files are where you expect, under /opt/hello-world

bash
$ ls /opt/hello-world
app                        libpcre2-8.so.0
base_library.zip           libpcre.so.3
icons                      libpixman-1.so.0
libatk-1.0.so.0            libpng16.so.16
libatk-bridge-2.0.so.0     libpython3.9.so.1.0
etc.

Next try and run the application from the menu/dock -- you can search for "Hello World" and the application will be found (thanks to the .desktop file).

Hello world in Ubuntu search Application shows up in the Ubuntu search panel, and will also appear in menus on other environments.

If you run the application, the icons will show up as expected.

Application, running in the dock Application runs and all icons show up as expected.

Scripting the build

We've walked through the steps required to build an installable Ubuntu .deb package from a PyQt5 application. There isn't that much too it, but if you have to do it more than once it'll quickly get quite tedious and prone to mistakes. To avoid problems I recommend scripting this with a simple bash script & fpm own automation tool.

In this section I'll give you scripts that automate the build we've done for our Hello World application.

package.sh

Save in your project root and chmod +x to make it executable.

sh
#!/bin/sh
# Create folders.
[ -e package ] && rm -r package
mkdir -p package/opt
mkdir -p package/usr/share/applications
mkdir -p package/usr/share/icons/hicolor/scalable/apps

# Copy files (change icon names, add lines for non-scaled icons)
cp -r dist/hello-world package/opt/hello-world
cp icons/penguin.svg package/usr/share/icons/hicolor/scalable/apps/hello-world.svg
cp hello-world.desktop package/usr/share/applications

# Change permissions
find package/opt/hello-world -type f -exec chmod 755 -- {} +
find package/usr/share -type f -exec chmod 644 -- {} +


.fpm file

fpm allows you to store the configuration for the packaging in a configuration file. The file name must be .fpm and it must be in the folder you run the fpm tool. Our configuration is as follows.

sh
-C package
-s dir
-t deb
-n "hello-world"
-v 0.1.0
-p hello-world.deb

You can override any of the options you like when executing fpm by passing command line arguments as normal.

Executing the build

With these scripts in place our application can be packaged reproducibly with the commands:

bash
pyinstaller hello-world.spec
./package.sh
fpm

Feel free to customize these build scripts further yourself to suit your own project!

Wrapping up

In this tutorial we've covered how to build your PyQt5 applications into a Linux executable using PyInstaller, including adding data files along with your code. Then we walked through the process of creating a Ubuntu .deb package to distribute your app to others. Following these steps you should be able to package up your own applications and make them available to other people.

For an in-depth guide to building GUIs with Python see my PyQt book.

C++23 Will Be Really Awesome

C++23 is feature complete and on track to be released next year. While many people are complaining that it’s, after all, a “minor” release (as the pandemic made the Committee work very difficult), C++23 still has a few very significant changes.

In this blog post, I want to talk about what I think is my favorite feature of C++23: the really keyword. This is a brand new keyword that can be used in a number of different scenarios. Let’s explore them together to understand the usefulness of it.

For Function Arguments

Consider a function like:

void f(int x);

Unfortunately, we all know too well that C++ allows you to call it with non-integer arguments:

f(1234);  // OK, intended: called with an int
f(3.14);  // Not really intended (called with double), still legal

If you’re lucky, your compiler may give you a warning. Otherwise, the calls with non-int (double, long, …) will be valid, and this may cause unexpected data losses and/or undefined behavior. Could it be possible to have a compile-time error instead?

C++11 and later gave us different strategies:

// C++11:
void f(int x);
template <typename T> void f(T) = delete;

f(42);   // OK
f(3.14); // ERROR

If you don’t want to re-declare your functions twice, you can also use SFINAE in C++11 or, in C++20, concepts:

// C++11/14, SFINAE
template <typename T>
std::enable_if_t<std::is_same_v<T, int>> f(T x);

// C++20: concepts
void f(std::same_as<int> auto x);

While the C++20 version is a huge readability improvement over the SFINAE version, it’s still a mouthful. Also, this makes the function a function template, so now it can’t be out-of-line, it impacts our compilation times, etc.

In C++23 this will be possible by adding really:

void f(really int x);

f(42);   // OK
f(3.14); // ERROR

Simple and effective! You can also apply it to ordinary variables or to return types:

really int i = getInt();    // OK
really int j = getDouble(); // ERROR

As a Deeper const

Consider this code:

class String {
    const char *data;
    int len;

public:
    void a(int) const;
    void b() const;
    void c();
};

void String::c()
{
    a(len * 42);
    a(len * 42);
    b();
}

It may look a bit silly, but stay with me. The disassembly of c() (with optimizations enabled) is:

String::c():
        push    rbp
        imul    esi, DWORD PTR [rdi+8], 42
        mov     rbp, rdi
        call    String::a(int) const
        imul    esi, DWORD PTR [rbp+8], 42
        mov     rdi, rbp
        call    String::a(int) const
        mov     rdi, rbp
        pop     rbp
        jmp     String::b() const

The compiler is doing something quite peculiar here: it’s multiplying len by 42, calling a() with the result of the calculation, then it multiplies len again in order to call a() a second time. How come it’s doing the same (relatively) expensive operation twice, rather than simply storing the result and using it for both calls? a() is a const member function, after all, so it can’t change len -- or can it?

Well, the short answer is that the compiler is not allowed to do otherwise: the value of len could’ve been indeed changed by the call to a(). Maybe a() ends up calling something else, and that something else changes *this. Or maybe a() is just doing a const_cast<String *>(this) and then mutating the result. It doesn’t change the result: the compiler is forced to reload the data members after a member function call as it can’t assume that they haven’t been changed. The fact that the target member function is const does not help the compiler in any way.

However, if we declare a() as really const:

class String {
    const char *data;
    int len;

public:
    void a(int) really const;
    void b() const;
    void c();
};

Then we get this improved codegen:

String::c():
        push    r12
        push    rbp
        mov     rbp, rdi
        sub     rsp, 8
        imul    r12d, DWORD PTR [rdi+8], 42
        mov     esi, r12d
        call    String::a(int) const
        mov     rdi, rbp
        mov     esi, r12d
        call    String::a(int) const
        add     rsp, 8
        mov     rdi, rbp
        pop     rbp
        pop     r12
        jmp     String::b() const

It may look longer, but we have a couple of extra mov in lieu of an imul. The multiplication result is stored and reused for the second call. And that’s a huge win!

What really const does is makes the compiler consider a work on an object declared const. Such objects cannot be mutated, not even by stripping their constness away (that’s undefined behavior). This causes better codegen; now the compiler knows that calling a cannot possibly mutate *this.

Closed Enumerations

Suppose you have an enumeration like this:

enum class MyEnum {
  E1, E2, E3
};

Now, suppose you have a function that takes a value of this enumeration. Correctly, you use a switch over the value, in order to handle all the possible cases:

int calculate(MyEnum e)
{
  switch (e) {
  case MyEnum::E1: return 42;
  case MyEnum::E2: return 51;
  case MyEnum::E3: return 123;
  }
}

Now, for some reason, your compiler will complain about calculate. Specifically written like this, you will get a warning regarding the possibility for control to reach the end of the function without hitting a return statement, and the function does not return void.

How in the world is that possible? There’s clearly a switch in there that is covering all the possible enumerators!

Defeated, you’ll change your function to something like this:

int calculate(MyEnum e)
{
  switch (e) {
  case MyEnum::E1: return 42;
  case MyEnum::E2: return 51;
  case MyEnum::E3: return 123;
  // for the love of kittens, DO NOT add a default: !!!
  }

  assert(false);
  return -1; // impossible
}

(Yes, do NOT add a default: to the switch — a default label will prevent the compiler from warning you that the switch is no longer covering all the enumerators, should you decide some day to extend the enumeration.)

So how is “the impossible” actually possible? Is the compiler wrong?

No, the compiler is right. There is the possibility of passing a value which isn’t handled by the switch; and that’s because C++ allows casting integers to enumerations, as long as the integer “fits” in the enumerator’s range. See the wording in [expr.static.cast]:

That means that this is 100% legal C++:

void bad() {
  MyEnum nonsense = static_cast<MyEnum>(-1);
  int result = calculate(nonsense); // whops!
}

But just how often is this feature really useful? While, in general, allowing conversions from integer to enumerations makes sense, people are not really supposed to cast arbitrary integers into an enumeration’s type.

Enter really enum, or, of course, enum really as it needs to be spelled:

enum class really MyEnum {
  E1, E2, E3
};

int calculate(MyEnum e)
{
  switch (e) {
  case MyEnum::E1: return 42;
  case MyEnum::E2: return 51;
  case MyEnum::E3: return 123;
  }
  // no warning, all cases are handled
}

void bad() {
  MyEnum nonsense = static_cast<MyEnum>(-1); // UB, -1 is not an enumerator
}

Basically, a really enum is just like an ordinary enum (or enum class, like in this case), except that converting integers to it requires the integer to match the value of one of the enumerators — otherwise, undefined behavior. The UB is not really worse than what we had before (walking out of a function without returning) or crashing due to the failed assert. On top if that, thanks to things like ubsan, we can catch the problem (= the illegal conversion) at its source rather than later, when it causes havok.

For Deduced rvalue References

Let’s face it: templates are hard. And deduction rules for function templates, combined with value categories, are even harder!

But let’s start with a simple example. Any proficient C++ developer should know that it’s possible to overload a function f for lvalues and rvalues:

void f(const std::string &s); // 1) lvalue reference
void f(std::string &&s);      // 2) rvalue reference

std::string str = "hello";
f(str);             // calls 1
f("hello"s);        // calls 2
f(std::move(str));  // calls 2

Now suppose you want to make your f generic, so it can work on std::string but also on other string types. That, of course, means turning f into a function template.

Easy enough, right? Well, raise your hand if you ever fell into this trap:

template <typename T> void f(const T &obj); // 1) lvalue reference
template <typename T> void f(T &&obj);      // 2) rvalue reference...?

The double ampersand still means rvalue reference, doesn’t it? Turns out that, well, yes, but actually no. Since T is deduced, it doesn’t; it now means forwarding (or universal) reference. This code:

std::string s = "hello";
f(s);  // call f on a lvalue

is now actually calling overload n. 2 (!!!), after deducing T = std::string &.

This is incredibly annoying and error-prone: the very same syntax, without deduced template arguments, works correctly. Reusing the same syntax with totally different meanings is a major annoying point (just think about the teachability of such a feature…).

In order to force the second overload only to take rvalue references, we have to again deploy SFINAE or similar techniques:

template <typename T> void f(const T &obj); // lvalue reference

template <typename T> 
  std::enable_if_t<!std::is_reference_v<T>>
f(T &&obj);      // T must not be a reference => T&& is a rvalue reference

I mean… really?! (pun intended) I bet that if we had a time machine, someone would surely propose the usage of different syntax (a triple ampersand?) to specify forwarding references and remove the entire problem.

Luckily for us, in C++23, we can do this:

template <typename T> void f(const T really &obj);  // 1) lvalue reference
template <typename T> void f(T really &&obj);       // 2) rvalue reference! not forwarding reference

This brings back the “traditional” rules without any of the verbose SFINAE syntax. Technically speaking, it’s not needed on 1), but I like the symmetry.

Really Operator Auto

A problem that sometimes affects Qt users is the following:

QString getString();
void print(QString);

void f() {
  auto result = QString("The result is: ") + getString();
  print(result);
}

What’s the type of result? You may think it’s a QString — after all, it’s being obtained by concatenating (via operator+) two QString objects.

That’s actually correct in some cases, but in (many) others, it’s not accurate. Generally speaking, result is actually an object of type QStringBuilder.

What’s QStringBuilder, you ask? It’s a private class in Qt that is employed to optimize string concatenations. A concatenation like this:

QString a, b, c, d;
QString result = a + b + c + d;

may cause up to 3 memory allocations, if done naively: first, calculate a + b (allocate); then, append c to the result (allocate again); then, append d to the final result (allocate again).

QStringBuilder instead considers the entire “sequence” of operations. When QStringBuilder is in use, a + b does not return a QString directly. It returns a QStringBuilder that remembers a and b (It simply internally stores references to them). Note that no allocations (for the result) or copies are taking place yet.

Then, appending c to the intermediate QStringBuilder yields another QStringBuilder that now remembers c, too. Finally, d is appended, yielding one final QStringBuilder.

This final object is then converted to a QString. That’s where the magic kicks in: QStringBuilder can now perform one memory allocation of the right size (by asking all the other strings about their sizes, as it’s keeping references to all of them), copy their contents in order to perform the concatenation, and produce result.

So what’s wrong in the first example? The answer is the usage of auto:

auto result = QString("The result is: ") + getString(); // whops!

Here, a QStringBuilder is produced in order to concatenate the two strings. But it’s never converted to a QString. Instead, result is an object of type QStringBuilder itself!

That is a problem because, as mentioned before, QStringBuilder merely references the strings that it is supposed to concatenate. These strings are temporaries that will get destroyed at the end of the statement. That means that we now have a QStringBuilder loaded with dangling references!

Again, with a time machine, the solution could be to allow like this:

class QStringBuilder
{
  // invoked when assigning the result to `auto` variable...?
  operator auto() const { return QString(*this); }
};

Although operator auto() exists, it’s not what the comment above says. operator auto is nothing but an “ordinary” conversion operator, combined with auto deduction for the return type of a function. In other words, the above is just declaring an operator QString() const — but we already have that one inside QStringBuilder!

Instead, in C++23 we can do this:

class QStringBuilder
{
  // invoked when assigning the result to `auto` variable.
  really operator auto() const { return QString(*this); }
};

Note that the position of really is important. Otherwise, it applies to the conversion target type, just like I’ve shown before:

class C
{
   operator really int() const;
};

C c;
int    i = c;    // OK
double d = c; // ERROR

And that’s it! I hope you liked this little presentation about the new keyword. I can’t wait to start working on C++23 projects to try this one out (and also the other goodies, which you can check out on cppreference).

If you want to know more about really, feel free to check out the original proposal here.

See you next time!

About KDAB

If you like this article and want to read similar material, consider subscribing via our RSS feed.

Subscribe to KDAB TV for similar informative short video content.

KDAB provides market leading software consulting and development services and training in Qt, C++ and 3D/OpenGL. Contact us.

The post C++23 Will Be Really Awesome appeared first on KDAB.

Embed a Custom Qt View in Native Android and iOS Applications with Felgo and Qt

The flexibility to integrate Qt and QML with any platform or system is a powerful advantage if you want to move towards cross-platform development. For example, you can extend your Android or iOS applications with a shared implementation for certain new features, or transition to cross-platform by replacing your native code step-by-step. 

With Native App Integration for Android and iOS, Felgo makes it easy to start using Qt in existing mobile applications that you created with Android Studio or Xcode. The Felgo 3.9.2 release of Native App Integration now provides more flexibility on Android to use a custom app Activity and to run Qt in a light-weight Android View instead of a Fragment. It also includes all the latest features and components of Qt 5.15.2 and Felgo 3.9.2.

How to write clean Qml code in 2022?

There is no argue that Qml is an amazing technology to develop an outstanding user interface matching today’s trends. The Qml language has beautiful and easy to learn syntax, but the code doesn’t structure itself that. By its nature, it can easily get messy. Therefore, you need to learn how to write clean Qml code in order to keep your Qt software project easily maintainable. Ergo, to make savings on money and time.

The Qt Less Traveled

Qt is a high quality library, but it is also massive in number of features. This massive number results in a higher likelihood that there are bugs that go unnoticed lying around in its less frequently used features or combination of features.

Our friends at qgis.org hired us to solve some of those issues that had been affecting them for a while. QGIS is an open source geospatial information system (an application which allows users to design maps, analyze spatial data, and perform various ETL tasks) that makes use of the Qt library for its GUI components and relies on the QPainter framework for all of its map rendering and exporting functionality. Instead of trying to workaround these issues in their downstream project, the QGIS leadership instead invested some project funds to contract KDAB to fix these rare issues which were specifically affecting the QGIS application. Keep reading to follow our adventures through those less often traveled paths.

SVG and Length Definitions

SVG, Scalable Vector Graphics, is commonly used to share graphics that need to be drawn at arbitrary resolutions. We fixed two issues in that area regarding the parsing of elements:

SVG has several specifications, 2.0, 1.1, and Tiny 1.2, each supporting different features. The Qt documentation is relatively clear that it supports a subset of features of Tiny 1.2: https://doc.qt.io/qt-5/svgrendering.html. That’s fine when you control the SVG that your application renders; your designers are a bit sad but can usually adapt to it. It gets much more complicated, however, when your application can use SVG files provided by the users. It’s relatively harder to explain the distinctions of the several SVG specifications to a normal user.

You can choose multiple units to represent SVG elements. Take, for example, the x,y properties of a text element. The definition in the specification (https://www.w3.org/TR/SVGTiny12/text.html#TextElement) is a bit long. But, after a few clicks, you end up with something that says:

A length is a distance measurement. The format of a <length> is a <number> optionally
followed by a unit identifier. 
If the <length> is expressed as a value without a unit identifier (e.g., '48'), then the <length>
represents a distance in the current user coordinate system. 
SVG Tiny 1.2 only supports optional units on the 'width' and 'height' attributes on the 'svg' element.
These can specify values in any of the following units: in, cm, mm, pt, pc, px and %.

If we do the same for the SVG 1.1 spec, it doesn’t have that restriction of optional units only being valid for the width and height properties.

Here, we colored a bit outside the lines because the Qt code was fine. It said that it supports SVG Tiny 1.2 and, as such, misrendered files that use unsupported features (like saying x=”15pt”), but given that supporting that feature was less than 10 lines. See https://codereview.qt-project.org/c/qt/qtsvg/+/376065 and https://codereview.qt-project.org/c/qt/qtsvg/+/376066. We applied the Robustness principle and allowed ourselves to be a bit more flexible in what we accept.

QFont Ignored Stretch Value When Used Together with styleName

Now, let’s talk about one of those combinations of two features that are probably not often used together.

You can tell QFont to stretch the font to a certain width:

You can also tell QFont to change the style of a given font, in this example to bold:

However, if you tell it to do both things, Qt forgets to do the stretching. After some digging, the fix ended up being a single line of code: https://codereview.qt-project.org/c/qt/qtbase/+/373727.

QPainterPath Did Not Respect SmallCaps in Some Situations

This bug was 10 years old! Let’s compare the old rendering (first below) and the fixed one (second below)

SmallCaps is that feature that says, “render lowercase characters using (smaller) capital letters.”

The old rendering was half right, the letters positions were correct, but they where not “small”. This means that the letters ended up overlapping each other.

Like the previous fix, this was also a one liner: https://codereview.qt-project.org/c/qt/qtbase/+/373741 (The commit log is much longer than the actual fix). We also added some tests, in hopes of preventing regression: https://codereview.qt-project.org/c/qt/qtbase/+/374874.

QFontComboBox and Broken Fonts

QFontComboBox is a class used to select fonts. It shows the name of the font rendered in the font itself so you can see how it looks. Rendering the name of the font in its font is not always possible since there are some “Symbol” fonts that don’t have all the characters. In such cases, Qt renders the name of the font in your normal font, followed by adding a few characters of the font itself.

For that feature to work, the font has to correctly identify itself as a Symbol font. Sadly, that’s not always the case, as it’s not the case with the D050000L font that comes with most Linux distributions, for example. Here’s how that font name is rendered:

It’s definitely not easy to figure out what all those pencils are.

Our first attempt at fixing that was saying, “Well, the type of font is not being properly detected, so let’s add a function so we can override the detected type.” We suggested a function called QFontDatabase::replaceWritingSystems to override the detected font type and get us this:

After some back and forth with the reviewers, we ended up discarding that in favor of a more generic solution. That consisted of adding two new functions, QFontComboBox::setDisplayFont and QFontComboBox::setSampleText, that allow you to set which font and text will be used to display the preview of the font in question. Thus, we are able to workaround the initial issue while also adding extra functionality — a win for everyone! 🙂

 

Getting Others to Fix Bugs for You

Sometimes when you try to fix a bug, your fix is not good enough. Fortunately, this doesn’t mean that all hope is lost. Your analysis of the situation may help the reviewers come up with a better solution.

This has happened twice in this series of bug fixes; let’s look at them.

Brush Transformations When Printing to a PDF File

Since brush transformations were not supported, I added a flag to the PDF printing engine that says so: https://codereview.qt-project.org/c/qt/qtbase/+/373978. Unfortunately, that flag regressed other features (the flags that mention which features are supported are not super detailed), so it had to be abandoned. But Eirik Aavitsland from The Qt Company came to the rescue and actually implemented support of brush transformations when printing to a PDF file with a one liner (as seen already a few times in this blog): https://codereview.qt-project.org/c/qt/qtbase/+/374484.

QDockArea Sizing When Starting an Application

This one took a while to investigate. At the end, the problem was identified as:

QMainWindow remembers a few things so it can restore its state when the application restarts. Among them are:

  • the size of the window when it was in “normal” state
  • the size of the window when it was in “maximized” state
  • whether the last state was maximized or normal
  • the sizes and positions of the dock widgets

Now when starting the application, something like this happens:

  • Creation of Window
  • Window is shown
  • Window says, “Oh, it’s the first time I’m being shown; let’s restore my previous state.”
  • If it was maximized when the application was closed and upon restoring the previous state, it will say “I was maximized; I want to be maximized again, please.”
  • After that, it will restore the sizes and positions of the dock widgets.

The problem is that “I want to be maximized” is not always a synchronous function (e.g. Linux/X11). Because of this, you sometimes still have a small-ish window when you get to the “restore dock widgets sizes and positions” phase and the dock widget sizes don’t make sense. The window says, “Oh, you silly. I can’t give you 500 pixels of width for this dock widget; I’m only 400 pixels wide myself.” So, the dock widget gets a smaller size. Eventually, the window will get maximized, but your original dock widget size is already lost.

I tried a simple approach (https://codereview.qt-project.org/c/qt/qtbase/+/374272), but it was clearly not enough. The Qt Company’s Volker Hilsheimer ended up adding https://codereview.qt-project.org/c/qt/qtbase/+/375521. Despite its having a timer to workaround the asynchronous resizing, it seems to work reliably well.

About KDAB

If you like this article and want to read similar material, consider subscribing via our RSS feed.

Subscribe to KDAB TV for similar informative short video content.

KDAB provides market leading software consulting and development services and training in Qt, C++ and 3D/OpenGL. Contact us.

The post The Qt Less Traveled appeared first on KDAB.

Qt for WebAssembly on mobile devices

Qt for WebAssembly on mobile devices, specifically phones, has lacked an essential feature - support for the native keyboard. It may or may not have worked. If it worked, it did not work very well. The tricky issue is opening the keyboard when needed and closing when it wasn't. There is no simple API for doing that on any platform we target - iOS, Android and Windows.

Although it feels a bit hacky to me to open the keyboard using javascript , we use Emscripten's C++ interface to create a hidden javascript text input element and set focus to that, which opens the platform keyboard.

Emscripten on Android had one other issue - the usual Emscripten input event API was not working. Nothing being typed on the native virtual keyboard was being handled like on other platforms. I found I could utilize the hidden input element that is used to pop up the keyboard, to listen for input characters and then send them to Qt. 

As it is, this patch adds support for native mobile keyboard for iOS, Android and Windows on Qt for WebAssembly. (now merged into dev as 66a76a5def46d0e4a330f7130ad440c639b87cf7), too late to make it into 6.3.

Other issues on mobile are memory (as usual). Up until recently, Safari limits browser memory to much less than the other browsers. 

Other areas that Qt covers for mobile devices that do not work on Qt for WebAssembly yet are sensors and bluetooth connectivity. Although there is a patch for some sensors support, it has not been merged and probably needs updating. It may not work at all, either.

Bluetooth connectivity for javascript is currently only supported on the Chrome browser and is currently experimental. While I could probably add this support to Qt WebAssembly, it would not get merged and the API might be too changey.

Any areas you find Qt WebAssembly is lacking on mobile, please report to https://bugreports.qt.io/


Display numpy and pandas tables in PySide6 QTableView (updated for PySide6)

In the previous chapter we covered an introduction to the Model View architecture. However, we only touched on one of the model views — QListView. There are two other Model Views available in Qt5 — QTableView and QTreeView which provide tabular (Excel-like) and tree (file directory browser-like) views using the same QStandardItemModel.

In this tutorial we'll look at how to use QTableView from PySide, including how to model your data, format values for display and add conditional formatting.

You can use model views with any data source, as long as your model returns that data in a format that Qt can understand. Working with tabular data in Python opens up a number of possibilities for how we load and work with that data. Here we'll start with a simple nested list of list and then move onto integrating your Qt application with the popular numpy and pandas libraries. This will provide you with a great foundation for building data-focused applications.

Introduction to QTableView

QTableView is a Qt view widget which presents data in a spreadsheet-like table view. Like all widgets in the Model View Architecture, this uses a separate model to provide data and presentation information to the view. Data in the model can be updated as required, and the view notified of these changes to redraw/display the changes. By customising the model it is possible to have a huge amount of control over how the data is presented.

To use the model we'll need a basic application structure and some dummy data. A simple working example is shown below, which defines a custom model working with a simple nested-list as a data store.

We'll go into alternative data structures in detail a bit later.

python
import sys
from PySide6 import QtCore, QtGui, QtWidgets
from PySide6.QtCore import Qt


class TableModel(QtCore.QAbstractTableModel):
    def __init__(self, data):
        super(TableModel, self).__init__()
        self._data = data

    def data(self, index, role):
        if role == Qt.DisplayRole:
            # See below for the nested-list data structure.
            # .row() indexes into the outer list,
            # .column() indexes into the sub-list
            return self._data[index.row()][index.column()]

    def rowCount(self, index):
        # The length of the outer list.
        return len(self._data)

    def columnCount(self, index):
        # The following takes the first sub-list, and returns
        # the length (only works if all rows are an equal length)
        return len(self._data[0])


class MainWindow(QtWidgets.QMainWindow):
    def __init__(self):
        super().__init__()

        self.table = QtWidgets.QTableView()

        data = [
          [4, 9, 2],
          [1, 0, 0],
          [3, 5, 0],
          [3, 3, 2],
          [7, 8, 9],
        ]

        self.model = TableModel(data)
        self.table.setModel(self.model)

        self.setCentralWidget(self.table)


app=QtWidgets.QApplication(sys.argv)
window=MainWindow()
window.show()
app.exec_()

As in our earlier model view examples, we create the QTableView widget, then create an instance of our custom model (which we've written to accept the data source as a parameter) and then we set the model on the view. That's all we need to do — the view widget now uses the model to get the data, and determine how to draw it.

Basic QTableView example Basic QTableView example

Nested list as a 2-dimensional data store

For a table you need a 2D data structure, with columns and rows. As shown in the example above you can model a simple 2D data structure using a nested Python list. We'll take a minute to look at this data structure, and it's limitations, below —

python
table = [
  [4, 1, 3, 3, 7],
  [9, 1, 5, 3, 8],
  [2, 1, 5, 3, 9],
]

The nested list is a "list of lists of values" — an outer list containing a number of sub-lists which themselves contain the values. With this structure, to index into individual values (or "cells") you must index twice, first to return one of the inner list objects and then again to index into that list.

The typical arrangement is for the outer list to hold the rows and each nested list to contain the values for the columns. With this arrangement when you index, you index first by row, then by column — making our example table a 3 row, 5 column table. Helpfully, this matches the visual layout in the source code.

The first index into the table will return a nested sub-list —

python
row = 2
col = 4

>>> table[row]
[2, 1, 5, 3, 9]

Which you then index again to return the value —

python
>>> table[row][col]
9

Note that using this type of structure you can't easily return an entire column, you would instead need to iterate all the rows. However, you are of course free to flip things on their head and use the first index as column depending on whether accessing by column or row is more useful to you.

python
table = [
  [4, 9, 2],
  [1, 1, 1],
  [3, 5, 5],
  [3, 3, 2],
  [7, 8, 9],
]

row = 4  # reversed
col = 2  # reversed

>>> table[col]
[3, 5, 5]

>>> table[col][row]
9

Nothing about this data structure enforces equal row or column lengths — one row can be 5 elements long, another 200. Inconsistencies can lead to unexpected errors on the table view. See the alternative data stores later if you're working with large or complex data tables.

Next we'll look in a bit more detail at our custom TableModel and see how it works with this simple data structure to display the values.

Writing a custom QAbstractTableModel

In the Model View Architecture the model is responsible for providing both the data and presentation metadata for display by the view. In order to interface between our data object and the view we need to write our own custom model, which understands the structure of our data.

To write our custom model we can create a subclass of QAbstractTableModel. The only required methods for a custom table model are data, rowCount and columnCount. The first returns data (or presentation information) for given locations in the table, while the latter two must return a single integer value for the dimensions of the data source.

python
class TableModel(QtCore.QAbstractTableModel):

    def __init__(self, data):
        super(TableModel, self).__init__()
        self._data = data

    def data(self, index, role):
        if role == Qt.DisplayRole:
            # See below for the nested-list data structure.
            # .row() indexes into the outer list,
            # .column() indexes into the sub-list
            return self._data[index.row()][index.column()]

    def rowCount(self, index):
        # The length of the outer list.
        return len(self._data)

    def columnCount(self, index):
        # The following takes the first sub-list, and returns
        # the length (only works if all rows are an equal length)
        return len(self._data[0])

QtCore.QAbstractTableModel is an abstract base class meaning it does not have implementations for the methods. If you try and use it directly, it will not work. You must sub-class it.

In the __init__ constructor we accept a single parameter data which we store as the instance attribute self._data so we can access it from our methods. The passed in data structure is stored by reference, so any external changes will be reflected here.

To notify the model of changes you need to trigger the model's layoutChanged signal, using self.model.layoutChanged.emit(). See the previous ModelView tutorial for more information.

The data method is called with two values index and role. The index parameter gives the location in the table for which information is currently being requested, and has two methods .row() and .column() which give the row and column number in the view respectively. In our example the data is stored as a nested list, and the row and column indices are used to index as follows data[row][column].

The view has no knowledge of the structure of the source data, and is the responsibility of the model to translate between the view's row and column and the relevant positions in your own data store.

The role parameter describes what kind of information the method should return on this call. To get the data to display the view calls this model method with the role of Qt.DisplayRole. However, role can have many other values including Qt.BackgroundRole, Qt.CheckStateRole, Qt.DecorationRole, Qt.FontRole, Qt.TextAlignmentRole and Qt.ForegroundRole, which each expect particular values in response (see later).

Qt.DisplayRole actually expects a string to be returned, although other basic Python types including float, int and bool will also be displayed using their default string representations. However, formatting these types to your strings is usually preferable.

Basic QTableView example Basic QTableView example

We'll cover how to use these other role types later, for now it is only necessary to know that you must check the role type is Qt.DisplayRole before returning your data for display.

The two custom methods columnCount and rowCount return the number of columns and rows in our data structure. In the case of a nested list of list in the arrangement we're using here, the number of rows is simply the number of elements in the outer list, and the number of columns is the number of elements in one of the inner lists — assuming they are all equal.

If these methods return values that are too high you will see out of bounds errors, if they return values that are too low, you'll see the table cut off.

Formatting numbers and dates

The data returned by the model for display is expected to be a string. While int and float values will also be displayed, using their default string representation, complex Python types will not. To display these, or to override the default formatting of float , int or bool values, you must format these to strings yourself.

You might be tempted to do this by converting your data to a table of strings in advance. However, by doing this you make it very difficult to continue working with the data in your table, whether for calculations or for updates.

Instead, you should use the model's data method to perform the string conversion on demand. By doing this you can continue to work with the original data, yet have complete control over how it is presented to the user — including changing this on the fly while through configuration.

Below is a simple custom formatter which looks up the values in our data table, and displays them in a number of different ways depending on the Python type of the data.

python
def data(self, index, role):
    if role == Qt.DisplayRole:
        # Get the raw value
        value = self._data[index.row()][index.column()]

        # Perform per-type checks and render accordingly.
        if isinstance(value, datetime):
            # Render time to YYY-MM-DD.
            return value.strftime("%Y-%m-%d")

        if isinstance(value, float):
            # Render float to 2 dp
            return "%.2f" % value

        if isinstance(value, str):
            # Render strings with quotes
            return '"%s"' % value

        # Default (anything not captured above: e.g. int)
        return value

Use this together with the modified sample data below to see it in action.

python
data = [
    [4, 9, 2],
    [1, -1, 'hello'],
    [3.023, 5, -5],
    [3, 3, datetime(2017,10,1)],
    [7.555, 8, 9],
]

QTableView data formatting QTableView data formatting

So far we've only looked at how we can customize how the data itself is formatted. However, the model interface gives you far more control over the display of table cells including colors and icons. In the next part we'll look at how to use the model to customise QTableView appearance.

Styles & Colours with Roles

Using colors and icons to highlight cells in data tables can help make data easier to find and understand, or help users to select or mark data of interest. Qt allows for complete control of all of these from the model, by responding to the relevant role on the data method.

The types expected to be returned in response to the various role types are shown below.

Role Type
Qt.BackgroundRole QBrush (also QColor)
Qt.CheckStateRole Qt.CheckState
Qt.DecorationRole QIcon, QPixmap, QColor
Qt.DisplayRole QString (also int, float, bool)
Qt.FontRole QFont
Qt.SizeHintRole QSize
Qt.TextAlignmentRole Qt.Alignment
Qt.ForegroundRole QBrush (also QColor)=

By responding to a particular combination of role and index we can modify the appearance of particular cells, columns or rows in the table — for example, setting a blue background for all cells in the 3rd column.

python
def data(self, index, role):
    # existing `if role == Qt.DisplayRole:` block hidden
    # hidden for clarity.

    if role == Qt.BackgroundRole and index.column() == 2:
        # See below for the data structure.
        return QtGui.QColor('blue')

By using the index to lookup values from our own data, we can also customise appearance based on values in our data. We'll go through some of the more common use-cases below.

Text alignment

In our previous formatting examples we had used text formatting to display float down to 2 decimal places. However, it's also common when displaying numbers to right-align them, to make it easier to compare across lists of numbers. This can be accomplished by returning Qt.AlignRight in response to Qt.TextAlignmentRole for any numeric values.

The modified data method is shown below. We check for role == Qt.TextAlignmentRole and look up the value by index as before, then determine if the value is numeric. If it is we can return Qt.AlignVCenter + Qt.AlignRight to align in the middle vertically, and on the right horizontally.

python
def data(self, index, role):
    # existing `if role == Qt.DisplayRole:` block hidden
    # hidden for clarity.

    if role == Qt.TextAlignmentRole:
        value = self._data[index.row()][index.column()]

        if isinstance(value, int) or isinstance(value, float):
            # Align right, vertical middle.
            return Qt.AlignVCenter + Qt.AlignRight

Other alignments are possible, including Qt.AlignHCenter to align centre horizontally. You can combine them together by adding them together e.g. Qt.AlignBottom + Qt.AlignRight.

QTableView cell alignment QTableView cell alignment

Text colors

If you've used spreadsheets like Excel you might be familiar with the concept of conditional formatting. These are rules you can apply to cells (or rows, or columns) which change text and background colors of cells depending on their value.

This can be useful to help visualise data, for example using red for negative numbers or highlighting ranges of numbers (e.g. low … high) with a gradient of blue to red.

First, the below example implements a Qt.ForegroundRole handler which checks if the value in the indexed cell is numeric, and below zero. If it is, then the handler returns the text (foreground) color red.

python
def data(self, index, role):
    # existing `if role == Qt.DisplayRole:` block hidden
    # hidden for clarity.

    if role == Qt.ForegroundRole:
        value = self._data[index.row()][index.column()]

        if (
            (isinstance(value, int) or isinstance(value, float))
            and value < 0
        ):
            return QtGui.QColor('red')

If you add this to your model's data handler, all negative numbers will now appear red.

QTableView text formatting, with red negative numbers QTableView text formatting, with red negative numbers

Number range gradients

The same principle can be used to apply gradients to numeric values in a table to, for example, highlight low and high values. First we define our color scale, which is taken from colorbrewer2.org.

python
COLORS = ['#053061', '#2166ac', '#4393c3', '#92c5de', '#d1e5f0', '#f7f7f7', '#fddbc7', '#f4a582', '#d6604d', '#b2182b', '#67001f']

Next we define our custom handler, this time for Qt.BackgroundRole. This takes the value at the given index, checks that this is numeric then performs a series of operations to constrain it to the range 0…10 required to index into our list.

python
def data(self, index, role):
    # existing `if role == Qt.DisplayRole:` block hidden
    # hidden for clarity.

    if role == Qt.BackgroundRole:
        value = self._data[index.row()][index.column()]
        if (isinstance(value, int) or isinstance(value, float)):
            value = int(value)  # Convert to integer for indexing.

            # Limit to range -5 ... +5, then convert to 0..10
            value = max(-5, value) # values < -5 become -5
            value = min(5, value)  # valaues > +5 become +5
            value = value + 5     # -5 becomes 0, +5 becomes + 10

            return QtGui.QColor(colors[value])

The logic used here for converting the value to the gradient is very basic, cutting off high/low values, and not adjusting to the range of the data. However, you can adapt this as needed, as long as the end result of your handler is to return a QColor or QBrush

QTableView with number-range color gradients QTableView with number-range color gradients

Icon & Image decoration

Each table cell contains a small decoration area which can be used to display icons, images or a solid block of color, on the left hand side next to the data. This can be used to indicate data type, e.g. calendars for dates, ticks and crosses for bool values, or for a more subtle conditional-formatting for number ranges.

Below are some simple implementations of these ideas.

Indicating bool/date data types with icons

For dates we'll use Python's built-in datetime type. First, add the following import to the top of your file to import this type.

python
from datetime import datetime

Then, update the data (set in the MainWindow.__init__) to add datetime and bool (True or False values), for example.

python
data = [
    [True, 9, 2],
    [1, 0, -1],
    [3, 5, False],
    [3, 3, 2],
    [datetime(2019, 5, 4), 8, 9],
]

With these in place, you can update your model data method to show icons and formatted dates for date types, with the following.

python
#  icons indicating data type

def data(self, index, role):
    if role == Qt.DisplayRole:
        value = self._data[index.row()][index.column()]
        if isinstance(value, datetime):
            return value.strftime('%Y-%m-%d')

        return value

    if role == Qt.DecorationRole:
        value = self._data[index.row()][index.column()]
        if isinstance(value, datetime):
            return QtGui.QIcon('calendar.png')

QTableView formatted dates with indicator icon QTableView formatted dates with indicator icon

The following shows how to use ticks and cross for boolean True and False values respectively.

python
# ticks and crosses for `bool`values
def data(self, index, role):
    # existing `if role == Qt.DisplayRole:` block hidden
    # hidden for clarity.

    if role == Qt.DecorationRole:
        value = self._data[index.row()][index.column()]
        if isinstance(value, bool):
            if value:
                return QtGui.QIcon('tick.png')

            return QtGui.QIcon('cross.png')


You can of course combine the above together, or any other mix of Qt.DecorationRole and Qt.DisplayRole handlers. It's usually simpler to keep each type grouped under the same role if branch, or as your model becomes more complex, to create sub-methods to handle each role.

QTableView boolean indicators QTableView boolean indicators

Colour blocks

If you return a QColor for Qt.DecorationRole a small square of color will be displayed on the left hand side of the cell, in the icon location. This is identical to the earlier Qt.BackgroundRole conditional formatting example, except now handling and responding to Qt.DecorationRole.

python
# color blocks
if role == Qt.DecorationRole:
    value = self._data[index.row()][index.column()]
    if (isinstance(value, int) or isinstance(value, float)):
        value = int(value)

        # Limit to range -5 ... +5, then convert to 0..10
        value = max(-5, value)  # values < -5 become -5
        value = min(5, value)   # valaues > +5 become +5
        value = value + 5       # -5 becomes 0, +5 becomes + 10

        return QtGui.QColor(COLORS[value])

QTableView color block decorations QTableView color block decorations

Alternative Python data structures

So far in our examples we've used simple nested Python lists to hold our data for display. This is fine for simple tables of data, however if you're working with large data tables there are some other better options in Python, which come with additional benefits. In the next parts we'll look at two Python data table libraries — numpy and pandas — and how to integrate these with Qt.

Numpy

Numpy is a library which provides support for large multi-dimensional arrays or matrix data structures in Python. The efficient and high-performance handling of large arrays makes numpy ideal for scientific and mathematical applications. This also makes numpy arrays an good data store for large, single-typed, data tables in PySide.

Using numpy as a data source

To support numpy arrays we need to make a number of changes to the model, first modifying the indexing in the data method, and then changing the row and column count calculations for rowCount and columnCount.

The standard numpy API provides element-level access to 2D arrays, by passing the row and column in the same slicing operation, e.g. _data[index.row(), index.column()]. This is more efficient than indexing in two steps, as for the list of list examples.

In numpy the dimensions of an array are available through .shape which returns a tuple of dimensions along each axis in turn. We get the length of each axis by selecting the correct item from this tuple, e.g. _data.shape[0] gets the size of the first axis.

The following complete example shows how to display a numpy array using Qt's QTableView via a custom model.

python
import sys
from PySide6 import QtCore, QtGui, QtWidgets
from PySide6.QtCore import Qt
import numpy as np


class TableModel(QtCore.QAbstractTableModel):

    def __init__(self, data):
        super(TableModel, self).__init__()
        self._data = data

    def data(self, index, role):
        if role == Qt.DisplayRole:
            # Note: self._data[index.row()][index.column()] will also work
            value = self._data[index.row(), index.column()]
            return str(value)

    def rowCount(self, index):
        return self._data.shape[0]

    def columnCount(self, index):
        return self._data.shape[1]


class MainWindow(QtWidgets.QMainWindow):

    def __init__(self):
        super().__init__()

        self.table = QtWidgets.QTableView()

        data = np.array([
          [1, 9, 2],
          [1, 0, -1],
          [3, 5, 2],
          [3, 3, 2],
          [5, 8, 9],
        ])

        self.model = TableModel(data)
        self.table.setModel(self.model)

        self.setCentralWidget(self.table)


app=QtWidgets.QApplication(sys.argv)
window=MainWindow()
window.show()
app.exec_()

While simple Python types such as int and float are displayed without converting to strings, numpy uses it's own types (e.g. numpy.int32) for array values. In order for these to be displayed we must first convert them to strings.

QTableView with numpy array QTableView with numpy array

With QTableView only 2D arrays can be displayed, however if you have a higher dimensional data structure you can combine the QTableView with a tabbed or scrollbar UI, to allow access to and display of these higher dimensions.

Pandas

Pandas is a Python library commonly used for data manipulation and analysis. It provides a nice API for loading 2D tabular data from various data sources and performing data analysis on it. By using the pandas DataTable as your QTableView model you can use these APIs to load and analyse your data from right within your application.

Using Pandas as a data source

The modifications of the model to work with pandas are fairly minor, requiring changes to the indexing in the data method and modifications to rowCount and columnCount. The changes for rowCount and columnCount are identical to numpy with pandas using a _data.shape tuple to represent the dimensions of the data.

For indexing we use the pandas .iloc method, for indexed locations — i.e. lookup by column and/or row index. This is done by passing the row, and then column to the slice _data.iloc[index.row(), index.column()] .

The following complete example shows how to display a pandas data frame using Qt QTableView via a custom model.

python
import sys
from PySide6 import QtCore, QtGui, QtWidgets
from PySide6.QtCore import Qt
import pandas as pd


class TableModel(QtCore.QAbstractTableModel):

    def __init__(self, data):
        super(TableModel, self).__init__()
        self._data = data

    def data(self, index, role):
        if role == Qt.DisplayRole:
            value = self._data.iloc[index.row(), index.column()]
            return str(value)

    def rowCount(self, index):
        return self._data.shape[0]

    def columnCount(self, index):
        return self._data.shape[1]

    def headerData(self, section, orientation, role):
        # section is the index of the column/row.
        if role == Qt.DisplayRole:
            if orientation == Qt.Horizontal:
                return str(self._data.columns[section])

            if orientation == Qt.Vertical:
                return str(self._data.index[section])


class MainWindow(QtWidgets.QMainWindow):

    def __init__(self):
        super().__init__()

        self.table = QtWidgets.QTableView()

        data = pd.DataFrame([
          [1, 9, 2],
          [1, 0, -1],
          [3, 5, 2],
          [3, 3, 2],
          [5, 8, 9],
        ], columns = ['A', 'B', 'C'], index=['Row 1', 'Row 2', 'Row 3', 'Row 4', 'Row 5'])

        self.model = TableModel(data)
        self.table.setModel(self.model)

        self.setCentralWidget(self.table)


app=QtWidgets.QApplication(sys.argv)
window=MainWindow()
window.show()
app.exec_()

An interesting extension here is to use the table header of the QTableView to display row and pandas column header values, which can be taken from DataFrame.index and DataFrame.columns respectively.

QTableView pandas DataTable, with column and row headers QTableView pandas DataTable, with column and row headers

For this we need to implement a Qt.DisplayRole handler in a custom headerData method. This receives section, the index of the row/column (0…n), orientation which can be either Qt.Horizontal for the column headers, or Qt.Vertical for the row headers, and role which works the same as for the data method.

The headerData method also receives other roles, which can be used to customise the appearance of the headers further.

Conclusion

In this tutorial we've covered the basics of using QTableView and a custom model to display tabular data in your applications. This was extended to demonstrate how to format data and decorate cells with icons and colors. Finally, we demonstrated using QTableView with tabular data from numpy and pandas data structures, including displaying custom column and row headers.

For an in-depth guide to building GUIs with Python see my PySide6 book.

Packaging PySide6 applications for Windows with PyInstaller & InstallForge (updated for 2022)

There is not much fun in creating your own desktop applications if you can't share them with other people — whether than means publishing it commercially, sharing it online or just giving it to someone you know. Sharing your apps allows other people to benefit from your hard work!

The good news is there are tools available to help you do just that with your Python applications which work well with apps built using PyQt6. In this tutorial we'll look at the most popular tool for packaging Python applications: PyInstaller.

This tutorial is broken down into a series of steps, using PyInstaller to build first simple, and then increasingly complex PySide6 applications into distributable EXE files on Windows. You can choose to follow it through completely, or skip ahead to the examples that are most relevant to your own project.

We finish off by using InstallForge to create a distributable Windows installer.

You always need to compile your app on your target system. So, if you want to create a Mac .app you need to do this on a Mac, for an EXE you need to use Windows.

Example Installer for Windows Example Installer for Windows

If you're impatient, you can download the Example Installer for Windows first.

Requirements

PyInstaller works out of the box with Qt for Python PySide6 and, as of writing, current versions of PyInstaller are compatible with Python 3.6+. Whatever project you're working on, you should be able to package your apps.

You can install PyInstaller using pip.

bash
pip3 install PyInstaller

If you experience problems packaging your apps, your first step should always be to update your PyInstaller and hooks package the latest versions using

bash
pip3 install --upgrade PyInstaller pyinstaller-hooks-contrib

The hooks module contains package-specific packaging instructions for PyInstaller which is updated regularly.

Install in virtual environment (optional)

You can also opt to install PySide6 and PyInstaller in a virtual environment (or your applications virtual environment) to keep your environment clean.

bash
python3 -m venv packenv

Once created, activate the virtual environment by running from the command line —

bash
call packenv\scripts\activate.bat

Finally, install the required libraries.

python
pip3 install PySide6 PyInstaller

Getting Started

It's a good idea to start packaging your application from the very beginning so you can confirm that packaging is still working as you develop it. This is particularly important if you add additional dependencies. If you only think about packaging at the end, it can be difficult to debug exactly where the problems are.

For this example we're going to start with a simple skeleton app, which doesn't do anything interesting. Once we've got the basic packaging process working, we'll extend the application to include icons and data files. We'll confirm the build as we go along.

To start with, create a new folder for your application and then add the following skeleton app in a file named app.py. You can also download the source code and associated files

python
from PySide6 import QtWidgets

import sys

class MainWindow(QtWidgets.QMainWindow):

    def __init__(self):
        super().__init__()

        self.setWindowTitle("Hello World")
        l = QtWidgets.QLabel("My simple app.")
        l.setMargin(10)
        self.setCentralWidget(l)
        self.show()

if __name__ == '__main__':
    app = QtWidgets.QApplication(sys.argv)
    w = MainWindow()
    app.exec_()

This is a basic bare-bones application which creates a custom QMainWindow and adds a simple widget QLabel to it. You can run this app as follows.

bash
python app.py

This should produce the following window (on Windows 11).

Simple skeleton app in PySide6 Simple skeleton app in PySide6

Building a basic app

Now we have our simple application skeleton in place, we can run our first build test to make sure everything is working.

Open your terminal (command prompt) and navigate to the folder containing your project. You can now run the following command to run the PyInstaller build.

python
pyinstaller app.py

You'll see a number of messages output, giving debug information about what PyInstaller is doing. These are useful for debugging issues in your build, but can otherwise be ignored. The output that I get for running the command on Windows 11 is shown below.

bash
C:\Users\Gebruiker\pyinstaller\pyside6>pyinstaller app.py
235 INFO: PyInstaller: 4.7
235 INFO: Python: 3.7.6
237 INFO: Platform: Windows-10-10.0.22000-SP0
238 INFO: wrote C:\Users\Gebruiker\pyinstaller\pyside6\app.spec
240 INFO: UPX is not available.
243 INFO: Extending PYTHONPATH with paths
['C:\\Users\\Gebruiker\\pyinstaller\\pyside6']
574 INFO: checking Analysis
574 INFO: Building Analysis because Analysis-00.toc is non existent
575 INFO: Initializing module dependency graph...
579 INFO: Caching module graph hooks...
590 INFO: Analyzing base_library.zip ...
4047 INFO: Caching module dependency graph...
4198 INFO: running Analysis Analysis-00.toc
4214 INFO: Adding Microsoft.Windows.Common-Controls to dependent assemblies of final executable
  required by c:\users\gebruiker\appdata\local\programs\python\python37\python.exe
4433 INFO: Analyzing C:\Users\Gebruiker\pyinstaller\pyside6\app.py
4600 INFO: Processing module hooks...
4601 INFO: Loading module hook 'hook-difflib.py' from 'c:\\users\\gebruiker\\appdata\\local\\programs\\python\\python37\\lib\\site-packages\\PyInstaller\\hooks'...
4602 INFO: Loading module hook 'hook-encodings.py' from 'c:\\users\\gebruiker\\appdata\\local\\programs\\python\\python37\\lib\\site-packages\\PyInstaller\\hooks'...
4667 INFO: Loading module hook 'hook-heapq.py' from 'c:\\users\\gebruiker\\appdata\\local\\programs\\python\\python37\\lib\\site-packages\\PyInstaller\\hooks'...
4668 INFO: Loading module hook 'hook-pickle.py' from 'c:\\users\\gebruiker\\appdata\\local\\programs\\python\\python37\\lib\\site-packages\\PyInstaller\\hooks'...
4669 INFO: Loading module hook 'hook-PySide6.py' from 'c:\\users\\gebruiker\\appdata\\local\\programs\\python\\python37\\lib\\site-packages\\PyInstaller\\hooks'...
5083 INFO: Loading module hook 'hook-PySide6.QtNetwork.py' from 'c:\\users\\gebruiker\\appdata\\local\\programs\\python\\python37\\lib\\site-packages\\PyInstaller\\hooks'...
5558 INFO: Loading module hook 'hook-PySide6.QtWidgets.py' from 'c:\\users\\gebruiker\\appdata\\local\\programs\\python\\python37\\lib\\site-packages\\PyInstaller\\hooks'...
5782 INFO: Loading module hook 'hook-xml.py' from 'c:\\users\\gebruiker\\appdata\\local\\programs\\python\\python37\\lib\\site-packages\\PyInstaller\\hooks'...
5988 INFO: Loading module hook 'hook-PySide6.QtCore.py' from 'c:\\users\\gebruiker\\appdata\\local\\programs\\python\\python37\\lib\\site-packages\\PyInstaller\\hooks'...
6061 INFO: Loading module hook 'hook-PySide6.QtGui.py' from 'c:\\users\\gebruiker\\appdata\\local\\programs\\python\\python37\\lib\\site-packages\\PyInstaller\\hooks'...
6253 INFO: Looking for ctypes DLLs
6257 INFO: Analyzing run-time hooks ...
6259 INFO: Including run-time hook 'c:\\users\\gebruiker\\appdata\\local\\programs\\python\\python37\\lib\\site-packages\\PyInstaller\\hooks\\rthooks\\pyi_rth_pkgutil.py'
6262 INFO: Including run-time hook 'c:\\users\\gebruiker\\appdata\\local\\programs\\python\\python37\\lib\\site-packages\\PyInstaller\\hooks\\rthooks\\pyi_rth_win32api.py'
6284 INFO: Processing pre-find module path hook distutils from 'c:\\users\\gebruiker\\appdata\\local\\programs\\python\\python37\\lib\\site-packages\\PyInstaller\\hooks\\pre_find_module_path\\hook-distutils.py'.
6285 INFO: distutils: retargeting to non-venv dir 'c:\\users\\gebruiker\\appdata\\local\\programs\\python\\python37\\lib'
6340 INFO: Including run-time hook 'c:\\users\\gebruiker\\appdata\\local\\programs\\python\\python37\\lib\\site-packages\\PyInstaller\\hooks\\rthooks\\pyi_rth_inspect.py'
6341 INFO: Including run-time hook 'c:\\users\\gebruiker\\appdata\\local\\programs\\python\\python37\\lib\\site-packages\\PyInstaller\\hooks\\rthooks\\pyi_rth_pyside6.py'
6353 INFO: Looking for dynamic libraries
7417 INFO: Looking for eggs
7417 INFO: Using Python library c:\users\gebruiker\appdata\local\programs\python\python37\python37.dll
7417 INFO: Found binding redirects:
[]
7420 INFO: Warnings written to C:\Users\Gebruiker\pyinstaller\pyside6\build\app\warn-app.txt
7453 INFO: Graph cross-reference written to C:\Users\Gebruiker\pyinstaller\pyside6\build\app\xref-app.html
7465 INFO: checking PYZ
7466 INFO: Building PYZ because PYZ-00.toc is non existent
7466 INFO: Building PYZ (ZlibArchive) C:\Users\Gebruiker\pyinstaller\pyside6\build\app\PYZ-00.pyz
7902 INFO: Building PYZ (ZlibArchive) C:\Users\Gebruiker\pyinstaller\pyside6\build\app\PYZ-00.pyz completed successfully.
7912 INFO: checking PKG
7912 INFO: Building PKG because PKG-00.toc is non existent
7912 INFO: Building PKG (CArchive) app.pkg
7937 INFO: Building PKG (CArchive) app.pkg completed successfully.
7939 INFO: Bootloader c:\users\gebruiker\appdata\local\programs\python\python37\lib\site-packages\PyInstaller\bootloader\Windows-64bit\run.exe
7939 INFO: checking EXE
7939 INFO: Building EXE because EXE-00.toc is non existent
7939 INFO: Building EXE from EXE-00.toc
7940 INFO: Copying bootloader EXE to C:\Users\Gebruiker\pyinstaller\pyside6\build\app\app.exe
7946 INFO: Copying icon to EXE
7946 INFO: Copying icons from ['c:\\users\\gebruiker\\appdata\\local\\programs\\python\\python37\\lib\\site-packages\\PyInstaller\\bootloader\\images\\icon-console.ico']
7949 INFO: Writing RT_GROUP_ICON 0 resource with 104 bytes
7949 INFO: Writing RT_ICON 1 resource with 3752 bytes
7949 INFO: Writing RT_ICON 2 resource with 2216 bytes
7950 INFO: Writing RT_ICON 3 resource with 1384 bytes
7950 INFO: Writing RT_ICON 4 resource with 37019 bytes
7950 INFO: Writing RT_ICON 5 resource with 9640 bytes
7951 INFO: Writing RT_ICON 6 resource with 4264 bytes
7951 INFO: Writing RT_ICON 7 resource with 1128 bytes
7953 INFO: Copying 0 resources to EXE
7953 INFO: Emedding manifest in EXE
7954 INFO: Updating manifest in C:\Users\Gebruiker\pyinstaller\pyside6\build\app\app.exe
7958 INFO: Updating resource type 24 name 1 language 0
7960 INFO: Appending PKG archive to EXE
9144 INFO: Building EXE from EXE-00.toc completed successfully.
9146 INFO: checking COLLECT
9146 INFO: Building COLLECT because COLLECT-00.toc is non existent
9147 INFO: Building COLLECT COLLECT-00.toc
11774 INFO: Building COLLECT COLLECT-00.toc completed successfully.

If you look in your folder you'll notice you now have two new folders dist and build.

build & dist folders created by PyInstaller build & dist folders created by PyInstaller

Below is a truncated listing of the folder content, showing the build and dist folders.

bash
.
&boxvr&boxh&boxh app.py
&boxvr&boxh&boxh app.spec
&boxvr&boxh&boxh build
&boxv   &boxur&boxh&boxh app
&boxv       &boxvr&boxh&boxh Analysis-00.toc
&boxv       &boxvr&boxh&boxh COLLECT-00.toc
&boxv       &boxvr&boxh&boxh EXE-00.toc
&boxv       &boxvr&boxh&boxh PKG-00.pkg
&boxv       &boxvr&boxh&boxh PKG-00.toc
&boxv       &boxvr&boxh&boxh PYZ-00.pyz
&boxv       &boxvr&boxh&boxh PYZ-00.toc
&boxv       &boxvr&boxh&boxh app.exe
&boxv       &boxvr&boxh&boxh app.exe.manifest
&boxv       &boxvr&boxh&boxh base_library.zip
&boxv       &boxvr&boxh&boxh warn-app.txt
&boxv       &boxur&boxh&boxh xref-app.html
&boxur&boxh&boxh dist
    &boxur&boxh&boxh app
        &boxvr&boxh&boxh MSVCP140.dll
        &boxvr&boxh&boxh PySide6
        &boxvr&boxh&boxh app.exe
        &boxvr&boxh&boxh app.exe.manifest
        &boxvr&boxh&boxh Qt6Core.dll
        ...

The build folder is used by PyInstaller to collect and prepare the files for bundling, it contains the results of analysis and some additional logs. For the most part, you can ignore the contents of this folder, unless you're trying to debug issues.

The dist (for "distribution") folder contains the files to be distributed. This includes your application, bundled as an executable file, together with any associated libraries (for example PySide6) and binary .dll files.

Everything necessary to run your application will be in this folder, meaning you can take this folder and "distribute" it to someone else to run your app.

You can try running your app yourself now, by running the executable file, named app.exe from the dist folder. After a short delay you'll see the familiar window of your application pop up as shown below.

Simple app, running after being packaged Simple app, running after being packaged

You may also notice a console/terminal window pop up as your application runs. We'll cover how to stop that happening shortly.

In the same folder as your Python file, alongside the build and dist folders PyInstaller will have also created a .spec file. In the next section we'll take a look at this file, what it is and what it does.

The Spec file

The .spec file contains the build configuration and instructions that PyInstaller uses to package up your application. Every PyInstaller project has a .spec file, which is generated based on the command line options you pass when running pyinstaller.

When we ran pyinstaller with our script, we didn't pass in anything other than the name of our Python application file. This means our spec file currently contains only the default configuration. If you open it, you'll see something similar to what we have below.

python
# -*- mode: python ; coding: utf-8 -*-

block_cipher = None


a = Analysis(['app.py'],
             pathex=[],
             binaries=[],
             datas=[],
             hiddenimports=[],
             hookspath=[],
             runtime_hooks=[],
             excludes=[],
             win_no_prefer_redirects=False,
             win_private_assemblies=False,
             cipher=block_cipher,
             noarchive=False)
pyz = PYZ(a.pure, a.zipped_data,
             cipher=block_cipher)
exe = EXE(pyz,
          a.scripts,
          [],
          exclude_binaries=True,
          name='app',
          debug=False,
          bootloader_ignore_signals=False,
          strip=False,
          upx=True,
          console=True )
coll = COLLECT(exe,
               a.binaries,
               a.zipfiles,
               a.datas,
               strip=False,
               upx=True,
               upx_exclude=[],
               name='app')


The first thing to notice is that this is a Python file, meaning you can edit it and use Python code to calculate values for the settings. This is mostly useful for complex builds, for example when you are targeting different platforms and want to conditionally define additional libraries or dependencies to bundle.

If you generate a .spec file on Windows the path separator will be \\. To use this same .spec file on macOS you'll need to switch the separators to /. Thankfully / also works on Windows.

Once a .spec file has been generated, you can pass this to pyinstaller instead of your script to repeat the previous build process. Run this now to rebuild your executable.

bash
pyinstaller app.spec

The resulting build will be identical to the build used to generate the .spec file (assuming you have made no changes). For many PyInstaller configuration changes you have the option of passing command-line arguments, or modifying your existing .spec file. Which you choose is up to you.

Tweaking the build

So far we've created a simple first build of a very basic application. Now we'll look at a few of the most useful options that PyInstaller provides to tweak our build. Then we'll go on to look at building more complex applications.

Naming your app

One of the simplest changes you can make is to provide a proper "name" for your application. By default the app takes the name of your source file (minus the extension), for example main or app. This isn't usually what you want.

You can provide a nicer name for PyInstaller to use for the executable (and dist folder) either by editing the .spec file to add a name= under the app block.

python
exe = EXE(pyz,
          a.scripts,
          [],
          exclude_binaries=True,
          name='Hello World',
          debug=False,
          bootloader_ignore_signals=False,
          strip=False,
          upx=True,
          console=False  # False = do not show console.
         )

Alternatively, you can re-run the pyinstaller command and pass the -n or --name configuration flag along with your app.py script.

bash
pyinstaller -n "Hello World" app.py
# or
pyinstaller --name "Hello World" app.py

The resulting EXE file will be given the name Hello World.exe and placed in the folder dist\Hello World\.

Application with custom name "Hello World" Application with custom name "Hello World"

The name of the .spec file is taken from the name passed in on the command line, so this will also create a new spec file for you, called Hello World.spec in your root folder.

Hiding the console window

When you run your packaged application you will notice that a console window runs in the background. If you try and close this console window your application will also close. You almost never want this window in a GUI application and PyInstaller provides a simple way to turn this off.

Application running with terminal in background Application running with terminal in background

You can fix this in one of two ways. Firstly, you can edit the previously created .spec file setting console=False under the EXE block as shown below.

python
exe = EXE(pyz,
          a.scripts,
          [],
          exclude_binaries=True,
          name='app',
          debug=False,
          bootloader_ignore_signals=False,
          strip=False,
          upx=True,
          console=False  # False = do not show console.
         )

Alternatively, you can re-run the pyinstaller command and pass the -w, --noconsole or --windowed configuration flag along with your app.py script.

bash
pyinstaller -w app.py
# or
pyinstaller --windowed app.py
# or
pyinstaller --noconsole app.py


There is no difference between any of the options.

Re-running pyinstaller will re-generate the .spec file. If you've made any other changes to this file these will be lost.

One File Build

On Windows PyInstaller has the ability to create a one-file build, that is, a single EXE file which contains all your code, libraries and data files in one. This can be a convenient way to share simple applications, as you don't need to provide an installer or zip up a folder of files.

To specify a one-file build provide the --onefile flag at the command line.

bash
pyinstaller --onefile app.py

Result of a one-file build Result of a one-file build

Note that while the one-file build is easier to distribute, it is slower to execute than a normally built application. This is because every time the application is run it must create a temporary folder to unpack the contents of the executable. Whether this trade-off is worth the convenience for your app is up to you!

Using the --onefile option makes quite a few changes to the .spec file. You can make these changes manually, but it's much simpler to use the command line switch when first creating your .spec

Since debugging a one file app is much harder, you should make sure everything is working with a normal build before you create a one-file package. We're going to continue this tutorial with a folder-based build for clarity.

Setting an application Icon

By default PyInstaller EXE files come with the following icon in place.

Default PyInstaller application icon, on app.exe Default PyInstaller application icon, on app.exe

You will probably want to customize this to make your application more recognisable. This can be done easily using the --icon=<filename> command-line switch to PyInstaller. On Windows the icon should be provided as an .ico file.

bash
pyinstaller --windowed --icon=hand.ico app.py

The portable version of IcoFx is a good free tool to create icons on Windows.

Or, by adding the icon= parameter to your .spec file.

python
exe = EXE(pyz,
          a.scripts,
          [],
          exclude_binaries=True,
          name='blarh',
          debug=False,
          bootloader_ignore_signals=False,
          strip=False,
          upx=True,
          console=False,
          icon='hand.ico')

If you now re-run the build (by using the command line arguments, or running with your modified .spec file) you'll see the specified icon file is now set on your application's EXE file.

Custom application icon (a hand) on app.exe Custom application icon (a hand) on app.exe

However, if you run your application, you're going to be disappointed.

The custom EXE icon is not applied to the window The custom EXE icon is not applied to the window

The specified icon is not showing up on the window, and it will also not appear on your taskbar.

Why not? Because the icon used for the window isn't determined by the icons in the executable file, but by the application itself. To show an icon on our window we need to modify our simple application a little bit, to add a call to .setWindowIcon().

python
from PySide6 import QtWidgets, QtGui
import sys


class MainWindow(QtWidgets.QMainWindow):

    def __init__(self):
        super().__init__()

        self.setWindowTitle("Hello World")
        l = QtWidgets.QLabel("My simple app.")
        l.setMargin(10)
        self.setCentralWidget(l)
        self.show()

if __name__ == '__main__':
    app = QtWidgets.QApplication(sys.argv)
    app.setWindowIcon(QtGui.QIcon('hand.ico'))
    w = MainWindow()
    app.exec()

Here we've added the .setWindowIcon call to the app instance. This defines a default icon to be used for all windows of our application. You can override this on a per-window basis if you like, by calling .setWindowIcon on the window itself.

If you run the above application you should now see the icon appears on the window.

Window showing the custom hand icon Window showing the custom hand icon

Even if you don't see the icons, keep reading!

Dealing with relative paths

There is a gotcha here, which might not be immediately apparent. To demonstrate it, open up a shell and change to the folder where our script is located. Run it with

bash
python3 app.py

If the icons are in the correct location, you should see them. Now change to the parent folder, and try and run your script again (change <folder> to the name of the folder your script is in).

bash
cd ..
python3 <folder>/app.py

Window with icon missing Window with icon missing.

The icons don't appear. What's happening?

We're using relative paths to refer to our data files. These paths are relative to the current working directory -- not the folder your script is in. So if you run the script from elsewhere it won't be able to find the files.

One common reason for icons not to show up, is running examples in an IDE which uses the project root as the current working directory.

This is a minor issue before the app is packaged, but once it's installed you don't know what the current working directory will be when it is run -- if it's wrong your app won't be able to find anything. We need to fix this before we go any further, which we can do by making our paths relative to our application folder.

In the updated code below, we define a new variable basedir, using os.path.dirname to get the containing folder of __file__ which holds the full path of the current Python file. We then use this to build the relative paths for icons using os.path.join().

Since our app.py file is in the root of our folder, all other paths are relative to that.

python
from PySide6 import QtWidgets, QtGui
import sys, os

basedir = os.path.dirname(__file__)


class MainWindow(QtWidgets.QMainWindow):

    def __init__(self):
        super().__init__()

        self.setWindowTitle("Hello World")
        l = QtWidgets.QLabel("My simple app.")
        l.setMargin(10)
        self.setCentralWidget(l)
        self.show()

if __name__ == '__main__':
    app = QtWidgets.QApplication(sys.argv)
    app.setWindowIcon(QtGui.QIcon(os.path.join(basedir, 'hand.ico')))
    w = MainWindow()
    app.exec()

Try and run your app again from the parent folder -- you'll find that the icon now appear as expected, no matter where you launch the app from.

Taskbar Icons

Unfortunately, even if the icon is showing on the window, it may still not show on the taskbar.

If it does for you, great! But it may not work when you distribute your application, so it's probably a good idea to follow the next steps anyway.

Custom icon is not shown on the toolbar Custom icon is not shown on the toolbar

The final tweak we need to make to get the icon showing on the taskbar is to add some cryptic incantations to the top of our Python file.

When you run your application, Windows looks at the executable and tries to guess what "application group" it belongs to. By default, any Python scripts (including your application) are grouped under the same "Python" group, and so will show the Python icon. To stop this happening, we need to provide Windows with a different application identifier.

The code below does this, by calling ctypes.windll.shell32.SetCurrentProcessExplicitAppUserModelID() with a custom application id.

python
from PySide6 import QtWidgets, QtGui
import sys, os

basedir = os.path.dirname(__file__)

try:
    from ctypes import windll  # Only exists on Windows.
    myappid = 'mycompany.myproduct.subproduct.version'
    windll.shell32.SetCurrentProcessExplicitAppUserModelID(myappid)
except ImportError:
    pass


class MainWindow(QtWidgets.QMainWindow):

    def __init__(self):
        super().__init__()

        self.setWindowTitle("Hello World")
        l = QtWidgets.QLabel("My simple app.")
        l.setMargin(10)
        self.setCentralWidget(l)
        self.show()

if __name__ == '__main__':
    app = QtWidgets.QApplication(sys.argv)
    app.setWindowIcon(QtGui.QIcon(os.path.join(basedir, 'hand.ico')))
    w = MainWindow()
    app.exec()

The listing above shows a generic mycompany.myproduct.subproduct.version string, but you should change this to reflect your actual application. It doesn't really matter what you put for this purpose, but the convention is to use reverse-domain notation, com.mycompany for the company identifier.

With this added to your script, running it should now show the icon on your window and taskbar. The final step is to ensure that this icon is correctly packaged with your application and continues to be shown when run from the dist folder.

Try it, it wont.

The issue is that our application now has a dependency on a external data file (the icon file) that's not part of our source. For our application to work, we now need to distribute this data file along with it. PyInstaller can do this for us, but we need to tell it what we want to include, and where to put it in the output.

In the next section we'll look at the options available to you for managing data files associated with your app.

Data files and Resources

So far we successfully built a simple app which had no external dependencies. However, once we needed to load an external file (in this case an icon) we hit upon a problem. The file wasn't copied into our dist folder and so could not be loaded.

In this section we'll look at the options we have to be able to bundle external resources, such as icons or Qt Designer .ui files, with our applications.

Bundling data files with PyInstaller

The simplest way to get these data files into the dist folder is to just tell PyInstaller to copy them over. PyInstaller accepts a list of individual file paths to copy over, together with a folder path relative to the dist/<app name> folder where it should to copy them to.

As with other options, this can be specified by command line arguments, --add-data

bash
pyinstaller --windowed --icon=hand.ico --add-data="hand.ico;." app.py

You can provide `--add-data` multiple times. Note that the path separator is platform-specific, on Windows use `;` while on Linux or Mac use `:`

Or via the datas list in the Analysis section of the spec file, shown below.

python
a = Analysis(['app.py'],
             pathex=[],
             binaries=[],
             datas=[('hand.ico', '.')],  # Copy the file into the root folder of dist
             hiddenimports=[],
             hookspath=[],
             runtime_hooks=[],
             excludes=[],
             win_no_prefer_redirects=False,
             win_private_assemblies=False,
             cipher=block_cipher,
             noarchive=False)

And then execute the .spec file with

bash
pyinstaller app.spec

In both cases we are telling PyInstaller to copy the specified file hand.ico to the location . which means the output folder dist. We could specify other locations here if we wanted. On the command line the source and destination are separated by the path separator ;, whereas in the .spec file, the values are provided as a 2-tuple of strings.

If you run the build, you should see your .ico file now in the output folder dist ready to be distributed with your application.

The icon file copied to the dist folder The icon file copied to the dist folder

If you run your app from dist you should now see the icon on the window, and on the taskbar as expected.

The hand icon showing on the toolbar The hand icon showing on the toolbar

The file must be loaded in Qt using a relative path, and be in the same relative location to the EXE as it was to the .py file for this to work.

If your icon looks blurry it means you don't have large-enough icon variations in your .ico file. An .ico file can contain multiple different sized icons in the same file. Ideally you want to have 16x16, 32x32, 48x48 and 256x256 pixel sizes included, although fewer will still work.

The main advantage of using PyInstaller to bundle your files in this way is you can use Python in your .spec file to search and add the files to bundle. So for example, you could get a list of all files in a folder named icons and add them to the datas= parameter. Then, as you add more icons to that folder they would be bundled automatically.

Bundling data folders

Usually you will have more than one data file you want to include with your packaged file. The latest PyInstaller versions let you bundle folders just like you would files, keeping the sub-folder structure. For example, lets extend our app to add some additional icons, and put them under a folder.

python
from PySide6.QtWidgets import QMainWindow, QApplication, QLabel, QVBoxLayout, QPushButton, QWidget
from PySide6.QtGui import QIcon
import sys, os

basedir = os.path.dirname(__file__)

try:
    from ctypes import windll  # Only exists on Windows.
    myappid = 'mycompany.myproduct.subproduct.version'
    windll.shell32.SetCurrentProcessExplicitAppUserModelID(myappid)
except ImportError:
    pass


class MainWindow(QMainWindow):

    def __init__(self):
        super().__init__()

        self.setWindowTitle("Hello World")
        layout = QVBoxLayout()
        label = QLabel("My simple app.")
        label.setMargin(10)
        layout.addWidget(label)

        button = QPushButton("Push")
        button.setIcon(QIcon(os.path.join(basedir, "icons", "lightning.png")))
        button.pressed.connect(self.close)
        layout.addWidget(button)

        container = QWidget()
        container.setLayout(layout)

        self.setCentralWidget(container)

        self.show()

if __name__ == '__main__':
    app = QApplication(sys.argv)
    app.setWindowIcon(QIcon(os.path.join(basedir, "icons", "hand.ico")))
    w = MainWindow()
    app.exec()

The icons (PNG files and an ICO file for the Windows file icon) are stored under a subfolder named 'icons'.

bash
.
&boxvr&boxh&boxh app.py
&boxur&boxh&boxh icons
    &boxur&boxh&boxh lightning.png
    &boxur&boxh&boxh hand.png
    &boxur&boxh&boxh hand.ico

If you run this you'll see the following window, with a Window icon and a button icon.

Two icons Window with two icons, and a button.

The paths are using the Unix forward-slash / convention, so they are cross-platform for macOS. If you're only developing for Windows, you can use \\

To copy the icons folder across to our build application, we just need to add the folder to our .spec file Analysis block. As for the single file, we add it as a tuple with the source path (from our project folder) and the destination folder under the resulting dist folder.

python
# -*- mode: python ; coding: utf-8 -*-


block_cipher = None


a = Analysis(['app.py'],
             pathex=[],
             binaries=[],
             datas=[('icons', 'icons')],   # tuple is (source_folder, destination_folder)
             hiddenimports=[],
             hookspath=[],
             hooksconfig={},
             runtime_hooks=[],
             excludes=[],
             win_no_prefer_redirects=False,
             win_private_assemblies=False,
             cipher=block_cipher,
             noarchive=False)
pyz = PYZ(a.pure, a.zipped_data,
             cipher=block_cipher)

exe = EXE(pyz,
          a.scripts,
          [],
          exclude_binaries=True,
          name='app',
          debug=False,
          bootloader_ignore_signals=False,
          strip=False,
          upx=True,
          console=False,
          disable_windowed_traceback=False,
          target_arch=None,
          codesign_identity=None,
          entitlements_file=None )
coll = COLLECT(exe,
               a.binaries,
               a.zipfiles,
               a.datas,
               strip=False,
               upx=True,
               upx_exclude=[],
               name='app')


If you run the build using this spec file you'll see the icons folder copied across to the dist folder. If you run the application from the folder, the icons will display as expected -- the relative paths remain correct in the new location.

Alternatively, you can bundle your data files using Qt's QResource architecture. See our tutorial for more information.

Building a Windows Installer with InstallForge

So far we've used PyInstaller to bundle applications for distribution, along with the associated data files. The output of this bundling process is a folder, named dist which contains all the files our application needs to run.

While you could share this folder with your users as a ZIP file it's not the best user experience. Desktop applications are normally distributed with installers which handle the process of putting the executable (and any other files) in the correct place, adding Start Menu shortcuts and the like.

Now we've successfully bundled our application, we'll next look at how we can take our dist folder and use it to create a Windows installer.

Making sure the build is ready.

If you've followed the tutorial so far, you'll already have your app ready in the /dist folder. If not, or yours isn't working you can also download the source code files for this tutorial which includes a sample .spec file. As above, you can run the same build using the provided app.spec file.

bash
pyinstaller app.spec

This packages everything up ready to distribute in the dist/app folder. Run the executable app.exe to ensure everything is bundled correctly, and you should the same window as before with icons visible.

Two icons Window with two icons, and a button.

The EXE section in the .spec has a name parameter where you can specify the name of the resulting EXE file. You may want to change this to the name of your application.

Creating an installer

Now we've successfully bundled our application, we'll next look at how we can take our dist folder and use it to create a functioning Windows installer.

To create our installer we'll be using a tool called InstallForge. InstallForge is free and you can download the installer from this page.

We'll now walk through the basic steps of creating an installer with InstallForge. If you're impatient, you can download the finished Installforge Installer here.

General

When you first run InstallForge you'll be presented with this General tab. Here you can enter the basic information about your application, including the name, program version, company and website.

InstallForge initial view, showing General settings InstallForge initial view, showing General settings

You can also select the target platforms for the installer, from various versions of Windows that are available. For desktop applications you currently probably only want to target Windows 7, 8 and 10.

Setup

Click on the left sidebar to open the "Files" page under "Setup". Here you can specify the files to be bundled in the installer.

Use "Add Files…" and select all the files in the dist/app folder produced by PyInstaller. The file browser that pops up allows multiple file selections, so you can add them all in a single go, however you need to add folders separately. Click "Add Folder…" and add any folders under dist/app such as the PySide6 folder and icons

InstallForge Files view, add all files & folders to be packaged InstallForge Files view, add all files & folders to be packaged

Once you're finished scroll through the list to the bottom and ensure that the folders are listed to be included. You want all files and folders under dist/app to be present. But the folder dist/app itself should not be listed.

The default install path can be left as-is. The values between angled brackets, e.g. <company> , are variables and will be filled automatically.

Next, it's nice to allow your users to uninstall your application. Even though it's undoubtedly awesome, they may want to remove it at some time in the future. You can do this under the "Uninstall" tab, simply by ticking the box. This will also make the application appear in "Add or Remove Programs".

InstallForge add Uninstaller for your app InstallForge add Uninstaller for your app

Dialogs

The "Dialogs" section can be used to show custom messages, splash screens or license information to the user. The "Finish" tab lets you control what happens once the installer is complete, and it's helpful here to give the user the option to run your program.

To do this you need to tick the box next to "Run program" and add your own application EXE into the box. Since <installpath>\ is already specified, we can just add app.exe.

InstallForge configure optional run program on finish install InstallForge configure optional run program on finish install

System

Under "System" select "Shortcuts" to open the shortcut editor. Here you can specify shortcuts for both the Start Menu and Desktop if you like.

InstallForge configure Shortcuts, for Start Menu and Desktop InstallForge configure Shortcuts, for Start Menu and Desktop

Click "Add…" to add new shortcuts for your application. Choose between Start menu and Desktop shortcuts, and fill in the name and target file. This is the path your application EXE will end up at once installed. Since <installpath>\ is already specified, you simply need to add your application's EXE name onto the end, here app.exe

InstallForge, adding a Shortcut InstallForge, adding a Shortcut

Build

With the basic settings in place, you can now build your installer.

At this point you can save your InstallForge project so you can re-build the installer from the same settings in future.

Click on the "Build" section at the bottom to open the build panel.

InstallForge, ready to build InstallForge, ready to build

Click on the large icon button to start the build process. If you haven't already specified a setup file location you will be prompted for one. This is the location where you want the completed installer to be saved.

Don't save it in your dist folder.

The build process will began, collecting and compressing the files into the installer.

InstallForge, build complete InstallForge, build complete

Once complete you will be prompted to run the installer. This is entirely optional, but a handy way to find out if it works.

Running the installer

The installer itself shouldn't have any surprises, working as expected. Depending on the options selected in InstallForge you may have extra panels or options.

InstallForge, running the resulting installer InstallForge, running the resulting installer

Step through the installer until it is complete. You can optionally run the application from the last page of the installer, or you can find it in your start menu.

Our demo app in the Start Menu on Windows 11 Our demo app in the Start Menu in the Start Menu on Windows 11

Wrapping up

In this tutorial we've covered how to build your PySide6 applications into a distributable EXE using PyInstaller, including adding data files along with your code. Then we walked through the process of building the application into a Windows Installer using InstallForge. Following these steps you should be able to package up your own applications and make them available to other people.

For a complete view of all PyInstaller bundling options take a look at the PyInstaller usage documentation.

For more, see the complete PyQt6 tutorial.