About developing one desktop application in Python

Hello. In this article I want to talk about the development of open-source code for the offline storage of notes — OutWiker. This project I do in my spare time the first version was released in late 2010, and OutWiker still continues to evolve. So historically, what's the code I'm doing almost alone (although occasionally get a useful separate patches), but users are actively involved in translating to your native language and sometimes send styles for pages that I include in the Assembly. And about how many interesting ideas they send and say no.

the

What is OutWiker

I wrote about this program on Habre, but it was so long ago that before speaking about the peculiarities of the internal structure and development process, I must say that is a program from the user's point of view. All links associated with the project are given at the end of the article. So, OutWiker is a program to store your notes in a tree view in the English-speaking Internet such software is usually called outliner (which is why the program has such a name). Among the more famous "colleagues" OutWiker with similar functionality can be called Zim, WikidPad, CherryTree, and many others (and, of course, org mode for Emacs). A logical question from the user's point of view — than OutWiker differs from other members tree notebooks. At this point, after this amount of time since the start of development, I'm not ready deployed to compare all this software. In the old days I tried dozens of outliner-s, five pieces were used for a long time, but always something was missing, I wanted another opportunity to take from WikidPad, one from viceversa that can work offline, etc. Therefore, the answer to this question will simply list the main features possessed by OutWiker.

All notes are stored in folders on disk. This is done for two reasons — for reliability, for example, with the gradual dying hard, all the notes go to the light together with a single file. And, in addition, it allows you to view and edit notes without OutWiker. Of course, this method of storage has its drawbacks, but nothing is given for granted.
For each note, you can attach any number of files (really folders, but this feature is not really advertised). One of the purposes of the attached files (mainly pictures) is the use of them in the text notes. Thus, by changing the attached picture, we see a new picture in page text.
From OutWiker no visual editor (I think he ever will, but while hands do not reach to it), but there are several types of pages — HTML pages, bicistronic and Markdown-pages (after installing the proper plugin). In OutWiker the emphasis is on bicistronic. Wikinotizie resembles pmWiki (not a very common engine for websites), but with some differences and peculiarities of wikinotizie and its implementation I will say below.
As you can see on the previous paragraph, the program is able to work with plugins, which are currently more 20. For example, with plugins for wiki pages you can add the coloring code for different programming languages, support for formulas in LaTeX format, ability to insert various counters (e.g., number of drawings), graphs and charts for your data, you can add templates for commonly entered phrases, create the pages by downloading them from the Internet and much more.
Each note can be tagged with an arbitrary number of tags. Tag cloud displayed in the sidebar window.

the Program is cross-platform, there are builds for Windows and Linux (with Mac OS X I don't add up). the

Source code is open and licensed under GPL 3.

Something about the inner workings of development

core libraries

Now I will tell you how OutWiker is arranged inside, and what problems had to face during development.

the Whole project is written in Python, and in Python 2.7, Python 3 migration.x is planned, but no time soon (why, I will tell later). To create the interface uses the wxPython library. At the beginning stages of the project as a library for the interface choosing between wxPython and PyQt, other libraries were rejected because they create an interface that looks alien or under Windows, or under Linux and can be there. As a result of staying in wxPython because of the fact that the program size was smaller. The exact numbers of the sizes I have would call, and the project have since then greatly expanded. Selection of wxPython I'm happy, although it was related to some unpleasant moments (they also say later).

Python has been chosen because he's great with the platform. It was originally planned to add the ability to create plugins, because the plugin is a normal Python scripts, there is no need to collect them separately under different operating systems. In General, the choice of language I do not regret, and brakes that would have arisen because of the language, was observed. There are sections of programs that run very quickly, but there are reasons for the other mainly in the necessity of reading a large number of files from disk.

as for cross-platform, not so many places, which depends on the operating system, and they are all connected with the interface. Perhaps the biggest difference is in the way of displaying HTML pages. Windows uses the Internet Explorer engine, and under Linux WebKit. Before OutWiker worked under wxPython 2.8 and engines of Internet Explorer and WebKit used by using hacks, which ceased to be relevant with the release of wxPython 3.0, and now the classes for working with these engines, built into the library. However, Internet Explorer I continue to use the hack with a call to the COM object, because such use gives more options for setting the behavior of the engine. In theory, the class wx.html2.WebView, which should be under Windows to work using Internet Explorer, and under Linux using WebKit, there is a method GetNativeBackend(), which should return a pointer to the "real" engine, but for some reason I have this method always returns None.

I Must admit that the project was influenced by the program WikidPad. I did it long enjoyed, it is also written in Python + wxPython, and sometimes I even looking in its source code, to see how some aspects of it are done. I must say that this spying has saved a lot of time.

Wikinature

For pages with registration are three types of pages: HTML pages, Markdown page (you need to add the appropriate plugin) and bicistronic. The fact that the program should be bicistronic I decided since the inception of the project. Unlike HTML, wikinotizie more concise and easy to extend by adding new commands. Another advantage of wikinotizie is the ability to convert the inline text description into a graphic object. For example, in OutWiker is the command to insert formulas in LaTeX format, inserting graphs, which are constructed either according from a text file or data that is inserted directly in the page, there is a plugin which creates a text description of description of charts, etc. in addition, since the wiki page will eventually be converted into HTML for display to the user, it can also be used, for example, in the articles on the website, if the engine is waiting for input in HTML format. For example, right now I'm writing this article in the form of wikinotizie, and in the end switch to the HTML tab and get ready HTML-code, which will only need a bit to adjust to the specific features of the site.

Result:

HTML:

as of wikinotizie I am guided by engine pmWiki, it's not very common, but I already had experience with it and I like the reasonableness of his notation. At some points in the implementation of the parser I moved away from the original notation, but generally try to stick to it. This wikinotizie very easy to add your command (:command_name:)...(:command_nameend:).

and Sometimes ask, why didn't I use Markdown originally, but in my opinion it's too limited to language, besides pmWiki more logical. For example, in Markdown, I always get confused with the order of arguments for links: [text](http://example.com/) — two brackets, more important, their order. In the use of wikinotizie a reference is made in the form of: [[text -> example.com]]. The arrow indicates the direction of the link. Actually there is a second link format, but personally I usually use this.

by the Way, OutWiker do not have to remember wikinotizie, all elements can be inserted through the menu or using the toolbar. For HTML and Markdown pages these buttons and menu items also available.

To create wikiparser used a very handy library pyparsing. Convenient it in the first place so that is only a single file pyparsing.py, which can be put in the source code and not to pull an extra dependency in the project. Using this library describes all of the tokens of wikinotizie, and then they are converted to HTML. About pyparsing has a detailed documentation with many examples and even a small book. In addition, the library author is actively involved in the forum on the website and ready to help you with using the library.

Localization

Over the OutWiker translation into other languages is actively working some users, for which they thank you. For a joint translation service is being used crowdin.com. It's quite a handy service where users can offer their own versions of translations, and discuss them.

currently, the program supports English, Russian, Ukrainian and Swedish languages. Unfortunately, support for some languages stopped. So, for example, need of translators, who will finish Italian localization (Italian guy who once actively supported this localization disappeared) and the German localization (now part of the German translation is German, and part of my wife as a practice of the German language, which she teaches. It is necessary that someone has checked and approved the second part of the translation. In addition, the German language is not translated plugins). There are several languages that somebody started to do translations, but they are not brought to mind (Bulgarian language and Brazilian Portuguese). And of course, any other translations are welcome.

for example, how does the Ukrainian and Swedish localization.

To use localizations inside OutWiker uses the standard gettext module.

cusp

Since Python is an interpreted programming language, in order to allow the user to run a program, theoretically it should have installed Python and all necessary libraries. Of course, to force ordinary users to install Python, wxPython and the other required libraries, but still need versions inhumane. The user needs to download the program, unpack it to install using the installer and the method of "Next — Next — Next", then run the icon on your desktop. Fortunately, programs written in Python, it's all done relatively simply. If we talk about Windows, the Python interpreter, the standard library fits in dll-ku slightly larger than 3 MB. There are several utilities which make the executables (exe-shnik, if we talk about Windows) from Python scripts. The most famous of them are cx_Freeze and pyInstaller. In features they are approximately equivalent (although pyInstaller allows you to create a single exe-shnik from a Python script, and cx_Freeze has no such ability), but they have a different approach to where to store the files *.pyc libraries, and about how to define build settings.

the same cx_Freeze / pyInstaller can be used for creating binary builds for Linux, the resulting Assembly could be run on different Linux distributions. Under Windows from the resulting Assembly, you create the installer using Inno Setup, Linux of such assemblies is created. deb packages (on virtual machines, which will be described below).

Another feature of the Assembly is how to run different build targets. It all began with a small Makefile, which gradually grew, and it became more and more. Mostly they were related to the cross — platform- you can maintain a single Makefile on Linux and Windows, it was hard. At the time, I got tired and I started to look for an alternative, it is desirable that it was written in Python. The alternative was found quickly enough in the form of a program Fabric. Now all the former targets the Makefile rewritten as Python functions, and indeed the build system has grown enormously. For example, now looks like a list of tasks Fabric.

$ fab -l
Available commands:

apiversion Print OutWiker current API versions
apiversions Print OutWiker current API versions
Create build artefacts for current version.
clear Remove artefacts after all assemblies
create_tree tree Create a wiki for the tests
Assemble the deb the deb packages
deb_binary Create binary deb package
deb_binary_clear Remove the binary deb package
deb_clear Remove the deb packages
deb_install Assemble the deb package for current Ubuntu release
deb_single Assemble the deb package for the current Ubuntu release
deb_sources_included Create files for uploading in PPA (including sources)
deploy to Upload the unstable version to site
doc Build documentation
linux_binary Assemble binary builds for Linux
linux_clear Remove binary builds for Linux
locale Update the localization file (outwiker.pot)
locale_plugin Create or update the localization file for pluginname plug-in
outwiker_changelog Generate OutWiker''s changelog for the site
plugin_changelog Generate plugin''s changelog for the site
plugin_locale Create or update the localization file for pluginname plug-in
plugins Create an archive with plugins (7z required)
plugins_clear Remove an archive with plugins (7z required)
Print plugins_list plugins list for th site
prepare_virtual Prepare virtual machine
run run from OutWiker sources
site_versions Compare OutWiker plugins and current versions with versions on the site
sources sources Create the archives
sources_clear Remove the sources archives.
test Run the unit tests
test_build Run the build, unit tests
upload_binary Upload the unstable version to site
upload_plugin Upload plugin to site
upload_plugins_pack Upload archive with all plugins to site
vm_halt Stop virtual machines for build
vm_linux_binary Create 32 - and 64-bit assembly on virtual machines
vm_prepare Prepare virtual machines for build
vm_remove_keys Remove local SSH keys for remote virual machines
vm_run Run virtual machines for build
vm_stop Stop virtual machines for build
vm_update Update the virtual machines
win Build OutWiker for Windows with cx_Freeze
win_clear Remove assemblies under Windows

As you can see, commands quite a lot, they are all described the documentation. This includes the command to build under Windows and Linux, run tests, work with virtual machines to build, upgrade, locations, uploading new versions of the website and others. Despite the fact that according to the documentation, more Fabric focused on work with remote servers (the conditional analogue of the Ansible), but as a replacement for the Makefile I like it very much.

War Ubuntu

wxPython was linked a few unpleasant moments, which, fortunately, has managed to quickly solve. All these problems were encountered under Ubuntu, which I am first and foremost guided in the development (due to the fact that I is the primary operating system). The first unpleasant moment was associated with the move from wxPython 2.8 to 3.0 (not to be confused with Python 3.x). The fact is that when upgrading from version 2.8 to 3.0 had some problems with backward compatibility, but fortunately, these two versions could be put at the same time from a python script to choose the correct version by using wxversion. (with the release of wxPython 4.0 this possibility no longer). Since Python is interpreted, and in Ubuntu it is always present, to run under this operating system there is no need to create the binary builds, and just when you install deb package sources are scattered in the correct folders and run the command python runoutwiker.py. All the required libraries to work wxPython and WebKit provided as a dependency. However, with the release of Ubuntu 16.04 LTS have two problems. The first was related to the fact that in this version of Ubuntu has removed wxPython 2.8, so we had to quickly migrate to wxPython 3.0, and to refuse to support another long-lived version of Ubuntu 14.04 LTS. But that's not all, it turned out that in Ubuntu 16.04 (next version it is fixed) wxPython somehow improperly compiled and without shamanism not working WebKit engine. Fortunately, here help advanced users OutWiker, which suggested that the symptom, the LD_PRELOAD and specify the path to one library relating to wxPython. As I said, the need for this crutch in later versions of Ubuntu anymore, but for the support of Ubuntu 16.04 it is necessary to use so far.

Other libraries

Besides the already mentioned libraries wxPython and pyparsing used in the project and other libraries. Briefly list them.

PyEnchant. Used for spell checking.
Pygments. Used in the plugin Source for coloring source pieces of code. At the entrance to the library, we serve source, the output is a beautifully designed HTML. Pygments understands a huge number of programming languages, including exotic and rarely used, and also allow you to choose a style (the library is attached to more than a dozen styles).
Python-Markdown. As the name implies, is designed to parse Markdown notation.
KaTeX. Used plugin TeXEquation. JavaScript library to render formulas in LaTeX format. Previously with the mimeTeX program, but the formula made it look not very aesthetically pleasing and had problems with the display of the formulas on pages with a dark background because mimeTeX creates a picture with black text.
Beautiful Soup. Used in the plugin WebPage designed to create pages from pages on the Internet (it downloads all the images, CSS styles and JavaScript). The library is used to parse and edit the downloaded HTML code. For example, to replace the original links to images in the HTML code on the path to the downloaded images.
Chardet. Used in the same plug-in WebPage, to determine the encoding of the downloaded pages.
Jinja2. Used in the Snippets plugin, which is designed to insert into the notes placeholder text. Templates are created in the format of Jinja (with some simplifications).
Blockdiag. Used Diagrammer plugin, which builds a text description of the diagram of blocks connected by arrows. Most of the work in the plug-in library performs Diagrammer. It can be considered as a simplified version of dot and Graphviz.

development Plans

Project OutWiker is actively developing, as evidenced by the statistics of commits on github:

Now the source code is about 90 MB, which includes nearly 4300 files, of which 1772 — python-scripts.

Plans and ideas a huge number (the number of issues on github is now a little less than 350). There are any wishes of users, even if in the near future to implement them hands just will not come. As code, I'm almost alone, that new features do not appear as quickly as we would like. Recently the stable version OutWiker go about once a year, but every month I post the unstable version. In fact, their instability is a philosophical question, because I don't openly post the broken version, and sometimes unstable version "more stable" stable in the sense that after release they fixed some found critical flaws of the latest release. But in the unstable version I can afford something does not have time to grind, and maybe something not to notice after addition of new features.

At the moment the work is in two areas — to make a more modern appearance of the program and reduce the number of files created for each note. As for the interface, I want to get rid of the existing toolbars and make their style Delphi. But most likely this global mess will leave to the next version 2.2.

In the light of the above-described forever for the problems Ubuntu want to try Linux to make a snap-pack or/and flatpak package.

you Need to write the documentation (I mean technical help is still there). In the documentation I describe the inner workings of the program, cusp, etc. Until almost all documentation is written in Russian, hopefully by release I'll get myself together and translate some of it into English.

References

program Page, where can I download the latest stable version.
Page unstable version.
Project on Github.
Technical documentation.

Article based on information from habrahabr.ru

Поиск по этому блогу

computer express