What is Scripty?

Abstract

This documentation is about the automatic translation system used in KDE, which is nicknamed Scripty.

Overview

Scripty is the automatic translation system used in KDE. It helps everybody involved with translations by doing routine tasks automatically, rather than requiring that the translators do those tasks manually.

Scripty is started once each European night and needs many hours to process all the tasks.

Scripty's most important task is to create the appropriate translation template files from the source code. (This process is called extracting.) The translation templates are the basis for the translation work.

A second task is called merging. It mixes the existing translation files with the appropriate translation template files, so that translators get the newest changed strings from code.

A third task is taking care of the .desktop and similar files. For those files, Scripty extracts the required information from the file, merges it with the existing translation files, and also copies the translation back into the original file.

Scripty is only partially used for documentation translation, mainly for the merging. However for example it does not generate the translated documentation.

Why the Nickname?

Scripty's nickname comes from the SVN account used to commit the files: scripty. The term "scripting" is often used for automation of tasks within Unix shell environments. (The "full name" is "Script Kiddy" - this is a word play for the disparaging name used for crackers.)

Notation

If full relative SVN paths are given (e.g. trunk/KDE/kde-common/makemessages) it means that only this path is used by Scripty not a similar path of any other branch. When no branch is given in the path (not even trunk), then it means the corresponding module of the processed branch.

By "trunk", we mean the trunk/ branch (i.e. the main development one, formerly known as KDE CVS HEAD) of the KDE SVN server.

Where Are The Logs?

If you are interested in the logs of Scripty, they are publicly available at ftp://i18n.kde.org/logs. The file names are of the format yymmdd.log where yy is the two digit year, mm the 2 digit month, dd the 2 digit day. In general, the size of the log is around a Megabyte; in extreme cases however, it can be as large as a few Megabytes.

Scripty runs take many hours. Logs become publicly available as soon as Scripty starts, and are added to during the run.

Technical Overview

Scripty's start script is the script trunk/KDE/kde-common/makemessages. That is the script that is called and from which the output is taken to make the log. However most of the processing code is not in this script but in other scripts that are called by it.

The most important scripts for Scripty are those in l10n/scripts.

In the background, the main programs used are part of the gettext package, mainly xgettext for extracting and msgmerge for merging.

Note: KDE does not use the standard xgettext program but a patched version. The patch is in trunk/KDE/kdesdk/scripts/gettext.patch. (The patch no longer applies as-is on any recent version of xgettext. Scripty still uses a patched version 0.10 of it.)

Note: some of these details are explained more in the I18N HOWTO.

Scripts

Apart the makefiles in the source code, Scripty uses the file kde-i18n/Makefile.am.in intensively. It is here that the extraction of .desktop files is done (target desktop), along with the re-injection of the translation back into the .desktop files (target desktops), and the merging of the translation template files to the translation files (target merge). (Some of the targets mainly consist of Perl scripts.)

Makefiles

Scripty makes an intensive use of makefiles to do its work, especially for the extraction.

Each Makefile.am file in KDE can have a target named messages that Scripty calls to extract the strings to be translated from source. (The strings to be translated are also called user visible strings or messages.) Another important target is rc.cpp which extracts the messages of .ui, .rc and .kcfg files in the current directory. (In fact rc.cpp is created as output of $EXTRACTRC *.ui *.rc *.kfcg when the messages target is defined like messages: rc.cpp)

Extracting XML

Extracting messages from XML is done by 2 scripts:

kdesdk/scripts/extractrc
kdesdk/scripts/extractattr

Thse scripts need to be called from the messages target of the directory's Makefile.am when needed.

The script kdesdk/scripts/extractrc is also used by the rc.cpp target.

Branches

Scripty normally processes two branches: trunk/l10n, and branches/stable/l10n. While trunk/l10n works with code in l10n, branches/stable/l10n works for stable versions, depending on the current stable (or almost stable) modules and applications.

Modifying Scripty

First important rule: be careful!

Second important rule: avoid doing it just before a release, except if it fixes a problem for the release.

To restate: you must be careful if you want to modify any component used by Scripty. A bad Scripty run can affect the translators, either because they will not get the changed messages from code or because the translations may not be merged back.

Otherwise Scripty's "way of life" is to output errors, warnings or other strings only when absolutely needed. (Imaging being the person having to check the log. You would surely prefer a few errors than 1000s lines of warnings.)

However some subtle bugs can only been shown if many data is given out. So sometimes it is a balance between too few and too much information. But static repeated information should be discarded if possible (for example: an output like "program version 1.1; making your files better" is something that should better be discarded, for example by filtering by grep).

Also missing files or similar errors should not block further processing. ("The show must go on!")

If you are not familiar with Scripty, better ask Stephan Kulow before committing anything.