Basic Engine localization

CityAceE

Engine BASIC speaks your language. It is available in English, German, French, Spanish and Japanese, with all system messages and help texts translated. National keyboards layouts for all supported languages are

I would like to know how much volume needs to be translated in order to add localization for another language. In this case, I mean my native language is Russian. At a minimum, I should be able to type messages in my language, that is, switch the keyboard layout and output messages in my language using the PRINT and INPUT commands.

Dmian

I can't speak for the system side of things, but from the font side of things, I added Cyrillic support to "Lexis", the default font for the system: https://github.com/damianvila/font-lexis
I don't know how well it works, so if you have any way to confirm if it works as intended and that the characters are correct, I will appreciate it (you can download the font and try it in your system, if you wish).

For what I see, the strings are in Poedit files in the /po directory, so maybe you can contribute the translations by creating the files for Russian?

As for the other font I made and is included (BESCII: https://github.com/damianvila/font-bescii) doesn't have Cyrillic characters, though it could be included eventually. I can't promise any timeframe, though, as I don't have the means or time at the moment. But I would be willing to do it if I can get some free time and can buy a license for the program I need.

Oh! I see the CPC 646 font is also included: https://github.com/damianvila/font-cpc464
If I include Cyrillic for BESCII, it would be rather trivial to include it in CPC646. The rest of the included fonts are Japanese. And you can load your own fonts too.

If you know a specific pixel 8x8 Cyrillic font that you want to include (for a specific retro system from Russia, like the Agat, for example), let me know, as I don't know much about that, but would like to know if there's a specific font that has special significance for the Russian people. Cheers!

CityAceE

Dmian I added Cyrillic support to "Lexis", the default font for the system:

It's a big surprise for me! Thank you very much!

10 PRINT "Горячий привет всем пользователям BASIC ENGINE из России!"
20 PRINT "Съешь ещё этих мягких французских булок да выпей же чаю!"

The second line contains all cyrillic symbols in one sentence.

I'll answer the rest later.

CityAceE

Dmian If you know a specific pixel 8x8 Cyrillic font that you want to include (for a specific retro system from Russia, like the Agat, for example), let me know

I've attached one of commone cyrillic fonts for retro computers. It's 8*8.

cyrillic.zip

1kB

CityAceE

Dmian For what I see, the strings are in Poedit files in the /po directory, so maybe you can contribute the translations by creating the files for Russian?

I started the translation. I already have a ready-made translation obtained using a neural network. I have already edited the system messages. I am currently editing the help file.

CityAceE

757 of 1378 strings are translated. 621 (46%) to go. Ugh!

CityAceE

I found another bug on Banana Pi M2 Zero with LT version. Steps for reproducing:

Switch the language to Japanese:
config 11,4
Type
help print
The system reboots each time

How can I check my translation?

I tried to replace the translation file (French and Japanese) with mine here:

basicengine-buildroot/output/build/enginebasic-sdl-jh/po

Next, I deleted this file:
basicengine-buildroot/output/build/enginebasic-sdl-jh/.stamp_built

And then make again. But the translation is not picked up.

CityAceE

I continued experimenting and was able to replace the help text for the French language with my Russian language file. However, I have not yet been able to replace the system messages.

CityAceE

CityAceE However, I have not yet been able to replace the system messages.

I was able to overcome this problem. Now there's a long test of translating the help for each command to read all the text in context.

CityAceE

I used the helptext_es.po file as a template, since it had the most lines. But when I translated everything (1378 lines) and started to check what I got, I saw some untranslated lines. I also saw that the *.po file itself had some formatting errors. As a result, I took the compiled helptext_en.json file that appears after compiling the project. It looks like it definitely has all the necessary strings. I made my own helptext_en.po file based on it. Now I have 350 more lines to translate. But at the same time the total number of lines to translate has decreased to 1102.

In the process of this research I found out that WARNING and IMPORTANT are highlighted in orange color only in English. Even in the author's native German language, these inscriptions are not colored differently from the other text. This is because in the script that translates *.po to *.json these words are hardcoded. But I found a way to make them colorized for Russian during translation.

uli

A few words on how I maintain translations for Engine BASIC:

First of all, everything is a machine translation, even German. I have tried several programs and services for that purpose, but the bulk of it has been done by Google Translate.

The script I'm currently using for that purpose collects untranslated strings and writes them into the clipboard as plain text separated by ===== and empty lines. I then paste that English text into Google Translate and have it translated into the target language, copy the text into the clipboard and paste it back into the script. Then I reverse the translation and have the result translated back again to a third language (usually German) and paste that back into the script as well. The script will then produce a new message catalog that includes both existing and new translations.

(BTW, the reason for the copy-and-paste stuff is that Google Translate in the browser produces much higher quality results than Google Translate when used through an API. Not to mention that the API is not really an API but web scraping or something and breaks all the time.)

I then copy that new message catalog into the git directory and use git cola to go through the messages one by one and check if the translations make sense. Since I don't speak all supported languages and cannot tell prima facie if a translation is correct or not I instead check if the original English text and the German re-translation both make sense. If so I assume that the "intermediate" target language translation also makes sense. When in doubt I drop the translation. (That's why there are missing translations, more in some languages than in others.)

The reason I have not included this translation stuff in the build system is that it's such a manual process. :/
One thing that is also missing is a provision for deleting unused strings.

CityAceE

uli A few words on how I maintain translations for Engine BASIC:

Thanks for sharing about the translation process. I've already figured out a lot of it by parsing your files and scripts.

I also use machine translation to translate into Russian. And I use three translators at once: DeepL, Yandex and Google. I like the options offered by DeepL the most. However, after machine translation, I sit and think about each phrase. Some phrases I have to come up with on my own, as the machine translation is no good. I also have to make the whole text uniform. And I like the result.

But so far I haven't found a way to get a complete po-file with all the strings. I almost made it, but I had to edit your help.py and then run the result through my own transcoder. A lot of time wasted on trial and error. And I don't even know if anyone needs the result of my work yet. There is really a lot of text to translate.

uli

These are the scripts that I'm currently using:

TMPMSGS=tmpmsgs.po
LANGS="de es fr ja"

sed 's/^E.*ERR_.*, "/_("/' <ttbasic/errdef.h >errdef_gettext.h

xgettext --foreign-user --no-wrap -k_ -k__ --from-code utf-8 errdef_gettext.h ttbasic/*.cpp ttbasic/*.c libraries/TTBAS_LIB/*.cpp ttbasic/*.h h3/*.cpp h3/*.h -s -o "$TMPMSGS"

rm -f errdef_gettext.h

# WTF can't you set the charset in the options??
# have to add a bogus translation or msgcat will refuse to recognize the definition
sed -i -e 's,charset=CHARSET,charset=utf-8,' -e 's,msgstr "",msgstr "===bogus===",g' -e 's,expected ,expected: ,' "$TMPMSGS"

for i in $LANGS ; do
        msgcat -u -s "$TMPMSGS" po/system_${i}.po >trans/totrans_${i}.po || exit 1
        python bakpof2txt.py trans/totrans_${i}.po >trans/totrans_${i}.txt
done

cd ttbasic
for i in $LANGS ; do
        cat *.cpp|sed -f scripts/bdoc_1.sed |python3 scripts/help.py /tmp/$$.json ${i}
        mv /tmp/bakpof.txt ../trans/totrans_help_${i}.txt
        rm -f /tmp/$$.json
done
cd ..

echo edit pause
read dummy

for i in $LANGS ; do
    for j in _help_ ; do # _
        xclip -i -selection clipboard <trans/totrans${j}${i}.txt
        wl-copy <trans/totrans${j}${i}.txt
        lines=`wc -l <trans/totrans${j}${i}.txt`
        lines=$((lines - 2))
        echo "Paste ${j}${i} translation, $lines lines:"
        rm -f trans/transed${j}${i}.txt
        while true ; do
                read line
                test "$line" == "EOT" && break
                echo "$line" >>trans/transed${j}${i}.txt
        done
        echo "Paste retrans:"
        rm -f trans/retrans${j}${i}.txt
        while true ; do
                read line
                test "$line" == "EOT" && break
                echo "$line" >>trans/retrans${j}${i}.txt
        done
   done
done

for i in $LANGS ; do
    for j in _ _help_ ; do
        python txts2frontpof.py trans/totrans${j}${i}.txt trans/transed${j}${i}.txt trans/retrans${j}${i}.txt trans/newtrans${j}${i}.po
        test "$j" == _ && outpre=system_ || outpre=helptext_
        msgcat po/${outpre}${i}.po trans/newtrans${j}${i}.po >trans/newfull${j}${i}.po
        #cp -p trans/newfull${j}${i}.po po/${outpre}${i}.po
    done
done

bakpof.py:

import sys
import polib
pofin = polib.pofile(sys.argv[1])
for e in pofin:
    if '<<' in e.msgid:
        continue
    print(e.msgid)
    print('\n=====\n')

txts2frontpof.py:

import sys
import polib
pofout = polib.POFile(encoding='utf-8')
pofout.metadata = {
    'Content-Type': 'text/plain; charset=utf-8',
}

in1_r = open(sys.argv[1], 'r').read().lstrip('=').rstrip('\n').split('\n\n=====\n\n')
in2_r = open(sys.argv[2], 'r').read().lstrip('=').rstrip('\n').split('\n\n=====\n\n')
in_re_r = open(sys.argv[3], 'r').read().lstrip('=').rstrip('\n').split('\n\n=====\n\n')

in1 = []
in2 = []
in_re = []
for i in range(len(in1_r)):
    if in1_r[i] in in1:
        continue
    in1.append(in1_r[i])
    in2.append(in2_r[i])
    in_re.append(in_re_r[i])
        
for idx in range(0, len(in2)):
    in_msg = in1[idx]
    out_msg = in2[idx]
    re_msg = in_re[idx]
    e = polib.POEntry(msgid=in_msg, msgstr=out_msg, comment=re_msg)
    pofout.append(e)

pofout.save(sys.argv[4])

Hope that helps.

CityAceE

uli These are the scripts that I'm currently using

Thank you very much!

I have plans to bring BASIC Engine to the attention of the Russian-speaking community and I have some resources for that.
I am the founder and administrator of the main Russian-language forum about ZX-Spectrum and other retro-compatibles (zx-pk.ru). But first I want to add Russian to the system and I want to do it in the best possible way. But, unfortunately, I have come across the fact that there are some errors in the translation code. And without fixing the code in the repository it is impossible to do what I want. But, as I realize, any changes will affect other languages as well.

I've been struggling for a long time to get a PO file with the most complete contents of the lines for translation. As a result, I came to your help.py script. This script uses the Python translate library to create json files for different languages, which are then used by the system to output help on commands. I captured everything that is sent to translate. This helped me to overcome some problems and find out why some strings were not being translated.

For example, the line

volume envelope [characters fromCHR$(0)toCHR$(15), default: constant volume]

is actually the string

volume envelope\t[characters fromCHR$(0)toCHR$(15), default: constant volume]

i.e. there is an extra tab character in the middle.

The line is cut off

* [[unclear]] Some key definitions are not well-defined. The kbd:[Pipe2] key is documented as belonging to a US layout keyboard, but US layouts do not have a separate kbd:[|] key.

by the “|” character, because it consider by script as tab.

There are three strings related to the table in the code:
\table \table header \endtable

This code will corrupt the “\table header” tag, turning it into an untranslatable “=== header” tag:

def colorize_code(d):

d = re.sub('`([^`]+)`', '\\\\\\\\Fn\g<1>\\\\\\\\Ff', d)
d = re.sub('<<([^`]+)>>', '\\\\\\\\Fn\g<1>\\\\\\\\Ff', d)
d = re.sub('(IMPORTANT:)', '\\\\\\\\Fp\g<1>\\\\\\\\Ff', d)
d = re.sub('(WARNING:)', '\\\\\\\\Fp\g<1>\\\\\\\\Ff', d)
d = re.sub('kbd:\[([^\]]*)]', '\\\\\\\\Fp[\g<1>]\\\\\\\\Ff', d)
d = re.sub('\\\\endtable', '===', d)
d = re.sub('\\\\table', '===', d)
d = re.sub('====\n----\n', '\\\\\\\\Fn', d)
d = re.sub('----\n====\n', '\\\\\\\\Ff\\n', d)
return d

And some strings are not sent for translation at all because of a script error, but are written to the json file as they are (in English).

Frankly speaking, my hands are down, because I see that without the participation of the author of the project I can not achieve a good result. And this forum is not the best place to solve issues promptly.

I found other problems and bugs that I didn't mention. I would like them to be fixed.

I think that while I have a lot of enthusiasm, we could work on the project together. If you are interested, write to me in Telegram @CityAceE

Dmian

CityAceE without the participation of the author of the project I can not achieve a good result

Just for the record, @uli is the author and sole developer of BASIC Engine.

Maybe it will help if changes are submitted to the GitHub repository?
I agree this forum is not the best way to improve the project. Bugs can be put as issues, and code as pull requests, for review.
I know Ulrich hasn’t a lot of time to dedicate, but I guess using proper development tools may be useful.
Cheers!

CityAceE

Dmian Just for the record, @uli is the author and sole developer of BASIC Engine.

I certainly know who Ulrich is. I am grateful to him for this project and for his enthusiasm.

Dmian Maybe it will help if changes are submitted to the GitHub repository?

Yeah, I was already thinking that would be better too. At the moment I'm running into problems in various places in the system. I think it's because the project is moving from one platform to another, but there are not enough users to test everything reliably.

uli

Dmian I know Ulrich hasn’t a lot of time to dedicate

I am, in fact, off for vacation very soon, so don't expect any immediate reactions for a while. Hope the last code push actually builds successfully... 😄

CityAceE

I forgot to attach the translation template yesterday, which absolutely contains all the strings that are sent for translation. If any strings aren't there, it means they won't be translated even if you add them manually. So, for example, there are no arguments for the BG command in this file.

{ "command": "BG", "tokens": [ "BG" ], "section": "bg", "type": "bc", "brief": "Defines a tiled background's properties.", "desc": "Using \\FnBG\\Ff, you can define a tiled background's map and tile size, tile set, window size and position, and priority, as well as turn it on or off.\n\\FnBG OFF\\Ff turns off all backgrounds.", "usage": "BG bg [TILES w, h] [PATTERN px, py, pw] [SIZE tx, ty] [WINDOW wx, wy, ww, wh]\n [PRIO priority] [<ON|OFF>]\nBG OFF", "args": { "bg": "background number [\\Fn0\\Ff to \\Fn{MAX_BG_m1}\\Ff]", "w": "background map width, in tiles", "h": "background map height, in tiles", "px": "tile set's X coordinate, pixels [\\Fn0\\Ff to \\FnPSIZE(0)-1\\Ff]", "py": "tile set's Y coordinate, pixels [\\Fn0\\Ff to \\FnPSIZE(2)-1\\Ff]", "tx": "tile width, pixels [\\Fn8\\Ff (default) to \\Fn255\\Ff]", "ty": "tile height, pixels [\\Fn8\\Ff (default) to \\Fn255\\Ff]", "wx": "window X coordinate, pixels [\\Fn0\\Ff (default) to \\FnPSIZE(0)-1\\Ff]", "wy": "window Y coordinate, pixels [\\Fn0\\Ff (default) to \\FnPSIZE(1)-1\\Ff]", "ww": "window width, pixels [\\Fn9\\Ff to \\FnPSIZE(0)-wx\\Ff (default)]", "wh": "window height, pixels [\\Fn0\\Ff to \\FnPSIZE(1)-wy\\Ff (default)]", "priority": "Background priority [\\Fn0\\Ff to \\Fn{MAX_BG_m1}\\Ff, default: \\Fnbg\\Ff]" }, "note": "* The \\FnBG\\Ff command's attributes can be specified in any order, but it is usually a good idea to place the \\FnON\\Ff attribute at the end if used.\n* A background cannot be turned on unless at least the \\FnTILES\\Ff attribute is set. In order to get sensible output, it will probably be necessary to specify the \\FnPATTERN\\Ff attribute as well.", "bugs": "If a background cannot be turned on, no error is generated.", "ref": [ "LOAD BG", "LOAD PCX", "MOVE BG", "SAVE BG" ] },
And it's not the only place in the program that currently can't be translated.

helptext-template.zip

22kB

CityAceE

uli Hope the last code push actually builds successfully... 😄

Yeah, I hope so, too. The major problems have been fixed. I'm starting the final polishing of the Russian translation.

Dmian

uli I am, in fact, off for vacation very soon

Have a great time, and enjoy your vacations! ☺️