Getting gettext & translation into Allacrost

For discussion of the code running behind the game

Moderator: Staff

User avatar
gorzuate
Developer
Posts: 2575
Joined: Thu Jun 17, 2004 3:03 am
Location: Hermosa Beach, CA
Contact:

Re: Getting gettext & translation into Allacrost

Postby gorzuate » Mon Feb 01, 2010 9:07 am

Updated the developer section on the Localization page on the wiki and also renamed it something more intuitive. Read it over and let me know if you have any questions or if you think something should be added (to the developer section, I'm still working on the translator section).

http://allacrost.sourceforge.net/wiki/index.php/Language_Translations

roots asked in another thread why the Translate function was in utils and not system. system does initialize gettext and handle the setting of the languages' environment variables. The Translate function already existed in utils when I started work on it (I think mooshicow put it there). I dunno, I think it's fine where it is, but if you want to move it to system, be my guest. In my opinion, it is a utility.
Image
rujasu
Developer
Posts: 758
Joined: Sun Feb 25, 2007 5:40 am
Location: Maryland, USA

Re: Getting gettext & translation into Allacrost

Postby rujasu » Mon Feb 01, 2010 2:29 pm

Looks great! :approve:
User avatar
Roots
Dictator
Posts: 8665
Joined: Wed Jun 16, 2004 6:07 pm
Location: Austin TX
Contact:

Re: Getting gettext & translation into Allacrost

Postby Roots » Mon Feb 01, 2010 4:31 pm

I read the whole page and I agree, it looks great. I think it might be useful to add a mini-FAQ for the developer (and maybe another one for translators) because this documentation still doesn't answer all my questions. But I don't want to replicate stuff that can be found in the gettext manual, so I think a lot of the answers can simply be a link to the appropriate section in the manual. My questions include...

  • Since the string itself serves as a look-up key, is it alright if you have the same string in two different places in a file? (I noticed this yesterday with "Attack" used for two different initialization functions in battle_windows.cpp)
  • What about initializing data structures like arrays with strings? (I found the answer

Also is there a preferred "way" for us to define our translatable strings in the code? For example, having all strings used in a file defined at the top along with their gettext calls, and then reference the string variables where they are used in the code? :shrug:

gorzuate wrote:roots asked in another thread why the Translate function was in utils and not system. system does initialize gettext and handle the setting of the languages' environment variables. The Translate function already existed in utils when I started work on it (I think mooshicow put it there). I dunno, I think it's fine where it is, but if you want to move it to system, be my guest. In my opinion, it is a utility.


I prefer that we keep the same color eggs in the same basket (gettext calls all located in the same area). I don't have a huge problem with it being located in utils.h instead of system.h, but I do think system.h is a more natural home for it. I disagree that its utility code. If you want to say that that is utility, you might as well say that the threading functions (which are in system.h) are utility as well, since they are a wrapper around a library call. So I'd prefer to move the call but if there isn't a consensus that agrees with me, I won't press the issue.
Image
User avatar
Roots
Dictator
Posts: 8665
Joined: Wed Jun 16, 2004 6:07 pm
Location: Austin TX
Contact:

Re: Getting gettext & translation into Allacrost

Postby Roots » Fri Feb 05, 2010 6:34 am

I'm sure I've asked this question before, but I forgot what the answer was and I can't find it. Why does Translate() return a string and not a ustring?

Code: Select all

/** \brief Returns a translated string (return value of the gettext() function)
*** \param text The string that needs to be translated
*** \return A translated text in the form of a ustring
***
*** This function is used to call the gettext() function, such that,
*** for different language modes different text will be displayed.
**/
std::string Translate(const std::string& text);
//hoa_utils::ustring Translate(const std::string& text);


That's in utils.h. It teases me thinking that there's a version for ustring's but its not implemented. I can understand why string is necessary I guess because Lua code needs to translate too and Lua is oblivious to the ustring class. In fact this brings to mind other questions. Should we make the ustring class available to Lua? If we have two translate functions, one for strings and one for ustrings, should they both be named Translate() or would that cause naming conflicts? What about using UTranslate() as a shorthand for "MakeUnicodeString(Translate())"?
Image
User avatar
gorzuate
Developer
Posts: 2575
Joined: Thu Jun 17, 2004 3:03 am
Location: Hermosa Beach, CA
Contact:

Re: Getting gettext & translation into Allacrost

Postby gorzuate » Fri Feb 05, 2010 6:41 am

Slow down. You have asked it before and I haven't forgotten about it. ;)

That comment you see is something left over from my debugging stuff. No such function currently exists. I tried it once obviously but can't remember why I chose one over the other so I'll have to run a test first before answering you. (I combed through this thread last night hoping for an answer but didn't find anything). I think it might be related to Lua though.

Translate() can be moved to system if you wish.
Image
User avatar
Roots
Dictator
Posts: 8665
Joined: Wed Jun 16, 2004 6:07 pm
Location: Austin TX
Contact:

Re: Getting gettext & translation into Allacrost

Postby Roots » Fri Feb 05, 2010 8:27 am

gorzuate wrote:I think it might be related to Lua though.


I'm pretty dang sure it is. Lua needs a bound C++ function for gettext to use and if Translate returns a ustring, well Lua doesn't know what to do with a ustring because it has no idea what it is. So you can't assign something like `name = hoa_utils.Translate("Claudius")` if Translate returns a ustring.

gorzuate wrote:Translate() can be moved to system if you wish.


I think I will do so.
Image
User avatar
gorzuate
Developer
Posts: 2575
Joined: Thu Jun 17, 2004 3:03 am
Location: Hermosa Beach, CA
Contact:

Re: Getting gettext & translation into Allacrost

Postby gorzuate » Sat Feb 06, 2010 9:57 pm

Added another section describing how gettext uses strings as keys (this should answer some of Roots' questions):
http://allacrost.sourceforge.net/wiki/index.php/Language_Translations#Strings_as_Keys

I don't feel like running the ustring/string test, so I'm just going to assume that we need a string version of Translate for Lua.

We need to decide on a comment symbol so gettext knows which comments are meant for translators. The current example on the wiki uses ///. Basically, it needs to be // followed by some other character(s).

There is nothing special that needs to be done with memory management of strings, C-strings, and ustrings. I know Roots asked that question too. gettext takes a char* string as input and returns a char* string. Since we use C++ strings, we just use the c_str() function to convert a std::string to a char* string to pass to gettext, then use the std::string(char* text) constructor to create a std::string from the returned char* string of gettext. Our MakeUnicodeString() and MakeStandardString() functions then handle everything needed to convert them back and forth.

I don't see the need to make ustrings available to Lua. Unless you just want one translating function that always returns unicode strings rather than two, one returning strings and the other ustrings. I think we're fine with 2 versions of the function.
Image
User avatar
Roots
Dictator
Posts: 8665
Joined: Wed Jun 16, 2004 6:07 pm
Location: Austin TX
Contact:

Re: Getting gettext & translation into Allacrost

Postby Roots » Sat Feb 06, 2010 11:30 pm

gorzuate wrote:Added another section describing how gettext uses strings as keys (this should answer some of Roots' questions):
http://allacrost.sourceforge.net/wiki/index.php/Language_Translations#Strings_as_Keys

gorzuate wrote:I don't feel like running the ustring/string test, so I'm just going to assume that we need a string version of Translate for Lua.


Fine with me.

gorzuate wrote:We need to decide on a comment symbol so gettext knows which comments are meant for translators. The current example on the wiki uses ///. Basically, it needs to be // followed by some other character(s).


Well both /// and //! are special comments for doxygen. How about //@ ? (Actually doxygen uses //@{ and //@}, but I'm not sure about //@).

Here's a question though. What about Lua? Lua comments are -- instead of "//" and "--[[ --]]" instead of "/* */". Would gettext commenting work if our Lua comments looked like this? "--//@"?

gorzuate wrote:I don't see the need to make ustrings available to Lua. Unless you just want one translating function that always returns unicode strings rather than two, one returning strings and the other ustrings. I think we're fine with 2 versions of the function.


As long as std::string can always hold the characters we need in any language (including non-Latin languages like Hebrew, Arabic, Chinese) I see no problem at all. The ustring class can simply be for the graphics code to render our translated strings appropriately.
Image
User avatar
gorzuate
Developer
Posts: 2575
Joined: Thu Jun 17, 2004 3:03 am
Location: Hermosa Beach, CA
Contact:

Re: Getting gettext & translation into Allacrost

Postby gorzuate » Sun Feb 07, 2010 5:57 am

Updated the wiki with the new make command that devs should be running and with descriptions for the new LINGUAS file and more on how comments work.

Currently comments do work in the cpp files if done like so:

Code: Select all

//TRANSLATORS: Hello there, fellow translator!
blahblah UTranslate(bloop) blahblah


Note no space between // and TRANSLATORS:.

So it looks like we don't need a special symbol, just that above.

No matter what I tried though, I could not get it to extract comments from the Lua files. I suspect it's due to the warnings xgettext spits out for the Lua files: "unrecognized file extension .lua, trying C".
So I think it's scanning the Lua files as if they were C files, so it's looking for C-style comments. Not Lua comments.
xgettext has an option to specify the language in use, but it only takes supported languages, and it doesn't support Lua. If the language isn't specified, it tries to guess based on the file extension.

The --add-comments=-- didn't work either.
Not sure what to do about this :|
Image
User avatar
prophile
Senior Member
Posts: 324
Joined: Fri Jan 27, 2006 7:18 pm
Location: Chaldon, Surrey, UK
Contact:

Re: Getting gettext & translation into Allacrost

Postby prophile » Sun Feb 07, 2010 7:27 pm

Why not patch xgettext?
Alastair Lynn / Resident Whinger / Allacrost
User avatar
gorzuate
Developer
Posts: 2575
Joined: Thu Jun 17, 2004 3:03 am
Location: Hermosa Beach, CA
Contact:

Re: Getting gettext & translation into Allacrost

Postby gorzuate » Sun Feb 07, 2010 10:09 pm

Updated the wiki with instructions for devs to see translations in game.
http://allacrost.sourceforge.net/wiki/index.php/Language_Translations#Viewing_Translations_in-Game

Being able to view the nice-looking quote characters (left and right quotes) automatically is done via the en@quot.po file, which is updated automatically via the make update-po command, and the System SingletonInitialize sets the default English language to that "translation".

prophile wrote:Why not patch xgettext?


That's for crazy old fools and 18-yr olds ;)

I did find a tool called lua-xgettext but can't download it as the server seems to be offline.
Image
User avatar
Roots
Dictator
Posts: 8665
Joined: Wed Jun 16, 2004 6:07 pm
Location: Austin TX
Contact:

Re: Getting gettext & translation into Allacrost

Postby Roots » Mon Feb 08, 2010 6:11 am

When I went through and made all the game modes call translate for their text, I noticed a few things about how we were using display text that don't seem appropriate. Here are some notes of the problems I encountered and possible ideas I thought about.


1) Constructing strings with dynamic numeric data.

For example: "HP: 22 / 45". We can't translate that string because its not static. The numeric values in this text change. We only want to translate the "HP: " part. Now in the code I went through, there were a lot of std::strings being constructed using string streams, then passing the constructed string into the MakeUnicodeString() function for displaying it. I had to remove this string construction and just use string concatenation with the different string segments instead (which is actually a little easier to do than managing all these string streams).

In retrospect I could have kept the string streams by just calling Translate() (not UTranslate()) on the static part that was added to the string stream and then done a MakeUnicodeString call around the constructed string when passing it to the display call.


2) Redundant strings

In some files there are a redundant number of the same string that have to be used in the construction of different text images and GUI displays. For example, in shop_buy.cpp the text "Buy" appears in multiple locations and I had to call translate on each of those. This is unfortunate because obviously the same text used in the same context uses the same translation, but if there's multiple translate calls at different locations in the file I think the translator has to translate it each time. (If I'm wrong about this, correct me Phil).


3) Idea: put all translatable text content at the top of every .cpp file

Issue #2 caused me to wonder if we shouldn't be listing all of our static string data that requires translation at the top of every .cpp file. This way we could re-use these string constants so that they wouldn't need to be translated multiple times. The disadvantage is it would make the code less readable since you are seeing the code display text like so: "_data_text.Draw(UTranslate(MY_STRING_DATA));" instead of "_data_text.Draw(UTranslate("Data value is valid"));".


4) Don't use ustrings, MakeUnicodeString(), or UTranslate() anywhere outside of video engine code.

Woah, now where did this crazy proposal come from? Well, I realized that if std::string can contain all the translated text by itself, then the only thing that the ustring class is good for is for displaying that text. And we (should be) replacing all of our "VideoManager->Text()->Draw("This line of text");" calls with _text_image objects or passing those strings into GUI objects. Both _text_image objects and all GUI objects keep an internally stored ustring to display, so these classes can take a std::string argument and convert and store it to a ustring by itself. This is pretty convenient for us, and it means that all the features in std::string are available for us to use in text manipulation. Maybe I'm forgetting something here, but I don't see a reason why we have to use ustrings outside of the video engine code.


5) Accounting for numerics in strings.

Here's a special case from map_treasure.cpp:

Code: Select all

_detail_textbox.SetDisplayText(UTranslate("With the additional ") + MakeUnicodeString(NumberToString(_treasure->_drunes)) +
     UTranslate(" drunes found in this treasure added, the party now holds a total of ") + MakeUnicodeString(NumberToString(GlobalManager->GetDrunes()))


As you can see, the static parts of the string are translated by the numerics are not. This isn't good because some languages will want to place the numerics at a different position. Or the order of the numerics may be reversed.

I read gettext's manual and it accounts for situations like this. We need to figure out how to use this feature for our side as well. Perhaps we put "%0" and "%1" into the string to represent the numeric arguments we put in there (I think that's what the gettext manual shows). Maybe the Translate/UTranslate function needs to take a ... argument to allow a variable number of additional arguments to write data into the string, similar to how printf() works.


-------

Finally, I'd like to make it clear that the current translation calls in the game mode code are messy. They don't follow the suggested ideas I outlined here. I just wanted to get this rolling so we could translate the game and come back and tidy up these string manipulations and translations once we have a better plan for how we're going to handle display text in the code.
Image
User avatar
gorzuate
Developer
Posts: 2575
Joined: Thu Jun 17, 2004 3:03 am
Location: Hermosa Beach, CA
Contact:

Re: Getting gettext & translation into Allacrost

Postby gorzuate » Mon Feb 08, 2010 6:59 am

Roots wrote:2) Redundant strings

In some files there are a redundant number of the same string that have to be used in the construction of different text images and GUI displays. For example, in shop_buy.cpp the text "Buy" appears in multiple locations and I had to call translate on each of those. This is unfortunate because obviously the same text used in the same context uses the same translation, but if there's multiple translate calls at different locations in the file I think the translator has to translate it each time. (If I'm wrong about this, correct me Phil).


Nope, you're wrong, so I'm correcting you. The translator only has to translate it once, because it only shows up in the PO file once. gettext is that good. It will only show up in the PO file more than once if the same string is used in more than one source code file, or they use different contexts (in which case they do need to be translated differently).
Reread this section, and if it still isn't clear then I need to reword it:
http://allacrost.sourceforge.net/wiki/index.php/Language_Translations#Strings_as_Keys

Roots wrote:3) Idea: put all translatable text content at the top of every .cpp file

Issue #2 caused me to wonder if we shouldn't be listing all of our static string data that requires translation at the top of every .cpp file. This way we could re-use these string constants so that they wouldn't need to be translated multiple times. The disadvantage is it would make the code less readable since you are seeing the code display text like so: "_data_text.Draw(UTranslate(MY_STRING_DATA));" instead of "_data_text.Draw(UTranslate("Data value is valid"));".


Nullified by my previous statement. Boo yeah :!:

Roots wrote:4) Don't use ustrings, MakeUnicodeString(), or UTranslate() anywhere outside of video engine code.

Woah, now where did this crazy proposal come from? Well, I realized that if std::string can contain all the translated text by itself, then the only thing that the ustring class is good for is for displaying that text. And we (should be) replacing all of our "VideoManager->Text()->Draw("This line of text");" calls with _text_image objects or passing those strings into GUI objects. Both _text_image objects and all GUI objects keep an internally stored ustring to display, so these classes can take a std::string argument and convert and store it to a ustring by itself. This is pretty convenient for us, and it means that all the features in std::string are available for us to use in text manipulation. Maybe I'm forgetting something here, but I don't see a reason why we have to use ustrings outside of the video engine code.


Fine by me. :approve:
Image
User avatar
gorzuate
Developer
Posts: 2575
Joined: Thu Jun 17, 2004 3:03 am
Location: Hermosa Beach, CA
Contact:

Re: Getting gettext & translation into Allacrost

Postby gorzuate » Sat Mar 06, 2010 9:17 pm

Image
User avatar
gorzuate
Developer
Posts: 2575
Joined: Thu Jun 17, 2004 3:03 am
Location: Hermosa Beach, CA
Contact:

Re: Getting gettext & translation into Allacrost

Postby gorzuate » Mon Mar 15, 2010 12:20 am

Roots wrote:4) Don't use ustrings, MakeUnicodeString(), or UTranslate() anywhere outside of video engine code.

Woah, now where did this crazy proposal come from? Well, I realized that if std::string can contain all the translated text by itself, then the only thing that the ustring class is good for is for displaying that text. And we (should be) replacing all of our "VideoManager->Text()->Draw("This line of text");" calls with _text_image objects or passing those strings into GUI objects. Both _text_image objects and all GUI objects keep an internally stored ustring to display, so these classes can take a std::string argument and convert and store it to a ustring by itself. This is pretty convenient for us, and it means that all the features in std::string are available for us to use in text manipulation. Maybe I'm forgetting something here, but I don't see a reason why we have to use ustrings outside of the video engine code.


This should take care of all the "headaches" you've come across, Roots. In the other thread where I said we've already chosen UTF-16, I was referring to the rendering code in the video engine, not gettext. Your proposal here for only the video engine needing to know about ustrings should work.
Image
User avatar
Roots
Dictator
Posts: 8665
Joined: Wed Jun 16, 2004 6:07 pm
Location: Austin TX
Contact:

Re: Getting gettext & translation into Allacrost

Postby Roots » Mon Mar 15, 2010 2:40 am

Ok cool. :approve:
Image

Return to “Programming”

Who is online

Users browsing this forum: No registered users and 2 guests