Getting gettext & translation into Allacrost

For discussion of the code running behind the game

Moderator: Staff

Linds
Developer
Posts: 145
Joined: Tue Jan 09, 2007 9:21 am
Location: Sydney, Australia

Re: Getting gettext & translation into Allacrost

Postby Linds » Fri Mar 06, 2009 10:10 am

Well in the off chance this is an endianness issue I've checked in code to use the correct endianness in the conversion. Its possible because of the existence of the Byte Order Mark there is a default conversion to little endian. Let me know how that goes for you.

As for the other error, it seems like one of the filenames we use is being converted to a ustring? As far as I know that shouldn't be the case as we aren't going to start naming our internal data files in high unicode any time soon.
User avatar
gorzuate
Developer
Posts: 2575
Joined: Thu Jun 17, 2004 3:03 am
Location: Hermosa Beach, CA
Contact:

Re: Getting gettext & translation into Allacrost

Postby gorzuate » Fri Mar 06, 2009 7:09 pm

Linds wrote:Well in the off chance this is an endianness issue I've checked in code to use the correct endianness in the conversion. Its possible because of the existence of the Byte Order Mark there is a default conversion to little endian. Let me know how that goes for you.


Um, well, I'll give it a try, but since I have an Intel chip I don't think it's an endian-related issue.

Linds wrote:As for the other error, it seems like one of the filenames we use is being converted to a ustring? As far as I know that shouldn't be the case as we aren't going to start naming our internal data files in high unicode any time soon.


Yeah, seems like either the map file itself or a file it references...
Image
User avatar
gorzuate
Developer
Posts: 2575
Joined: Thu Jun 17, 2004 3:03 am
Location: Hermosa Beach, CA
Contact:

Re: Getting gettext & translation into Allacrost

Postby gorzuate » Mon Mar 16, 2009 12:01 am

Linds wrote:Well in the off chance this is an endianness issue I've checked in code to use the correct endianness in the conversion. Its possible because of the existence of the Byte Order Mark there is a default conversion to little endian. Let me know how that goes for you.

As for the other error, it seems like one of the filenames we use is being converted to a ustring? As far as I know that shouldn't be the case as we aren't going to start naming our internal data files in high unicode any time soon.


Well, the change in the code seems straightforward, but it produced the following when I ran it (this is what I got as soon as the game got to the Boot screen):
Picture 2.png
Picture 2.png (724.98 KiB) Viewed 6258 times


When I selected New Game, it crashed with the following errors:

Code: Select all

MODE MANAGER: GameMode constructor invoked
WARNING:/Users/philip/allacrost/demo/src/engine/video/option.cpp:SetSelection:379: argument was invalid (out of bounds): 0
SCRIPT ERROR: ReadScriptDescriptor::OpenFile() could not open the file at/maps/demo_town.lua
cannot read at/maps/demo_town.lua: No such file or directory
BOOT: BootMode destructor invoked.
MODE MANAGER: GameMode destructor invoked
Program received signal:  “EXC_BAD_ACCESS”.


The filename it should be looking for should have either "dat" or "." in the beginning (can't remember which, although it's probably "."). Somehow the first character of I imagine any string converted from UTF8 to UTF16 is getting cut off.
Image
User avatar
prophile
Senior Member
Posts: 324
Joined: Fri Jan 27, 2006 7:18 pm
Location: Chaldon, Surrey, UK
Contact:

Re: Getting gettext & translation into Allacrost

Postby prophile » Mon Mar 16, 2009 12:12 am

This is why you should have used UTF8 :)
Alastair Lynn / Resident Whinger / Allacrost
Linds
Developer
Posts: 145
Joined: Tue Jan 09, 2007 9:21 am
Location: Sydney, Australia

Re: Getting gettext & translation into Allacrost

Postby Linds » Tue Mar 17, 2009 12:56 pm

gorzuate wrote:The filename it should be looking for should have either "dat" or "." in the beginning (can't remember which, although it's probably "."). Somehow the first character of I imagine any string converted from UTF8 to UTF16 is getting cut off.

Damnit it looks like the Mac version of gettext isn't giving us the BOM and the linux version is. Ok I'll check in the change detecting the BOM by its character ID(s).

prophile: I think it says more about the quality of the function I hacked in to test if it the text code was working properly ;)
Linds
Developer
Posts: 145
Joined: Tue Jan 09, 2007 9:21 am
Location: Sydney, Australia

Re: Getting gettext & translation into Allacrost

Postby Linds » Wed Mar 18, 2009 9:41 am

Ok, its done. Let me know how this goes for you gorz.
User avatar
gorzuate
Developer
Posts: 2575
Joined: Thu Jun 17, 2004 3:03 am
Location: Hermosa Beach, CA
Contact:

Re: Getting gettext & translation into Allacrost

Postby gorzuate » Thu Mar 19, 2009 7:04 am

Image :) :approve: :D :bow: Image

Woohoo :!: Hurrah :!: It works. Thanks Linds.

Alright, now I just have to figure out how to efficiently translate all desired strings and maybe do some debug clean up. Hmm, and some way to refresh/flush the system that's holding these text strings so the user can switch languages in the middle of running the game instead of right at the beginning (unless you guys don't think that's necessary, although it does seem a bit strange when you select a new language and nothing happens right away... :uhoh: )
Image
User avatar
gorzuate
Developer
Posts: 2575
Joined: Thu Jun 17, 2004 3:03 am
Location: Hermosa Beach, CA
Contact:

Re: Getting gettext & translation into Allacrost

Postby gorzuate » Sun Mar 22, 2009 4:43 am

Eek! Spoke too soon :(

This is what it looks like on a PPC Mac (Intel Macs work great):
Image

I'm still waiting to see if it works on Windows properly.
Image
Winter Knight
Contributor
Posts: 304
Joined: Fri Sep 21, 2007 12:35 pm
Contact:

Re: Getting gettext & translation into Allacrost

Postby Winter Knight » Mon Mar 23, 2009 7:30 pm

gorzuate wrote:Eek! Spoke too soon :(

OK. Could you elaborate on what doesn't work? That picture looks fine to me.
gorzuate wrote:I'm still waiting to see if it works on Windows properly.

How do you switch languages in Windows? For that matter, do we have a wiki page that says how to use, test, and help out with Allacrost gettext? We should.
User avatar
gorzuate
Developer
Posts: 2575
Joined: Thu Jun 17, 2004 3:03 am
Location: Hermosa Beach, CA
Contact:

Re: Getting gettext & translation into Allacrost

Postby gorzuate » Mon Mar 23, 2009 11:46 pm

Winter Knight wrote:
gorzuate wrote:Eek! Spoke too soon :(

OK. Could you elaborate on what doesn't work? That picture looks fine to me.


Erm, well, the apostrophes should not be there. They should be accents over the 'e' coming after them like this: é

Winter Knight wrote:
gorzuate wrote:I'm still waiting to see if it works on Windows properly.

How do you switch languages in Windows? For that matter, do we have a wiki page that says how to use, test, and help out with Allacrost gettext? We should.


There is a wiki page though it's still a work in progress. On the wiki search for localization. It doesn't really have anything specific to Windows on it because I don't know how to do any of this stuff on Windows. To switch languages just go to the Options->Language menu in the boot screen and select French. You will probably need this file (though it is a binary file, it may need to be created by yourself) and you will probably have to put it in txt/fr/LC_MESSAGES/.
Image
Linds
Developer
Posts: 145
Joined: Tue Jan 09, 2007 9:21 am
Location: Sydney, Australia

Re: Getting gettext & translation into Allacrost

Postby Linds » Tue Apr 07, 2009 11:51 am

gorzuate wrote:Eek! Spoke too soon :(

This is what it looks like on a PPC Mac (Intel Macs work great):
Image

I'm still waiting to see if it works on Windows properly.

What the heck!?

My only assumption could be the unicode creators thought it would be somehow poignant to have the UTF16 sequence for accented e be accent, e. That would make this some kind of strange error where somewhere down the chain UTF16 is being received where UTF8 is expected. But it seems too functional for this great a mistake...

Will have to examine the path further...
User avatar
Drakkoon
Developer
Posts: 173
Joined: Thu Jan 11, 2007 12:54 am
Location: Montréal, Qc

Re: Getting gettext & translation into Allacrost

Postby Drakkoon » Thu Apr 09, 2009 10:26 pm

At first I thought that it could have been caused by the fact that an è is written in 2 keystrokes `, then e on some french keyboards, but then I realised that it was written inquiétée which really is a single keystroke on any french keyboard. (I would have written inquiète, we seldom see this word used like that in such a context.)

So yeah, no real input here, since I'm not familiar with the new unicode system. :) Haa the joy of posting just for the fun of it.
User avatar
gorzuate
Developer
Posts: 2575
Joined: Thu Jun 17, 2004 3:03 am
Location: Hermosa Beach, CA
Contact:

Re: Getting gettext & translation into Allacrost

Postby gorzuate » Fri Apr 10, 2009 12:51 am

I have a feeling it's somehow related to the following since the problem is only occurring on PPCs :shrug: :
Linds wrote:Damnit it looks like the Mac version of gettext isn't giving us the BOM and the linux version is.
Image
rujasu
Developer
Posts: 758
Joined: Sun Feb 25, 2007 5:40 am
Location: Maryland, USA

Re: Getting gettext & translation into Allacrost

Postby rujasu » Tue Sep 08, 2009 8:26 pm

Aside from the bug on PPC Macs, is the gettext functionality fairly close to being usable at this point? Did we ever get any indication as to how well it works on Windows? If we can get it to work on all non-PPC platforms, that's basically the vast majority, and I'd rather release a demo with localization working on 98% of machines than one with only English supported. Does anyone know what more would need to be done, aside from actually writing the translated dialogue? And is anyone currently willing/able to work on it? I think it would probably take some time before I could get to this.
User avatar
gorzuate
Developer
Posts: 2575
Joined: Thu Jun 17, 2004 3:03 am
Location: Hermosa Beach, CA
Contact:

Re: Getting gettext & translation into Allacrost

Postby gorzuate » Wed Sep 09, 2009 6:07 am

rujasu wrote:Aside from the bug on PPC Macs, is the gettext functionality fairly close to being usable at this point?

Depends on your definition of usable. Going by the basic definition in the dictionary, yes. It can display translated strings. Going by something that means "robust enough not to get any complaints from the types that will be downloading and playing the demo" then I would say probably not, but I'm most likely just being a perfectionist.

Take a look at Wesnoth and try switching languages. Does the user have to do anything other than select the newly desired language? No. The game automatically refreshes its text images. This is not the case with Allacrost. I think right now if you start a new game in one language and then try to switch to another language, you won't actually see the change until you restart Allacrost.

rujasu wrote:Did we ever get any indication as to how well it works on Windows?

No indication whatsoever :axe:
rujasu wrote:If we can get it to work on all non-PPC platforms, that's basically the vast majority, and I'd rather release a demo with localization working on 98% of machines than one with only English supported.

I agree :)
rujasu wrote:Does anyone know what more would need to be done, aside from actually writing the translated dialogue?

See above about refreshing. There might also be some debugging stuff in there that could be removed. And probably updating the wiki so translators actually know how to do the translating.
rujasu wrote:And is anyone currently willing/able to work on it? I think it would probably take some time before I could get to this.

I can try to find anything that needs to be cleaned up and update the wiki within the next 2 days. If I knew how to do the refreshing thing, or could even remotely figure it out, it would have been done a long time ago.
Image
rujasu
Developer
Posts: 758
Joined: Sun Feb 25, 2007 5:40 am
Location: Maryland, USA

Re: Getting gettext & translation into Allacrost

Postby rujasu » Wed Sep 09, 2009 6:26 am

gorzuate wrote:Depends on your definition of usable. Going by the basic definition in the dictionary, yes. It can display translated strings. Going by something that means "robust enough not to get any complaints from the types that will be downloading and playing the demo" then I would say probably not, but I'm most likely just being a perfectionist.

Take a look at Wesnoth and try switching languages. Does the user have to do anything other than select the newly desired language? No. The game automatically refreshes its text images. This is not the case with Allacrost. I think right now if you start a new game in one language and then try to switch to another language, you won't actually see the change until you restart Allacrost.


Hmm, okay. Personally, I'm willing to put it out there despite this issue -- if people are complaining, then at least they're showing interest. Of course, the sooner we can resolve this, the better.

rujasu wrote:Did we ever get any indication as to how well it works on Windows?

No indication whatsoever :axe:


Okay. Hopefully we can find someone willing to test things out on Windows in the near future.

I can try to find anything that needs to be cleaned up and update the wiki within the next 2 days. If I knew how to do the refreshing thing, or could even remotely figure it out, it would have been done a long time ago.


Okay, sounds good. Getting some documentation on the Wiki would be a great idea. :approve:
User avatar
Jetryl
Artist
Posts: 1485
Joined: Fri Aug 26, 2005 7:35 am
Location: Southern Minnesota, USA

Re: Getting gettext & translation into Allacrost

Postby Jetryl » Wed Sep 09, 2009 10:48 am

rujasu wrote:Aside from the bug on PPC Macs, is the gettext functionality fairly close to being usable at this point? Did we ever get any indication as to how well it works on Windows? If we can get it to work on all non-PPC platforms, that's basically the vast majority, and I'd rather release a demo with localization working on 98% of machines than one with only English supported. Does anyone know what more would need to be done, aside from actually writing the translated dialogue? And is anyone currently willing/able to work on it? I think it would probably take some time before I could get to this.



Rain and I still have PPC macs, but I wouldn't kill yourself trying to fix bugs for PPC. The usage-share of PPC macs is collapsing. I myself, for example, already have an intel laptop. By the time you get allacrost done (let's be honest, it's easily gonna be 2-4 years), PPC macs will be a thing of the past, and rain and I will have moved on to intel machines.

If it's easy, go ahead and do it, but if it's something intractable, your time is better spent actually getting the 'substance' of the game done.
Image
Winter Knight
Contributor
Posts: 304
Joined: Fri Sep 21, 2007 12:35 pm
Contact:

Re: Getting gettext & translation into Allacrost

Postby Winter Knight » Wed Sep 09, 2009 3:14 pm

I've asked this before, and I think gorzuate answered, but I forgot. How do we test gettext and translation? I can test it on Windows (and Linux), and report if it works or not.
User avatar
gorzuate
Developer
Posts: 2575
Joined: Thu Jun 17, 2004 3:03 am
Location: Hermosa Beach, CA
Contact:

Re: Getting gettext & translation into Allacrost

Postby gorzuate » Fri Sep 11, 2009 7:27 am

I'll let you know, Winter Knight, when I've got everything ready for testing. I have to update some of the trial translations for the new map now in use... and I won't be able to get to it until next week.
Image
User avatar
Roots
Dictator
Posts: 8666
Joined: Wed Jun 16, 2004 6:07 pm
Location: Austin TX
Contact:

Re: Getting gettext & translation into Allacrost

Postby Roots » Mon Sep 14, 2009 2:48 pm

I don't think it would be too terribly difficult to get the refresh thing working. There are two types of rendered strings in Allacrost: text images and instant text. Text images use an OpenGL texture that stores the rendered text, and when drawn this type of text just displays that texture. Instant text has to render the text completely from scratch every frame in order to draw it. And FYI, we eventually want to move 99% of our text rendering to text images, since the performance penalty is much lower (only render once instead of every frame). Instant text is really only meant for debugging purposes.

To get refresh working for text images...
  • All text images store an internal ustring, so we'd need to go through every text image and give it the new ustring value containing the text in the new language
  • We'd have to go through all text images and tell them to re-render their image texture

That's it. Its basically a single for loop that will iterate through all text images to both update the internal string member and re-render the texture. The hard part is figuring out how to get the new translated ustring that we want to render.


To get refresh working for instant text...
Actually I think this should already work. Since we re-render the text every frame, if we change the value of the ustring that we are using to render, the corresponding rendered text should reflect the new translated string.
Image

Return to “Programming”

Who is online

Users browsing this forum: No registered users and 1 guest