8 replies to this topic

[Goofy]

This topic is dedicated to discussion and maybe realization of the idea as initially displayed here
http://babelwiki.babelzilla.org/index.php?...testing_machine

I am glad to say that this dream may have a little chance to come to reality, since I received encouraging feedback from Davide Ficano (dafi) and Martijn Kooij (Captain Caveman) not to mention Fenian himself. Of course it is certainly a huge work and I don't expect it can be done as quickly as an extension.

I paste here some significant observations. Everyone is welcome to comment, suggest, and why not, contribute


From dafi

what you say can be easily done (at least 90%) directly on BZ using the scripts I've written to parse install.rdf, it is easy to add support for chrome.manifest.
I can try to develop a webapp, I'm not sure developing an extension is the correct way, in this case a server side application sound better.

From Captain Caveman

Practically:
In my oh so humble opinion this would be a bad idea. Because it puts the responsibility of a good localization away from the localizer into the hands of the developer. So if at all, it would seem better to create an automated way for a localizer to test his/her own work. And since a localizer should already feel responsible to test his own work, I don't think this process can be made any easier for him/her.

Theoretically:
- I think a prerequisite must be some extension being installed in the testing Firefox to make a couple of things easier (installing addons, reporting problems)
- An application (either web/client) could process the xpi, and parse the required information as needed, that should as Davide said, not be the hard part.
- The application could start Firefox with a command parameter indicating the extension to be installed, our pre installed helper extension would then capture this parameter and install the extension indicated, writes an install succeeded message in a logfile the main application monitors, and closes Firefox.
- Next the application would start Firefox with the command parameter -UILocale and so setting the correct locale to test.
- Our helper extension could check for errors in the console (Firefox 3 only?), report these errors or a test succeeded message in the logfile, and closes Firefox again.
- The main application (which was monitoring the logfile) processes the result, and continues accordingly.

But that quite a lot of work I think...

[DaveG]

Writing an extension to sanity check another extension's locale files shouldn't be too hard. Automating the setup of each sounds a bit overkill though. I think that it could test the whole set at once and then the users could check the actual showing of each locale themselves.

While I do agree that the server-side scripts should be doing some of this, an extension makes sense for translators to test their addition of their locale for their own purposes.

[Goofy]

I think that it could test the whole set at once and then the users could check the actual showing of each locale themselves.

Do you mean smthg like: all locales are tested in one pass, and a report is done (saying e.g "en-US -- ok it-IT--failed ...)
mmmh if every locale is tested, the app must restart every time with an appropriate general.useragent.locale, right ?

While I do agree that the server-side scripts should be doing some of this, an extension makes sense for translators to test their addition of their locale for their own purposes.

Sure extension translators will always appreciate some automatic checking. But my idea was more to be useful to developers at the moment when they update/upload their new version on AMO with a bunch of locales, or moreover it could be used by AMO reviewers and spare their time.

[DaveG]

Do you mean smthg like: all locales are tested in one pass, and a report is done (saying e.g "en-US -- ok it-IT--failed ...)

Yeah.

mmmh if every locale is tested, the app must restart every time with an appropriate general.useragent.locale, right ?

No. I'm thinking maybe just load the files manually and check to make sure everything is where it should be and no entities are missing. You'd still have to set the correct pref and check yourself to be sure. (i.e. with quick locale switcher)

Sure extension translators will always appreciate some automatic checking. But my idea was more to be useful to developers at the moment when they update/upload their new version on AMO with a bunch of locales, or moreover it coyuld be used by AMO reviewers and spare their time.

Yes, that sounds like a good idea.

[Goofy]

No. I'm thinking maybe just load the files manually and check to make sure everything is where it should be and no entities are missing. You'd still have to set the correct pref and check yourself to be sure. (i.e. with quick locale switcher)

Sure Quick Locale Switcher is a very useful tool, but I was dreaming of a more complete automation, with almost no "human" intervention: once fed with the xpi, the "testing machine" would just know which lang pref to set and cycle through locales, restarting Fx as many times as necessary... (I know, it is big challenge here)

[DaveG]

Sure Quick Locale Switcher is a very useful tool, but I was dreaming of a more complete automation, with almost no "human" intervention: once fed with the xpi, the "testing machine" would just know which lang pref to set and cycle through locales, restarting Fx as many times as necessary... (I know, it is big challenge here)

Nothing automated will be able to tell the user if something just looks wrong. If what you're suggesting is that it just load each up for the user to check, thus saving them a step, then that makes sense.

[Goofy]

Nothing automated will be able to tell the user if something just looks wrong. If what you're suggesting is that it just load each up for the user to check, thus saving them a step, then that makes sense.

hey it is not here a question of "looking wrong" (you are right, nothing can automatize translation proofreading), but to detect xul errors on install/start or on trigerring the options.

[Goofy]

Hey I also have an answer and suggestions from Axel Hecht

my question was

I discovered this set of scripts you created
> http://mxr.mozilla.org/mozilla/source/test...s/l10n/scripts/
>
> Coul not these tools be used to check extension locales as well, or how
> could they be modified to have some kind of automatic checking?

His answer:

Yes, and to some extent they're designed to do so. I like your idea to
look at the chrome.manifest to gather locales.

The code is currently on one of my hg repos,
http://hg.mozilla.org/users/axel_mozilla.com/tooling/, and some of my
peers are rewriting the backend to be more capable in a project called
silme. http://hg.mozilla.org/users/zbraniecki_mozilla.com/silme/ has the
code, though unstable still.

Those libs are good for some source code checks. They're not catching
all kinds of errors yet, and might never completely do so.

Thus, adding some runtime testing on top of that might be a valuable
idea. To efficiently do so, I'd suggest to look at mozmill, a testing
environment developed by Mikeal Rogers and our QA folks. Read more about
that on Mikeal's blog, http://www.mikealrogers.com/. Somewhere beneath
you'd find https://wiki.mozilla.org/QA/TDAI/Gristmill/Mozmill_Tutorial
explaining how to write tests, which the extension authors should
provide, at least beyond a certain point.

HTH

Axel

[daveschaefer]

Hey everyone,

I found this thread and wanted to mention: I have created a script in python to help validate localization data for Mozilla extensions. I created a thread for it over here - http://www.babelzilla.org/forum/index.php?showtopic=7641 - , but thought I would reply in this thread as well. It seems like most of the scripts and tools mentioned here have not been worked on in quite a while - is that correct?

My script is called 'checkloc'. It uses python, and it's easy to run:

>git clone https://github.com/daveschaefer/checkloc.git
>pip install lxml
>python checkloc/checkloc.py path/to/your/plugin/chrome/locale


You can check it out on GitHub:
https://github.com/daveschaefer/checkloc


It currently tests the following things:

1. No localization has extra files
2. No localization is missing files
3. Each localization has at least one key
4. No localization has extra keys
5. No localization is missing keys
6. No key is blank
7. DTD keys contain no invalid characters, including "!@#$%^&*<>[](){} ?'
8. DTD values contain no invalid characters, including "%<&
(you can use the HTML character entity codes &quot;, %, &lt;, and &amp; if you need to use those characters inside a DTD value)
9. DTD values are not empty: "" or ''
10. DTD comments contain no double hyphens --
11. .properties values are valid, meaning either:
11 a) no % on a line
11 b) double %% to escape and print a regular %
11 c) %S or %n$S , where n is a number, for formatted string replacement.
12. No files contain the Byte Order Marker (BOM)
13. No localization has duplicate keys defined in the same .properties file
(the same key defined in different .properties files is okay - presumably they will be loaded and used in different string bundles)
14. .properties values use the same count and type of string substitutions across languages. e.g.:
* If one language uses four %S they all should
* If one language uses %1$S, %2$S, and %3$S, they all should
(order of substitutions may of course differ between languages)
15. .properties keys contain no spaces
16. No .properties value contains more than 10 unique string substitutions - either %S-style, numbered %1$S-style, or combined.
(It is of course valid to re-use any one %1$S-style numbered substitution as many times as you want)
17. Check the manifest files, chrome.manifest and install.rdf:
17 a) Check every chrome.manifest entry to make sure a matching locale folder exists on disk
17 b) Check that every locale folder has a matching entry in chrome.manifest
17 c) Check that no locale in chrome.manifest is defined more than once
17 d) Check if the locales listed in chrome.manifest are valid known Mozilla locale codes
17 e) Check that install.rdf is valid XML
17 f) Check every install.rdf locale entry to make sure a matching locale folder exists on disk
17 g) Check that every locale folder has a matching entry in install.rdf
17 h) Check that no locale in install.rdf is defined more than once
17 i) Check if the locales listed in install.rdf are valid known Mozilla locale codes

Feedback is welcome if you give it a try.
Cheers!