Discussion:
Pootle migration
(too old to reply)
Rimas Kudelis
2011-03-28 20:54:03 UTC
Permalink
Hi,

right now I'm trying to migrate all the data from our current Pootle
server to the new one. This means that if you make any changes in Pootle
past this point, they will not be reflected in the new installation, so
please don't edit your translations in Pootle until further notice, or
at least make yourself a backup to restore from when done.

Once I'm finished setting Pootle up, we'll all have a few days to test
it and decide whether or not its performance is acceptable. If it is,
we'll just continue using it, and if not, we'll see how to proceed from
there.

So, the bottom line is: if you'll find your latest changes missing at
some poing, don't be too surprised – it's sort of expected.

Regards,
Rimas


--
Unsubscribe instructions: E-mail to l10n+***@libreoffice.org
List archive: http://listarchives.libreoffice.org/www/l10n/
*** All po
Sérgio Marques
2011-03-28 21:12:22 UTC
Permalink
Hi Rimas

For Portuguese language, pootle doesn´t have the strings merged from
openoffice (ui and help) nor extensions. Is it possible to add them in new
server?

Regards

2011/3/28 Rimas Kudelis <***@akl.lt>

> Hi,
>
> right now I'm trying to migrate all the data from our current Pootle
> server to the new one. This means that if you make any changes in Pootle
> past this point, they will not be reflected in the new installation, so
> please don't edit your translations in Pootle until further notice, or
> at least make yourself a backup to restore from when done.
>
> Once I'm finished setting Pootle up, we'll all have a few days to test
> it and decide whether or not its performance is acceptable. If it is,
> we'll just continue using it, and if not, we'll see how to proceed from
> there.
>
> So, the bottom line is: if you'll find your latest changes missing at
> some poing, don't be too surprised – it's sort of expected.
>
> Regards,
> Rimas
>
>
> --
> Unsubscribe instructions: E-mail to l10n+***@libreoffice.org
> List archive: http://listarchives.libreoffice.org/www/l10n/
> *** All posts to this list are publicly archived for eternity ***
>



--
Sérgio Marques

--
Unsubscribe instructions: E-mail to l10n+***@libreoffice.org
List archive: http://listarchives.libreoffice.org/www/l10n/
*** All posts to this list are publicly archived for eternity ***
Rimas Kudelis
2011-03-28 22:05:15 UTC
Permalink
Hi Sérgio,

2011.03.29 00:12, Sérgio Marques rašė:
> For Portuguese language, pootle doesn´t have the strings merged from
> openoffice (ui and help) nor extensions. Is it possible to add them in new
> server?

Yes. We'll ask Andras to do the merging stuff, and I'll upload the
resulting files to Pootle.

Darn, we (I?) should keep a TODO on the wiki...

Good night, :)
Rimas


--
Unsubscribe instructions: E-mail to l10n+***@libreoffice.org
List archive: http://listarchives.libreoffice.org/www/l10n/
*** All posts to this list are publicly archived for eternity
Andras Timar
2011-03-29 07:45:29 UTC
Permalink
2011/3/29 Rimas Kudelis <***@akl.lt>:
> Hi Sérgio,
>
> 2011.03.29 00:12, Sérgio Marques rašė:
>> For Portuguese language, pootle doesn´t have the strings merged from
>> openoffice (ui and help) nor extensions. Is it possible to add them in new
>> server?
>
> Yes. We'll ask Andras to do the merging stuff, and I'll upload the
> resulting files to Pootle.
>

Yes, when Rimas tells me that new Pootle server is ready, I'll process
all the backlog (Lao, Portuguise etc.). Thanks for your patience. :)

Andras

--
Unsubscribe instructions: E-mail to l10n+***@libreoffice.org
List archive: http://listarchives.libreoffice.org/www/l10n/
*** All posts to this list are publicly archive
Ankitkumar Rameshchandra Patel
2011-03-29 08:41:56 UTC
Permalink
On 03/29/2011 01:15 PM, Andras Timar wrote:
> 2011/3/29 Rimas Kudelis<***@akl.lt>:
>> Hi Sérgio,
>>
>> 2011.03.29 00:12, Sérgio Marques rašė:
>>> For Portuguese language, pootle doesn´t have the strings merged
>>> from openoffice (ui and help) nor extensions. Is it possible to
>>> add them in new server?
>>
>> Yes. We'll ask Andras to do the merging stuff, and I'll upload the
>> resulting files to Pootle.
>>
>
> Yes, when Rimas tells me that new Pootle server is ready, I'll
> process all the backlog (Lao, Portuguise etc.). Thanks for your
> patience. :)
>
> Andras
>

Dear Andras/Rimas,

Please add Gujarati (gu) in your list as well for LibreOffice UI and
Help migration, it isn't available in current pootle server.

Moreover, I would suggest to have all languages with at least
LibreOffice UI and Help configured by default, as this is LibreOffice
pootle translation server.


Thanks!
--
Regards,
Ankit Patel
http://www.ankit644.com/

--
Unsubscribe instructions: E-mail to l10n+***@libreoffice.org
List archive: http://listarchives.libreoffice.org/www/l10n/
*** All posts to this list are publicly archived for eternity ***
d***@drouizig.org
2011-03-31 06:30:59 UTC
Permalink
Hello Andras ,
When the new server is ready Breton should be set in UI 3.4.
Should I do it ? Do I have rights ? Or do you ?
Is the new server already working ?
Let us know when it's ok.
Thanks
Denis


> 2011/3/29 Rimas Kudelis <***@akl.lt>:
>> Hi Sérgio,
>>
>> 2011.03.29 00:12, Sérgio Marques rašė:
>>> For Portuguese language, pootle doesn´t have the strings merged from
>>> openoffice (ui and help) nor extensions. Is it possible to add them in
>>> new
>>> server?
>>
>> Yes. We'll ask Andras to do the merging stuff, and I'll upload the
>> resulting files to Pootle.
>>
>
> Yes, when Rimas tells me that new Pootle server is ready, I'll process
> all the backlog (Lao, Portuguise etc.). Thanks for your patience. :)
>
> Andras
>
> --
> Unsubscribe instructions: E-mail to l10n+***@libreoffice.org
> List archive: http://listarchives.libreoffice.org/www/l10n/
> *** All posts to this list are publicly archived for eternity ***
>




--
Unsubscribe instructions: E-mail to l10n+***@libreoffice.org
Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
List archive: http://listarchives.libreoffice.org/www/l10n/
*All messages sent to this list will be publicly archived and cannot be deleted*
Andras Timar
2011-03-31 11:21:57 UTC
Permalink
2011/3/31 <***@drouizig.org>:
> Hello Andras ,
> When the new server is ready Breton should be set in UI 3.4.
> Should I do it ? Do I have rights ? Or do you ?
> Is the new server already working ?
> Let us know when it's ok.
> Thanks
> Denis

Rimas knows... I'm not involved in the server migration. It would be
great, if we could start to work on 3.4 translations next week.

Andras

--
Unsubscribe instructions: E-mail to l10n+***@libreoffice.org
Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
List archive: http://listarchives.libreoffice.org/www/l10n/
*All messages sent to this list will
Andras Timar
2011-04-04 12:35:33 UTC
Permalink
2011.03.31. 13:21 keltezéssel, Andras Timar írta:
> 2011/3/31 <***@drouizig.org>:
>> Hello Andras ,
>> When the new server is ready Breton should be set in UI 3.4.
>> Should I do it ? Do I have rights ? Or do you ?
>> Is the new server already working ?
>> Let us know when it's ok.
>> Thanks
>> Denis
>
> Rimas knows... I'm not involved in the server migration. It would be
> great, if we could start to work on 3.4 translations next week.
>

Hi,

What is the status of this migration? It seems that it happened. But it
was not announced. Please let me know, because we need to start 3.4
translations ASAP.

Thanks,
Andras

--
Unsubscribe instructions: E-mail to l10n+***@libreoffice.org
Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
List archive: http://listarchives.libreoffice.org/www/l10n/
All messages sent to this list will be publicly archived and cannot be deleted
Rimas Kudelis
2011-04-04 12:58:04 UTC
Permalink
Hi,

2011.04.04 15:35, Andras Timar rašė:
> 2011.03.31. 13:21 keltezéssel, Andras Timar írta:
>> 2011/3/31<***@drouizig.org>:
>>> Hello Andras ,
>>> When the new server is ready Breton should be set in UI 3.4.
>>> Should I do it ? Do I have rights ? Or do you ?
>>> Is the new server already working ?
>>> Let us know when it's ok.
>>> Thanks
>>> Denis
>> Rimas knows... I'm not involved in the server migration. It would be
>> great, if we could start to work on 3.4 translations next week.
>>
> What is the status of this migration? It seems that it happened. But it
> was not announced. Please let me know, because we need to start 3.4
> translations ASAP.

Right. I've migrated the database and files, and updated the database at
some later point, before switching DNS entries to point to the Virtual
machine.

Now we need to import 3.4 files into the new Pootle and then start
localizing (aka testing the installation). How can we proceed with this?

Stuff to consider: Pootle may run slower on this machine, and we may
experience other problems, at least for now. I wasn't able to convince
the admin that Pootle needs more resources, so we have what we have.
Maybe if we manage to give enough load to the server, he'll change his
mind (or we'll find other ways to deal with the problem).

So, let's start importing files for 3.4, shall we?

Rimas


--
Unsubscribe instructions: E-mail to l10n+***@libreoffice.org
Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
List archive: http://listarchives.libreoffice.org/www/l10n/
All messages sent to this
Andras Timar
2011-04-04 14:45:37 UTC
Permalink
2011/4/4 Rimas Kudelis <***@akl.lt>:
> Right. I've migrated the database and files, and updated the database at
> some later point, before switching DNS entries to point to the Virtual
> machine.
>
> Now we need to import 3.4 files into the new Pootle and then start
> localizing (aka testing the installation). How can we proceed with this?
>
> Stuff to consider: Pootle may run slower on this machine, and we may
> experience other problems, at least for now. I wasn't able to convince the
> admin that Pootle needs more resources, so we have what we have. Maybe if we
> manage to give enough load to the server, he'll change his mind (or we'll
> find other ways to deal with the problem).
>
> So, let's start importing files for 3.4, shall we?
>

I think first step should be to migrate those languages that already
have 'full' translations. Rimas, is it possible to copy libo33x_ui and
libo33x_help files to libo34x_ui and libo34_help respectively, and
update them using the libo34x_ui and libo34_help templates? I can add
other languages via the web interface later. I updated the templates
for 3.4 now.

Thanks,
Andras

--
Unsubscribe instructions: E-mail to l10n+***@libreoffice.org
Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
List archive: http://listarchives.libreoffice.org/www/l10n/
All messages sent to this list will be publi
Christian Lohmaier
2011-04-04 20:28:43 UTC
Permalink
Hi *,

On Mon, Apr 4, 2011 at 2:58 PM, Rimas Kudelis <***@akl.lt> wrote:
>
> Stuff to consider: Pootle may run slower on this machine, and we may
> experience other problems, at least for now. I wasn't able to convince the
> admin that Pootle needs more resources, so we have what we have.

Again, as you apparently still don't understand what I already wrote many times:
* Adding more resources will /NOT/ make pootle run faster than it does now.
The VM already has way more resources assigned than necessary. It is
/idle/ almost all the time.
* The only thing that is slow (when executed the first time) is
generation of the zips. So when you as translator request a zip: Don't
click the link multiple times because you don't immediately get the
zip. It can take 10 seconds for the files to be generated. Again:
* Adding more resources will /not/ make that time shorter. It is a
single-threaded process that can only uses one single CPU, thus
assigning more CPUs won't help at all (the VM has 4CPUs assigned
already)
Requesting that same zip another time (or different zips of the
project belonging to the same language is fast/instant, but requesting
the zip for another language again may take some seconds for the first
request (or again after the files did change in between).
* Pootle has a memory leak when creating the zips. It won't release
memory after processing the files.
This would be the only time where the assigned resources may run out
(the VM has 1GB or RAM assigned): Multiple different languages request
the zip at the same time. Then memory usage increases, memory runs out
and either it is crawling along or the process gets killed.
* I will NOT assign RAM to a VM (and thus block that ram for other
use) to satisfy a memory leak, when that RAM is unused 99% of the
time.
* The effects of the memory leak can be nullified by just restarting
the worker processes more frequently. Thus again:
* Adding more resources will /NOT/ make the VM run faster, it will
/NOT/ allow it to handle more requests

Pootle is idling almost all of the time. There is less than one apache
request per second on average (and for regular requests (i.e.
non-"generate-a-zip" actions) it can easily serve >>50 simultaneous
requests per second.)

> Maybe if we
> manage to give enough load to the server, he'll change his mind (or we'll
> find other ways to deal with the problem).

No, I won't change my mind, but depending on the load/effects of the
memory leak I'll reduce lifetime of the server-processes further.

Again:
The only time where pootle is "slow" is:
* Creation of zips for the first time / after files have changed.
This is CPU intensive process, and the CPU cannot be made faster by
assigning more resources. Live with it. Redesign pootle to use another
method (i.e. a multithreaded one that can use multiple CPUs at once)
or whatever, but this one problem is not solvable by assigning more
resources.

* This also is only noteworthy when the files are big/numerous.
* Requesting the other zips in the same project & language is fast.

So there is no point in clicking all zip-URLs on a page at once (on
the contrary, than the request will all cause CPU to be burnt, while
all that is only needed one single time). If you want to download more
than one zip of a page, click the first one, wait until it is
generated and handed over to your browser, then click as many others
on the same page and get all of them quickly.

* Any other time, doing in-place-translation, just browsing along is
supposed to be fast.
If it is not, it needs to be investigated - i.e. please report it with
a description of what you did, when the problem occurred (and of
course what you definition of "slow" is)

The server is under close monitoring using munin, so bottlenecks
should be easily identified.

But with an (daily) average load of 0.09 and an average of 0.08 apache
requests/second (to which munin also contributes) are no way reason to
assign more resources.

Again:
* The VM has plenty of resources, with much reserves for more traffic/accesses.
* Creation of ZIPs is CPU-intense and thus take some seconds (for the
large packages up to 10 seconds) Be patient when requesting a zip for
the first time.
* When you see "premature end of script headers" or similar error
message - report it. This means the memory-leak did exceed the limits
and the lifetime parameters need to be tweaked.
* If you experience any "slowness" on other operations than requesting
zips: Report them.

(for users in Europe, it might now even be faster, as the data
doesn't need to cross the Atlantic anymore - well, not that humans
will notice that difference, but still :-))

ciao
Christian

--
Unsubscribe instructions: E-mail to l10n+***@libreoffice.org
Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
List archive: http://listarchives.libreoffice.org/www/l10n/
All messages sent to this list will be publicly archived and cannot be deleted
Andras Timar
2011-04-05 07:42:23 UTC
Permalink
Hi Christian,

2011.04.04. 22:28 keltezéssel, Christian Lohmaier írta:

> * If you experience any "slowness" on other operations than requesting
> zips: Report them.
>

Many thanks for your insightful explanations. I find the new server more
responsive than the old one. However, some operations are still slow.
When I add a new language to a project (e.g. LibreOffice 3.4.x UI) and
click Overview tab of that new language, I have to wait ~2-3 minutes.
Then I upload translations to that language and I have to wait another
~5 minutes. This is not a convenient way to upload 100 languages. Can it
be done in a batch process from the shell?

Thanks,
Andras

--
Unsubscribe instructions: E-mail to l10n+***@libreoffice.org
Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
List archive: http://listarchives.libreoffice.org/www/l10n/
All messages sent to this list will be publicly archived and cannot be deleted
Christian Lohmaier
2011-04-05 13:20:25 UTC
Permalink
Hi Andras, *,

On Tue, Apr 5, 2011 at 9:42 AM, Andras Timar <***@gmail.com> wrote:
> 2011.04.04. 22:28 keltezéssel, Christian Lohmaier írta:
>
>> * If you experience any "slowness" on other operations than requesting
>> zips: Report them.
>
> Many thanks for your insightful explanations. I find the new server more
> responsive than the old one. However, some operations are still slow.
> When I add a new language to a project (e.g. LibreOffice 3.4.x UI) and
> click Overview tab of that new language, I have to wait ~2-3 minutes.

You don't write when exactly you did it, but from munin stats I see
there is high CPU utilization between around the time you wrote that
post (i.e. between 9 and 10). But that high utilization is again only
on one single core, i.e. 100% out of 400% are used only.
This is pootles's processing that is slow here (CPU-bound, thus
nothing fixable unless you rewrite pootle to use multithreaded
approach).

> Then I upload translations to that language and I have to wait another
> ~5 minutes. This is not a convenient way to upload 100 languages. Can it
> be done in a batch process from the shell?

Oh, large-scale updates surely should be possible using the shell, but
I don't know really know pootle, thus I cannot tell for sure (but I
guess Rimas is doing just that right now...)

ciao
Christian

--
Unsubscribe instructions: E-mail to l10n+***@libreoffice.org
Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
List archive: http://listarchives.libreoffice.org/www/l10n/
All messages sent to this list will be publicly archived and cannot be deleted
Dwayne Bailey
2011-04-05 15:30:28 UTC
Permalink
On 2011-04-05 15:20, Christian Lohmaier wrote:
> Hi Andras, *,
>
> On Tue, Apr 5, 2011 at 9:42 AM, Andras Timar<***@gmail.com> wrote:
>> 2011.04.04. 22:28 keltezéssel, Christian Lohmaier írta:
>>
>>> * If you experience any "slowness" on other operations than requesting
>>> zips: Report them.
>> Many thanks for your insightful explanations. I find the new server more
>> responsive than the old one. However, some operations are still slow.
>> When I add a new language to a project (e.g. LibreOffice 3.4.x UI) and
>> click Overview tab of that new language, I have to wait ~2-3 minutes.
> You don't write when exactly you did it, but from munin stats I see
> there is high CPU utilization between around the time you wrote that
> post (i.e. between 9 and 10). But that high utilization is again only
> on one single core, i.e. 100% out of 400% are used only.
> This is pootles's processing that is slow here (CPU-bound, thus
> nothing fixable unless you rewrite pootle to use multithreaded
> approach).
When adding a new language we don't do any caching until you go to the
overview page. Thus when first viewing we are doing all the updating
for search and check failures. A way around this would be to run
refresh_stats manage.py command.

I doubt multithreading would help here because of Python's poor
multithreading ability. So more invasive changes to display stats as
they are available would probably be the correct way to make the UI more
responsive. We're relying on Apache to start each mod_wsgi instance so
while this is slow and hogging that one core it wouldn't prevent someone
from translating elsewhere in Pootle.

>> Then I upload translations to that language and I have to wait another
>> ~5 minutes. This is not a convenient way to upload 100 languages. Can it
>> be done in a batch process from the shell?
> Oh, large-scale updates surely should be possible using the shell, but
> I don't know really know pootle, thus I cannot tell for sure (but I
> guess Rimas is doing just that right now...)
When uploading translations a few things are happening.

1. Unzipping if needed
2. Parsing the files that you supplied
3. Merging your uploads with the current data in the db
4. Refreshing the stats

We do this to prevent any data loss, but as you can imagine its a lot of
work. One thing worth trying on the LO Pootle server is to make use of
the C PO parser, its much faster but not as widely tested as the Python
PO parser.

You can do this from the command line update_stores followed by
refresh_stats. This of course if you know that the files on the files
system are the ones that you want as it will override anything in the
database.
> ciao
> Christian
>


--
regards
Dwayne


--
Unsubscribe instructions: E-mail to l10n+***@libreoffice.org
Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
List archive: http://listarchives.libreoffice.org/www/l10n/
All messages sent to this list will be publicly archived and cannot be deleted
Dwayne Bailey
2011-04-06 07:58:27 UTC
Permalink
Sent to the list on behalf of Friedel:

Hi Christian, everybody

Please CC me on any replies as I'm not on the list.

I'm one of the developers in the Translate project, and have been trying
to help Rimas a bit with this deployment of Pootle. Thanks for your work
on the server for Pootle! Please allow me a few comments:

Background:
Pootle simply isn't coded to run in small 16MB processes. It is a full
featured web application written on a heavy framework (Django) in a
programming language that isn't very frugal with memory use (Python). We
didn't specifically optimise for memory use over other things when we
programmed Pootle, and Libreoffice is running a system with over 5000
files for the libo34x_ui project alone. When looking at help, the files
are very big by any standard in the world of FOSS. Of course, the help
has many of these big files per language. This is all fine. Pootle runs
fine with loads like this, as (obviously) visible from the OOo project.


The rest of my comments are inline...


> -------- Original Message --------
> Subject:
> Re: [libreoffice-l10n] Pootle
> migration
> Date:
> Mon, 4 Apr 2011 22:28:43 +0200
> From:
> Christian Lohmaier
>
>
>
>
Hi *,
>
> On Mon, Apr 4, 2011 at 2:58 PM, Rimas Kudelis <***@akl.lt> wrote:
> >
> > Stuff to consider: Pootle may run slower on this machine, and we may
> > experience other problems, at least for now. I wasn't able to
convince the
> > admin that Pootle needs more resources, so we have what we have.
>
> Again, as you apparently still don't understand what I already wrote
many times:
> * Adding more resources will /NOT/ make pootle run faster than it
does now.
> The VM already has way more resources assigned than necessary. It is
> /idle/ almost all the time.

As far as I know the server hasn't really been used yet, so I guess
we'll be collecting data from now on to see how things go. During the
setup of the server, we make tradeoffs between performance and memory
use. If there is no memory available, we'll obviously try to optimise at
all cost for minimising memory use, and that is what I understand that
Rimas said: things might be slower than necessary, since we are not
optimising for performance, but for memory use.



> * The only thing that is slow (when executed the first time) is
> generation of the zips. So when you as translator request a zip: Don't
> click the link multiple times because you don't immediately get the
> zip. It can take 10 seconds for the files to be generated. Again:
> * Adding more resources will /not/ make that time shorter. It is a
> single-threaded process that can only uses one single CPU, thus
> assigning more CPUs won't help at all (the VM has 4CPUs assigned
> already)
> Requesting that same zip another time (or different zips of the
> project belonging to the same language is fast/instant, but requesting
> the zip for another language again may take some seconds for the first
> request (or again after the files did change in between).
> * Pootle has a memory leak when creating the zips. It won't release
> memory after processing the files.
> This would be the only time where the assigned resources may run out
> (the VM has 1GB or RAM assigned): Multiple different languages request
> the zip at the same time. Then memory usage increases, memory runs out
> and either it is crawling along or the process gets killed.

Some stuff that is slow to load is cached for later use. This is done
for performance optimisation. This is one of the reasons you won't see
the memory use go down immediately after generating a ZIP file. Another
reason is the way the garbage collector works in Python.

Deciding to cache something is a tradeoff. So we can disable or minimize
some of the caching, which will simply make a few things slower,
hopefully not by much, but we're guessing while the server hasn't been
used much yet.

I suggested some customisations to the parse pool (to do exactly this).
That affects the number of cached files and search indexes, both of
which are very large on your server.



> * I will NOT assign RAM to a VM (and thus block that ram for other
> use) to satisfy a memory leak, when that RAM is unused 99% of the
> time.

I believe what you are seeing is the caching, not a memory leak.

We haven't seen the server used much yet. My educated guess from having
worked on a few Pootle installations is that the RAM isn't enough, but
let's keep an eye on things and see how it goes. I assumed we'll want a
nice fast server supporting several concurrent users during the build-up
to a release, but we can still tune things down a bit more, I guess.


> * The effects of the memory leak can be nullified by just restarting
> the worker processes more frequently. Thus again:

...at the cost of making things slower, since more stuff needs to be
loaded in memory afresh every time you restart a process.


> * Adding more resources will /NOT/ make the VM run faster

It most probably will, since we are sacrificing performance to minimise
memory use. For example: we opted for more threads, rather than
processes, that is known not to perform as well in Python, especially
for CPU intensive tasks.


> it will
> /NOT/ allow it to handle more requests

It most probably will, since we reduced the number of processes to
minimise memory use, and slower serving of requests necessarily affects
the number of requests you can serve in any given time.


> Pootle is idling almost all of the time. There is less than one apache
> request per second on average (and for regular requests (i.e.
> non-"generate-a-zip" actions) it can easily serve >>50 simultaneous
> requests per second.)

Let's keep an eye on things when people actually start to use the
server.


> > Maybe if we
> > manage to give enough load to the server, he'll change his mind (or
we'll
> > find other ways to deal with the problem).
>
> No, I won't change my mind, but depending on the load/effects of the
> memory leak I'll reduce lifetime of the server-processes further.

I hope you will be reasonable and look at the data as it becomes
available, and at least consider changing your mind. As mentioned, I'm
pretty sure there is no memory leak. If you reduce the lifetime of the
server processes, you are just making performance worse, which is all
that Rimas warned the users for.


> Again:
> The only time where pootle is "slow" is:
> * Creation of zips for the first time / after files have changed.
> This is CPU intensive process, and the CPU cannot be made faster by
> assigning more resources. Live with it. Redesign pootle to use another
> method (i.e. a multithreaded one that can use multiple CPUs at once)
> or whatever, but this one problem is not solvable by assigning more
> resources.

I agree. Generating the ZIP files is slow. Doing it multithreaded, will
limit the performance for more users while doing that.


> * This also is only noteworthy when the files are big/numerous.

Which is the norm on this server, unfortunately.


> * Requesting the other zips in the same project & language is fast.
>
> So there is no point in clicking all zip-URLs on a page at once (on
> the contrary, than the request will all cause CPU to be burnt, while
> all that is only needed one single time). If you want to download more
> than one zip of a page, click the first one, wait until it is
> generated and handed over to your browser, then click as many others
> on the same page and get all of them quickly.

Yes, we have optimised for several cases here that are likely, and I
suggested some workarounds for some of the issues we're likely to hit
with the little bit of RAM as Rimas has already started doing, as far as
I know.


> * Any other time, doing in-place-translation, just browsing along is
> supposed to be fast.

... as long as we're not hitting the current imposed limits of
concurrency.


Let's see how well we can make this work.

Keep well
Friedel



--
Recently on my blog:
http://translate.org.za/blogs/friedel/en/content/better-lies-about-gnome-localisation

--
Unsubscribe instructions: E-mail to l10n+***@libreoffice.org
Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
List archive: http://listarchives.libreoffice.org/www/l10n/
All messages sent to this list will be publicly archived and cannot be deleted
Christian Lohmaier
2011-04-06 08:50:34 UTC
Permalink
Hi *,

On Wed, Apr 6, 2011 at 9:58 AM, Dwayne Bailey <***@translate.org.za> wrote:
>
> Please CC me on any replies as I'm not on the list.

Keeping this, so others will follow the request as well.

>> Mon, 4 Apr 2011 22:28:43 +0200
>>                              From:
>> Christian Lohmaier
>> On Mon, Apr 4, 2011 at 2:58 PM, Rimas Kudelis <***@akl.lt> wrote:
>> Again, as you apparently still don't understand what I already wrote many
>> times:
>> * Adding more resources will /NOT/ make pootle run faster than it does
>> now.
>> The VM already has way more resources assigned than necessary. It is
>> /idle/ almost all the time.
>
> As far as I know the server hasn't really been used yet, so I guess
> we'll be collecting data from now on to see how things go.

Also from the avialble data from the old pootle server. But yes, the
new one doesn't have much data yet.

> During the
> setup of the server, we make tradeoffs between performance and memory
> use. If there is no memory available, we'll obviously try to optimise at
> all cost for minimising memory use,

Please explain those settings. Which settings, what is the effect, how
to see the effect (i.e. what UI actions to perform)

> and that is what I understand that
> Rimas said: things might be slower than necessary, since we are not
> optimising for performance, but for memory use.

No, and I repeat again: Memory is not an issue. The memory leak is.

>> * The only thing that is slow (when executed the first time) is
>> generation of the zips. So when you as translator request a zip: Don't
>> click the link multiple times because you don't immediately get the
>> zip. It can take 10 seconds for the files to be generated. Again:
>> * Adding more resources will /not/ make that time shorter. It is a
>> single-threaded process that can only uses one single CPU, thus
>> assigning more CPUs won't help at all (the VM has 4CPUs assigned
>> already)
>> Requesting that same zip another time (or different zips of the
>> project belonging to the same language is fast/instant, but requesting
>> the zip for another language again may take some seconds for the first
>> request (or again after the files did change in between).
>> * Pootle has a memory leak when creating the zips. It won't release
>> memory after processing the files.
>> This would be the only time where the assigned resources may run out
>> (the VM has 1GB or RAM assigned): Multiple different languages request
>> the zip at the same time. Then memory usage increases, memory runs out
>> and either it is crawling along or the process gets killed.
>
> Some stuff that is slow to load is cached for later use.

"Some stuff" maybe, but that little is not what I'm talking about.

> This is done
> for performance optimisation. This is one of the reasons you won't see
> the memory use go down immediately after generating a ZIP file.

No, this has nothing to do with caching. It is a memory leak. Rimas is
very good at not forwarding relevant information it seems. So here I
just copy and paste what I wrote to Rimas already.
#############################
>>> [Rimas]
>>> After generating, the zip files are cached on disk, so before
>>> redownloading the same file, you'll probably want to execute
>>> $ find /var/www/pootle/po/POOTLE_EXPORT/ -type f -iname *.zip -exec rm {}
>>> \;

No, this is not enough to do the same as what pootle does when
requesting the files the first time.
The first time it takes long (100% CPU while processing), and that is
also the time where the memory leak occurs. After it is finished
producing the file, the memory is not freed.

Requesting one of the zips interestingly also prepares the other files
at once, at least when selecting any of the other files on the same
language and project, the file will be served immediately, without the
lengthy processing step.
(And those requests also don't increase the memory usage further)

But switching to another language, and requesting a file results in
the processing with memory leak again.

But that memory is not needed at all, as is obvious when the process
has reached it's max-requests and is replaced by a new one, then it
serves all of the already covered languages without that increase in
memory-usage.

With that info at hand, I tweaked the apache-settings a bit (basically
reduced the amount of concurrent wsgi processes, and reducing their
lifetime) to make the accumulation less likely.

The machine will still run out-of memory when different language zips
are requested in a short amount of time with not enough regular
requests in between. In case people see too many "premature end of
script" or similar (i.e. the worker has been killed by the OOM
killer), reduce the number of requests before the worker is restarted.
Currently it is at 400, which is still pretty high, considering the
more or less non-existant regular load (but then again, this might be
because people know the server is in transition and because 3.3 is
done already)

Similarily, the very first page-load after a reboot also takes "ages",
but after that it runs smoothly,

1GB is enough for Pootle The rest is a matter of
configuration/tweaking. (or best: fixing the memory leak)
#######################
> Another
> reason is the way the garbage collector works in Python.

Well - how long should one wait for the memory to be freed? If it is
not freed when memory is about to run out, or after a hour or so, then
I don't assume it is a problem of running the garbage collector. (and
again that memory is not used/needed for further request of the same
kind, so I don't take your "it is caching stuff to accelerate future
actions" for granted.)

Please give me instructions on stuff to do in pootle to notice a difference.

> Deciding to cache something is a tradeoff. So we can disable or minimize
> some of the caching, which will simply make a few things slower,
> hopefully not by much, but we're guessing while the server hasn't been
> used much yet.

Numbers and examples please.

Caching for a more or less one-time operation that impacts the whole
server all the time surely is the wrong way to go. Even more so when
despite that "caching" (I still call it leak, as it doesn't cache
anything) the requesting of the zips is still CPU bound and slow.

> I suggested some customisations to the parse pool (to do exactly this).
> That affects the number of cached files and search indexes, both of
> which are very large on your server.

Please be more detailed on this. What setting, in what way to tweak,
and what should the effect be?
Numbers?

>> * I will NOT assign RAM to a VM (and thus block that ram for other
>> use) to satisfy a memory leak, when that RAM is unused 99% of the
>> time.
>
> I believe what you are seeing is the caching, not a memory leak.

And I strongly disagree, see above.

> We haven't seen the server used much yet. My educated guess from having
> worked on a few Pootle installations is that the RAM isn't enough,

And I strongly disagree. See above.

>> * The effects of the memory leak can be nullified by just restarting
>> the worker processes more frequently. Thus again:
>
> ...at the cost of making things slower, since more stuff needs to be
> loaded in memory afresh every time you restart a process.

I'm starting to belive you're in the same league as Rimas when it
comes to server administration (i.e. very much guesswork).

The starting/initializing of the processes is negligible. Just fire up
an ab bechmark and request 20000 with concurrency of 30 and look at
the distribution.
I'd gladly live with a few hundred of milliseconds, if that will save
hundreds of memory that can be freed by just restarting the worker
process.

>> * Adding more resources will /NOT/ make the VM run faster
>
> It most probably will, since we are sacrificing performance to minimise
> memory use.

No. As the times where it is slow is not bound by memory, but by CPU.
Please please please: Read what I wrote. I repeated myself so many
times, it really gets annoying.

> For example: we opted for more threads, rather than
> processes, that is known not to perform as well in Python, especially
> for CPU intensive tasks.

Who is "we"? Yes, I decided to (for now) only run with 2 wsgi
processes. But then again: how would that make execution faster?

If you provide me with hard numbers, concrete settings and
instructions on what to do in the UI to reproduce and compare the
settings, I'm willing to try them out.

>>  it will
>> /NOT/ allow it to handle more requests
>
> It most probably will, since we reduced the number of processes to
> minimise memory use, and slower serving of requests necessarily affects
> the number of requests you can serve in any given time.

Again this is wrong. it might allow 4 simultaneous "request the zip"
requests, but all 4 would be taking ages, and all would have the
memory leak, thus needing more RAM (that I'm not wiling to assign,
since regular operation just doesn't need it).
(and the users that are just using regular webinterface have to wait
then as well). Allowing more processes doesn't make sense when pootle
doesn't queue/limit those lengthy processes by itself.

>> Pootle is idling almost all of the time. There is less than one apache
>> request per second on average (and for regular requests (i.e.
>> non-"generate-a-zip" actions) it can easily serve >>50 simultaneous
>> requests per second.)
>
> Let's keep an eye on things when people actually start to use the
> server.

Whether the requests are generated by a benchmark tool or by regular
users doesn't make any difference. The number it can handle is way
beyond any real load to expect.

>> > Maybe if we
>> > manage to give enough load to the server, he'll change his mind (or
>> > we'll
>> > find other ways to deal with the problem).
>>
>> No, I won't change my mind, but depending on the load/effects of the
>> memory leak I'll reduce lifetime of the server-processes further.
>
> I hope you will be reasonable and look at the data as it becomes
> available, and at least consider changing your mind.

Again this is out of the question, but all *data*, real facts I
gathered so far don't suggest that it would be any different from my
current opinion.

> As mentioned, I'm
> pretty sure there is no memory leak.

I disagree, see above. Alternatively give a reasonable explanation on
on why the export still works fine even when the process that created
it (and thus the memory it occupied, and all possible caching it could
have down in its own space) is gone.

> If you reduce the lifetime of the
> server processes, you are just making performance worse, which is all
> that Rimas warned the users for.

Numbers/examples on how performance can be worse. Otherwise I cannot
take your comments seriously, as my measurements did show otherwise.

> [...]
> I agree. Generating the ZIP files is slow. Doing it multithreaded, will
> limit the performance for more users while doing that.

But the time they have to wait will be shorter.

>> * Requesting the other zips in the same project & language is fast.
>>
>> So there is no point in clicking all zip-URLs on a page at once (on
>> the contrary, than the request will all cause CPU to be burnt, while
>> all that is only needed one single time). If you want to download more
>> than one zip of a page, click the first one, wait until it is
>> generated and handed over to your browser, then click as many others
>> on the same page and get all of them quickly.
>
> Yes, we have optimised for several cases here that are likely, and I
> suggested some workarounds for some of the issues we're likely to hit
> with the little bit of RAM as Rimas has already started doing, as far as
> I know.

Don't suggest them only to Rimas, as he is not communicating well with
such technical matters it seems.

>> * Any other time, doing in-place-translation, just browsing along is
>> supposed to be fast.
>
> ... as long as we're not hitting the current imposed limits of
> concurrency.

Again: Numbers, no foggy guesswork please.

ciao
Christian

--
Unsubscribe instructions: E-mail to l10n+***@libreoffice.org
Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
List archive: http://listarchives.libreoffice.org/www/l10n/
All messages sent to this list will be publicly archived and cannot be deleted
Rimas Kudelis
2011-04-06 09:03:34 UTC
Permalink
Latest news! Pootle is also leaking disk space, cause now we're out of
this resource too.


--
Unsubscribe instructions: E-mail to l10n+***@libreoffice.org
Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
List archive: http://listarchives.libreoffice.org/www/l10n/
All messages sent to this list will be publicly archived and cannot be deleted
Christian Lohmaier
2011-04-06 10:34:18 UTC
Permalink
On Wed, Apr 6, 2011 at 11:03 AM, Rimas Kudelis <***@akl.lt> wrote:
> Latest news! Pootle is also leaking disk space, cause now we're out of this
> resource too.

This is easily solvable (and not sure whether you can call it "leaking"
It's definitely using more than was expected (more than 8GB of
po-files on disk, and almost 4GB of mysql database)

But if people would have told beforehand that the values on the
dedicated pootle server are not representative ( <3GB for www-dir/pos
and < 1GB of mysqldb), then appropriate disk-space could have been
assigned beforehand.

I'm currently moving stuff pootle over to an additional virtual disk though...

ciao
Christian

--
Unsubscribe instructions: E-mail to l10n+***@libreoffice.org
Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
List archive: http://listarchives.libreoffice.org/www/l10n/
All messages sent to this list will be publicly archived and cannot be deleted
Rimas Kudelis
2011-04-06 10:52:18 UTC
Permalink
2011.04.06 13:34, Christian Lohmaier rašė:
> On Wed, Apr 6, 2011 at 11:03 AM, Rimas Kudelis<***@akl.lt> wrote:
>> Latest news! Pootle is also leaking disk space, cause now we're out of this
>> resource too.
> This is easily solvable (and not sure whether you can call it "leaking"
> It's definitely using more than was expected (more than 8GB of
> po-files on disk, and almost 4GB of mysql database)
>
> But if people would have told beforehand that the values on the
> dedicated pootle server are not representative (<3GB for www-dir/pos
> and< 1GB of mysqldb), then appropriate disk-space could have been
> assigned beforehand.
>
> I'm currently moving stuff pootle over to an additional virtual disk though...

Thanks, please ping us when you're done.

By the way, maybe we should take this discussion of the l10n list? I
don't think many localizers are interested in technical difficulties
we're facing.

Rimas


--
Unsubscribe instructions: E-mail to l10n+***@libreoffice.org
Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
List archive: http://listarchives.libreoffice.org/www/l10n/
All messages sent to this list will be publicly archived and cannot be delet
Christian Lohmaier
2011-04-06 11:27:14 UTC
Permalink
Hi Rimas, *,

On Wed, Apr 6, 2011 at 12:52 PM, Rimas Kudelis <***@akl.lt> wrote:
> 2011.04.06 13:34, Christian Lohmaier rašė:
>> [...]
>> I'm currently moving stuff pootle over to an additional virtual disk
>> though...
>
> Thanks, please ping us when you're done.

Done.

> By the way, maybe we should take this discussion of the l10n list? I don't
> think many localizers are interested in technical difficulties we're facing.

Well, I want to keep it on a list - and the l10n list seems just
suitable for it. While the localizers might not be interested in every
technical detail, having the discussion on list saves them the
question "what is wrong with pootle?" or "is pootle down? When can we
work with pootle" etc.

Having it all off-list basically results in more work in the end. (or
more annoyed users, since they try in vain to do some work, and don't
know the reason for it.

I as a user of a webservice would like to know why it is not
working/when I can expect it to be online again...

ciao
Christian

--
Unsubscribe instructions: E-mail to l10n+***@libreoffice.org
Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
List archive: http://listarchives.libreoffice.org/www/l10n/
All messages sent to this list will b
Rimas Kudelis
2011-04-06 12:05:15 UTC
Permalink
Hi,

2011.04.06 14:27, Christian Lohmaier rašė:
>>> [...]
>>> I'm currently moving stuff pootle over to an additional virtual disk
>>> though...
>> Thanks, please ping us when you're done.
> Done.

Thanks again.

>> By the way, maybe we should take this discussion of the l10n list? I don't
>> think many localizers are interested in technical difficulties we're facing.

<...>
> I as a user of a webservice would like to know why it is not
> working/when I can expect it to be online again...

Well, I wrote to the list yesterday that it won't work for a while and
to stay tuned for further announcements. I think this should be enough. ;)

Rimas

--
Unsubscribe instructions: E-mail to l10n+***@libreoffice.org
Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
List archive: http://listarchives.libreoffice.org/www/l10n/
All messages sent to this list will be publicly archived and
Christian Lohmaier
2011-04-06 17:25:13 UTC
Permalink
Hi *

On Wed, Apr 6, 2011 at 6:11 PM, F Wolff <***@translate.org.za> wrote:
> Op Wo, 2011-04-06 om 10:50 +0200 skryf Christian Lohmaier:
>
> I find your responses rude and unhelpful.

Sorry for that. I'm just tired of writing the same stuff over and over again.

> Somehow we manage to provide this software on hundreds of sites running
> fine, ranging from hosting on a few hundred megabytes of RAM to machines
> with several gigabytes. All accidents?

No, on the contrary, especially because it is run on so many servers
with limited resources, I'm not willing to waste RAM on pootle-VM
alone, when pootle doesn't need it at all.
RAM assigned to the VM is not available to the host or other VMs, even
if the RAM is not used inside the VM.

> I could show you the specific lines of code caching big objects
> (expected to be tens to hundreds of megabytes in size), but I guess that
> won't convince you either. It is a leak, because someone who (I guess)
> never looked at the Pootle code, says so.

I clearly explained why I'm convinced that it is a memory leak. It
doesn't free the memory, even when the machine is idling for hours. It
doesn't need that memory, since when the worker is replaced by a fresh
one, the functionality still works.

And You're top-posting and fullquoting. This (again from my long
email/mailinglist experience) emprically is a sign that people don't
actually read what was written, don't answer the questions, and
basically discuss different topics all the way, leading to repetition
over and over again.

> Why can't you at least start
> by assuming that I might have an idea of how Pootle works?

That's not the point. Please take your time to actually read the post.

> When you are willing to discuss things under the good faith assumption
> that I _might_ not be talking nonsense, we can continue the
> conversation. I've been trying to help you in my free time based on my
> experience, but it seems you'd prefer to assume I don't know what I'm
> talking about.

Well - All I heard so far, and that was not from you personally, but
mainly via Rimas was just guesswork, no facts. And that guesswork is
contradicting hard numbers, real benchmarks. And I just hate when I
have to write the same thing over and over again. It surely did have
an impact on the tone of my response, apologies for that. But that
doesn't change anything.

> In the meanwhile, here is some recommended reading:
> http://effbot.org/pyfaq/why-doesnt-python-release-the-memory-when-i-delete-a-large-object.htm

Sorry, but the information content of that article is near zero
compared to what has been written here already. It contains a hint for
programmers regarding the "builtin memory leak" when working with
large number of integers or floats. I also don't care whether it is
"python itself" or the allocator that doesn't give back memory, or
that memory cannot be returned because of memory fragmentation. In the
end it makes the system unusable. Not returning memory is a memory
leak, no matter whether it is by design or not. If you cannot free
memory, you have to spawn a childprocess and dismiss that child to
keep a runnable system. It is /not/ acceptable to accumulate more and
more memory and just say "but it is not my fault, it is the memory
allocators that don't return the memory to the system"
It's not a few kBs here and there we're talking about, but tens to
hundreds of MB.

In my definition when a (long-living) process doesn't return unneeded
memory. that process is leaking memory. Whether it is by design or not
doesn't matter to me.
* It consumes more and more memory (it would no problem if creating
the zips for another language would just reuse the memory that was
allocated when creating the zip for the first language)
* It doesn't release that memory when idle (it would also be no
problem if that memory would be returned to the system after a while -
now it has to be enforced by restarting the server-process itself)
* It doesn't release the memory when the machine runs out of available
memory (thus people cry "the machine needs more RAM", but there is no
need, as it can be easily circumvented)
* the allocated memory is not needed to perform any operation after
the memory has been used once. (it doesn't accelerate anything, having
that memory or not allocated does not make any difference at all to
subsequent requests. It is just blocking system resources - as is
obvious by replacing the process with a fresh one)


Again:
* I don't have a problem with pootle taking half a day to import the
files for a new release (one time thing, no big deal)
* I don't have a problem with pootle taking very long for the very
first hit of the frontpage after a reboot (system is not rebooted that
often, also no big deal)
* I don't have a problem with restarting pootle server-processes to
workaround the memory-leak (whether you call it leak or not, I
definitely call it leak). It limits the amount of concurrent worker
processes, but that is not a problem since even with just the two as
in the VM, you can easily serve 50 concurrent requests per second
while benchmarking (resulting in being able to handle about 200
request/second of the frontpage, thus much reserves available. No
problem)
* I don't have a problem with you
* I don't have a problem with Rimas (or anyone else here)

* I see a problem in pootle not having a queue or other limitation on
the "create zips" case. It is very easy to take a server out for at
least a couple of minutes by requesting the zips of a huge project for
multiple projects at once. 8 CPU system with 8 workers → just request
8 different languages and all workers will be using 100% each for
multiple minutes, additionally fighting a little over disk-i/o and
will not be available for other stuff, and the preparations are slow
(let alone the waste of memory resulting from not releasing it again).

* I have a problem with top-post & fullquote. As (with a very high
confidence) this is a clear sign of the person not reading the post,
not trying to understand the post
* I have a problem with having to write the same stuff over and over
again. I get angry when I have to, and my tone might then no longer be
100% appropriate.

ciao
Christian

--
Unsubscribe instructions: E-mail to l10n+***@libreoffice.org
Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
List archive: http://listarchives.libreoffice.org/www/l10n/
All messages sent to this list will be publicly archived and cannot be
Andre Schnabel
2011-04-07 11:37:55 UTC
Permalink
Hi Christian, *


>
> No, on the contrary, especially because it is run on so many servers
> with limited resources, I'm not willing to waste RAM on pootle-VM
> alone, when pootle doesn't need it at all.

Please leave the technical discussion aside for a minute.

What *we* as a team of localizers need is a working and performant
setup to do our translation work. All data that you retreaved from
the brazilian pootle server are just peanuts and not relevant for
what we need to have in the next few weeks for several reasons:

- we had no full localization on the server (but we will have to provide
this for 3.4 localization)

- hardly any team did full translation on the server, we "only" did
bugfixes

There were some good reasons to have a rather good equipped server for
pootle. We knew, that pootle might be not the perfect solution regarding
memory usage, multithreading ... That we now cannot use the initialy
planned setup is very unfortunate. But we still need a system that
provides similar performance.


>
> I clearly explained why I'm convinced that it is a memory leak. It
> doesn't free the memory, even when the machine is idling for hours. It
> doesn't need that memory, since when the worker is replaced by a fresh
> one, the functionality still works.

No - it is just that this is totally irrelevant in the current situation.
Even if pootle has a memory leak, we won't fix this within the next few
days. But we need to start translating asap!

So - if there is any way to provide more ressources (memory) we should do
this and analyze the root cause later (I'm sure, pootle developers are
interested to help with this).

>
> Again:
> * I don't have a problem with pootle taking half a day to import the
> files for a new release (one time thing, no big deal)
> * I don't have a problem with pootle taking very long for the very
> first hit of the frontpage after a reboot (system is not rebooted that
> often, also no big deal)
> * I don't have a problem with restarting pootle server-processes to
> workaround the memory-leak (whether you call it leak or not, I
> definitely call it leak). It limits the amount of concurrent worker
> processes, but that is not a problem since even with just the two as
> in the VM, you can easily serve 50 concurrent requests per second
> while benchmarking (resulting in being able to handle about 200
> request/second of the frontpage, thus much reserves available. No
> problem)

Maybe you don't have, but the translators will have a problem with this.
So either we can provide more ressources to the VM or we need a
different solution.

All other discussion is very academic at the moment.

regards,

André


--
Unsubscribe instructions: E-mail to l10n+***@libreoffice.org
Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
List archive: http://listarchives.libreoffice.org/www/l10n/
All messages sent to this
Christian Lohmaier
2011-04-07 13:10:30 UTC
Permalink
Hi Andre, *,

On Thu, Apr 7, 2011 at 1:37 PM, Andre Schnabel <***@gmx.net> wrote:

> Please leave the technical discussion aside for a minute.

No, cannot do that, as this is directly related to your point:

> What *we* as a team of localizers need is a working and performant
> setup to do our translation work.

Yes, and I fully agree, and that of course is also my highest measuremen/goal.

It is my true belief that the current setup will handle this just fine.

> - we had no full localization on the server (but we will have to provide
> this for 3.4 localization)

See the other messages. What is timeconsuming (and no memory will help
here, as it is is CPU bound) is importing of the new data, and time
until it is ready after a server-restart. Nothing to do about it,
apart from the admins actually doing the import using differently
formed comments that make use of all of the assigned CPUs.

> - hardly any team did full translation on the server, we "only" did
> bugfixes

See my post. I'm /sure/ the server can handle the full
online-translation within pootle.
I'm sure that the server will not get more than 10 apache requests per
seconds on average, and the server can handle about 200

> There were some good reasons to have a rather good equipped server for
> pootle.

A memory leak/wasteful setup is not a good reason for this.

> We knew, that pootle might be not the perfect solution regarding
> memory usage, multithreading ... That we now cannot use the initialy
> planned setup is very unfortunate. But we still need a system that
> provides similar performance.

Again: If performance is not satisfactory for translation work, I'll
reconsider. But again: I don't see any evidence that performance could
be increased by assigning more RAM to the VM.

>> I clearly explained why I'm convinced that it is a memory leak. It
>> doesn't free the memory, even when the machine is idling for hours. It
>> doesn't need that memory, since when the worker is replaced by a fresh
>> one, the functionality still works.
>
> No - it is just that this is totally irrelevant in the current situation.

No it is not.

> Even if pootle has a memory leak, we won't fix this within the next few
> days. But we need to start translating asap!

Again: It is /perfeclty working/ when restarting the worker threads.
And the translator will not notice that the worker-thread has been
replaced. The translator will not notice whether the few milliseconds
he has to wait are
* because the keep-alive expired and a new http-connection has to be negotiated
* a server process expired and thus is restarted
* seeking through the actual file for next/previous match took a little longer.

> So - if there is any way to provide more ressources (memory) we should do
> this and analyze the root cause later (I'm sure, pootle developers are
> interested to help with this).

/IF/ there are memory related problems, I'll assign more RAM. But
again: everything I experienced so far says: More memory won't help at
all.

>> Again:
>> * I don't have a problem with pootle taking half a day to import the
>> files for a new release (one time thing, no big deal)
>> * I don't have a problem with pootle taking very long for the very
>> first hit of the frontpage after a reboot (system is not rebooted that
>> often, also no big deal)
>> * I don't have a problem with restarting pootle server-processes to
>> workaround the memory-leak (whether you call it leak or not, I
>> definitely call it leak). It limits the amount of concurrent worker
>> processes, but that is not a problem since even with just the two as
>> in the VM, you can easily serve 50 concurrent requests per second
>> while benchmarking (resulting in being able to handle about 200
>> request/second of the frontpage, thus much reserves available. No
>> problem)
>
> Maybe you don't have, but the translators will have a problem with this.
> So either we can provide more ressources to the VM or we need a
> different solution.

Again: Those stuff that takes ages are /NOT/ solvable by assigning
more ressources to the VM. Those are CPU-bound, and all of them are
single-threaded.
If the process maxes out a single CPU, that is as much speed as you
can get, no matter how many RAM is sitting idle.

* Pootle has a very, very poor first-start performance.
→ not a problem as the server will not be rebooted every other day.
And in case this wasn't clear: Restarting the worker-processes will
/not/ have the same effect, the user will not notice anything,

* Pootle has poor performance when generating the zips for download
(fist trigger per language and project)
→ This again is CPU bound, and again: More RAM will not help.

This is the only case where the user can have a problem (and is part
of the problem).
It doesn't help to click multiple download links on one page to get
the zips faster, on the contrary. Click one, wait for the processing
to be done, and when you got the first one, feel free to download all
the remaining ones.
When multiple users request huge zips at the same time, all server
processes are busy. Currently there are two, can be increased to 4,
but that's not a big difference. It is CPU bound (and also does a
little disk i/o) and I repeat myself once again: MORE RAM WILL NOT
HELP!!!

If you want a dedicated pootle server, than order a 64 core system.
With that, you can hide pootle's weak spots. It will be idle 99% of
the time, but for the point of time where 10+ people get the idea to
download the zips at the same time, you won't run out of CPUs.

> All other discussion is very academic at the moment.

No - it is just very tiring that people still stick with their guesswork.

Again: The only problem is creation of zips for download. This takes
(in terms of web-responsiveness) /ages/, and blocks the server's
process while it is performed.
It is CPU-bound, and thus the amount of workers one can have is very limited.
Pootle must be reconfigured to limiit these actions, or (more simple
to implement as a workaround just create all the zips once per day in
a cronjob (or better distributed during the day, as creating them all
at once would again create the CPU-bottleneck) and only allow to
download those. Thus you don't get 100%up-to-date versions of the
file.
But when you download the zips, you're not interested in the current
state anyway, as just after you download someone else could have
edited in pootle thus creating the same issue.

Again: * Don't request additional zips of a project before the first
one is delivered to you.

ciao
Christian

--
Unsubscribe instructions: E-mail to l10n+***@libreoffice.org
Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
List archive: http://listarchives.libreoffice.org/www/l10n/
All messages sent to this list will be publicly archived and cannot be dele
Christian Lohmaier
2011-04-07 17:38:56 UTC
Permalink
Hi *,

On Thu, Apr 7, 2011 at 3:10 PM, Christian Lohmaier
<lohmaier+***@googlemail.com> wrote:
> On Thu, Apr 7, 2011 at 1:37 PM, Andre Schnabel <***@gmx.net> wrote:
>
>> What *we* as a team of localizers need is a working and performant
>> setup to do our translation work.
>
> Yes, and I fully agree, and that of course is also my highest measuremen/goal.

I enabled caching of static content (images, the javascript, css), so
while that has no impact on the speed of the server itself, it has
huge impact on the feel of the site for users, as now the huge part of
a request doesn't have to be re-downloaded each time (the html is less
than 20% of a typical page request). Now the browser only needs to
download the html and can use the other files from its cache.

> [...]
> * Pootle has poor performance when generating the zips for download
> (fist trigger per language and project)
> → This again is CPU bound, and again: More RAM will not help.
> [...]
> When multiple users request huge zips at the same time, all server
> processes are busy. Currently there are two, can be increased to 4,
> but that's not a big difference. It is CPU bound (and also does a
> little disk i/o) and I repeat myself once again: MORE RAM WILL NOT
> HELP!!!

As this is the real (and only) problem with the server (a few users
reuqest the zips and thus block the workers that then cannot handle
other requests until generating the zips is finished) I kept thinking
about it and the easiest solution seems to just seperate the two, i.e.
create a seperate WSGIDaemonProcess group and use that for the
export-requests.

Thus only the exporters can run out - but then only the others users
wanting the zips will have to wait, those who just want to
review/translate in pootle itself can continue their work.

Thus especially to Friedel and Dwayne and Rimas: Is there a problem with adding

WSGIDaemonProcess pootle-export threads=3 stack-size=1048576
maximum-requests=5 inactivity-timeout=1200 display-name=%{GROUP}

<Location /*/*/export/zip>
WSGIProcessGroup pootle-export
</Location>

to the vhost's config? In turn the process lifetime of the regular
threads can then be increased again. How many of the export workers
can be allowed then is more a question of stress concurrent ones add
to the i/o (be it mysql related or raw disk i/o) and less of memory
wastage.

ciao
Christian

--
Unsubscribe instructions: E-mail to l10n+***@libreoffice.org
Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
List archive: http://listarchives.libreoffice.org/www/l10n/
All messages sent to this list
Rimas Kudelis
2011-04-07 20:16:21 UTC
Permalink
Hi,

2011.04.07 20:38, Christian Lohmaier rašė:
> Thus especially to Friedel and Dwayne and Rimas: Is there a problem with adding
>
> WSGIDaemonProcess pootle-export threads=3 stack-size=1048576
> maximum-requests=5 inactivity-timeout=1200 display-name=%{GROUP}
>
> <Location /*/*/export/zip>
> WSGIProcessGroup pootle-export
> </Location>
>
> to the vhost's config? In turn the process lifetime of the regular
> threads can then be increased again. How many of the export workers
> can be allowed then is more a question of stress concurrent ones add
> to the i/o (be it mysql related or raw disk i/o) and less of memory
> wastage.

I like the idea, but I'll leave it to Friedel and Dwayne to judge if
it's problematic or not.

Rimas

--
Unsubscribe instructions: E-mail to l10n+***@libreoffice.org
Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
List archive: http://listarchives.libreoffice.org/www/l10n/
All messages sent to this
Christian Lohmaier
2011-04-10 16:51:26 UTC
Permalink
Hi *,

On Thu, Apr 7, 2011 at 10:16 PM, Rimas Kudelis <***@akl.lt> wrote:
> 2011.04.07 20:38, Christian Lohmaier rašė:
>>
>> Thus especially to Friedel and Dwayne and Rimas: Is there a problem with
>> adding
>>
>> WSGIDaemonProcess pootle-export threads=3 stack-size=1048576
>> maximum-requests=5 inactivity-timeout=1200 display-name=%{GROUP}
>>
>> <Location /*/*/export/zip>
>>     WSGIProcessGroup pootle-export
>> </Location>
> [...]
> I like the idea, but I'll leave it to Friedel and Dwayne to judge if it's
> problematic or not.

As there has been no reply, I just did add that (with LocationMatch
instead of Location, as it's not just lang/project/export/zip, but
<variable-length>/export/zip) and also limited the number of
concurrent export jobs to 1, (otherwise requesting zips for the same
language and project would consume ~twice the amount of time - if the
user waits until first processing is finished, and then requests the
other zip, the user will get that second zip instantly.

There don't seem to be any drawbacks, so that's the way to go - this
way regular translation will not be blocked/affected by the exporting
blocking all available worker slots or all RAM or all CPU

ciao
Christian

--
Unsubscribe instructions: E-mail to l10n+***@libreoffice.org
Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
List archive: http://listarchives.libreoffice.org/www/l10n/
All messages sent to this list will be publicly archived and cannot b
Christian Lohmaier
2011-04-11 14:42:22 UTC
Permalink
Hi *,

On Sun, Apr 10, 2011 at 6:51 PM, Christian Lohmaier
<lohmaier+***@googlemail.com> wrote:
> On Thu, Apr 7, 2011 at 10:16 PM, Rimas Kudelis <***@akl.lt> wrote:
>> 2011.04.07 20:38, Christian Lohmaier rašė:
> [seperate worker for export/zip]
> There don't seem to be any drawbacks, so that's the way to go - this
> way regular translation will not be blocked/affected by the exporting
> blocking all available worker slots or all RAM or all CPU

Unfortunately, real-usage also lets the regular workers grow insanely
(from standard of around 80/90MB to700MB) - thus I had to reduce the
lifetime of the regular worker again as well.

ciao
Christian

--
Unsubscribe instructions: E-mail to l10n+***@libreoffice.org
Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
List archive: http://listarchives.libreoffice.org/www/l10n/
All messages sent to this list will be publicly archiv
Anton Meixome
2011-04-12 13:49:51 UTC
Permalink
Hello,
I'm experiencing a horrible performance in Pootle since a couple of days.
Is not possible only affects me

- strings not translated but I can't access to them
- upload de .po is too slow (more than 2 minutos for 45KB)
- fatal performance of navegation

... Please,

Errors:



Server error!

The server encountered an internal error and was unable to complete
your request.

Error message:
Premature end of script headers: wsgi.py

If you think this is a server error, please contact the webmaster.

Error 500

translations.documentfoundation.org
Tue Apr 12 15:39:06 2011
Apache

2011/4/11 Christian Lohmaier <lohmaier+***@googlemail.com>:
> Hi *,
>
> On Sun, Apr 10, 2011 at 6:51 PM, Christian Lohmaier
> <lohmaier+***@googlemail.com> wrote:
>> On Thu, Apr 7, 2011 at 10:16 PM, Rimas Kudelis <***@akl.lt> wrote:
>>> 2011.04.07 20:38, Christian Lohmaier rašė:
>> [seperate worker for export/zip]
>> There don't seem to be any drawbacks, so that's the way to go - this
>> way regular translation will not be blocked/affected by the exporting
>> blocking all available worker slots or all RAM or all CPU
>
> Unfortunately, real-usage also lets the regular workers grow insanely
> (from standard of around 80/90MB to700MB) - thus I had to reduce the
> lifetime of the regular worker again as well.
>
> ciao
> Christian
>
> --
> Unsubscribe instructions: E-mail to l10n+***@libreoffice.org
> Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
> List archive: http://listarchives.libreoffice.org/www/l10n/
> All messages sent to this list will be publicly archived and cannot be deleted
>



--
Antón Méixome - Blog about Galician Office Suite
Galician community OOo.org & LibO
http://blog.openoffice.gl // http://blog.libreoffice.gl

--
Unsubscribe instructions: E-mail to l10n+***@libreoffice.org
Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
List archive: http://listarchives.libreoffice.org/www/l10n/
All messages sent to this list will be publicly archived and cannot be delet
Andras Timar
2011-04-12 14:06:49 UTC
Permalink
2011/4/12 Anton Meixome <***@certima.net>:
> Hello,
> I'm experiencing a horrible performance in Pootle since a couple of days.
> Is not possible only affects me
>
> - strings not translated but I can't access to them

I experienced the same. Solution: on Settings page you can set how
many rows are displayed on one page. When I set it to 50 I had the
same problem. I set it to 30 and now it is fine.

> - upload  de .po is too slow (more than 2 minutos for 45KB)
> - fatal performance of navegation
>

I hope when the help update processes - which has been running in the
background since morning - finish, the site will be more responsive.

Best regards,
Andras

--
Unsubscribe instructions: E-mail to l10n+***@libreoffice.org
Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
List archive: http://listarchives.libreoffice.org/www/l10n/
All messages sent to this list will be pub
Christian Lohmaier
2011-04-12 15:31:36 UTC
Permalink
Hi *,

On Tue, Apr 12, 2011 at 4:06 PM, Andras Timar <***@gmail.com> wrote:
> 2011/4/12 Anton Meixome <***@certima.net>:
>> Hello,
>> I'm experiencing a horrible performance in Pootle since a couple of days.
>> Is not possible only affects me
>>
>> - strings not translated but I can't access to them
>
> I experienced the same. Solution: on Settings page you can set how
> many rows are displayed on one page. When I set it to 50 I had the
> same problem. I set it to 30 and now it is fine.
>
>> - upload  de .po is too slow (more than 2 minutos for 45KB)
>> - fatal performance of navegation
>
> I hope when the help update processes - which has been running in the
> background since morning - finish, the site will be more responsive.

The problem is that the update is not run as much in the background
than it was though, on the contrary it pretty much blocks regular
operation to to the heavy i/o load it causes.

This is partially to blame on a bad database design used by pootle,
and to the fact that the mysqldb currently uses myisam, and that locks
the whole database table, thus limits concurrency.

ciao
Christian

--
Unsubscribe instructions: E-mail to l10n+***@libreoffice.org
Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
List archive: http://listarchives.libreoffice.org/www/l10n/
All messages sent to this list will be publicly archived and cannot be deleted
Anton Meixome
2011-04-12 16:09:45 UTC
Permalink
At this moment :-(


Pootle: Install

Error: "(1040, 'Too many connections')" while attempting to access the
Pootle database, will try to initialize database.





2011/4/12 Christian Lohmaier <lohmaier+***@googlemail.com>:
> Hi *,
>
> On Tue, Apr 12, 2011 at 4:06 PM, Andras Timar <***@gmail.com> wrote:
>> 2011/4/12 Anton Meixome <***@certima.net>:
>>> Hello,
>>> I'm experiencing a horrible performance in Pootle since a couple of days.
>>> Is not possible only affects me
>>>
>>> - strings not translated but I can't access to them
>>
>> I experienced the same. Solution: on Settings page you can set how
>> many rows are displayed on one page. When I set it to 50 I had the
>> same problem. I set it to 30 and now it is fine.
>>
>>> - upload  de .po is too slow (more than 2 minutos for 45KB)
>>> - fatal performance of navegation
>>
>> I hope when the help update processes - which has been running in the
>> background since morning - finish, the site will be more responsive.
>
> The problem is that the update is not run as much in the background
> than it was though, on the contrary it pretty much blocks regular
> operation to to the heavy i/o load it causes.
>
> This is partially to blame on a bad database design used by pootle,
> and to the fact that the mysqldb currently uses myisam, and that locks
> the whole database table, thus limits concurrency.
>
> ciao
> Christian
>
> --
> Unsubscribe instructions: E-mail to l10n+***@libreoffice.org
> Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
> List archive: http://listarchives.libreoffice.org/www/l10n/
> All messages sent to this list will be publicly archived and cannot be deleted
>
>



--
Antón Méixome - Blog about Galician Office Suite
Galician community OOo.org & LibO
http://blog.openoffice.gl // http://blog.libreoffice.gl

--
Unsubscribe instructions: E-mail to l10n+***@libreoffice.org
Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
List archive: http://listarchives.libreoffice.org/www/l10n/
All messages sent to this list will be publicly archived and cannot be deleted
Christian Lohmaier
2011-04-12 16:39:34 UTC
Permalink
On Tue, Apr 12, 2011 at 6:09 PM, Anton Meixome <***@certima.net> wrote:
> At this moment :-(
>
>
> Pootle: Install
>
> Error: "(1040, 'Too many connections')" while attempting to access the
> Pootle database, will try to initialize database.

Yes, this is beacuse it wasn't as much a background task as it was
intended, but did put way too much load on the system, mainly on mysql
- the process has now been interrupted so that mysql can recover.

That limit as it turns out was too high already, as it allowed for too
many extensive operations to be queued up that are all fighting for
ressources, getting over the critical point where the system could not
keep up.

I put the site into maintenance mode to give it a chance to recover.

Please check again in 10 minutes or so.

ciao
Christian

--
Unsubscribe instructions: E-mail to l10n+***@libreoffice.org
Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
List archive: http://listarchives.libreoffice.org/www/l10n/
All messages sent to this list will be publicly archived and cannot be deleted
Christian Lohmaier
2011-04-12 17:48:48 UTC
Permalink
Hi *,

On Tue, Apr 12, 2011 at 6:39 PM, Christian Lohmaier
<lohmaier+***@googlemail.com> wrote:
>
> I put the site into maintenance mode to give it a chance to recover.

Oups - almost for got to unlock it again. Pootle is ready for use
again, sorry for the inconvenience

ciao
Christian

--
Unsubscribe instructions: E-mail to l10n+***@libreoffice.org
Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
List archive: http://listarchives.libreoffice.org/www/l10n/
All messages sent to this list will be publicly archived and cannot be deleted
Olivier Hallot
2011-04-12 17:56:09 UTC
Permalink
Wow... a breeze now....

Thanks indeed.

Olivier

Em 12-04-2011 14:48, Christian Lohmaier escreveu:
> Hi *,
>
> On Tue, Apr 12, 2011 at 6:39 PM, Christian Lohmaier
> <lohmaier+***@googlemail.com> wrote:
>>
>> I put the site into maintenance mode to give it a chance to recover.
>
> Oups - almost for got to unlock it again. Pootle is ready for use
> again, sorry for the inconvenience
>
> ciao
> Christian
>

--
Olivier Hallot
Founder, Steering Commitee Member - The Document Foundation
Voicing the enterprise needs
LibreOffice translation leader for Brazilian Portuguese

--
Unsubscribe instructions: E-mail to l10n+***@libreoffice.org
Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
List archive: http://listarchives.libreoffice.org/www/l10n/
All messages sent to this list will be publicly archived and cannot be deleted
F Wolff
2011-04-06 16:11:29 UTC
Permalink
Op Wo, 2011-04-06 om 10:50 +0200 skryf Christian Lohmaier:
> Hi *,


Hi Christian

I find your responses rude and unhelpful. I'll try to assume that
communication isn't going smoothly since we might both be using our
second language.

Somehow we manage to provide this software on hundreds of sites running
fine, ranging from hosting on a few hundred megabytes of RAM to machines
with several gigabytes. All accidents?

I could show you the specific lines of code caching big objects
(expected to be tens to hundreds of megabytes in size), but I guess that
won't convince you either. It is a leak, because someone who (I guess)
never looked at the Pootle code, says so. Why can't you at least start
by assuming that I might have an idea of how Pootle works? I don't have
time to explain each optimisation and feature we have in the code. My
time is limited, and I was hoping we can work together instead of giving
lectures about programming and system administration.

When you are willing to discuss things under the good faith assumption
that I _might_ not be talking nonsense, we can continue the
conversation. I've been trying to help you in my free time based on my
experience, but it seems you'd prefer to assume I don't know what I'm
talking about.

With mutual respect, we can take this forward, but not without it. And
that includes respect for the hard work that Rimas is doing.


In the meanwhile, here is some recommended reading:
http://effbot.org/pyfaq/why-doesnt-python-release-the-memory-when-i-delete-a-large-object.htm


Keep well
Friedel



> On Wed, Apr 6, 2011 at 9:58 AM, Dwayne Bailey <***@translate.org.za> wrote:
> >
> > Please CC me on any replies as I'm not on the list.
>
> Keeping this, so others will follow the request as well.
>
> >> Mon, 4 Apr 2011 22:28:43 +0200
> >> From:
> >> Christian Lohmaier
> >> On Mon, Apr 4, 2011 at 2:58 PM, Rimas Kudelis <***@akl.lt> wrote:
> >> Again, as you apparently still don't understand what I already wrote many
> >> times:
> >> * Adding more resources will /NOT/ make pootle run faster than it does
> >> now.
> >> The VM already has way more resources assigned than necessary. It is
> >> /idle/ almost all the time.
> >
> > As far as I know the server hasn't really been used yet, so I guess
> > we'll be collecting data from now on to see how things go.
>
> Also from the avialble data from the old pootle server. But yes, the
> new one doesn't have much data yet.
>
> > During the
> > setup of the server, we make tradeoffs between performance and memory
> > use. If there is no memory available, we'll obviously try to optimise at
> > all cost for minimising memory use,
>
> Please explain those settings. Which settings, what is the effect, how
> to see the effect (i.e. what UI actions to perform)
>
> > and that is what I understand that
> > Rimas said: things might be slower than necessary, since we are not
> > optimising for performance, but for memory use.
>
> No, and I repeat again: Memory is not an issue. The memory leak is.
>
> >> * The only thing that is slow (when executed the first time) is
> >> generation of the zips. So when you as translator request a zip: Don't
> >> click the link multiple times because you don't immediately get the
> >> zip. It can take 10 seconds for the files to be generated. Again:
> >> * Adding more resources will /not/ make that time shorter. It is a
> >> single-threaded process that can only uses one single CPU, thus
> >> assigning more CPUs won't help at all (the VM has 4CPUs assigned
> >> already)
> >> Requesting that same zip another time (or different zips of the
> >> project belonging to the same language is fast/instant, but requesting
> >> the zip for another language again may take some seconds for the first
> >> request (or again after the files did change in between).
> >> * Pootle has a memory leak when creating the zips. It won't release
> >> memory after processing the files.
> >> This would be the only time where the assigned resources may run out
> >> (the VM has 1GB or RAM assigned): Multiple different languages request
> >> the zip at the same time. Then memory usage increases, memory runs out
> >> and either it is crawling along or the process gets killed.
> >
> > Some stuff that is slow to load is cached for later use.
>
> "Some stuff" maybe, but that little is not what I'm talking about.
>
> > This is done
> > for performance optimisation. This is one of the reasons you won't see
> > the memory use go down immediately after generating a ZIP file.
>
> No, this has nothing to do with caching. It is a memory leak. Rimas is
> very good at not forwarding relevant information it seems. So here I
> just copy and paste what I wrote to Rimas already.
> #############################
> >>> [Rimas]
> >>> After generating, the zip files are cached on disk, so before
> >>> redownloading the same file, you'll probably want to execute
> >>> $ find /var/www/pootle/po/POOTLE_EXPORT/ -type f -iname *.zip -exec rm {}
> >>> \;
>
> No, this is not enough to do the same as what pootle does when
> requesting the files the first time.
> The first time it takes long (100% CPU while processing), and that is
> also the time where the memory leak occurs. After it is finished
> producing the file, the memory is not freed.
>
> Requesting one of the zips interestingly also prepares the other files
> at once, at least when selecting any of the other files on the same
> language and project, the file will be served immediately, without the
> lengthy processing step.
> (And those requests also don't increase the memory usage further)
>
> But switching to another language, and requesting a file results in
> the processing with memory leak again.
>
> But that memory is not needed at all, as is obvious when the process
> has reached it's max-requests and is replaced by a new one, then it
> serves all of the already covered languages without that increase in
> memory-usage.
>
> With that info at hand, I tweaked the apache-settings a bit (basically
> reduced the amount of concurrent wsgi processes, and reducing their
> lifetime) to make the accumulation less likely.
>
> The machine will still run out-of memory when different language zips
> are requested in a short amount of time with not enough regular
> requests in between. In case people see too many "premature end of
> script" or similar (i.e. the worker has been killed by the OOM
> killer), reduce the number of requests before the worker is restarted.
> Currently it is at 400, which is still pretty high, considering the
> more or less non-existant regular load (but then again, this might be
> because people know the server is in transition and because 3.3 is
> done already)
>
> Similarily, the very first page-load after a reboot also takes "ages",
> but after that it runs smoothly,
>
> 1GB is enough for Pootle The rest is a matter of
> configuration/tweaking. (or best: fixing the memory leak)
> #######################
> > Another
> > reason is the way the garbage collector works in Python.
>
> Well - how long should one wait for the memory to be freed? If it is
> not freed when memory is about to run out, or after a hour or so, then
> I don't assume it is a problem of running the garbage collector. (and
> again that memory is not used/needed for further request of the same
> kind, so I don't take your "it is caching stuff to accelerate future
> actions" for granted.)
>
> Please give me instructions on stuff to do in pootle to notice a difference.
>
> > Deciding to cache something is a tradeoff. So we can disable or minimize
> > some of the caching, which will simply make a few things slower,
> > hopefully not by much, but we're guessing while the server hasn't been
> > used much yet.
>
> Numbers and examples please.
>
> Caching for a more or less one-time operation that impacts the whole
> server all the time surely is the wrong way to go. Even more so when
> despite that "caching" (I still call it leak, as it doesn't cache
> anything) the requesting of the zips is still CPU bound and slow.
>
> > I suggested some customisations to the parse pool (to do exactly this).
> > That affects the number of cached files and search indexes, both of
> > which are very large on your server.
>
> Please be more detailed on this. What setting, in what way to tweak,
> and what should the effect be?
> Numbers?
>
> >> * I will NOT assign RAM to a VM (and thus block that ram for other
> >> use) to satisfy a memory leak, when that RAM is unused 99% of the
> >> time.
> >
> > I believe what you are seeing is the caching, not a memory leak.
>
> And I strongly disagree, see above.
>
> > We haven't seen the server used much yet. My educated guess from having
> > worked on a few Pootle installations is that the RAM isn't enough,
>
> And I strongly disagree. See above.
>
> >> * The effects of the memory leak can be nullified by just restarting
> >> the worker processes more frequently. Thus again:
> >
> > ...at the cost of making things slower, since more stuff needs to be
> > loaded in memory afresh every time you restart a process.
>
> I'm starting to belive you're in the same league as Rimas when it
> comes to server administration (i.e. very much guesswork).
>
> The starting/initializing of the processes is negligible. Just fire up
> an ab bechmark and request 20000 with concurrency of 30 and look at
> the distribution.
> I'd gladly live with a few hundred of milliseconds, if that will save
> hundreds of memory that can be freed by just restarting the worker
> process.
>
> >> * Adding more resources will /NOT/ make the VM run faster
> >
> > It most probably will, since we are sacrificing performance to minimise
> > memory use.
>
> No. As the times where it is slow is not bound by memory, but by CPU.
> Please please please: Read what I wrote. I repeated myself so many
> times, it really gets annoying.
>
> > For example: we opted for more threads, rather than
> > processes, that is known not to perform as well in Python, especially
> > for CPU intensive tasks.
>
> Who is "we"? Yes, I decided to (for now) only run with 2 wsgi
> processes. But then again: how would that make execution faster?
>
> If you provide me with hard numbers, concrete settings and
> instructions on what to do in the UI to reproduce and compare the
> settings, I'm willing to try them out.
>
> >> it will
> >> /NOT/ allow it to handle more requests
> >
> > It most probably will, since we reduced the number of processes to
> > minimise memory use, and slower serving of requests necessarily affects
> > the number of requests you can serve in any given time.
>
> Again this is wrong. it might allow 4 simultaneous "request the zip"
> requests, but all 4 would be taking ages, and all would have the
> memory leak, thus needing more RAM (that I'm not wiling to assign,
> since regular operation just doesn't need it).
> (and the users that are just using regular webinterface have to wait
> then as well). Allowing more processes doesn't make sense when pootle
> doesn't queue/limit those lengthy processes by itself.
>
> >> Pootle is idling almost all of the time. There is less than one apache
> >> request per second on average (and for regular requests (i.e.
> >> non-"generate-a-zip" actions) it can easily serve >>50 simultaneous
> >> requests per second.)
> >
> > Let's keep an eye on things when people actually start to use the
> > server.
>
> Whether the requests are generated by a benchmark tool or by regular
> users doesn't make any difference. The number it can handle is way
> beyond any real load to expect.
>
> >> > Maybe if we
> >> > manage to give enough load to the server, he'll change his mind (or
> >> > we'll
> >> > find other ways to deal with the problem).
> >>
> >> No, I won't change my mind, but depending on the load/effects of the
> >> memory leak I'll reduce lifetime of the server-processes further.
> >
> > I hope you will be reasonable and look at the data as it becomes
> > available, and at least consider changing your mind.
>
> Again this is out of the question, but all *data*, real facts I
> gathered so far don't suggest that it would be any different from my
> current opinion.
>
> > As mentioned, I'm
> > pretty sure there is no memory leak.
>
> I disagree, see above. Alternatively give a reasonable explanation on
> on why the export still works fine even when the process that created
> it (and thus the memory it occupied, and all possible caching it could
> have down in its own space) is gone.
>
> > If you reduce the lifetime of the
> > server processes, you are just making performance worse, which is all
> > that Rimas warned the users for.
>
> Numbers/examples on how performance can be worse. Otherwise I cannot
> take your comments seriously, as my measurements did show otherwise.
>
> > [...]
> > I agree. Generating the ZIP files is slow. Doing it multithreaded, will
> > limit the performance for more users while doing that.
>
> But the time they have to wait will be shorter.
>
> >> * Requesting the other zips in the same project & language is fast.
> >>
> >> So there is no point in clicking all zip-URLs on a page at once (on
> >> the contrary, than the request will all cause CPU to be burnt, while
> >> all that is only needed one single time). If you want to download more
> >> than one zip of a page, click the first one, wait until it is
> >> generated and handed over to your browser, then click as many others
> >> on the same page and get all of them quickly.
> >
> > Yes, we have optimised for several cases here that are likely, and I
> > suggested some workarounds for some of the issues we're likely to hit
> > with the little bit of RAM as Rimas has already started doing, as far as
> > I know.
>
> Don't suggest them only to Rimas, as he is not communicating well with
> such technical matters it seems.
>
> >> * Any other time, doing in-place-translation, just browsing along is
> >> supposed to be fast.
> >
> > ... as long as we're not hitting the current imposed limits of
> > concurrency.
>
> Again: Numbers, no foggy guesswork please.
>
> ciao
> Christian
--
Recently on my blog:
http://translate.org.za/blogs/friedel/en/content/better-lies-about-gnome-localisation


--
Unsubscribe instructions: E-mail to l10n+***@libreoffice.org
Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
List archive: http://listarchives.libreoffice.org/www/l10n/
All messages sent to this list will be publicly archived and cannot be deleted
Sérgio Marques
2011-03-29 10:06:28 UTC
Permalink
HI Rimas

Thanks for you reply. When merged, I´ll try to update all strings for 3.4

Regards

2011/3/28 Rimas Kudelis <***@akl.lt>

> Hi Sérgio,
>
> 2011.03.29 00:12, Sérgio Marques rašė:
> > For Portuguese language, pootle doesn´t have the strings merged from
> > openoffice (ui and help) nor extensions. Is it possible to add them in
> new
> > server?
>
> Yes. We'll ask Andras to do the merging stuff, and I'll upload the
> resulting files to Pootle.
>
> Darn, we (I?) should keep a TODO on the wiki...
>
> Good night, :)
> Rimas
>
>
> --
> Unsubscribe instructions: E-mail to l10n+***@libreoffice.org
> List archive: http://listarchives.libreoffice.org/www/l10n/
> *** All posts to this list are publicly archived for eternity ***
>



--
Sérgio Marques

--
Unsubscribe instructions: E-mail to l10n+***@libreoffice.org
List archive: http://listarchives.libreoffice.org/www/l10n/
*** All posts to th
Leif Lodahl
2011-03-29 11:32:59 UTC
Permalink
May I suggest that you close the old server in this periode?

Cheers,
Leif Lodahl

2011/3/28 Rimas Kudelis <***@akl.lt>

> Hi,
>
> right now I'm trying to migrate all the data from our current Pootle
> server to the new one. This means that if you make any changes in Pootle
> past this point, they will not be reflected in the new installation, so
> please don't edit your translations in Pootle until further notice, or
> at least make yourself a backup to restore from when done.
>
> Once I'm finished setting Pootle up, we'll all have a few days to test
> it and decide whether or not its performance is acceptable. If it is,
> we'll just continue using it, and if not, we'll see how to proceed from
> there.
>
> So, the bottom line is: if you'll find your latest changes missing at
> some poing, don't be too surprised – it's sort of expected.
>
> Regards,
> Rimas
>
>
> --
> Unsubscribe instructions: E-mail to l10n+***@libreoffice.org
> List archive: http://listarchives.libreoffice.org/www/l10n/
> *** All posts to this list are publicly archived for eternity ***
>

--
Unsubscribe instructions: E-mail to l10n+***@libreoffice.org
List archive: http://listarchives.libreoffice.org/www/l10n/
*** All posts to this list are publicly archived for eternity ***
Rimas Kudelis
2011-04-05 09:02:12 UTC
Permalink
Hi all,

I'm importing files for 3.4 into Pootle now. To make this process
faster, please don't work on it just yet. We'll post here when the
import is finished.

Rimas


--
Unsubscribe instructions: E-mail to l10n+***@libreoffice.org
Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
List archive: http://listarchives.libreoffice.org/www/l10n/
All messages sent to this l
Christian Lohmaier
2011-04-05 13:24:12 UTC
Permalink
Hi *,

2011/4/5 Rimas Kudelis <***@akl.lt>:
>
> I'm importing files for 3.4 into Pootle now. To make this process faster,
> please don't work on it just yet. We'll post here when the import is
> finished.

This import also is not very efficient resource-wise. It stresses disk
i/o, thus wa time makes up much of the load.

Maybe (I don't know) it is more efficient to run multiple manage
processes in parallel, with the task to update one language only
instead of starting one and giving it the task to update all languages
at once.

ciao
Christian

--
Unsubscribe instructions: E-mail to l10n+***@libreoffice.org
Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
List archive: http://listarchives.libreoffice.org/www/l10n/
All messages sent to this list will be publicly archived and cannot be deleted
Christian Lohmaier
2011-04-05 13:27:14 UTC
Permalink
On Tue, Apr 5, 2011 at 3:24 PM, Christian Lohmaier
<lohmaier+***@googlemail.com> wrote:
>
> Maybe (I don't know) it is more efficient to run multiple manage
> processes in parallel, with the task to update one language only
> instead of starting one and giving it the task to update all languages
> at once.

..or if it does similar weirdo stuff like when creating zip, request
one language only at first, and then see whether processing an
additional one is (much) faster than the initial language)

ciao
Christian

--
Unsubscribe instructions: E-mail to l10n+***@libreoffice.org
Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
List archive: http://listarchives.libreoffice.org/www/l10n/
All messages sent to this list will be publicly archived and cannot be deleted
Rimas Kudelis
2011-04-07 14:23:15 UTC
Permalink
Hi all,

Pootle is now unlocked, and you can start working on libo34x_ui and
libo34x_help projects. As usual, the word counts vary a bit, especially
in libo34x_help. Ignore that – the actual numbers should be the same
(checked them in command line), with a little exception of
fr/libo34x_ui, which Andras will take a look at this evening.

Best regards,
Rimas


--
Unsubscribe instructions: E-mail to l10n+***@libreoffice.org
Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
List archive: http://listarchives.libreoffice.org/www/l10n/
All messages sent to this list will be publicly archived and c
Andras Timar
2011-04-07 17:08:03 UTC
Permalink
Hi Sophie,

2011.04.07. 16:23 keltezéssel, Rimas Kudelis írta:
> Pootle is now unlocked, and you can start working on libo34x_ui and
> libo34x_help projects. As usual, the word counts vary a bit, especially
> in libo34x_help. Ignore that – the actual numbers should be the same
> (checked them in command line), with a little exception of
> fr/libo34x_ui, which Andras will take a look at this evening.


I found corrupted lines in formula\source\core\resource.po. I fixed
them, so your wordcount is now the same as other's.

Please note however, that there are the following differences between
LibreOffice source and formula\source\core\resource.po:

Pootle:
#:
core_resource.src#RID_STRLIST_FUNCTION_NAMES.SC_OPCODE_BAHTTEXT.string.text
msgid "BAHTTEXT"
msgstr "BAHTTEXT"

Source:
#:
core_resource.src#RID_STRLIST_FUNCTION_NAMES.SC_OPCODE_BAHTTEXT.string.text
msgid "BAHTTEXT"
msgstr "BAHTTEXTE"

Pootle:
#:
core_resource.src#RID_STRLIST_FUNCTION_NAMES.SC_OPCODE_CHISQ_INV.string.text
msgid "CHISQINV"
msgstr "KHIDEUX.INVERSE"

Source:
#:
core_resource.src#RID_STRLIST_FUNCTION_NAMES.SC_OPCODE_CHISQ_INV.string.text
msgid "CHISQINV"
msgstr "LOI.KHIDEUX.INVERSE"

Pootle:
#:
core_resource.src#RID_STRLIST_FUNCTION_NAMES.SC_OPCODE_TABLE_OP.string.text
msgid "MULTIPLE.OPERATIONS"
msgstr "OPERATION.MULTIPLE"

Source:
#:
core_resource.src#RID_STRLIST_FUNCTION_NAMES.SC_OPCODE_TABLE_OP.string.text
msgid "MULTIPLE.OPERATIONS"
msgstr "OPERATIONS.MULTIPLES"

Please review these strings in Pootle.

Cheers,
Andras

--
Unsubscribe instructions: E-mail to l10n+***@libreoffice.org
Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
List archive: http://listarchives.libreoffice.org/www/l10n/
All messages sent to this list will be publicly arch
Sophie Gautier
2011-04-07 17:50:03 UTC
Permalink
Hi Andras,
On 07/04/2011 20:08, Andras Timar wrote:
> Hi Sophie,
>
> 2011.04.07. 16:23 keltezéssel, Rimas Kudelis írta:
>> Pootle is now unlocked, and you can start working on libo34x_ui and
>> libo34x_help projects. As usual, the word counts vary a bit, especially
>> in libo34x_help. Ignore that – the actual numbers should be the same
>> (checked them in command line), with a little exception of
>> fr/libo34x_ui, which Andras will take a look at this evening.
>
>
> I found corrupted lines in formula\source\core\resource.po. I fixed
> them, so your wordcount is now the same as other's.

Thanks a lot, that's really great :)
>
> Please note however, that there are the following differences between
> LibreOffice source and formula\source\core\resource.po:
>
> Pootle:
> #:
> core_resource.src#RID_STRLIST_FUNCTION_NAMES.SC_OPCODE_BAHTTEXT.string.text
> msgid "BAHTTEXT"
> msgstr "BAHTTEXT"
>
> Source:
> #:
> core_resource.src#RID_STRLIST_FUNCTION_NAMES.SC_OPCODE_BAHTTEXT.string.text
> msgid "BAHTTEXT"
> msgstr "BAHTTEXTE"
>
> Pootle:
> #:
> core_resource.src#RID_STRLIST_FUNCTION_NAMES.SC_OPCODE_CHISQ_INV.string.text
> msgid "CHISQINV"
> msgstr "KHIDEUX.INVERSE"
>
> Source:
> #:
> core_resource.src#RID_STRLIST_FUNCTION_NAMES.SC_OPCODE_CHISQ_INV.string.text
> msgid "CHISQINV"
> msgstr "LOI.KHIDEUX.INVERSE"
>
> Pootle:
> #:
> core_resource.src#RID_STRLIST_FUNCTION_NAMES.SC_OPCODE_TABLE_OP.string.text
> msgid "MULTIPLE.OPERATIONS"
> msgstr "OPERATION.MULTIPLE"
>
> Source:
> #:
> core_resource.src#RID_STRLIST_FUNCTION_NAMES.SC_OPCODE_TABLE_OP.string.text
> msgid "MULTIPLE.OPERATIONS"
> msgstr "OPERATIONS.MULTIPLES"
>
> Please review these strings in Pootle.

Ok, again thanks a lot for your work on this.

Kind regards
Sophie
--
Founding member of The Document Foundation

--
Unsubscribe instructions: E-mail to l10n+***@libreoffice.org
Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
List archive: http://listarchives.libreoffice.org/www/l10n/
All messages sent to this list will be publicly archived a
Continue reading on narkive:
Loading...