[Harbour] hbmk2 .po generation real life testing - speed issue

[Szakáts Viktor]

>> See my above thoughs. Do you think it's possible to do the 'reverse'
>> logic I was referring to? To put it simply, I'd like to be able to  
>> update
>> my full .po (containing translation) with one simple .pot file. In  
>> this
>> mode only new strings should be added. In another (full-sync) mode,
>> which can only occure with merged .pot files, deleted items are
>> also dropped (this is basically what we already have). So what we'd
>> need is a new method to update .po with one source .pot.
>
> I think that you missed one think. You do not have to use autotrans
> at all. The functionality you are asking have existed from beginning
> and it's simple -m option. The only one think you have to  make is:
>
>   hbi18n -m mergedtranslation.{pot,po} <new_pot_files,...>
>
> and inside mergedtranslation.{pot,pot} you will have all existing
> translation and if sth new has appeared in <new_pot_files,...> it
> will be added to mergedtranslation.{pot,po}.
> The -m option works like -a and it tries to save any existing  
> translations
> in any of the given .po[t] files.

Hm, that's exactly what I need. I'll try to implement this
method in hbmk2, but probably only sometimes next week.

>> [ Here the compromise is that the .po file will contain even deleted
>> strings until we don't issue a full sync, but in practice this  
>> seems to
>> be okay to me. ]
>
> Exactly and this is the result of above command. It's also possible  
> that
> line numbers will be replicated when translation strings will be moved
> in source code due to some other modifications.
> If you want to remove all garbage information like old unused  
> translations
> then it's enough to make:
>   hbi18n -a cleanmergedtrans.{pot,po} mergedtranslation.{pot,po,hbl}

Okay, I'll try to add that to hbmk2 as a separate command.

The only remaining issue here is how to ensure that all source
.pots are present. One option is to run a harbour -j -s on all
sources in a separate pass when this option is chosen, or
maybe we could also tweak -j option to *always* create a .pot file,
even if there isn't any translatable strings (in this case an
empty one). This way it's easy to rebuild what's needed on
demand, and the whole pass can be done incrementally. It wouldn't
raise disk space requirements on modern FSes.

> Maybe I'm missing sth but for me it looks that you already have  
> everything
> what you are asking for.

It seems to me also, except the missing .pot uncertainty issue.
I'll know for sure after I reimplemented hese parts using above
information.

> There is also one thing related to -j+ option we should discuss.
> Should it merge existing and new data or only append new one?
> The merge can be time consumed because compiler will have to open
> existing files and decode all existing strings then append new ones
> respecting already existing strings and translations (we will have
> to store also existing translations) and then the whole file will
> have to be generated from scratch. For very long .pot files it can
> take a lot of time reducing the compilation performance.
> We can also append new information without decoding the old one.

Few j+ scenarios:

a) multiple/single file(s), single output, addition
    - useful, but not without potential drawbacks for incremental  
build situations.
      Consecutive compile sessions can make the .pot grow very large
      until forcing a clean rebuild. Difficult to guesstimate
      whether we save more time by eliminating .pot merge in hbmk2,
      than we lose by dealing with size growth of single output .pot.
      My guess is that it will for normal situation, but it's just a  
guess.
      This feature will certainly force periodic rebuilds in order to
      perform well.

b) multiple/single files(s), single output, merge
    - not usable for incremental build situations.
      Old entries may be deleted in a partial project compile session.
    - usable for small projects as it can make hbi18n unnecessary.

c) multiple files, multiple output, addition
    - has no advantage over simply recreating the .pot.

d) multiple files, multiple output, merge
    - eliminates one hbi18n step, but hbi18n is still required.

Only a) and b) seems useful but for other situations each.
b) still requires aid from hbmk2 or hbi18n in order to create
a real-life workflow, so probably we should go for a), and
it's also much simpler to implement.

Later if we want to introduce merge operation we can still
add it using additional -j modifier switch.

To me though it looks that if we'd add a switch to create
empty .pot for non-translatable sources (f.e. -j! switch),
plus I implement incremental merge + separate clean option,
we will end up with a more optimal solution.

> In such case we should only add internal merging for 1-st given .pot
> file, f.e. LoadFiles() should be modified a little bit:
>
> STATIC FUNCTION LoadFiles( aFiles )
>   LOCAL aTrans, aTrans2
>   LOCAL hIndex
>   LOCAL cErrorMsg
>   LOCAL n
>
>   aTrans := {}
>   FOR n := 1 TO Len( aFiles )
>      aTrans2 := __I18N_potArrayLoad( aFiles[ n ], @cErrorMsg )
>      IF aTrans2 == NIL
>         ErrorMsg( cErrorMsg )
>      ENDIF
>      __I18N_potArrayJoin( aTrans, aTrans2, @hIndex )
>   NEXT
>
>   RETURN aTrans
>
> As you can see the code is even simpler though it has one additional  
> call
> to __I18N_potArrayJoin(). Anyhow it should not be very expensive in
> comparison to other operations. But above gives also yet another  
> feature.
> You can simply append many different .pot files into single one, f.e.
>   cat *.pot > all_trans.po
> and then make:
>   hbi18n -m all_trans.po
> to merge all repeated translations inside all_trans.po so we will have
> additional functionality.

I did this exact modification to LoadFiles() present in hbmk2 [rev  
11002],
thinking it's a functionally equivalent modification. Do you think I  
broke it?

Brgds,
Viktor