VrrtepCLI

Started by Tirea Aean, May 22, 2011, 03:40:58 PM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

Swoka Ikran

I'm not even sure what changes I need to make...you attached changes for v2, but I need changes for the older 1.94 codebase, since that's what Windows runs.

v2.x vrrtep.py looks nothing like 1.94's, and grammar.py does not exist in 1.94, making backporting virtually impossible.

There will not be a version 2 for Windows (at least for now). I just don't have time. Maybe over winter break I'll start looking at it, but I don't know.

@TA: Can you make the changes needed in the files found here? https://code.google.com/p/vrrtepcli/source/browse/trunk/Windows/vcli_win/

If so, I can grab the SVN and recompile it when I get the time.

Alternately, you could add code to bulk replace semicolons in the definitions with another character ( , or / maybe?) in the dictionary compiler, undo the linux changes, and keep using ; as the delimiter...
2010 was the year of the Na'vi.Vivar 'ivong Na'vi!


 
Avatray | NWOTD Sigbars | Sacred's Sigbar Tool | My collection of Avatar merchandise

Tirea Aean

#421
Ohhhh. Right. I'll do that when I can.

I don't want to replace every semicolon with something. That would introduce the same problem all over again. The field delimiter will be found up to two times inside a field.

I only want to change the field delimiter. And I did. But vrrtep counts on having the delimiter be semicolon. I'll go make changes to the 1.94 and repost here.

And I'll commit changes to 1.94 later as 1.95

Tirea Aean

Quote from: Swoka Ikran on December 09, 2012, 10:29:18 AM
@TA: Can you make the changes needed in the files found here? https://code.google.com/p/vrrtepcli/source/browse/trunk/Windows/vcli_win/

If so, I can grab the SVN and recompile it when I get the time.

I have just done so. Committed minutes ago. I edited your Windows src/vrrtepcli.py to read the new tab dictionaries.

I have also updated all of my Linux code. for whatever that's worth. I also seem to have lost a lot of personal Linux version changes since reinstalling my os. oh well. It was mainly cleaning up code anyway. nothing really new.

Swoka Ikran

Update is finished. :)

Windows users with 1.94.1 and newer can run vrrtepcli -u --all to get 1.95.0 + new dictionary.

If anyone has 1.94.0 or older, they need to download the zip file manually and install it again.

Only change from 1.94.2 is the new dictionary format support.
2010 was the year of the Na'vi.Vivar 'ivong Na'vi!


 
Avatray | NWOTD Sigbars | Sacred's Sigbar Tool | My collection of Avatar merchandise

Blue Elf

#424
tested on WinXP -> updated correctly, new data works with updated vrrtepcli. Many thanks!
At home I'll try to update it on Win7.

Edit: on Win7 no problem too
Oe lu skxawng skxakep. Slä oe nerume mi.
"Oe tasyätxaw ulte koren za'u oehu" (Limonádový Joe)


Puvomun

Ma Tirea, as I am messing with the dictionary-files for wotd I discovered this in the metawords:

4204   hìmtxew   n.   1
9368   hìmtxew   n.   1
Krr a lì'fya lam sraw, may' frivìp utralit.

Ngopyu ayvurä.

Blue Elf

Quote from: Puvomun on December 28, 2012, 02:16:37 PM
Ma Tirea, as I am messing with the dictionary-files for wotd I discovered this in the metawords:

4204   hìmtxew   n.   1
9368   hìmtxew   n.   1

One of them would be hìmtxan - it would be solved by next data files update (see dictionary thread)
Oe lu skxawng skxakep. Slä oe nerume mi.
"Oe tasyätxaw ulte koren za'u oehu" (Limonádový Joe)


Tirea Aean

4204   hìmtxew   hɪm.ˈt'ɛw   \N   n.
9368   hìmtxew   hɪm.ˈt'ɛw   \N   n.

Yeah, 4204 should be hìmtxan. It was once, pretty sure.

Puvomun

Quote from: Tirea Aean on December 31, 2012, 08:45:09 AM
4204   hìmtxew   hɪm.ˈt'ɛw   \N   n.
9368   hìmtxew   hɪm.ˈt'ɛw   \N   n.

Yeah, 4204 should be hìmtxan. It was once, pretty sure.
Odd that it changed somewhere along the line. But irayo, I changed it in my db table. (Yes, I got it loaded!)
Krr a lì'fya lam sraw, may' frivìp utralit.

Ngopyu ayvurä.

Tirea Aean

Quote from: Puvomun on December 31, 2012, 04:15:38 PM
Quote from: Tirea Aean on December 31, 2012, 08:45:09 AM
4204   hìmtxew   hɪm.ˈt'ɛw   \N   n.
9368   hìmtxew   hɪm.ˈt'ɛw   \N   n.

Yeah, 4204 should be hìmtxan. It was once, pretty sure.
Odd that it changed somewhere along the line. But irayo, I changed it in my db table. (Yes, I got it loaded!)

ah so you have a local copy of Eana Eltu as well. Well. looks like you'll never have to wait for me to dump vrrtepcli dictionaries ever again ;D

Here are my dump codes:



######## My internal copy of the mysql database is called navi ########

#### From the shell or in a shell script ####
#### Dump metaWords.txt and localizedWords.txt ####

mysqldump -u<user> -p<password> -h localhost --default-character-set=utf8 -T/tmp/ navi metaWords --fields-terminated-by="\t"

mysqldump -u<user> -p<password> -h localhost --default-character-set=utf8 -T/tmp/ navi localizedWords --fields-terminated-by="\t"

#### In MySQL OR in a .sql script as < input ####
#### Dump [lang_code].txt and naviwords.txt ####

USE navi;

SELECT navi FROM metaWords INTO OUTFILE '/tmp/naviwords.txt';

SELECT localized FROM localizedWords WHERE languageCode = 'est' INTO OUTFILE '/tmp/est.txt';

SELECT localized FROM localizedWords WHERE languageCode = 'de' INTO OUTFILE '/tmp/de.txt';

SELECT localized FROM localizedWords WHERE languageCode = 'eng' INTO OUTFILE '/tmp/eng.txt';

SELECT localized FROM localizedWords WHERE languageCode = 'hu' INTO OUTFILE '/tmp/hu.txt';

SELECT localized FROM localizedWords WHERE languageCode = 'nl' INTO OUTFILE '/tmp/nl.txt';

SELECT localized FROM localizedWords WHERE languageCode = 'sv' INTO OUTFILE '/tmp/sv.txt';

SELECT localized FROM localizedWords WHERE languageCode = 'ptbr' INTO OUTFILE '/tmp/ptbr.txt';

Blue Elf

Oh, some posts disappeared after database crash, so repeating what I found as a reminder:
- vrrtepcli dictionary can't be updated as skxawng.lu is offline
- vrrtepcli au shows as a result just drum (made of skin), but not exclamation of consternation
C:\Windows\System32>vrrtepcli au
Vrrtep CLI v1.95.0 by Tirea Aean
Windows version by Swoka Ikran
Standalone version

n. drum (made of skin)

C:\Windows\System32>vrrtepcli -l exclamation
Vrrtep CLI v1.95.0 by Tirea Aean
Windows version by Swoka Ikran
Standalone version

Query matches:
part. nang
intj. sau
intj. au

I remember something like this was discussed already - only first match is returned?
Oe lu skxawng skxakep. Slä oe nerume mi.
"Oe tasyätxaw ulte koren za'u oehu" (Limonádový Joe)


Tirea Aean

#431
Quote from: Blue Elf on February 06, 2013, 01:39:43 PM
Oh, some posts disappeared after database crash, so repeating what I found as a reminder:
- vrrtepcli dictionary can't be updated as skxawng.lu is offline
- vrrtepcli au shows as a result just drum (made of skin), but not exclamation of consternation
C:\Windows\System32>vrrtepcli au
Vrrtep CLI v1.95.0 by Tirea Aean
Windows version by Swoka Ikran
Standalone version

n. drum (made of skin)

C:\Windows\System32>vrrtepcli -l exclamation
Vrrtep CLI v1.95.0 by Tirea Aean
Windows version by Swoka Ikran
Standalone version

Query matches:
part. nang
intj. sau
intj. au

I remember something like this was discussed already - only first match is returned?

I am working on a complete re-write of all the code. I plan to implement an option to return all. In this new developer code, the first, smallest root is returrned by default. Works nearly every time.

For the version you are running, I think yes, the first is returned. As you can see by your second run, it's in there, but not being returned because only the first is. hmmm.

OH And I am migrating alll my stuff to a new host and will implement automatic updating system so I don't have to do it by hand. (I thouht I already made this post. maybe it didn't go through.)

Tsyalatun te Eyktan Txuratu'itan

Quote from: Blue Elf on February 06, 2013, 01:39:43 PM
Oh, some posts disappeared after database crash, so repeating what I found as a reminder:
- vrrtepcli dictionary can't be updated as skxawng.lu is offline
Kaltxì ma frapo.

I have some good news concerning the dictionary updates.  ;D

This is something I've been working on restoring for Tirea over the course of the last 24 hours.

The improvements are:

  • automated generation of dictionary data (requires no human interaction)
  • vrrtepcli dictionary available as a single zip file download, which vrrtepcli can then unpack itself
  • dictionary files on the website all update together.

The way these updates are generated has changed, though I'm still using the same source data that the original used.  It's just the method (and scripts which do it) which has changed, and the technology behind it. :)

Everything is now in place for the updates to be resumed for this.  There's a hold-up at the moment with releasing the new website address as the replacement site is not fully functional yet.  :(

In the mean time, anyone using the Windows Python (not "Standalone version") and Unix versions of vrrtepcli who would like to beta test it, please PM me for details.  Please include details of the platform you're running it on so I know ahead of time what to expect from you!

Irayo!

Blue Elf

Good news, thanks for it!
Oe lu skxawng skxakep. Slä oe nerume mi.
"Oe tasyätxaw ulte koren za'u oehu" (Limonádový Joe)


Tsyalatun te Eyktan Txuratu'itan

Quote from: Tsyalatun te Eyktan Txuratu'itan on February 18, 2013, 01:20:56 PM
Kaltxì ma frapo.

I have some good news concerning the dictionary updates.  ;D

This is something I've been working on restoring for Tirea over the course of the last 24 hours.

The improvements are:

  • automated generation of dictionary data (requires no human interaction)
  • vrrtepcli dictionary available as a single zip file download, which vrrtepcli can then unpack itself
  • dictionary files on the website all update together.

The way these updates are generated has changed, though I'm still using the same source data that the original used.  It's just the method (and scripts which do it) which has changed, and the technology behind it. :)

Everything is now in place for the updates to be resumed for this.  There's a hold-up at the moment with releasing the new website address as the replacement site is not fully functional yet.  :(

In the mean time, anyone using the Windows Python (not "Standalone version") and Unix versions of vrrtepcli who would like to beta test it, please PM me for details.  Please include details of the platform you're running it on so I know ahead of time what to expect from you!

Irayo!
Tam.  Tirea has now announced his new website, so I can go ahead and announce the new vrrtepcli dictionaries now.

As the web address for the vrrtepcli dictionary distribution has changed, one file in vrrtepcli needs to be updated.  You can do this by one of the following commands on Unix-based systems (as ever, omit the $ character in the following at the command prompt):


Alternatively, point your browser at http://tirea.learnnavi.org/source/vrrtepcli_update.sh, and save the file over the installed version.

Then, you can update the dictionary using the normal command:
$ vrrtepcli -u

If you have 'unzip' installed, you will find that the update process is a lot shorter than before; it now fetches just one file (a zip file), unzips it for you and installs the dictionary files as before.  If you don't have 'unzip' then it will fall back to downloading the files individually.

At the time of this posting, the dictionary version is 12.89:
$ vrrtepcli -v
Vrrtep CLI v2.0
Dictionary Version:
12.89


Finally, please send feedback on the new dictionary update process to me.  Also, any mistakes in the compilation of the vrrtepcli dictionary from the main dictionary are probably down to bugs in my scripts, so please let me know if there's an error there which isn't in the main DictionaryNavi.pdf file.

Irayo ma eylan!

Swoka Ikran

#435
For Windows users to use the new update location:

Download the attached vrrtepcli_update.cmd.

Copy it into %userprofile%\.vrrtepcli (overwrite the old one).

Update as normal. Note that this update just changes the path to the downloaded files since the old location is no longer active. The dicts are still downloaded as 11 separate files due to the lack of a native unzip command in Windows.

EDIT: The below file can also be had from the SVN (r49).
2010 was the year of the Na'vi.Vivar 'ivong Na'vi!


 
Avatray | NWOTD Sigbars | Sacred's Sigbar Tool | My collection of Avatar merchandise

Blue Elf

Thanks for update - finally we are in sync again :)
Oe lu skxawng skxakep. Slä oe nerume mi.
"Oe tasyätxaw ulte koren za'u oehu" (Limonádový Joe)


Tsyalatun te Eyktan Txuratu'itan

This morning, the new vrrtepcli dictionary scripts reported to me that an update happened, and they updated all the files that you download.  What they didn't report was any change in dictionary data.

So, is this a success or failure?  Well, the dictionary data hasn't actually changed.  The dictionary version in the dictversion.txt file is generated from the Navi Dictionary PDF, and the PDF dictionary had a change in one of the appendices - see the changelog in appendix J for the details.  So, although there's a dictionary update, the underlying data remains the same, and there's no need to update to 12.891.  (However, you can if you so desire.)  That seems to be a success!

(I would have said that last sentence in Na'vi, but I'm pretty sure "Tsakem lam säflä nang!" is wrong... bah my Na'vi sentence creation still sucks after 10 months of studying.)

I'll continue to monitor these new scripts closely for a while - especially when the next dictionary data update happens.

Blue Elf

according changelog there was change only in 'ekxinum definition - so word list didn't change, just English definition of one word (current version is 12.892)
QuoteThat seems to be a success!
I'd translate it as:
Lam oeru fwa tsakem flä -> Appears to me that that action succeed.
Oe lu skxawng skxakep. Slä oe nerume mi.
"Oe tasyätxaw ulte koren za'u oehu" (Limonádový Joe)


Tsyalatun te Eyktan Txuratu'itan

Quote from: Blue Elf on March 01, 2013, 12:56:13 AM
according changelog there was change only in 'ekxinum definition - so word list didn't change, just English definition of one word (current version is 12.892)
QuoteThat seems to be a success!
I'd translate it as:
Lam oeru fwa tsakem flä -> Appears to me that that action succeed.

Looking at it this morning, it's isn't working correctly.  It seems that the SQL data is updated after the dictionary PDFs.  This means it's possible for the vrrtep dictionary generation to use stale data.

So, at 6am yesterday 12.891 in the dictionary changelog says that tìftang si was updated in 12.891.  The dictionary PDF download at 6am yesterday said 12.891.  The SQL data did not contain any change to tìftang si.

The next update was at midnight last night to 12.892 - 'ekxinum.  Again, the PDF says 12.892, but the SQL data now contained only the change to tìftang si and not 'ekxinum.

At 7am this morning, it appears that thte dictionary PDF is again updated, this time 12.893.  Looking at the timestamp on the SQL data, that was updated some 2 hours later.

So, what this means is there is no sane way to ensure that the vrrtep data files are correctly versioned.  I'm going to modify the scripts so that it pulls the entire SQL data every hour and regenerates the vrrtep dictionary data if the SHA1 sum of the SQL file changes.  This will mean the "dictversion.txt" file will be useless for telling what version of the data you have.

The only place this can be fixed is on eanaeltu.leannavi.org by incorporating a version into the SQL data dump.