Time  Nick     Message
02:26 ToinS    hi all
03:14 hdl      hell
03:15 hdl      o
03:15 chris    hello hdl
03:16 ToinS    hello hdl
03:16 ToinS    hello chros
03:16 ToinS    chris !
03:16 chris    :) hi toins
03:17 hdl      how are you chris ?
03:18 chris    doing well thanks, i enjoyed watching france win on sunday morning (nz time)
03:18 chris    how are you?
03:21 hdl      good.
03:22 chris    the third public library in NZ went live with koha on saturday
03:22 hdl      My work for 23 libraries is quite hard. because It takes a LONG time to do migration and there are quite demanding on data quality, going from a low quality data provided.
03:22 hdl      Great
03:23 chris    ahhh yeah, i know what you mean there
03:23 chris    somehow people think we can magically make bad data good :)
04:53 hdl      hi tumer
04:54 tumer    hi hdl
04:55 tumer    hdl check the posting i have done on koha-devel regarding this issue
04:57 tumer    hdl:accented character matching for zebra i have committed to koha-devel. I contains all turkish chars and some french. Use that for starting point
04:58 hdl      Unfortunately, it is in PERL in itself.
04:58 hdl      when matching Pr.*f
04:58 hdl      it matches Préf
04:59 hdl      But if I use $val=Pr.*f and then $val=~/\.\s*$val.*/ then pattern matching fails.
04:59 hdl      Thx for your commit.
05:01 tumer    hdl:i can not comment on that since i do not that much perl
09:34 kados    paul: you around?
09:35 kados    ToinS: can you join koha-devel list?
09:35 ToinS    hi kados
09:35 tumer[A] kados:did you get my mail
09:35 kados    ToinS: otherwise I"ll need to approve every email you send there :-)
09:35 kados    tumer[A]: yes, I need to test that
09:35 kados    tumer[A]: and look at the line you mention
09:35 ToinS    how can i join ?
09:35 kados    tumer[A]: because it's strange I haven't encountered that problem before
09:36 ToinS    i believe i have already join ....
09:36 kados    ToinS: http://lists.nongnu.org/mailman/listinfo/koha-devel
09:36 ToinS    oh no i know !
09:36 tumer[A] kados:it occurs with accented characters only
09:36 ToinS    because i have send the mail with the bad email adress
09:36 kados    ToinS: :-)
09:37 kados    tumer[A]: I'm gonna go consult the source ... brb
09:38 tumer[A] kados: the issue of normalizing is another matter that nobody is mentioning but is important as well
09:39 tumer    kados:i need the devel-week space to upload a new test version of koha. whendo you merge?
09:40 kados    devel-week isn't for test versions of Koha
09:41 kados    at least not right now
09:41 kados    we still don't have a stable version of zebra
09:41 tumer    i know but when it is merged it does not have any other purpose
09:41 kados    so that's first priority for me ... we actually have to get the thing running in some production environments (besides cyprus :-))
09:42 kados    well HEAD is too unstable for production
09:42 kados    so if someone is going to use Koha with zebra in production, it has to be devel-week
09:42 kados    what test versions are you planning to commit?
09:42 kados    (maybe they could be committed to HEAD?)
09:43 tumer    kados:yes but i will not be able to continue with devel week. I modified koha to use MARC only and ZEBRA
09:43 kados    tumer: put that in HEAD :-)
09:44 kados    (and it's only 11:00am :()
09:44 tumer    kados: i no longer have biblioitems, only biblioserver,itemserver,authorityserver all on zebra plus backups of marcs in sql
09:44 kados    tumer: that's exactly what I want for 3.0
09:45 kados    tumer: so HEAD is the perfect place for that
09:45 tumer    well i thought/me on phone
09:47 kados    all I have done is:
09:47 kados    1. Search.pm routines for searching zebra
09:47 kados    2. new opac-zoomsearch.pl and tmpl
09:48 kados      demonstrates how to use the new Search.pm routines
09:48 kados    but ...
09:48 kados    for now, all the Search.pm routines are still stored in opac-zoomsearch.pl
09:48 kados    because it's easier for me to refine things that way
09:48 kados    3. some configuration files
09:49 kados      new bib1.att, new record.abs, etc.
09:50 kados    ToinS: how many files were affected by your merge of rel_2_2 and HEAD?
09:50 tumer    kados: i thought may be version 3.5 for this as its a big leap and it may delay your release of 3
09:50 ToinS    kados: a lot !
09:50 ToinS    i don't know exactly
09:51 tumer    kados: did you merge?
09:51 kados    I haven't merged anything
09:51 kados    toins has merged rel_2_2 and HEAD on his local machine
09:51 kados    he's plannig to commit it soon
09:51 ToinS    and i have no commited yet
09:52 tumer    Toins:does it work?
09:52 kados    right
09:52 kados    I can't see how HEAD could work at all
09:52 kados    the zebra stuff in there is pretty busted IIRC
09:52 ToinS    i'm trying all scripts...
09:52 ToinS    all seems good execpt scripts about catalogue
09:53 tumer    kados: another question: wth npl templates cloning a subfield where does it appear? before or after wher it has been cloned?
09:53 kados    ToinS++
09:53 ToinS    because of zebra intregration
09:53 ToinS    :)
09:53 kados    ToinS: can you commit the non-catalog stuff?
09:53 ToinS    ok
09:53 kados    tumer: before ... but it puts the value in both places IIRC
09:53 tumer    ToinS: if you need help on Zebra you may ask me whenever you like
09:54 kados    tumer: if ToinS commits non-catalog stuff to HEAD, can you commit your new catalog stuff?
09:54 ToinS    but i've to make some test to see if the other scripts are ok
09:54 kados    tumer: we also need to alter the database defs in HEAD
09:54 kados    tumer: and we need to think about how best to create an install script
09:55 tumer    kados: if I am to commit what i said earlier, there are lors of database commits to be done
09:55 kados    hmmm
09:56 tumer    kados:upgrading will also be a big issue.Convert everything to marc and to utf8 create marc items separately etc.
09:56 kados    yep
09:57 kados    tumer: do you have some time today to help me get my dev-week system to add/edit records?
09:57 tumer    kados:we will need a programmers installation guide
09:57 kados    tumer: if I can get that working, maybe I will be able to focus a bit on the new stuff :-)
09:57 kados    tumer: instead of panicking :-)
09:58 tumer    kados: calm  down whats to pamic about except for UTF8 ,breaking MARC records broken authorities, editor, Zebra quirks....
09:58 kados    tumer: at some  point, you sent me a file containing your db defs
09:59 kados    yea, that's a whole other issue
09:59 tumer    kados:db defs should be on dev_week
10:00 kados    tumer: do you remember where?
10:01 tumer    kados:chris put it somewhere i dont remember
10:01 kados    tumer: you committed a database dump ...
10:01 kados    tumer: or maybe you sent it via email ...
10:02 tumer    kados: but since your zebra works db_defs should be ok
10:02 kados    no ...
10:02 kados    zebra only works for searching
10:02 kados    adding records, editing records, circ
10:02 kados    don't work
10:03 kados    ok
10:03 kados    I checked the log:
10:03 kados    10:24 < tumer> committed koha.sql and addbiblio.pl10:27 <@kados> thx
10:04 kados    so it looks like koha.sql has your db defs
10:04 tumer    kados:yes it has
10:06 kados    tumer: in dev-week code, we still require the data to exist in old koha tables?
10:07 tumer    kados:yes cause it does searching both ways
10:07 kados    tumer: this means upgrading is a nightmare :-)
10:07 tumer    kados:but in new version NO
10:08 tumer    kados:adding and editing should still work with only marc being there
10:09 kados    hmmm
10:09 tumer    kados:but lots of other code breaks  without them
10:10 kados    tumer: i've just sent you a link to the zoomkoha admin interface
10:10 tumer    it was ment to be a populated database, create marcs and import them in
10:12 kados    when I try to add a record I get the following in the zebra logs:
10:12 kados    11:11:57-03/07 zebrasrv(339) [warn] No keys generated for record
10:12 kados    11:11:57-03/07 zebrasrv(339) [warn]  The file is probably empty
10:12 kados    11:11:58-03/07 zebrasrv(339) [request] Extended Service: Update (failed)
10:12 kados    11:11:58-03/07 zebrasrv(339) [request] EsRequest  ERROR 224 update_record failed
10:13 kados    and nothing shows up in specialUpdate
10:13 tumer    kados: it should at least write .xml with nothing in it
10:13 kados    hmmm
10:13 kados    does it try to update zebra or mysql first?
10:15 tumer    mysql first
10:15 kados    ok
10:16 kados    tumer: biblio_framework.sql
10:16 kados    tumer: in misc
10:16 kados    tumer: is that the stuff that chris committed?
10:17 tumer    kados:may be thd
10:17 kados    hmmm
10:18 thd      I have not committed anything specifically for Zebra
10:18 kados    DBD::mysql::st execute failed: Unknown column 'lcsort' in 'field list' at /koha/intranet/modules/C4/Biblio.pm line 1805.
10:18 kados    SERVER:biblioserver at /koha/intranet/modules/C4/Biblio.pm line 2780.
10:18 thd      Ohh
10:18 kados    Error-biblioserver   specialUpdate 171715 /errcode:, 224, /MSG:,ES: immediate execution failed,update_record failed
10:19 kados    tumer: what is marc_biblio table?
10:20 thd      kados: yes I had committed the standard MARC 21 biblio framework but I thought that you knew that
10:20 kados    thd: yes, we aren't speaking of that
10:20 tumer    kados: marc_biblio is the old rel_2_2 table
10:20 kados    tumer: my installation doesn't seem to have it
10:20 tumer    kados: you should have something in SpecialUpdate now
10:20 kados    tumer: so I can't run update biblio,marc_biblio set biblio.frameworkcode=marc_biblio.frameworkcode where marc_biblio.biblionumber=biblio.biblionumber;
10:21 kados    nope
10:22 tumer    you seem to have a biblionumber 171715 in your error meaning sql is updated
10:25 tumer    kados: regarding your sql updating framework. it is corerct you can run that
10:26 kados    tumer: I updated the sql
10:26 kados    tumer: then, created and saved a record
10:26 kados    tumer: but now I can't find it
10:26 kados    tumer: in the OPAC
10:26 tumer    well who needs a record anyway
10:26 kados    hehe
10:26 kados    http://zoomkoha.liblime.com/cgi-bin/koha/detail.pl?bib=171715
10:27 kados    http://zoomopac.liblime.com/cgi-bin/koha/opac-detail.pl?bib=171715
10:27 kados    well, it shows up
10:27 tumer    it shows in MARC view
10:28 kados    when I do a simple search
10:28 kados    can't find it :(
10:28 kados    so it's there ...
10:28 kados    but not indexed or something
10:29 kados    sigh
10:29 tumer    well probably differnt bib-1 attribute to what i am searching
10:30 kados    I also get some errors:
10:30 kados    DBD::mysql::st execute failed: Table 'zoomkoha.marc_subfield_table' doesn't exist at /koha/intranet/modules/C4/SearchMarc.pm line 601.
10:30 kados    tumer: record.abs is used right?
10:31 tumer    kados:yes record.abs is used but the marc_subfield_table should be there as part of rel_2_2 unmodified
10:33 kados    yes it should
10:34 kados    so why is it not?
10:34 kados    (does it matter if it's there?)
10:34 kados    (not used anymore, right?)
10:34 tumer    kados:no it is used for mapping marc to kohafields
10:35 kados    arrg
10:35 kados    well that's got to be it then
10:35 tumer    kados:thats where we define the frameworks
10:35 kados    zebra doesn't know which marc to search for
10:35 tumer    yep
10:35 kados    or to index on
10:35 kados    but ...
10:35 kados    why does it work for records I manually imported?
10:35 tumer    well not exactly
10:36 tumer    koha does not know what title is
10:36 kados    so I put in a title
10:36 kados    Joshua Ferraro tests zebra
10:36 tumer    you should be able to reach your record on yaz client
10:36 kados    it was in the MARC editor
10:37 kados    245$a
10:37 tumer    if it is indexed at all
10:37 kados    well if it is indexed in zebra, my search would have found it
10:37 kados    the opac search uses pqf
10:37 tumer    well unless search.pm crashed before the search
10:38 kados    but it works fine for other records
10:38 kados    http://zoomopac.liblime.com/
10:38 dewey    well, http://zoomopac.liblime.com/ is working now wher do i look
10:38 kados    you can see the record I made here:
10:38 kados    http://zoomopac.liblime.com/cgi-bin/koha/opac-detail.pl?bib=171715
10:38 tumer    kados:ok i see search works, so this record never got indexed, any zebra errors reported
10:38 kados    the MARC view looks right
10:39 kados    I didn't see zebra errors
10:39 tumer    marc is reda from sql after search
10:39 kados    ahh ...
10:39 kados    I thought it was from zebra
10:42 kados    tumer: http://zoomopac.liblime.com/zebra.log
10:42 kados    tumer: that's what zebra does when I save a record
10:42 kados    tumer: ahh ...
10:42 kados    tumer: I have shadows enabled
10:42 kados    tumer: does the code support that?
10:43 tumer    kados: its commented out in zebraop
10:43 kados    shoot
10:43 kados    that must be it
10:43 kados    why is it commented out?
10:44 tumer    kados:try a manual commit from command line and see whether we find this record
10:44 kados    ok
10:45 tumer    kados:sorry its not commented out.It needs a parameter in koha.xml
10:45 kados    got a strange error on commit:
10:45 kados    11:45:17-03/07 zebraidx(23572) [warn] Failed to open /koha/zebradb/biblios/shadow/reci-i-0.mf [Permission denied]
10:45 kados    I suppose the user was wrong
10:46 kados    maybe owned by root or something
10:46 tumer    <biblioservershadow>1</biblioservershadow>
10:46 kados    ok ...
10:46 kados    11:45:45-03/07 zebraidx(23577) [log] commit start
10:46 kados    11:45:47-03/07 zebraidx(23577) [log][app2] zebra_stop
10:46 kados    yay
10:46 kados    it worked
10:46 kados    ok, I enable shadow in the config
10:46 kados    then test again
10:47 tumer    kados:did you get the shadow thing i wrote
10:47 kados    yea
10:47 kados    tumer: internal server error :-)
10:48 tumer    :-(
10:50 kados    still nothing in specialUpdate
10:50 tumer    kados:just for the sake of this test can you just turn off shadowing from zebra.cfg and restart zebrasrv
10:51 tumer    kados:file is being indexed but not committed so nothing in specialupdate
10:52 tumer    i do not use shadowing so i have to look into commit script and test that later
10:53 kados    tumer: Not a CODE reference at /koha/intranet/modules/C4/Biblio.pm line 2830.
10:53 kados    tumer: that mean anything to you?
10:53 tumer    no i dont understand that
10:53 kados    that line is:
10:53 kados     $Zpackage->("commit");
10:54 kados    so it's to do with ZOOM
10:54 tumer    aha
10:54 tumer    that line should read :
10:54 tumer    $Zpackage->send('commit')
10:54 kados    ok
10:55 kados    now a different internal server error :-)
10:55 kados    wait
10:55 kados    my bad
10:55 kados    it seems to have worked
10:56 kados    but ...!
10:56 tumer    ohh!
10:56 kados    each time i refreshed the page
10:56 kados    it apparantly indexed the record
10:56 kados    in zebra
10:56 kados    so now I have four duplicates
10:56 kados    that's no good
10:57 kados    search on 'testing for a third time' :-)
10:57 tumer    are we talking about addbiblio
10:57 kados    yes
10:57 kados    here's what happened:
10:57 kados    I added a record
10:57 kados    addbiblio threw a error page
10:57 kados    I changed something
10:57 kados    refreshed the page, send the post request in
10:58 kados    addbiblio threw another error
10:58 kados    etc.
10:58 kados    then ... finally it worked
10:58 kados    now, I have four identical records :-)
10:58 kados    the old Koha used to check for duplicates
10:58 tumer    kados:addbiblio.pm and addbiblio.tmpl can be the newest versions it does not matter
10:58 kados    does your code check?
10:59 tumer    yes it uses the same Findduplicate routine
11:00 kados    add item seems to work
11:01 kados    work
11:01 kados    works even
11:02 kados    hmmm
11:02 kados    if I add a item with the same barcode as another item
11:02 kados    there is no warning
11:02 kados    item delete works
11:03 tumer    kados:kados something is probably braking along the line, no matching fields etc
11:04 kados    if I add an item with the same barcode as another item, zebra is never invoked
11:04 kados    I watched the log as I did it
11:04 tumer    you will probably find some DBD errors in your log all related to sql due to mess of tables
11:05 tumer    barcode search is done on items table
11:05 tumer    old code
11:05 kados    ahh
11:05 kados    so it's probably not making it into items or something
11:06 tumer    on dev week only biblio search is done on zebra everything else is old code
11:06 kados    this error:
11:06 kados    BD::mysql::st execute failed: Table 'zoomkoha.marc_subfield_table' doesn't exist at /koha/intranet/modules/C4/SearchMarc.pm line 664.
11:06 kados    might be the prob
11:06 kados    gotcha
11:06 tumer    not populating items table is your problem
11:06 kados    ok ...
11:06 kados    so the question is
11:07 kados    if I've got a rel_2_2 db
11:07 kados    how do I update to dev_week :-)
11:07 kados    of course, I've documented this partly
11:07 tumer    unless marc_subfield_table exists you cannot populate any table
11:07 kados    but what I'm asking is, how can future Koha users do it?
11:07 kados    what's the best plan?
11:08 tumer    $100 question
11:08 kados    so the steps are:
11:08 tumer    they keep all they have
11:08 kados    1. run latest updatedatabase from rel_2_2
11:08 kados    2. run a zoomupdatedatabase that has zoom-specific stuff
11:09 kados    3. run fix_090
11:09 kados    4. export MARC records
11:09 kados    5. index in zebra
11:09 kados    6. start zebra
11:09 kados    maybe that's it?
11:10 kados    tumer: any opinions?
11:10 tumer    1- have to check rel_2_2 update database
11:10 tumer    it may be missing fields
11:10 kados    well
11:10 kados    it shouldn't include zebra-specific fields
11:10 tumer    dev_week i thought has an updatedatabase
11:10 kados    those are handled in zoomupdatedatabase script
11:10 kados    which I propose to write :-)
11:11 tumer    ok when do you convert to utf8?
11:11 kados    on export
11:11 kados    or you can preprocess them with a separate script
11:11 kados    I've got code to do either
11:11 tumer    the line of logic seem logical
11:12 kados    so we need to check rel_2_2 db and compare it with dev-week's db
11:12 kados    to see what zoomupdatedatabase needs to have in it
11:13 tumer    well dev_week code has fields for updating items for circulation as well
11:13 kados    ok
11:14 tumer    my problem is i cannot run installer.pm on windows so i cannot check these
11:14 kados    right
11:14 tumer    i do manual installation all the time
11:14 kados    well we just need some manual sql statements
11:15 kados    that can take a rel_2_2 db
11:15 kados    and turn it into dev-wek
11:15 tumer    dont forget the commit fix we done on biblio.pm (to commit i mean)
11:15 kados    already done :-)
11:15 kados    ahh ...
11:16 kados    we need to convert the tables to utf8
11:16 tumer    yes in updatadatabase
11:16 tumer    updatedatabase i mean
11:17 tumer    all tables need alter table sonadso charset utf8 collate utf8_general_ci
11:17 kados    tumer: which tables do I need to check for changes in?
11:17 kados    biblio, biblioitem, items ... any others?
11:17 kados    sonadso?
11:18 tumer    thats all, any extra tables i have may be ommitted
11:18 kados    ok
11:18 tumer    soanso=table name
11:18 kados    ok
11:18 kados    tumer: before or after ALTER TABLE $table[0] CHARACTER SET UTF8";?
11:19 tumer    at the end of this line add collate utf8_general_ci
11:19 kados    ok
11:21 tumer    kados:we have to populate framework from marc_biblio to biblio
11:22 kados    ls
11:22 kados    update biblio,marc_biblio set biblio.frameworkcode=marc_biblio.frameworkcode where marc_biblio.biblionumber=biblio.biblionumber;
11:22 tumer    yep
11:22 kados    got that stuff
11:22 kados    I think we have all the pieces
11:22 kados    just need to string them together
11:23 kados    I'll do that after lunch
11:23 kados    tumer: thanks for your help
11:23 tumer    ok thanks as well-waiting your response on USMARC.pm
11:23 kados    yep, I"ll get there :-)