Time  Nick        Message
15:48 hdl         kados
15:49 hdl         I checked his problem and has the same on 226 version for utf-8 management
15:49 hdl         taht is,
15:51 hdl         if I choose an accented term for 700 or 650 in a bibliographic record from an authority (say égypte for instance), it wil result into a strange encoding when blind-detail-biblio occurs
16:27 kados       hdl: yes?
16:28 kados       interesting
16:28 kados       what is blind-detail-biblio?
16:30 hdl         kados : it is the page that fills the bibliographic datas with the Authority input.
16:31 kados       hmmm
16:38 kados       another encoding problem has turned up
16:38 kados       with the rel_2_2 version
16:38 kados       if you attempt to mysqldump your utf8 records out
16:38 kados       and then import them
16:38 kados       characters are mangled
16:38 kados       accented characters outside the ascii range
16:39 kados       it makes me question the chosen platform !
16:40 thd         hdl: do you have an example of the problem and an example of what the 'blind detail' page is itself with and without encoding problems?
16:40 chris       that wont be a big thing, it'll be the commandline client not supporting utf-8
16:40 thd         kados: you did not seem to have any shortage of hair to me
16:41 thd         chris: when would you eve need to use UTF( on the command line?
16:41 chris       Using MySQLDump, make sure to use the '-r <filename>' option, instead of redirecting command-line output to a file ('> <filename>'). Redirection does not capture UTF characters properly, while the -r option does.
16:41 chris       Example:
16:41 chris       >mysqldump -r dbdump.sql
16:41 chris       Using MySQL, source the sql script.
16:41 chris       Example:
16:41 chris       MySQL>source dbdump.sql
16:41 thd         s/UTF(/UTF8/
16:42 thd         chris: but I see no UTF8 characters in your command
16:43 chris       there arent
16:44 chris       im saying that the redirect to a file doesnt work with utf8
16:44 chris       use -r instead
16:44 thd         chris: I see, I missed your -r option posting while I was typing
16:44 hdl         thd: Try on i3.hdlaurent.paulpoulain.com
16:45 hdl         and you will be able to see a problem when creating a new Biblio and getting accented authorities
16:47 chris        mysql  -e 'source /patch-to-backup/backup-file.sql'
16:47 chris       to load it back in
16:48 thd         hdl: I am uncertain just what I should do to observe this problem
16:49 hdl         try and click on the ... on field 632 then search egypt*
16:50 hdl         and select the first accented value.
16:50 kados       chris: thx for the tips, I'll try that
16:50 hdl         chris : thx too
16:51 chris       no problem ive done it with maori data and its worked .. so hopefully it should fix the problems you are having with import/export too
16:55 kados       chris: I tried the -r option, but when I do 'cat koha.sql |more' I still see mangled characters
16:55 chris       i wouldnt expect that to work
16:56 chris       is it mangled if you import it back in
16:56 kados       I'll give it a shot
16:56 chris       or if you open it in jed?
16:58 chris       or some editor we know parses utf8 ok
16:59 kados       mangled in vi
16:59 kados       which I know does utf8 ok
16:59 kados       because I can read raw MARC with it
17:01 chris       hmm
17:02 kados       diff -q kohawipo.utf8.sql kohawipo.utf8.sql2
17:02 kados       there is no diff between > and -r
17:02 kados       :(
17:03 chris       how odd
17:03 chris       what version of mysql do you have?
17:03 kados       debian sarge's : mysql  Ver 14.7 Distrib 4.1.11, for pc-linux-gnu (i386)
17:06 chris       -compatible=mysql40 make any difference?
17:06 kados       in mysqldump?
17:07 chris       yeah
17:07 kados       I woudln't think it would since 4.0 didn't support _any_ character sets :-)
17:07 chris       ok
17:07 chris       i have a theory, can you try this for me please
17:07 kados       btw:
17:07 kados       l$ echo $LANG
17:07 kados       en_US.UTF-8
17:08 chris       --default-character-set=latin1
17:08 chris       when you mysql dump it
17:08 chris       and just take a look what it looks like
17:08 kados       I've tried that too :-)
17:09 chris       and still mangled?
17:09 kados       it just mangles it in a different way :-)
17:09 kados       I also tried --default-character-set=utf8
17:09 chris       yeah thats its default
17:09 kados       well ... I've got to get going
17:09 kados       thanks for the help
17:10 kados       I'll see if I can lose some more hair on this tomorrow :-)
17:10 chris       you are aware changing a table to utf8 doesnt change the data in the table eh?
17:11 chris       this data was loaded in after the table was set to utf8 ?
17:11 kados       yes
17:11 chris       cool
17:11 chris       there will be something we are missing
17:11 kados       maybe that mysql can't handle charsets :-)
17:12 chris       no it can
17:12 chris       ppl all over the world are using it to do just that
17:15 chris       one thing to try would be in
17:15 chris       in mysql
17:16 chris       select * from tablename into outfile '/tmp/somefile';
17:16 chris       then take a look at that file
17:32 chris       interesting, i just mysqldumped some, and loaded them back in .. and they appear ok ... ill check in with you tomorrow joshua and see what we are doing different
17:40 thd         hdl: everything looks correct to me except that on my ISO 8859-1 environment system I do not see the accented character represented as a composed single character glyph but I see the correct characters for each of the two UTF8 bytes which will form the character.
17:43 hdl         thd: IMHO, you should see the accented character and not "as a composed single character glyph but I see the correct characters for each of the two UTF8 bytes which will form the character."
17:44 thd         hdl: I cannot save a test record on your test system to see that the end result is fine because hidden MARC tag for the record ID is not being automatically filled in the default  framework.
17:45 thd         hdl: are you saying that I ought to see the single UTF8 character in the record editor even if my environment is not UTF8?
17:47 hdl         Normally, the display is UTF-8 encoded (sent by apache.) And even with MY environment full UTF-8, it is ugly displayed.
17:47 bob         i've a question about the value_builder plugins do they work for repeatable fields?
17:48 bob         eg when editing a record does the popup get info from just one field or many?
17:50 thd         hdl: I thought tumer had reported that it was not a problem for him using Internet Explorer set to UTF8 preferred but only for the rest of us using Firefox or whatever else.
17:51 thd         bob: the value builder is often filling repeatable fields and filling them individually.
17:52 thd         bob: had you asked a question on koha-devel to day relating to value builder pop-up?
17:52 bob         so if you have a bunch of repeatable fields for 650a with info in them will clicking the ... grap the info from all or just the one
17:52 bob         ah ype
17:52 bob         yep
17:53 thd         bob: I answered above for the record editor.  Is your context the record editor?
17:53 bob         yep
17:54 thd         bob: I had difficulty understanding your question as asked on koha-devel
17:54 bob         i understand the popups can take a bunch of info and place it in right places in the marc record form
17:55 bob         but i'm wondering if you then edit the record using the popup
17:55 bob         will the popup be repopulated with the right info
17:55 bob         if you are using repeatable fields
17:57 thd         bob: so one confusion I had was how you were naming fields or subfields in asking your question.
17:58 bob         ah right
17:58 bob         i'm meaning tag 650 subfield a
17:59 thd         bob: I answered above about repeatable fields such as repeatable MARC 21 field 650 for topical subjects
18:00 thd         bob: so you mean the repeatable 650 of which the first subfield is $a and $a as we know for this case is non-repeatable while the 650 is repeatable?
18:01 bob         umm not sure
18:02 thd         bob: so you should be able to fill one 650 after another with different values and if you have a good version of the record editor you can fill all the subfields in the field at once and not just $a.
18:06 thd         bob: I think in future, multiple value builders will function so that the free floating subdivisions $z, $x, $y, and $v can be assigned using the value builder in context separately from $a if needed.
18:07 bob         right
18:08 thd         bob: this is a problem especially for subjects, where the value builder does not work as well as it might for filling subdivided subject headings in most cases..
18:09 bob         ta
18:09 thd         s/most cases/most cases with real authority records from real authority files/
18:11 thd         bob: the current value builder design which expects one value for all the subfields in the field works best for names in 1XX, 7XX, etc.
18:12 thd         tumer: have you come to save us from the scary UTF8 monsters?
18:20 thd         tumer: the scary UTF8 monsters dragged kados away.
18:24 thd         bob: what version of the record editor are you using?
18:24 bob         2.2.5 i think
18:25 tumer       hi thd
18:26 tumer       bob:what u are asking i belive is using authorities not value builder.
18:27 tumer       bob:thats how we fill tag 650 with data from somehere else
18:27 thd         bob: the best version is rel_2_2 just before paul's recent commits to fix a bug reverted things somewhat
18:28 bob         hmm, i looking at the files in the /value_builder/ sub dir
18:28 bob         and playing with them
18:28 bob         but not sure what i'm doing is right
18:28 tumer       thd:regarding utf8 i believe we have a bigger problem than anticipated. Revert to earliear version of MARC::Record
18:29 tumer       as long as you dont need MARC8 to UTF8 conversion you will see that all problems go away
18:29 thd         bob: yes are you using a set of authority records or are you just creating a few token values in an array?
18:30 thd         tumer: what is wrong with the current version of MARC::Record?
18:31 thd         tumer: I mean what is wrong that you had not fixed?
18:31 tumer       that is what is causing all this havoc of utf8 display problems
18:31 tumer       thd:i believe i have fixed it but kados does not
18:32 thd         tumer: where does kados find that the problem still exists which you do not see?
18:33 tumer       thd:well i think he did not still try it so i cannot see the difference on his server
18:34 thd         tumer: so if he does not have your fix installed he cannot test it :)
18:35 tumer       well i have posted it to devel expecting people to use and comment on it
18:36 chris       tumer: im working on getting some space to test it
18:36 chris       hopefully in the next few days
18:36 thd         tumer: he had another suggested solution today using Encode but he found problems persisted and I was uncertain why that code would solve anything.
18:36 chris       we had a public library go live on saturday, so ive been pretty busy
18:36 tumer       hi chris btw i hear wife expecting a baby?
18:37 chris       yep, december 19th
18:37 tumer       so is he/she started kicking yet?
18:37 chris       anytime in the next week or so, the kicking should start apparently :)
18:38 tumer       chris congragulations
18:38 thd         chris: and in December the screaming starts
18:38 chris       thank you
18:38 chris       :-) dont remind me
18:38 rach        howdy
18:38 tumer       well sleepless nights help koha:-)
18:38 chris       :-)
18:39 thd         chris: only pleasant screaming of course :)
18:39 tumer       rach:library.neu.edu.tr should be working
18:39 thd         tumer: yes, what records bring up the animated walking path to the book?
18:40 rach        thanks very much
18:40 tumer       click on the LC call number
18:40 thd         tumer: any call number? how complete is the map?
18:41 tumer       chris:what kados is proposing of using Encode for the query dows not work ( i tried them all)
18:41 tumer       what that does is to create the problem of query
18:41 chris       right
18:41 thd         tumer: It did make some difference in the display
18:42 tumer       but i already have utf8 characters on the template and they get broken
18:42 tumer       Encoding the whole Output.pm does not work either
18:43 tumer       i am currently using the fix i reported in production
18:44 tumer       library.neu.edu.tr uses this fix
18:44 chris       its looking ok to me
18:44 tumer       changing the language to turkish from the top of the page do you see any anomalies
18:45 chris       nope
18:45 chris       if i change encoding from utf-8 to anythign else i do
18:45 chris       but if im viewing the page as utf8 it looks fine
18:45 tumer       so i say we bug the Marc::Record maintainer and see what he says
18:46 chris       cant hurt
18:46 tumer       chris we are supposed to look and see in utf8 dont we?
18:46 chris       yes thats right tumer
18:47 chris       i was just checking that i was viewing it in utf8 by toggling off and on
18:47 thd         tumer: does the JavaScript for the call number map pop-up only work on Internet Explorer?
18:48 tumer       chris:i have checked characters hex by hex since this problem started used to have white hair now no hair at all
18:49 tumer       thd:is there a bug i only look at it in IE?
18:49 thd         tumer: kados and I still have extra hair
18:50 tumer       thd: can you see the little popup plan?
18:50 thd         tumer: I cannot get the map to pop-up on Firefox and I do not have Opera installed at the moment because Opera prevented me from upgrading x-windows libraries.
18:51 tumer       thd: ihave to check that and see what i can do.Firefox popups do not popup in IE so i change them
18:51 chris       i get a javascript error in firefox
18:52 tumer       same javascript supposedly!
18:52 thd         tumer: Is there not one set of JavaScript code to make the same function work for any browser?
18:52 tumer       chris:does it say what error is that
18:53 chris       it doesnt like the location function
18:53 chris       uncaught security exception
18:54 tumer       chris:i will look into that. its a simple localion.pl Perl script supposed to run in a window
18:55 bob         i found the problems with most javascript popups that work in firefox but not IE were due to people trying to give the popup window a name with a space in it
18:55 bob         which firefox can handle but IE could not
18:55 thd         tumer: you would never do that would you?
18:56 bob         eg window.open(url,name,...)
18:56 bob         where name was like 'koha popup'
18:56 chris       http://library.neu.edu.tr/cgi-bin/koha/opac-search.pl?op=do_search&type=opac&keyword=fish&ttype=exact&op=do_search
18:56 chris       take a look at that page bob, thats where it happens
18:57 thd         tumer: do any of your names have holes in them?
18:57 tumer       thd: i delete all the names the templates have in them as well otherwise it does not popup in IE
18:58 chris       line 232 it seems to be moaning about
18:59 chris       its the javascript its complaining about
18:59 thd         tumer: are you using a variable 'General Collection', just the sort of name with a space that bob cautioned about?
18:59 bob         says location is not a function
18:59 chris       yeah and before that do you get the error
19:00 chris       on page load
19:00 chris       uncaught exception bit bob?
19:00 chris       i think thats why location isnt a function
19:00 bob         me reloads
19:01 chris       the help popup works
19:01 bob         says it may not load or link to file:///C:/Program%20Files/Microsoft%20Office/Office10/fpclass/animate.js
19:02 chris       function help(help_topic) {
19:02 chris                var help_window = window.open('/cgi-bin/koha/opac_help.pl?help_template=' + help_topic ,'_blanck','height=400,width=400,status=no,toolbar=yes,menubar=no,location=no,resizable=yes,top=0,left=0,scrollbars=1');
19:02 chris                help_window.focus();
19:02 chris       function location(locationc,lib) {
19:02 chris                var help_window = window.open('/cgi-bin/koha/opac-location.pl?location=' + locationc +'&lib='+lib,'_blanck','height=300,width=640,status=no,toolbar=no,menubar=no,location=no,resizable=no,top=0,left=0,scrollbars=1');
19:02 chris                help_window.focus();
19:02 chris        }
19:02 chris       the help one seems to work, the location one doesnt
19:02 tumer       it just calls href="javascript: location('<!-- TMPL_VAR NAME="classification" -->','<!-- TMPL_VAR name="location-only" -->');">
19:03 tumer       and then..
19:03 tumer       function location(locationc,lib) {
19:03 tumer                var help_window = window.open('/cgi-bin/koha/opac-location.pl?location=' + locationc +'&lib='+lib,'_blanck','height=300,width=640,status=no,toolbar=no,menubar=no,location=no,resizable=no,top=0,left=0,scrollbars=1');
19:03 tumer                help_window.focus();
19:03 tumer       	}
19:03 chris       yep
19:03 chris       and for some reason that bit doesnt work .. the help function above it .. works just fine
19:03 chris       the only difference i can see
19:04 chris       is one is resizable and the other isnt
19:04 chris       but i might be missing something
19:04 bob         is location a reserved word?
19:04 bob         i know there one can call window.location
19:04 chris       hmm that might be it
19:04 bob         eg window.location.href
19:05 tumer       bob:good point
19:06 tumer       ii have renamed it to locationonly
19:06 tumer       any luck anyone?
19:08 chris       im still seeing location tumer
19:08 tumer       oops sorry wrong server
19:08 chris       :)
19:09 thd         :)
19:10 thd         tumer: you never modified the record detail display to support the map?
19:11 tumer       correct server now i hope
19:12 chris       sweet works now
19:12 chris       thats cool as hell tumer
19:12 thd         fantastic it works
19:12 tumer       thanks guys
19:13 bob         woo that is fancy
19:13 tumer       end of october it will take you to the correct face of the shelf- i hope
19:13 thd         rach: click on a call number in an OPAC search result set to see the best Koha demo ever :)
19:14 chris       very cool
19:14 bob         this line in the code though looks wrong -> <script language="JavaScript1.2" fptype="dynamicanimation" src="file:///C:/Program%20Files/Microsoft%20Office/Office10/fpclass/animate.js">
19:14 tumer       bob: i am cleaning those. left owers of previous anymation we tried
19:15 chris       ahh
19:15 bob         cool
19:15 thd         tumer: you had not added this to the record detail display it is only in the multi-record result set
19:16 tumer       thd: currently yes
19:16 tumer       its just a line of code which currently is too much
19:16 thd         :0
19:17 tumer       i am currently deeply modifying intranet
19:17 tumer       working on version 3.5!
19:18 thd         :)
19:18 thd         tumer: kados says that most of my suggestions are for version 4
19:18 tumer       well so you are ahead of me
19:20 thd         tumer: you can see the result of what kados had done with encode earlier
19:20 tumer       thd: just checking
19:21 chris       not me
19:22 chris       whats the problem?
19:22 dewey       the problem is probably if there is no data in the sort field zebra throws and error and does not even return the unsorted results
19:22 tumer       thd:ok this resolves the problem of query displaying correctly
19:22 kados       chris: you there?
19:22 tumer       dewey:upgrade zebra
19:22 dewey       tumer: i'm not following you...
19:22 kados       chris: you're right about the problem
19:23 thd         tumer: I find that the canonisation rules are not supporting Altug\x{0306} Osman,
19:23 kados       chris: just emailed you what the mysqldump looks like
19:23 kados       chris: now, the trick is, what's the solution? :-)
19:23 chris       ahh so i suspect that there is the problem eh
19:23 chris       yeah how to fix it
19:23 chris       i think what we have done is told the tabel to be utf8
19:24 thd         tumer: http://zoomopac.liblime.com/cgi-bin/koha/opac-zoomsearch.pl?query_form=ccl&op=get_results&ccl_query=O%CC%88rneklerle
19:24 tumer       kados: if you had accented characters in your template you will see that they break as well
19:24 chris       but all the columns are still latin1
19:24 kados       right
19:24 kados       so we need to convert them then, eh?
19:24 chris       hmmm i suspect so
19:25 kados       though it would probably be better to create them as utf8
19:25 kados       then import the data
19:25 chris       i suspect out convet_utf8.pl needs to be fixed
19:25 tumer       kados:your page looks correct on IE
19:25 chris       yeah that would be best joshua
19:25 thd         kados: yes if you create the table new it is not a problem but then you do not have your data
19:26 kados       chris: so what do I need to do to create all the Koha tables and columns correctly?
19:27 kados       chris: setting aside for the moment how to fix the databases of already existing Koha users
19:27 chris       good question
19:27 chris       ill have to google
19:27 thd         tumer: if everything is correct why does my posted author search finding nothing and failing to use the correct characters
19:27 tumer       thd:?
19:28 thd         tumer: http://zoomopac.liblime.com/cgi-bin/koha/opac-zoomsearch.pl?query_form=ccl&op=get_results&ccl_query=O%CC%88rneklerle
19:28 kados       tumer: in safari the O in that page works, but the s doesn't ... in firefox, neither one does :(
19:29 tumer       kados:i know and in IE they will. If you use my corrected version of USMARC.pm thay all will look correct
19:29 tumer       its a matter of firefox and safari beeing poorer in displaying decomposed characters
19:30 kados       tumer: so is it necessary to import using your modified USMARC.pm?
19:30 thd         kados: are you not referring to a mere browser problem if it is browser specific?
19:30 kados       tumer: or is it just the display?
19:30 kados       tumer: (ie, are they stored wrong in the db, or just being mangled on the way out?)
19:31 tumer       kados:XML definition and web character definition requires them to be composed
19:31 tumer       some browsers like IE can handle them ddecomposed
19:31 kados       tumer: does that mean we should entity encode them?
19:31 tumer       yes
19:31 kados       tumer: i have code that does that in Record.pm in HEAD
19:31 kados       tumer: _entity_encode() subroutine
19:31 kados       tumer: should do the trick
19:32 tumer       otherwise you will not be able to serach them from the keyboard
19:32 kados       ok ...
19:32 kados       so do we entity encode them before we import?
19:32 kados       or just while they are being pulled out for display?
19:32 tumer       our keyboards produce composed characters
19:32 thd         kados: we can find them in most cases by copying the string and pasting the query so they must be stored correctly.
19:33 tumer       we put them in zebra composed
19:33 tumer       i have to check your _entity_encode
19:33 tumer       i have to read entity_encode
19:34 thd         kados: yet my posted author search fails, why?
19:34 kados       tumer: it's quit short :-)
19:34 kados       tumer: quite even
19:35 tumer       is it NFC'ing the results?
19:35 kados       yep
19:35 tumer       then it should work
19:36 thd         kados: why is there nothing displayed for this record outside of the result set?  http://zoomopac.liblime.com/cgi-bin/koha/opac-detail.pl?bib=99999999
19:36 kados       thd: I've no idea
19:36 tumer       thd:copying and pasting you can search but not from keyboard
19:37 thd         kados: that is the record found by searching for yasa
19:38 tumer       thd:this record seems to exist only in zebra
19:39 thd         tumer: so if it is only in Zebra then there is no code to display the record detail or support author searching by clicking on the author link?
19:39 tumer       kados:still all my records are NFC'ed both in zebra and in Mysql
19:40 tumer       and all my templates do break with the new Record.pm but mine
19:40 thd         tumer: what is NFC?
19:40 tumer       s/Record/USMARC
19:41 tumer       composed character format
19:41 tumer       thd:from now on i use NFC for composed and NFD for decomposed characters
19:42 thd         tumer: what does the 'N' represent?
19:44 tumer       thd: these are Unicode terms i am not very familiar with
19:45 thd         tumer: must have been introduced with Klingon :)
19:53 thd         kados: do I understand correctly that Perl cannot manipulate the byte string correctly unless Encode::decode has been used?
20:04 thd         chis: do you understand when and where Encode::decode should be needed and when it would be unnecessary?
20:24 thd         N
20:25 thd         I want a different universe or at least a different OS
20:26 Burgundavia thd: what do you use?
20:29 thd         Burgunavia: the ordinary version of reality viewed through Debian GNU/Linux etch with more font and encoding problems than if I were using some system running Internet Explorer.  Yet I do not want any of the Internet Explorer OS's or Internet Explorer even.
20:36 kados       thd: I'm not entirely convinced it's a browser problem
20:38 thd         kados: what differences do you find between browsers if anything other than display?
20:40 thd         kados: why would Encode::decode be needed to pass the content of a string if there is no need to parse the actual characters in the string?
01:59 ToinS       hello world
02:59 hdl         hello world
03:46 chris       hi all
03:46 chris       big game today eh arnaud?
04:26 alaurin     hi all, yes, it's a big day today, for a lot of french
04:30 russ        see you tomorrow morning
04:31 chris       yep, i have to be up at 7am to see kickoff, so im going to sleep too
10:21 slef        Has anyone seen .lex files before?
10:23 slef        it's some sort of book data
10:23 slef        but trying to less it just ate my session :-/
10:23 slef        so I'm guessing binary and not nice
10:23 owen        http://filext.com/detaillist.php?extdetail=LEX
10:24 slef        yeah, it's none of those AFAICT