Time Nick Message 15:48 hdl kados 15:49 hdl I checked his problem and has the same on 226 version for utf-8 management 15:49 hdl taht is, 15:51 hdl if I choose an accented term for 700 or 650 in a bibliographic record from an authority (say égypte for instance), it wil result into a strange encoding when blind-detail-biblio occurs 16:27 kados hdl: yes? 16:28 kados interesting 16:28 kados what is blind-detail-biblio? 16:30 hdl kados : it is the page that fills the bibliographic datas with the Authority input. 16:31 kados hmmm 16:38 kados another encoding problem has turned up 16:38 kados with the rel_2_2 version 16:38 kados if you attempt to mysqldump your utf8 records out 16:38 kados and then import them 16:38 kados characters are mangled 16:38 kados accented characters outside the ascii range 16:39 kados it makes me question the chosen platform ! 16:40 thd hdl: do you have an example of the problem and an example of what the 'blind detail' page is itself with and without encoding problems? 16:40 chris that wont be a big thing, it'll be the commandline client not supporting utf-8 16:40 thd kados: you did not seem to have any shortage of hair to me 16:41 thd chris: when would you eve need to use UTF( on the command line? 16:41 chris Using MySQLDump, make sure to use the '-r <filename>' option, instead of redirecting command-line output to a file ('> <filename>'). Redirection does not capture UTF characters properly, while the -r option does. 16:41 chris Example: 16:41 chris >mysqldump -r dbdump.sql 16:41 chris Using MySQL, source the sql script. 16:41 chris Example: 16:41 chris MySQL>source dbdump.sql 16:41 thd s/UTF(/UTF8/ 16:42 thd chris: but I see no UTF8 characters in your command 16:43 chris there arent 16:44 chris im saying that the redirect to a file doesnt work with utf8 16:44 chris use -r instead 16:44 thd chris: I see, I missed your -r option posting while I was typing 16:44 hdl thd: Try on i3.hdlaurent.paulpoulain.com 16:45 hdl and you will be able to see a problem when creating a new Biblio and getting accented authorities 16:47 chris mysql -e 'source /patch-to-backup/backup-file.sql' 16:47 chris to load it back in 16:48 thd hdl: I am uncertain just what I should do to observe this problem 16:49 hdl try and click on the ... on field 632 then search egypt* 16:50 hdl and select the first accented value. 16:50 kados chris: thx for the tips, I'll try that 16:50 hdl chris : thx too 16:51 chris no problem ive done it with maori data and its worked .. so hopefully it should fix the problems you are having with import/export too 16:55 kados chris: I tried the -r option, but when I do 'cat koha.sql |more' I still see mangled characters 16:55 chris i wouldnt expect that to work 16:56 chris is it mangled if you import it back in 16:56 kados I'll give it a shot 16:56 chris or if you open it in jed? 16:58 chris or some editor we know parses utf8 ok 16:59 kados mangled in vi 16:59 kados which I know does utf8 ok 16:59 kados because I can read raw MARC with it 17:01 chris hmm 17:02 kados diff -q kohawipo.utf8.sql kohawipo.utf8.sql2 17:02 kados there is no diff between > and -r 17:02 kados :( 17:03 chris how odd 17:03 chris what version of mysql do you have? 17:03 kados debian sarge's : mysql Ver 14.7 Distrib 4.1.11, for pc-linux-gnu (i386) 17:06 chris -compatible=mysql40 make any difference? 17:06 kados in mysqldump? 17:07 chris yeah 17:07 kados I woudln't think it would since 4.0 didn't support _any_ character sets :-) 17:07 chris ok 17:07 chris i have a theory, can you try this for me please 17:07 kados btw: 17:07 kados l$ echo $LANG 17:07 kados en_US.UTF-8 17:08 chris --default-character-set=latin1 17:08 chris when you mysql dump it 17:08 chris and just take a look what it looks like 17:08 kados I've tried that too :-) 17:09 chris and still mangled? 17:09 kados it just mangles it in a different way :-) 17:09 kados I also tried --default-character-set=utf8 17:09 chris yeah thats its default 17:09 kados well ... I've got to get going 17:09 kados thanks for the help 17:10 kados I'll see if I can lose some more hair on this tomorrow :-) 17:10 chris you are aware changing a table to utf8 doesnt change the data in the table eh? 17:11 chris this data was loaded in after the table was set to utf8 ? 17:11 kados yes 17:11 chris cool 17:11 chris there will be something we are missing 17:11 kados maybe that mysql can't handle charsets :-) 17:12 chris no it can 17:12 chris ppl all over the world are using it to do just that 17:15 chris one thing to try would be in 17:15 chris in mysql 17:16 chris select * from tablename into outfile '/tmp/somefile'; 17:16 chris then take a look at that file 17:32 chris interesting, i just mysqldumped some, and loaded them back in .. and they appear ok ... ill check in with you tomorrow joshua and see what we are doing different 17:40 thd hdl: everything looks correct to me except that on my ISO 8859-1 environment system I do not see the accented character represented as a composed single character glyph but I see the correct characters for each of the two UTF8 bytes which will form the character. 17:43 hdl thd: IMHO, you should see the accented character and not "as a composed single character glyph but I see the correct characters for each of the two UTF8 bytes which will form the character." 17:44 thd hdl: I cannot save a test record on your test system to see that the end result is fine because hidden MARC tag for the record ID is not being automatically filled in the default framework. 17:45 thd hdl: are you saying that I ought to see the single UTF8 character in the record editor even if my environment is not UTF8? 17:47 hdl Normally, the display is UTF-8 encoded (sent by apache.) And even with MY environment full UTF-8, it is ugly displayed. 17:47 bob i've a question about the value_builder plugins do they work for repeatable fields? 17:48 bob eg when editing a record does the popup get info from just one field or many? 17:50 thd hdl: I thought tumer had reported that it was not a problem for him using Internet Explorer set to UTF8 preferred but only for the rest of us using Firefox or whatever else. 17:51 thd bob: the value builder is often filling repeatable fields and filling them individually. 17:52 thd bob: had you asked a question on koha-devel to day relating to value builder pop-up? 17:52 bob so if you have a bunch of repeatable fields for 650a with info in them will clicking the ... grap the info from all or just the one 17:52 bob ah ype 17:52 bob yep 17:53 thd bob: I answered above for the record editor. Is your context the record editor? 17:53 bob yep 17:54 thd bob: I had difficulty understanding your question as asked on koha-devel 17:54 bob i understand the popups can take a bunch of info and place it in right places in the marc record form 17:55 bob but i'm wondering if you then edit the record using the popup 17:55 bob will the popup be repopulated with the right info 17:55 bob if you are using repeatable fields 17:57 thd bob: so one confusion I had was how you were naming fields or subfields in asking your question. 17:58 bob ah right 17:58 bob i'm meaning tag 650 subfield a 17:59 thd bob: I answered above about repeatable fields such as repeatable MARC 21 field 650 for topical subjects 18:00 thd bob: so you mean the repeatable 650 of which the first subfield is $a and $a as we know for this case is non-repeatable while the 650 is repeatable? 18:01 bob umm not sure 18:02 thd bob: so you should be able to fill one 650 after another with different values and if you have a good version of the record editor you can fill all the subfields in the field at once and not just $a. 18:06 thd bob: I think in future, multiple value builders will function so that the free floating subdivisions $z, $x, $y, and $v can be assigned using the value builder in context separately from $a if needed. 18:07 bob right 18:08 thd bob: this is a problem especially for subjects, where the value builder does not work as well as it might for filling subdivided subject headings in most cases.. 18:09 bob ta 18:09 thd s/most cases/most cases with real authority records from real authority files/ 18:11 thd bob: the current value builder design which expects one value for all the subfields in the field works best for names in 1XX, 7XX, etc. 18:12 thd tumer: have you come to save us from the scary UTF8 monsters? 18:20 thd tumer: the scary UTF8 monsters dragged kados away. 18:24 thd bob: what version of the record editor are you using? 18:24 bob 2.2.5 i think 18:25 tumer hi thd 18:26 tumer bob:what u are asking i belive is using authorities not value builder. 18:27 tumer bob:thats how we fill tag 650 with data from somehere else 18:27 thd bob: the best version is rel_2_2 just before paul's recent commits to fix a bug reverted things somewhat 18:28 bob hmm, i looking at the files in the /value_builder/ sub dir 18:28 bob and playing with them 18:28 bob but not sure what i'm doing is right 18:28 tumer thd:regarding utf8 i believe we have a bigger problem than anticipated. Revert to earliear version of MARC::Record 18:29 tumer as long as you dont need MARC8 to UTF8 conversion you will see that all problems go away 18:29 thd bob: yes are you using a set of authority records or are you just creating a few token values in an array? 18:30 thd tumer: what is wrong with the current version of MARC::Record? 18:31 thd tumer: I mean what is wrong that you had not fixed? 18:31 tumer that is what is causing all this havoc of utf8 display problems 18:31 tumer thd:i believe i have fixed it but kados does not 18:32 thd tumer: where does kados find that the problem still exists which you do not see? 18:33 tumer thd:well i think he did not still try it so i cannot see the difference on his server 18:34 thd tumer: so if he does not have your fix installed he cannot test it :) 18:35 tumer well i have posted it to devel expecting people to use and comment on it 18:36 chris tumer: im working on getting some space to test it 18:36 chris hopefully in the next few days 18:36 thd tumer: he had another suggested solution today using Encode but he found problems persisted and I was uncertain why that code would solve anything. 18:36 chris we had a public library go live on saturday, so ive been pretty busy 18:36 tumer hi chris btw i hear wife expecting a baby? 18:37 chris yep, december 19th 18:37 tumer so is he/she started kicking yet? 18:37 chris anytime in the next week or so, the kicking should start apparently :) 18:38 tumer chris congragulations 18:38 thd chris: and in December the screaming starts 18:38 chris thank you 18:38 chris :-) dont remind me 18:38 rach howdy 18:38 tumer well sleepless nights help koha:-) 18:38 chris :-) 18:39 thd chris: only pleasant screaming of course :) 18:39 tumer rach:library.neu.edu.tr should be working 18:39 thd tumer: yes, what records bring up the animated walking path to the book? 18:40 rach thanks very much 18:40 tumer click on the LC call number 18:40 thd tumer: any call number? how complete is the map? 18:41 tumer chris:what kados is proposing of using Encode for the query dows not work ( i tried them all) 18:41 tumer what that does is to create the problem of query 18:41 chris right 18:41 thd tumer: It did make some difference in the display 18:42 tumer but i already have utf8 characters on the template and they get broken 18:42 tumer Encoding the whole Output.pm does not work either 18:43 tumer i am currently using the fix i reported in production 18:44 tumer library.neu.edu.tr uses this fix 18:44 chris its looking ok to me 18:44 tumer changing the language to turkish from the top of the page do you see any anomalies 18:45 chris nope 18:45 chris if i change encoding from utf-8 to anythign else i do 18:45 chris but if im viewing the page as utf8 it looks fine 18:45 tumer so i say we bug the Marc::Record maintainer and see what he says 18:46 chris cant hurt 18:46 tumer chris we are supposed to look and see in utf8 dont we? 18:46 chris yes thats right tumer 18:47 chris i was just checking that i was viewing it in utf8 by toggling off and on 18:47 thd tumer: does the JavaScript for the call number map pop-up only work on Internet Explorer? 18:48 tumer chris:i have checked characters hex by hex since this problem started used to have white hair now no hair at all 18:49 tumer thd:is there a bug i only look at it in IE? 18:49 thd tumer: kados and I still have extra hair 18:50 tumer thd: can you see the little popup plan? 18:50 thd tumer: I cannot get the map to pop-up on Firefox and I do not have Opera installed at the moment because Opera prevented me from upgrading x-windows libraries. 18:51 tumer thd: ihave to check that and see what i can do.Firefox popups do not popup in IE so i change them 18:51 chris i get a javascript error in firefox 18:52 tumer same javascript supposedly! 18:52 thd tumer: Is there not one set of JavaScript code to make the same function work for any browser? 18:52 tumer chris:does it say what error is that 18:53 chris it doesnt like the location function 18:53 chris uncaught security exception 18:54 tumer chris:i will look into that. its a simple localion.pl Perl script supposed to run in a window 18:55 bob i found the problems with most javascript popups that work in firefox but not IE were due to people trying to give the popup window a name with a space in it 18:55 bob which firefox can handle but IE could not 18:55 thd tumer: you would never do that would you? 18:56 bob eg window.open(url,name,...) 18:56 bob where name was like 'koha popup' 18:56 chris http://library.neu.edu.tr/cgi-bin/koha/opac-search.pl?op=do_search&type=opac&keyword=fish&ttype=exact&op=do_search 18:56 chris take a look at that page bob, thats where it happens 18:57 thd tumer: do any of your names have holes in them? 18:57 tumer thd: i delete all the names the templates have in them as well otherwise it does not popup in IE 18:58 chris line 232 it seems to be moaning about 18:59 chris its the javascript its complaining about 18:59 thd tumer: are you using a variable 'General Collection', just the sort of name with a space that bob cautioned about? 18:59 bob says location is not a function 18:59 chris yeah and before that do you get the error 19:00 chris on page load 19:00 chris uncaught exception bit bob? 19:00 chris i think thats why location isnt a function 19:00 bob me reloads 19:01 chris the help popup works 19:01 bob says it may not load or link to file:///C:/Program%20Files/Microsoft%20Office/Office10/fpclass/animate.js 19:02 chris function help(help_topic) { 19:02 chris var help_window = window.open('/cgi-bin/koha/opac_help.pl?help_template=' + help_topic ,'_blanck','height=400,width=400,status=no,toolbar=yes,menubar=no,location=no,resizable=yes,top=0,left=0,scrollbars=1'); 19:02 chris help_window.focus(); 19:02 chris function location(locationc,lib) { 19:02 chris var help_window = window.open('/cgi-bin/koha/opac-location.pl?location=' + locationc +'&lib='+lib,'_blanck','height=300,width=640,status=no,toolbar=no,menubar=no,location=no,resizable=no,top=0,left=0,scrollbars=1'); 19:02 chris help_window.focus(); 19:02 chris } 19:02 chris the help one seems to work, the location one doesnt 19:02 tumer it just calls href="javascript: location('<!-- TMPL_VAR NAME="classification" -->','<!-- TMPL_VAR name="location-only" -->');"> 19:03 tumer and then.. 19:03 tumer function location(locationc,lib) { 19:03 tumer var help_window = window.open('/cgi-bin/koha/opac-location.pl?location=' + locationc +'&lib='+lib,'_blanck','height=300,width=640,status=no,toolbar=no,menubar=no,location=no,resizable=no,top=0,left=0,scrollbars=1'); 19:03 tumer help_window.focus(); 19:03 tumer } 19:03 chris yep 19:03 chris and for some reason that bit doesnt work .. the help function above it .. works just fine 19:03 chris the only difference i can see 19:04 chris is one is resizable and the other isnt 19:04 chris but i might be missing something 19:04 bob is location a reserved word? 19:04 bob i know there one can call window.location 19:04 chris hmm that might be it 19:04 bob eg window.location.href 19:05 tumer bob:good point 19:06 tumer ii have renamed it to locationonly 19:06 tumer any luck anyone? 19:08 chris im still seeing location tumer 19:08 tumer oops sorry wrong server 19:08 chris :) 19:09 thd :) 19:10 thd tumer: you never modified the record detail display to support the map? 19:11 tumer correct server now i hope 19:12 chris sweet works now 19:12 chris thats cool as hell tumer 19:12 thd fantastic it works 19:12 tumer thanks guys 19:13 bob woo that is fancy 19:13 tumer end of october it will take you to the correct face of the shelf- i hope 19:13 thd rach: click on a call number in an OPAC search result set to see the best Koha demo ever :) 19:14 chris very cool 19:14 bob this line in the code though looks wrong -> <script language="JavaScript1.2" fptype="dynamicanimation" src="file:///C:/Program%20Files/Microsoft%20Office/Office10/fpclass/animate.js"> 19:14 tumer bob: i am cleaning those. left owers of previous anymation we tried 19:15 chris ahh 19:15 bob cool 19:15 thd tumer: you had not added this to the record detail display it is only in the multi-record result set 19:16 tumer thd: currently yes 19:16 tumer its just a line of code which currently is too much 19:16 thd :0 19:17 tumer i am currently deeply modifying intranet 19:17 tumer working on version 3.5! 19:18 thd :) 19:18 thd tumer: kados says that most of my suggestions are for version 4 19:18 tumer well so you are ahead of me 19:20 thd tumer: you can see the result of what kados had done with encode earlier 19:20 tumer thd: just checking 19:21 chris not me 19:22 chris whats the problem? 19:22 dewey the problem is probably if there is no data in the sort field zebra throws and error and does not even return the unsorted results 19:22 tumer thd:ok this resolves the problem of query displaying correctly 19:22 kados chris: you there? 19:22 tumer dewey:upgrade zebra 19:22 dewey tumer: i'm not following you... 19:22 kados chris: you're right about the problem 19:23 thd tumer: I find that the canonisation rules are not supporting Altug\x{0306} Osman, 19:23 kados chris: just emailed you what the mysqldump looks like 19:23 kados chris: now, the trick is, what's the solution? :-) 19:23 chris ahh so i suspect that there is the problem eh 19:23 chris yeah how to fix it 19:23 chris i think what we have done is told the tabel to be utf8 19:24 thd tumer: http://zoomopac.liblime.com/cgi-bin/koha/opac-zoomsearch.pl?query_form=ccl&op=get_results&ccl_query=O%CC%88rneklerle 19:24 tumer kados: if you had accented characters in your template you will see that they break as well 19:24 chris but all the columns are still latin1 19:24 kados right 19:24 kados so we need to convert them then, eh? 19:24 chris hmmm i suspect so 19:25 kados though it would probably be better to create them as utf8 19:25 kados then import the data 19:25 chris i suspect out convet_utf8.pl needs to be fixed 19:25 tumer kados:your page looks correct on IE 19:25 chris yeah that would be best joshua 19:25 thd kados: yes if you create the table new it is not a problem but then you do not have your data 19:26 kados chris: so what do I need to do to create all the Koha tables and columns correctly? 19:27 kados chris: setting aside for the moment how to fix the databases of already existing Koha users 19:27 chris good question 19:27 chris ill have to google 19:27 thd tumer: if everything is correct why does my posted author search finding nothing and failing to use the correct characters 19:27 tumer thd:? 19:28 thd tumer: http://zoomopac.liblime.com/cgi-bin/koha/opac-zoomsearch.pl?query_form=ccl&op=get_results&ccl_query=O%CC%88rneklerle 19:28 kados tumer: in safari the O in that page works, but the s doesn't ... in firefox, neither one does :( 19:29 tumer kados:i know and in IE they will. If you use my corrected version of USMARC.pm thay all will look correct 19:29 tumer its a matter of firefox and safari beeing poorer in displaying decomposed characters 19:30 kados tumer: so is it necessary to import using your modified USMARC.pm? 19:30 thd kados: are you not referring to a mere browser problem if it is browser specific? 19:30 kados tumer: or is it just the display? 19:30 kados tumer: (ie, are they stored wrong in the db, or just being mangled on the way out?) 19:31 tumer kados:XML definition and web character definition requires them to be composed 19:31 tumer some browsers like IE can handle them ddecomposed 19:31 kados tumer: does that mean we should entity encode them? 19:31 tumer yes 19:31 kados tumer: i have code that does that in Record.pm in HEAD 19:31 kados tumer: _entity_encode() subroutine 19:31 kados tumer: should do the trick 19:32 tumer otherwise you will not be able to serach them from the keyboard 19:32 kados ok ... 19:32 kados so do we entity encode them before we import? 19:32 kados or just while they are being pulled out for display? 19:32 tumer our keyboards produce composed characters 19:32 thd kados: we can find them in most cases by copying the string and pasting the query so they must be stored correctly. 19:33 tumer we put them in zebra composed 19:33 tumer i have to check your _entity_encode 19:33 tumer i have to read entity_encode 19:34 thd kados: yet my posted author search fails, why? 19:34 kados tumer: it's quit short :-) 19:34 kados tumer: quite even 19:35 tumer is it NFC'ing the results? 19:35 kados yep 19:35 tumer then it should work 19:36 thd kados: why is there nothing displayed for this record outside of the result set? http://zoomopac.liblime.com/cgi-bin/koha/opac-detail.pl?bib=99999999 19:36 kados thd: I've no idea 19:36 tumer thd:copying and pasting you can search but not from keyboard 19:37 thd kados: that is the record found by searching for yasa 19:38 tumer thd:this record seems to exist only in zebra 19:39 thd tumer: so if it is only in Zebra then there is no code to display the record detail or support author searching by clicking on the author link? 19:39 tumer kados:still all my records are NFC'ed both in zebra and in Mysql 19:40 tumer and all my templates do break with the new Record.pm but mine 19:40 thd tumer: what is NFC? 19:40 tumer s/Record/USMARC 19:41 tumer composed character format 19:41 tumer thd:from now on i use NFC for composed and NFD for decomposed characters 19:42 thd tumer: what does the 'N' represent? 19:44 tumer thd: these are Unicode terms i am not very familiar with 19:45 thd tumer: must have been introduced with Klingon :) 19:53 thd kados: do I understand correctly that Perl cannot manipulate the byte string correctly unless Encode::decode has been used? 20:04 thd chis: do you understand when and where Encode::decode should be needed and when it would be unnecessary? 20:24 thd N 20:25 thd I want a different universe or at least a different OS 20:26 Burgundavia thd: what do you use? 20:29 thd Burgunavia: the ordinary version of reality viewed through Debian GNU/Linux etch with more font and encoding problems than if I were using some system running Internet Explorer. Yet I do not want any of the Internet Explorer OS's or Internet Explorer even. 20:36 kados thd: I'm not entirely convinced it's a browser problem 20:38 thd kados: what differences do you find between browsers if anything other than display? 20:40 thd kados: why would Encode::decode be needed to pass the content of a string if there is no need to parse the actual characters in the string? 01:59 ToinS hello world 02:59 hdl hello world 03:46 chris hi all 03:46 chris big game today eh arnaud? 04:26 alaurin hi all, yes, it's a big day today, for a lot of french 04:30 russ see you tomorrow morning 04:31 chris yep, i have to be up at 7am to see kickoff, so im going to sleep too 10:21 slef Has anyone seen .lex files before? 10:23 slef it's some sort of book data 10:23 slef but trying to less it just ate my session :-/ 10:23 slef so I'm guessing binary and not nice 10:23 owen http://filext.com/detaillist.php?extdetail=LEX 10:24 slef yeah, it's none of those AFAICT