Time  Nick          Message
00:31 dcook         oleonard++
01:06 dcook         Anyone around in #koha who deals with Chinese, Japanese, Korean MARC data?
01:07 cait          kind of
01:07 dcook         Having some interesting issues where Koha (or more like the MARC libraries) are failing to fully parse the MARC data
01:07 dcook         We had to convert GB2312 to UTF-8 and that was fine, but loading it into Koha has been challenging
01:08 cait          hm we only use utf-8 and no issues as far as ai know
01:08 dcook         It is in ISO MARC which makes it even worse..
01:08 cait          iso marc? (like not xml?)
01:08 dcook         Right
01:08 cait          we always use that then
01:08 cait          no xml
01:08 dcook         Really? Interesting...
01:08 dcook         Transforming it from MARC to MARCXML didn't work with MarcEdit, and Koha fails as well
01:08 dcook         But only some characters
01:09 dcook         Like it'll turn 基督教历史地图集 into 基督教历史 with 3 rubbish bytes at the end
01:10 dcook         Almost like it created 3 rubbish bytes where it couldn't read the last 3 characters...
01:11 cait          if you have proper encoding, the format should not be an issue
01:11 dcook         Agreed
01:11 cait          [off] https://cybertower.bibliothek.goethe.de/cgi-bin/koha/opac-search.pl?idx=kw&op=and&idx=kw&op=and&idx=kw&limit=ln%2Crtrn%3Achi&sort_by=relevance&do=Suche
01:11 cait          there are some examples a bit lower on the result list
01:12 cait          ah and second page is very chinese
01:13 dcook         Yeah that makes sense
01:13 dcook         Maybe there's some issues with the transcoding from GB2312 to UTF-8 although it seems so fine
01:13 dcook         Maybe I'll try a simple script using MARC::Record to load it and maybe it'll tell me what the problem is
01:13 wahanui       i think the problem is it has several code issues
01:14 dcook         I can view the GB2312 and UTF8 just fine in Vim and Notepad++
01:14 dcook         And you use ISO MARC for yours?
01:17 cait          no xml yes
01:17 cait          at import time
01:17 cait          that's what we are talking about?
01:17 dcook         Yep
01:18 dcook         I just tried MARC::File::USMARC->decode($text) and it's doing it wrong...
01:18 cait          hm
01:18 dcook         Although I could play around with a few settings..
01:18 cait          we get the data as mrc from our union catalog
01:19 dcook         That really does seem to suggest some encoding issue on our end..
01:20 cait          we do conversions before importing, but i'd still suspect an encoding issue
01:24 cait          hm thinkig about it, we might convert to xml at some point when doing data manipulation... but the files i see on import have mrc ending. encoding never popped up outside of normalization (combined non-combined)
01:25 dcook         I think I see something looking at the hex
01:26 dcook         Hmm or maybe not..
01:26 dcook         I see the title, a record separator, two spaces, then a unit separator..
01:27 dcook         Well actually...
01:27 dcook         I think the problem might be that we're converting GB2312 to UTF8 as that might screw up the directories and such
01:27 dcook         That's probably what this is
01:28 dcook         Maybe we should be doing MARC to MARCXML as GB2312 and then convert to UTF8
01:28 dcook         That's logical...
01:28 dcook         I think I even suspected that yesterday but forgot
01:28 cait          worth trying?
01:39 dcook         It's certainly interesting
01:40 dcook         no mapping found for [0x9F] at position 1 in 基督教历史▒ g0=ASCII_DEFAULT g1=EXTENDED_LATIN at /usr/lib/perl5/site_perl/5.26.1/MARC/Charset.pm line 308.
01:40 dcook         I feel like I'm getting closer..
01:41 dcook         Like I just need to configure MARC::Charset correctly
01:41 dcook         Although it seems designed for marc8 to utf8 only...
01:41 dcook         But that can't be right..
01:50 dcook         Well I've been told to stop working on it, so unsolved for now
01:50 dcook         But I think that's probably what it is
01:51 dcook         I think GB2312 uses 2 bytes and UTF-8 uses 3 bytes
01:51 dcook         So converting GB2312 to UTF8 then trying to process the MARC won't work
01:51 dcook         But... I don't see a well supported way of parsing GB2312 ISO MARC with Perl...
01:52 dcook         Will probably have to hack something together
03:01 mtj           hi dcook, ive have good results using iconv and/or yaz-iconv for encoding stuff
03:03 mtj           iconv --from-code=GB2312 --to-code=UTF-8 in.mrc > utf8.mrc
03:13 dcook         mtj: Interesting. That's what I did but that created the problem I think
03:14 dcook         Like the encoding came across correctly
03:14 dcook         However, when MARC::Record parses the MARC, it doesn't parse it correctly
03:15 dcook         I could view in.mrc and utf8.mrc just fine in their respective encodings, but try to load that into Koha, MARC::Record, or MarcEdit and no good.
03:15 dcook         It would truncate fields and turn valid UTF8 data into garbage bytes
03:15 dcook         Even though it was find in the utf8.mrc file
03:15 dcook         My only guess at the moment is that the problem is going from a 2 byte encoding to a 3 byte encoding is causing the ISO MARC directory and such to become corrupt
03:16 dcook         As field lengths and such would change at a byte level, but that's just a guess
03:17 dcook         This theory also comes from how most of the Chinese is parsed correctly but it gets truncated
03:17 dcook         No more work time to spend on it but maybe after work time sometime...
03:23 mtj           dcook: hiya, i recall importing CJK records as marcxml to be a bit more "robust"
03:23 mtj           'Maybe we should be doing MARC to MARCXML as GB2312 and then convert to UTF8'
03:24 mtj           ^ you might be on the right track there :)
03:26 dcook         I was thinking I was on the right track but getting MARC::Record to work with GB2312 has been tough heh
03:26 dcook         But... I might give it a go another time
03:26 dcook         Doing the transcoding with MARCXML would be trivial I think
03:27 * dcook       shakes his fist at ISO 2709
03:28 dcook         I might give it another go with MarcEdit instead...
03:29 dcook         Mmm MarcEdit's conversion did the same job as iconv I think..
03:31 mtj           i found it pretty tough too, i basically tried every tool and option i could find, and diffed the output
03:31 dcook         Oh... but MarcEdit was able to open the UTF8 MARC. MARC::Record was struggling with that (or I was struggling to configure my script to do it correctly)
03:32 mtj           ...the difference in results was often subtle
03:34 dcook         Success!
03:34 dcook         Well at least generating valid MARCXML
03:35 dcook         I think I could do the character conversion with iconv or MarcEdit...
03:35 dcook         Then the key was to open it in MarcEdit.
03:35 dcook         And then save into the desired format
03:35 mtj           i found converting a problematic iso2709 file to marcxml, imported more goodly into koha
03:35 dcook         Which now?
03:37 mtj           yaz-marcdump  -i marc -o marcxml  file.mrc > file.xml
03:37 mtj           ^ twas what i used
03:37 dcook         Ooo interesting
03:38 dcook         So you did iconv to do the transcoding
03:38 mtj           ... according to my terrible notes :p
03:38 dcook         Then that to go from marc to marcxml?
03:38 mtj           yeah, i think thats the order
03:39 mtj           iconv has a --to-code=UTF-8//TRANSLIT option too..
03:40 dcook         looks like yaz-marcdump worked well
03:40 dcook         I wonder if MARC::Record would work too if I just set it up correctly, but I'm guessing not
03:41 dcook         Thanks for that help, mtj. Made a big difference.
03:42 mtj           np, the xml parser seems to handle 'problematic' files waay better
03:43 mtj           iconv --verbose  -c --to-code=UTF-8//TRANSLIT  ./file.mrc  > file-trans.mrc
03:44 mtj           ^ that looked useful for stripping glitched chars from a file
03:45 dcook         Oh I thought it worked but now it looks like yaz-marcdump didn't
03:46 dcook         Actually yaz-marcdump creates the wrong result too
03:46 dcook         When you open it in vim you see: ""testmarc.xml" [ILLEGAL BYTE in line 16] 20109L, 775824C"
03:46 mtj           oof
03:47 dcook         And now I can't recreate it in MarcEdit either although I did before lol
03:47 dcook         Might be due to file extension names..
03:48 dcook         Or maybe I used a different...
03:49 mtj           i would create a small test file of a few problem records, and experiment with iconv,yaz-iconv, yaz-marcdump on it
03:49 dcook         lol yeah MarcEdit is picky with its file extensions...
03:49 dcook         The sample collection is the small test file heh
03:50 mtj           i could never work out how to script/auotmate marcedit, so never really used it
03:50 dcook         Mmm fair enough
03:50 dcook         I find it handy for semi-automation where you still need a human set of eyes
03:51 dcook         Sure enough... if I do a MARC21 => MARCXML transform in MarcEdit it can't do it either
03:52 dcook         Same result as yaz-marcdump
03:52 mtj           the tools i mentioned got the job done, for me
03:52 dcook         But if I open the MRC file into the MRK format in MarcEdit and then export it... beautiful
03:53 dcook         I might try those additional iconv optoins you mentioned, mtj
03:54 dcook         But I don't think it
03:54 dcook         's an encoding issue
03:54 dcook         I think it's an ISO MARC issue
03:54 dcook         Although if you recall your exact commands, I'd give it a go
03:55 mtj           hmm, try...
03:55 mtj           iconv --from-code=GB2312 --to-code=UTF-8 in.mrc > utf8.mrc
03:55 mtj           iconv --from-code=GB2312  --to-code=UTF-8//TRANSLIT in.mrc > utf8-trans.mrc
03:56 mtj           compare those ^
03:56 dcook         iconv: conversion from `code=GB2312' is not supported
03:56 dcook         And actually I did have to use a newer charset for the chinese to get it to work with iconv
03:57 dcook         If I tried GB2312 I would get this:
03:57 dcook         iconv -f GB2312 -t UTF-8 testmarc.mrc -o TestUTF8.mrc
03:57 dcook         iconv: illegal input sequence at position 35414
03:57 dcook         Even though it is supposedly GB2312
03:58 mtj           hmm, iconv -l has CSGB2312 too
03:59 dcook         Yeah that's no good either
03:59 dcook         Ohh this is interesting...
03:59 dcook         source marc has the following for the record length: 00651
03:59 mtj           hmm, id try yaz-inconv for good measure
03:59 dcook         But on my good XML export, it is actually 00701
03:59 dcook         Yeah, I really don't think the problem is the encoding
03:59 dcook         I think it's the ISO MARC leader
04:00 dcook         Although I suppose that length would change with marcxml potentially...
04:00 dcook         But anyway I found something that works
04:03 dcook         I reckon the "Record label" and "Directory" are compromised by the transcoding
04:04 mtj           iconv --from-code=GB2312 works ok on deb10/buster
04:06 dcook         Yeah this is OpenSUSE. It would be interesting to try on Debian.
04:07 dcook         Looking at https://metacpan.org/release/MARC-Record/source/lib/MARC/File/USMARC.pm, I don't see how it could work
04:08 dcook         As it's dealing with bytes
04:08 dcook         And it's going from a 2 byte encoding to a 3 byte encoding I think
04:08 dcook         For yaz-marcdump, I'd have to check their source
04:09 dcook         But I'm guessing MarcEdit's Editor is more cautious about the record label and directory in ISO MARC
04:11 dcook         Hmm actually maybe I was misusing MARC::Record..
04:11 dcook         I was trying to decode a whole file rather than one record..
04:19 dcook         Got a bit further with my MARC::Record script but yeah it's not working correctly... because I think it's misinterpreting the ISO MARC
04:25 dcook         Ahh and checking $record->warnings() proves a lot of that
04:27 dcook         mtj: What version of yaz-marcdump did you use?
04:27 dcook         I have YAZ 5.8.1
04:28 dcook         https://github.com/indexdata/yaz/blob/master/util/marcdump.c is so challenging to read imho...
04:28 dcook         Ubuntu 18.04 has YAZ 5.30.2.
04:28 dcook         I wouldn't be surprised if newer versions of yaz-marcdump don't trust the directory in ISO MARC...
04:28 mtj           yaz 5.28.0-1.indexdata
04:28 dcook         Interesting!
04:29 dcook         Ok I'm going to try a newer version
04:29 mtj           deb http://ftp.indexdata.dk/debian stretch main
04:30 mtj           aah, there is a newer version for stretch too ..  Candidate: 5.30.2-1.indexdata
04:31 mtj           yeah, def. grab a newish yaz dcook :)
04:31 dcook         Hmm yep no luck
04:31 dcook         Same rseult
04:32 dcook         result*
04:33 dcook         yaz-iconv just used iconv and yeah yaz-marcdump seems to be lost by the leader too
04:34 dcook         although yaz-marcdump -v didn't show anything..
04:35 dcook         oh wait no it did lol
04:35 dcook         It just printed it to STDOUT instead of STDERR
05:05 dcook         @later tell cait I resolved the problem in the end by doing the GB18030 -> UTF8 transcoding with iconv, opened the ISO MARC in MarcEdit into its MRK format, and then saved as MARCXML, and that did the trick. I think the issue was going from a 2 byte to 3 byte encoding and that confused MARC::File::USMARC.
05:05 huginn        dcook: The operation succeeded.
06:12 KK            Hi everyone, I'm new here.. We're looking to implement Koha for a library management system that we're working on.. How far does Koha allow for the customization of UI?
06:22 KK            IDENTIFY
06:23 KK            HELP SENDPASS
06:38 mtj           @later tell joubu: hiya, you should be able to install master koha from here...  http://apt.kohaaloha.com/koha-nightly/
06:38 huginn        mtj: The operation succeeded.
06:41 karthikkumars Hi everyone, I'm new here.. We're looking to use Koha for a library management system that we're working on.. My question is how far does Koha allow UI customization?
06:43 karthikkumars I see that the administration section allows me to provide HTML to pre-built blocks.. Is there any way I could override these blocks and make my own UI components?
07:37 nsk           hi guys
07:37 nsk           anyone around?
08:05 Joubu         mtj: yes, but the idea was to have a "clone" of what would be released. We would need a version switch and the PO files (at least, cannot think of something else right now)
09:16 Joubu         @later tell kidclamp ping bug 21591
09:16 huginn        Joubu: The operation succeeded.
09:54 kidclamp      replied Joubu
10:09 Joubu         kidclamp: thanks, will have a look again then
10:09 * cait        waves
10:09 Joubu         we need bug 25473 and bug 25563 for 20.05
10:09 huginn        Bug http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25473 major, P5 - low, ---, oleonard, Needs Signoff , Can't add order from MARC file, save button does nothing
10:09 huginn        Bug http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25563 major, P5 - low, ---, jonathan.druart, Needs Signoff , Cannot sub "add order from MARC file" form after alert
10:09 Joubu         we even need them for next stable releases actually
10:10 cait          hm I can trya and have a look in an hour or so?
10:10 Joubu         would be great yes
10:11 kidclamp      I didn't realize you were submitting on 25563 Joubu, I put a patch there too ;-)
10:11 Joubu         arf sorry
10:11 Joubu         I am not sure my patch is godd
10:11 Joubu         how did you fix it?
10:11 kidclamp      no worries, they are almost exactly the same - I just make the button not be a submit button so we don't have to prevent default
10:12 Joubu         exactly like me
10:12 Joubu         :)
10:12 Joubu         good
10:12 kidclamp      almost, you leave it as 'submit'
10:12 Joubu         I don't have a strong preference, but I think it's better as submit, isn't it?
10:13 Joubu         at least mine has a test plan :D
10:13 Joubu         I let you decide
10:14 cait          wahtever you decide, i want a test plan
10:15 Joubu         there is one
10:15 Joubu         Patches from both bug reportrs should be tested together
10:17 Joubu         did: around?
10:52 Joubu         Thanks kidclamp!
11:02 Joubu         @later tell tcohen please have a look at bug 25513 comments 12 and 13. This is super weird. I know it's not what we are testing, and it is a quite minor issue, but... should we hide the problem and set the integers to a given value to make sure the exponent formatting won't be displayed?
11:02 huginn        Joubu: The operation succeeded.
11:17 oleonard      o/
11:19 cait          hi oleonard :)
11:28 nsk1          hi all
11:28 nsk1          i have a doubt
11:28 nsk1          can anyone help?
11:29 nsk1          i have installed the software
11:29 nsk1          is it possible to replace the Koha logo?
11:32 nsk1          no one around?
11:32 tcohen        jpña
11:32 tcohen        hola
11:32 nsk1          hi cohen
11:34 oleonard      nsk1 which Koha logo? OPAC?
11:34 nsk1          staff
11:35 nsk1          this one and the ones inside:
11:35 nsk1          https://intranet.bywatersolutions.com/
11:35 nsk1          thats just a sample
11:36 nsk1          i assume this is called the staff login
11:37 pastebot      "oleonard" at 127.0.0.1 pasted "nsk1: CSS controlling display of the logo in the staff client" (16 lines) at http://paste.koha-community.org/20554
11:38 oleonard      That paste shows the parts of the CSS which control display of the logos
11:38 oleonard      You can use the  IntranetUserCSS preference to override that CSS
11:41 nsk1          i see...lemme check it out
12:53 cait          Joubu: bit late, but both done
12:53 cait          do you have another urgent one on oyur list? missing overview myself right now
12:56 Joubu         tcohen: 25513 - we should force the formatting when we display integers/floats for the REST API
12:56 Joubu         cait: bug 24229
12:56 huginn        Bug http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24229 normal, P5 - low, ---, tomascohen, Signed Off , /items API tests fail on Ubuntu 18.04
12:56 Joubu         watch rel_20_05_candidate
12:57 Joubu         this is the only one waiting for QA
12:57 cait          hm api
12:57 cait          I think that one might not be for me :(
12:58 Joubu         you just need to run the tests :)
12:58 tcohen        6 of them
12:58 Joubu         and ask Tomas to open a separate bug report :D
12:59 cait          for?
13:00 Joubu         see second to last comment
13:01 cait          tcohen: can you just file the separate bug without me having to read? ;)
13:01 tcohen        I will, I'm kind of overloaded at the moment sorry
13:10 nsk1          hi guys
13:11 nsk1          i have installed the library and created a staff via the admin
13:11 nsk1          but the staff is unable to login
13:11 nsk1          it says that the user does not have enough permissions to access this page..when they try to login
13:11 nsk1          why?
13:14 cait          did you assign permissions?
13:14 Joubu         it should be a superlibrarian user
13:14 Joubu         Query your DB with: select userid, flags from borrowers;
13:14 cait          go to their patron account in staff and check more > permissions
13:15 Joubu         userid is the "login", and flags must be 1
13:15 nsk1          ok, lemme check...can we do this checking via admin?
13:16 Joubu         I thought you were not able to login
13:16 Joubu         but actually you did
13:16 Joubu         so cait is right, you should set the permissions for the new user
13:17 nsk1          i set the permission as superlibrarian
13:17 nsk1          but still unable to access
13:18 nsk1          never mind...i think i made a mistake...lemme check
13:18 nsk1          got it
13:18 nsk1          thanks
13:28 cait          hi khall :)
13:29 khall         hi cait !
13:31 Joubu         bug 25567
13:31 huginn        Bug http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25567 critical, P5 - low, ---, jonathan.druart, ASSIGNED , borrower_attribute_types.category_code must be set to undef if not set
13:32 Joubu         there is a patch to test
14:01 tuxayo        Joubu: Hi o/ I'm looking at it
14:23 nsk1          how can i access the OPAC? i am able to see only the admin/staff login
14:25 Joubu         how did you install?
14:25 Joubu         you should find it in the apache config (search for VirtualHost)
14:26 Joubu         it => the host name I meant
14:27 nsk1          i see
14:27 nsk1          lemme check
16:46 koha-jenkins  Project Koha_19.11_D8 build #155: SUCCESS in 36 min: https://jenkins.koha-community.org/job/Koha_19.11_D8/155/
16:47 koha-jenkins  Project Koha_19.11_U18 build #154: STILL UNSTABLE in 37 min: https://jenkins.koha-community.org/job/Koha_19.11_U18/154/
17:35 koha-jenkins  Project Koha_19.11_D9 build #153: SUCCESS in 48 min: https://jenkins.koha-community.org/job/Koha_19.11_D9/153/
18:31 tuxayo        Does anyone see any bug particularly worth to sign off on this last day before release?
18:31 caroline      Anyone doing checkout deliveries? If so how how do you manage it?
19:19 cait          checkout deliveries?
19:23 * tuxayo      plans to keep reading the wiki pages about Rmaint, development, bug reporting, etc. As it seems that there is no more work related to the release that they could do.
19:25 cait          setting up your environment for rmainting is probably a good idea :)
19:31 caroline      cait: like ppl place holds via the opac, you check them out to their file and you deliver them to their house
19:32 cait          hm delivery i haven't heard from form our libraries, but pick-up
19:32 cait          they get the items ready and you can schedule a pick-up time - contact less basically
19:34 cait          sorry for the typos
19:36 tuxayo        cait: Thanks for the advice :)
21:25 koha-jenkins  Project Koha_19.11_D9 build #154: SUCCESS in 34 min: https://jenkins.koha-community.org/job/Koha_19.11_D9/154/
21:45 koha-jenkins  Yippee, build fixed!
21:45 wahanui       Congratulations!
21:45 koha-jenkins  Project Koha_19.11_U18 build #155: FIXED in 50 min: https://jenkins.koha-community.org/job/Koha_19.11_U18/155/
21:53 koha-jenkins  Project Koha_19.11_D8 build #156: SUCCESS in 27 min: https://jenkins.koha-community.org/job/Koha_19.11_D8/156/
22:50 talljoy       @later tell aleisha 19.11.x branch is all yours!!  hope i left it in good shape for you.  please let me know if i can answer any questions.
22:50 huginn        talljoy: The operation succeeded.
22:50 aleisha       hello talljoy!
22:50 talljoy       ohai!!!
22:50 talljoy       just rolled up my last release on 19.11.x
22:51 aleisha       fab! i'm looking forward to getting started :)
22:51 talljoy       kidclamp babysat me during the last 6 months, so the branch should be in great shape.  i don't think i did too much damage!
22:52 koha-jenkins  Project Koha_19.11_D8 build #157: UNSTABLE in 27 min: https://jenkins.koha-community.org/job/Koha_19.11_D8/157/
22:52 aleisha       :)
23:15 koha-jenkins  Project Koha_19.11_U18 build #156: SUCCESS in 50 min: https://jenkins.koha-community.org/job/Koha_19.11_U18/156/
23:26 koha-jenkins  Project Koha_19.11_D9 build #155: SUCCESS in 34 min: https://jenkins.koha-community.org/job/Koha_19.11_D9/155/
23:39 dcook         @later tell caroline I haven't used it but is that what the HouseboundModule is for?
23:39 huginn        dcook: The operation succeeded.
23:51 koha-jenkins  Yippee, build fixed!
23:51 wahanui       Congratulations!
23:51 koha-jenkins  Project Koha_19.11_D8 build #158: FIXED in 36 min: https://jenkins.koha-community.org/job/Koha_19.11_D8/158/