IRC log for #koha, 2020-05-21

All times shown according to UTC.

Time S Nick Message
00:27 inlibro joined #koha
00:31 dcook oleonard++
01:06 dcook Anyone around in #koha who deals with Chinese, Japanese, Korean MARC data?
01:07 cait kind of
01:07 dcook Having some interesting issues where Koha (or more like the MARC libraries) are failing to fully parse the MARC data
01:07 dcook We had to convert GB2312 to UTF-8 and that was fine, but loading it into Koha has been challenging
01:08 cait hm we only use utf-8 and no issues as far as ai know
01:08 dcook It is in ISO MARC which makes it even worse..
01:08 cait iso marc? (like not xml?)
01:08 dcook Right
01:08 cait we always use that then
01:08 cait no xml
01:08 dcook Really? Interesting...
01:08 dcook Transforming it from MARC to MARCXML didn't work with MarcEdit, and Koha fails as well
01:08 dcook But only some characters
01:09 dcook Like it'll turn 基督教历史地图集 into 基督教历史 with 3 rubbish bytes at the end
01:10 dcook Almost like it created 3 rubbish bytes where it couldn't read the last 3 characters...
01:11 cait if you have proper encoding, the format should not be an issue
01:11 dcook Agreed
01:11 cait there are some examples a bit lower on the result list
01:12 cait ah and second page is very chinese
01:13 dcook Yeah that makes sense
01:13 dcook Maybe there's some issues with the transcoding from GB2312 to UTF-8 although it seems so fine
01:13 dcook Maybe I'll try a simple script using MARC::Record to load it and maybe it'll tell me what the problem is
01:13 wahanui i think the problem is it has several code issues
01:14 dcook I can view the GB2312 and UTF8 just fine in Vim and Notepad++
01:14 dcook And you use ISO MARC for yours?
01:17 cait no xml yes
01:17 cait at import time
01:17 cait that's what we are talking about?
01:17 dcook Yep
01:18 dcook I just tried MARC::File::USMARC->decode($text) and it's doing it wrong...
01:18 cait hm
01:18 dcook Although I could play around with a few settings..
01:18 cait we get the data as mrc from our union catalog
01:19 dcook That really does seem to suggest some encoding issue on our end..
01:20 cait we do conversions before importing, but i'd still suspect an encoding issue
01:24 cait hm thinkig about it, we might convert to xml at some point when doing data manipulation... but the files i see on import have mrc ending. encoding never popped up outside of normalization (combined non-combined)
01:25 dcook I think I see something looking at the hex
01:26 dcook Hmm or maybe not..
01:26 dcook I see the title, a record separator, two spaces, then a unit separator..
01:27 dcook Well actually...
01:27 inlibro joined #koha
01:27 dcook I think the problem might be that we're converting GB2312 to UTF8 as that might screw up the directories and such
01:27 dcook That's probably what this is
01:28 dcook Maybe we should be doing MARC to MARCXML as GB2312 and then convert to UTF8
01:28 dcook That's logical...
01:28 dcook I think I even suspected that yesterday but forgot
01:28 cait worth trying?
01:39 dcook It's certainly interesting
01:40 dcook no mapping found for [0x9F] at position 1 in 基督教历史▒ g0=ASCII_DEFAULT g1=EXTENDED_LATIN at /usr/lib/perl5/site_perl/5.26.1/MARC/ line 308.
01:40 dcook I feel like I'm getting closer..
01:41 dcook Like I just need to configure MARC::Charset correctly
01:41 dcook Although it seems designed for marc8 to utf8 only...
01:41 dcook But that can't be right..
01:50 dcook Well I've been told to stop working on it, so unsolved for now
01:50 dcook But I think that's probably what it is
01:51 dcook I think GB2312 uses 2 bytes and UTF-8 uses 3 bytes
01:51 dcook So converting GB2312 to UTF8 then trying to process the MARC won't work
01:51 dcook But... I don't see a well supported way of parsing GB2312 ISO MARC with Perl...
01:52 dcook Will probably have to hack something together
01:54 liliputech_asu joined #koha
02:27 inlibro joined #koha
02:30 AndrewFH joined #koha
03:01 mtj hi dcook, ive have good results using iconv and/or yaz-iconv for encoding stuff
03:03 mtj iconv --from-code=GB2312 --to-code=UTF-8 in.mrc > utf8.mrc
03:13 dcook mtj: Interesting. That's what I did but that created the problem I think
03:14 dcook Like the encoding came across correctly
03:14 dcook However, when MARC::Record parses the MARC, it doesn't parse it correctly
03:15 dcook I could view in.mrc and utf8.mrc just fine in their respective encodings, but try to load that into Koha, MARC::Record, or MarcEdit and no good.
03:15 dcook It would truncate fields and turn valid UTF8 data into garbage bytes
03:15 dcook Even though it was find in the utf8.mrc file
03:15 dcook My only guess at the moment is that the problem is going from a 2 byte encoding to a 3 byte encoding is causing the ISO MARC directory and such to become corrupt
03:16 dcook As field lengths and such would change at a byte level, but that's just a guess
03:17 dcook This theory also comes from how most of the Chinese is parsed correctly but it gets truncated
03:17 dcook No more work time to spend on it but maybe after work time sometime...
03:23 mtj dcook: hiya, i recall importing CJK records as marcxml to be a bit more "robust"
03:23 mtj 'Maybe we should be doing MARC to MARCXML as GB2312 and then convert to UTF8'
03:24 mtj ^ you might be on the right track there :)
03:26 dcook I was thinking I was on the right track but getting MARC::Record to work with GB2312 has been tough heh
03:26 dcook But... I might give it a go another time
03:26 dcook Doing the transcoding with MARCXML would be trivial I think
03:27 * dcook shakes his fist at ISO 2709
03:27 inlibro joined #koha
03:28 dcook I might give it another go with MarcEdit instead...
03:29 dcook Mmm MarcEdit's conversion did the same job as iconv I think..
03:31 mtj i found it pretty tough too, i basically tried every tool and option i could find, and diffed the output
03:31 dcook Oh... but MarcEdit was able to open the UTF8 MARC. MARC::Record was struggling with that (or I was struggling to configure my script to do it correctly)
03:32 mtj ...the difference in results was often subtle
03:34 dcook Success!
03:34 dcook Well at least generating valid MARCXML
03:35 dcook I think I could do the character conversion with iconv or MarcEdit...
03:35 dcook Then the key was to open it in MarcEdit.
03:35 dcook And then save into the desired format
03:35 mtj i found converting a problematic iso2709 file to marcxml, imported more goodly into koha
03:35 dcook Which now?
03:37 mtj yaz-marcdump  -i marc -o marcxml  file.mrc > file.xml
03:37 mtj ^ twas what i used
03:37 cait1 joined #koha
03:37 dcook Ooo interesting
03:38 dcook So you did iconv to do the transcoding
03:38 mtj ... according to my terrible notes :p
03:38 dcook Then that to go from marc to marcxml?
03:38 mtj yeah, i think thats the order
03:39 mtj iconv has a --to-code=UTF-8//TRANSLIT option too..
03:40 dcook looks like yaz-marcdump worked well
03:40 dcook I wonder if MARC::Record would work too if I just set it up correctly, but I'm guessing not
03:41 dcook Thanks for that help, mtj. Made a big difference.
03:42 mtj np, the xml parser seems to handle 'problematic' files waay better
03:43 mtj iconv --verbose  -c --to-code=UTF-8//TRANSLIT  ./file.mrc  > file-trans.mrc
03:44 mtj ^ that looked useful for stripping glitched chars from a file
03:45 dcook Oh I thought it worked but now it looks like yaz-marcdump didn't
03:46 dcook Actually yaz-marcdump creates the wrong result too
03:46 dcook When you open it in vim you see: ""testmarc.xml" [ILLEGAL BYTE in line 16] 20109L, 775824C"
03:46 mtj oof
03:47 dcook And now I can't recreate it in MarcEdit either although I did before lol
03:47 dcook Might be due to file extension names..
03:48 dcook Or maybe I used a different...
03:49 mtj i would create a small test file of a few problem records, and experiment with iconv,yaz-iconv, yaz-marcdump on it
03:49 dcook lol yeah MarcEdit is picky with its file extensions...
03:49 dcook The sample collection is the small test file heh
03:50 mtj i could never work out how to script/auotmate marcedit, so never really used it
03:50 dcook Mmm fair enough
03:50 dcook I find it handy for semi-automation where you still need a human set of eyes
03:51 dcook Sure enough... if I do a MARC21 => MARCXML transform in MarcEdit it can't do it either
03:52 dcook Same result as yaz-marcdump
03:52 mtj the tools i mentioned got the job done, for me
03:52 dcook But if I open the MRC file into the MRK format in MarcEdit and then export it... beautiful
03:53 dcook I might try those additional iconv optoins you mentioned, mtj
03:54 dcook But I don't think it
03:54 dcook 's an encoding issue
03:54 dcook I think it's an ISO MARC issue
03:54 dcook Although if you recall your exact commands, I'd give it a go
03:55 mtj hmm, try...
03:55 mtj iconv --from-code=GB2312 --to-code=UTF-8 in.mrc > utf8.mrc
03:55 mtj iconv --from-code=GB2312  --to-code=UTF-8//TRANSLIT in.mrc > utf8-trans.mrc
03:56 mtj compare those ^
03:56 dcook iconv: conversion from `code=GB2312' is not supported
03:56 dcook And actually I did have to use a newer charset for the chinese to get it to work with iconv
03:57 dcook If I tried GB2312 I would get this:
03:57 dcook iconv -f GB2312 -t UTF-8 testmarc.mrc -o TestUTF8.mrc
03:57 dcook iconv: illegal input sequence at position 35414
03:57 dcook Even though it is supposedly GB2312
03:58 mtj hmm, iconv -l has CSGB2312 too
03:59 dcook Yeah that's no good either
03:59 dcook Ohh this is interesting...
03:59 dcook source marc has the following for the record length: 00651
03:59 mtj hmm, id try yaz-inconv for good measure
03:59 dcook But on my good XML export, it is actually 00701
03:59 dcook Yeah, I really don't think the problem is the encoding
03:59 dcook I think it's the ISO MARC leader
04:00 dcook Although I suppose that length would change with marcxml potentially...
04:00 dcook But anyway I found something that works
04:03 dcook I reckon the "Record label" and "Directory" are compromised by the transcoding
04:04 mtj iconv --from-code=GB2312 works ok on deb10/buster
04:05 oleonard-away_ joined #koha
04:06 dcook Yeah this is OpenSUSE. It would be interesting to try on Debian.
04:07 dcook Looking at[…]RC/File/, I don't see how it could work
04:08 dcook As it's dealing with bytes
04:08 oleonard_ joined #koha
04:08 dcook And it's going from a 2 byte encoding to a 3 byte encoding I think
04:08 dcook For yaz-marcdump, I'd have to check their source
04:09 dcook But I'm guessing MarcEdit's Editor is more cautious about the record label and directory in ISO MARC
04:11 dcook Hmm actually maybe I was misusing MARC::Record..
04:11 dcook I was trying to decode a whole file rather than one record..
04:17 kathryn joined #koha
04:19 dcook Got a bit further with my MARC::Record script but yeah it's not working correctly... because I think it's misinterpreting the ISO MARC
04:25 dcook Ahh and checking $record->warnings() proves a lot of that
04:27 dcook mtj: What version of yaz-marcdump did you use?
04:27 dcook I have YAZ 5.8.1
04:28 dcook[…]r/util/marcdump.c is so challenging to read imho...
04:28 inlibro joined #koha
04:28 dcook Ubuntu 18.04 has YAZ 5.30.2.
04:28 dcook I wouldn't be surprised if newer versions of yaz-marcdump don't trust the directory in ISO MARC...
04:28 mtj yaz 5.28.0-1.indexdata
04:28 dcook Interesting!
04:29 dcook Ok I'm going to try a newer version
04:29 mtj deb stretch main
04:30 mtj aah, there is a newer version for stretch too ..  Candidate: 5.30.2-1.indexdata
04:31 mtj yeah, def. grab a newish yaz dcook :)
04:31 dcook Hmm yep no luck
04:31 dcook Same rseult
04:32 dcook result*
04:33 dcook yaz-iconv just used iconv and yeah yaz-marcdump seems to be lost by the leader too
04:34 dcook although yaz-marcdump -v didn't show anything..
04:35 dcook oh wait no it did lol
04:35 dcook It just printed it to STDOUT instead of STDERR
05:05 dcook @later tell cait I resolved the problem in the end by doing the GB18030 -> UTF8 transcoding with iconv, opened the ISO MARC in MarcEdit into its MRK format, and then saved as MARCXML, and that did the trick. I think the issue was going from a 2 byte to 3 byte encoding and that confused MARC::File::USMARC.
05:05 huginn dcook: The operation succeeded.
05:28 inlibro joined #koha
05:47 KK joined #koha
06:12 KK Hi everyone, I'm new here.. We're looking to implement Koha for a library management system that we're working on.. How far does Koha allow for the customization of UI?
06:28 inlibro joined #koha
06:34 karthikkumars joined #koha
06:38 mtj @later tell joubu: hiya, you should be able to install master koha from here...
06:38 huginn mtj: The operation succeeded.
06:40 did joined #koha
06:40 alexbuckley joined #koha
06:41 karthikkumars Hi everyone, I'm new here.. We're looking to use Koha for a library management system that we're working on.. My question is how far does Koha allow UI customization?
06:43 karthikkumars I see that the administration section allows me to provide HTML to pre-built blocks.. Is there any way I could override these blocks and make my own UI components?
07:13 nsk joined #koha
07:28 inlibro joined #koha
07:37 nsk hi guys
07:37 nsk anyone around?
08:05 Joubu mtj: yes, but the idea was to have a "clone" of what would be released. We would need a version switch and the PO files (at least, cannot think of something else right now)
08:28 inlibro joined #koha
09:07 karthikkumars joined #koha
09:16 Joubu @later tell kidclamp ping bug 21591
09:16 huginn Joubu: The operation succeeded.
09:28 inlibro joined #koha
09:54 kidclamp replied Joubu
10:03 cait joined #koha
10:08 khall joined #koha
10:09 Joubu kidclamp: thanks, will have a look again then
10:09 * cait waves
10:09 Joubu we need bug 25473 and bug 25563 for 20.05
10:09 huginn Bug[…]_bug.cgi?id=25473 major, P5 - low, ---, oleonard, Needs Signoff , Can't add order from MARC file, save button does nothing
10:09 huginn Bug[…]_bug.cgi?id=25563 major, P5 - low, ---, jonathan.druart, Needs Signoff , Cannot sub "add order from MARC file" form after alert
10:09 Joubu we even need them for next stable releases actually
10:10 cait hm I can trya and have a look in an hour or so?
10:10 Joubu would be great yes
10:11 kidclamp I didn't realize you were submitting on 25563 Joubu, I put a patch there too ;-)
10:11 Joubu arf sorry
10:11 Joubu I am not sure my patch is godd
10:11 Joubu how did you fix it?
10:11 kidclamp no worries, they are almost exactly the same - I just make the button not be a submit button so we don't have to prevent default
10:12 Joubu exactly like me
10:12 Joubu :)
10:12 Joubu good
10:12 kidclamp almost, you leave it as 'submit'
10:12 Joubu I don't have a strong preference, but I think it's better as submit, isn't it?
10:13 Joubu at least mine has a test plan :D
10:13 Joubu I let you decide
10:14 cait wahtever you decide, i want a test plan
10:15 Joubu there is one
10:15 Joubu Patches from both bug reportrs should be tested together
10:17 Joubu did: around?
10:29 inlibro joined #koha
10:43 khall joined #koha
10:52 Joubu Thanks kidclamp!
11:02 Joubu @later tell tcohen please have a look at bug 25513 comments 12 and 13. This is super weird. I know it's not what we are testing, and it is a quite minor issue, but... should we hide the problem and set the integers to a given value to make sure the exponent formatting won't be displayed?
11:02 huginn Joubu: The operation succeeded.
11:12 davidnind left #koha
11:17 oleonard o/
11:19 cait hi oleonard :)
11:28 nsk1 joined #koha
11:28 nsk1 hi all
11:28 nsk1 i have a doubt
11:28 nsk1 can anyone help?
11:29 inlibro joined #koha
11:29 nsk1 i have installed the software
11:29 nsk1 is it possible to replace the Koha logo?
11:32 nsk1 no one around?
11:32 tcohen jpña
11:32 tcohen hola
11:32 nsk1 hi cohen
11:34 oleonard nsk1 which Koha logo? OPAC?
11:34 nsk1 staff
11:35 nsk1 this one and the ones inside:
11:35 nsk1
11:35 nsk1 thats just a sample
11:36 nsk1 i assume this is called the staff login
11:37 pastebot "oleonard" at pasted "nsk1: CSS controlling display of the logo in the staff client" (16 lines) at
11:38 oleonard That paste shows the parts of the CSS which control display of the logos
11:38 oleonard You can use the  IntranetUserCSS preference to override that CSS
11:41 nsk1 i see...lemme check it out
12:14 Dyrcona joined #koha
12:29 inlibro joined #koha
12:31 Marie-Luce joined #koha
12:53 cait Joubu: bit late, but both done
12:53 cait do you have another urgent one on oyur list? missing overview myself right now
12:56 Joubu tcohen: 25513 - we should force the formatting when we display integers/floats for the REST API
12:56 Joubu cait: bug 24229
12:56 huginn Bug[…]_bug.cgi?id=24229 normal, P5 - low, ---, tomascohen, Signed Off , /items API tests fail on Ubuntu 18.04
12:56 Joubu watch rel_20_05_candidate
12:57 Joubu this is the only one waiting for QA
12:57 cait hm api
12:57 cait I think that one might not be for me :(
12:58 Joubu you just need to run the tests :)
12:58 tcohen 6 of them
12:58 Joubu and ask Tomas to open a separate bug report :D
12:59 cait for?
13:00 Joubu see second to last comment
13:00 khall joined #koha
13:01 cait tcohen: can you just file the separate bug without me having to read? ;)
13:01 tcohen I will, I'm kind of overloaded at the moment sorry
13:10 nsk1 joined #koha
13:10 nsk1 hi guys
13:11 nsk1 i have installed the library and created a staff via the admin
13:11 nsk1 but the staff is unable to login
13:11 khall joined #koha
13:11 nsk1 it says that the user does not have enough permissions to access this page..when they try to login
13:11 nsk1 why?
13:14 cait did you assign permissions?
13:14 Joubu it should be a superlibrarian user
13:14 Joubu Query your DB with: select userid, flags from borrowers;
13:14 cait go to their patron account in staff and check more > permissions
13:15 Joubu userid is the "login", and flags must be 1
13:15 nsk1 ok, lemme check...can we do this checking via admin?
13:16 Joubu I thought you were not able to login
13:16 Joubu but actually you did
13:16 Joubu so cait is right, you should set the permissions for the new user
13:17 nsk1 i set the permission as superlibrarian
13:17 nsk1 but still unable to access
13:18 nsk1 never mind...i think i made a mistake...lemme check
13:18 nsk1 got it
13:18 nsk1 thanks
13:19 wizzyrea joined #koha
13:19 khall joined #koha
13:28 cait hi khall :)
13:29 khall hi cait !
13:29 inlibro joined #koha
13:31 Joubu bug 25567
13:31 huginn Bug[…]_bug.cgi?id=25567 critical, P5 - low, ---, jonathan.druart, ASSIGNED , borrower_attribute_types.category_code must be set to undef if not set
13:32 Joubu there is a patch to test
14:01 tuxayo Joubu: Hi o/ I'm looking at it
14:23 nsk1 how can i access the OPAC? i am able to see only the admin/staff login
14:25 Joubu how did you install?
14:25 Joubu you should find it in the apache config (search for VirtualHost)
14:26 Joubu it => the host name I meant
14:27 nsk1 i see
14:27 nsk1 lemme check
14:29 inlibro joined #koha
14:42 georgew joined #koha
15:29 inlibro joined #koha
16:30 inlibro joined #koha
16:42 georgew joined #koha
16:46 koha-jenkins Project Koha_19.11_D8 build #155: SUCCESS in 36 min: https://jenkins.koha-community[…]oha_19.11_D8/155/
16:47 koha-jenkins Project Koha_19.11_U18 build #154: STILL UNSTABLE in 37 min: https://jenkins.koha-community[…]ha_19.11_U18/154/
16:47 bdonnahue2 joined #koha
16:48 bdonnahue3 joined #koha
16:51 khall_ joined #koha
16:56 bdonnahue1 joined #koha
17:00 bdonnahue2 joined #koha
17:17 CrispyBran joined #koha
17:30 inlibro joined #koha
17:35 koha-jenkins Project Koha_19.11_D9 build #153: SUCCESS in 48 min: https://jenkins.koha-community[…]oha_19.11_D9/153/
17:37 davidnind joined #koha
18:30 inlibro joined #koha
18:31 tuxayo Does anyone see any bug particularly worth to sign off on this last day before release?
18:31 caroline Anyone doing checkout deliveries? If so how how do you manage it?
19:19 cait checkout deliveries?
19:23 * tuxayo plans to keep reading the wiki pages about Rmaint, development, bug reporting, etc. As it seems that there is no more work related to the release that they could do.
19:25 cait setting up your environment for rmainting is probably a good idea :)
19:30 inlibro joined #koha
19:31 caroline cait: like ppl place holds via the opac, you check them out to their file and you deliver them to their house
19:32 cait hm delivery i haven't heard from form our libraries, but pick-up
19:32 cait they get the items ready and you can schedule a pick-up time - contact less basically
19:34 cait sorry for the typos
19:36 tuxayo cait: Thanks for the advice :)
20:00 bdonnahue3 joined #koha
20:18 lukeG left #koha
20:18 kathryn joined #koha
20:18 lukeG joined #koha
20:18 alexbuckley joined #koha
20:30 inlibro joined #koha
20:33 aleisha joined #koha
21:04 tosca joined #koha
21:14 hayley joined #koha
21:21 tosca_ joined #koha
21:25 koha-jenkins Project Koha_19.11_D9 build #154: SUCCESS in 34 min: https://jenkins.koha-community[…]oha_19.11_D9/154/
21:31 inlibro joined #koha
21:41 lukeG left #koha
21:45 koha-jenkins Yippee, build fixed!
21:45 wahanui Congratulations!
21:45 koha-jenkins Project Koha_19.11_U18 build #155: FIXED in 50 min: https://jenkins.koha-community[…]ha_19.11_U18/155/
21:53 koha-jenkins Project Koha_19.11_D8 build #156: SUCCESS in 27 min: https://jenkins.koha-community[…]oha_19.11_D8/156/
22:31 inlibro joined #koha
22:50 talljoy @later tell aleisha 19.11.x branch is all yours!!  hope i left it in good shape for you.  please let me know if i can answer any questions.
22:50 huginn talljoy: The operation succeeded.
22:50 aleisha hello talljoy!
22:50 talljoy ohai!!!
22:50 talljoy just rolled up my last release on 19.11.x
22:51 aleisha fab! i'm looking forward to getting started :)
22:51 talljoy kidclamp babysat me during the last 6 months, so the branch should be in great shape.  i don't think i did too much damage!
22:52 koha-jenkins Project Koha_19.11_D8 build #157: UNSTABLE in 27 min: https://jenkins.koha-community[…]oha_19.11_D8/157/
22:52 aleisha :)
23:15 koha-jenkins Project Koha_19.11_U18 build #156: SUCCESS in 50 min: https://jenkins.koha-community[…]ha_19.11_U18/156/
23:26 koha-jenkins Project Koha_19.11_D9 build #155: SUCCESS in 34 min: https://jenkins.koha-community[…]oha_19.11_D9/155/
23:31 inlibro joined #koha
23:39 dcook @later tell caroline I haven't used it but is that what the HouseboundModule is for?
23:39 huginn dcook: The operation succeeded.
23:51 koha-jenkins Yippee, build fixed!
23:51 wahanui Congratulations!
23:51 koha-jenkins Project Koha_19.11_D8 build #158: FIXED in 36 min: https://jenkins.koha-community[…]oha_19.11_D8/158/

| Channels | #koha index | Today | | Search | Google Search | Plain-Text | plain, newest first | summary