IRC log for #koha, 2016-09-04

All times shown according to UTC.

Time S Nick Message
00:20 JoshB joined #koha
01:09 dbs joined #koha
01:44 dbs joined #koha
02:15 NateC joined #koha
02:25 JoshB joined #koha
04:06 Francesca joined #koha
05:40 JoshB joined #koha
06:02 Francesca joined #koha
06:13 Francesca joined #koha
06:46 drojf joined #koha
06:46 drojf hi #koha
07:58 StomproJosh joined #koha
09:18 cait joined #koha
09:22 Francesca joined #koha
10:12 Francesca joined #koha
10:16 cait1 joined #koha
12:12 cait joined #koha
13:23 ashimema joined #koha
14:10 DrLang joined #koha
14:49 dbs joined #koha
15:35 dbs joined #koha
17:18 StomproJ joined #koha
17:19 JesseM_a_ joined #koha
17:23 barton|a1ay joined #koha
17:23 denials joined #koha
17:24 Mike-CSPL joined #koha
17:26 rangi joined #koha
17:26 druthb_away joined #koha
18:38 cdickinson_ joined #koha
20:02 ibeardslee joined #koha
20:09 archer121 joined #koha
20:10 archer121 Hi, our koha server is crashing occasionally for some reason. Where can I look to get an idea as to why this is happening? I have described my issue in detail over here:[…]e-proxy-503-error Please see the answer to the post too.
20:11 archer121 Fellas, extremely sorry, I posted the wrong link. Here is the correct one:[…]-error-503/801032
20:12 archer121 I don't see anything else in /var/logs/koha
20:14 cait1 hi archer121
20:14 cait1 still sunday in some parts of the world - quiet here usually
20:15 archer121 monday morning 2 AM here... :-)
20:15 cait1 did you see the answer on the mailing list?
20:17 archer121 yeah, but in my case, the backed service, ie koha has completely crashed.
20:17 archer121 backend*
20:18 archer121 Do you still think it will solve my issue?
20:21 cait1 i am not a sysadmin
20:22 cait1 sorry, probably not much help there
20:22 cait1 how did you install Koha? which OS? I think that's what you will be asked first
20:22 archer121 me neither, but thanks anyway
20:22 archer121 debain, apt
20:24 cait1 is there something special about your setup?
20:25 cait1 mostly feeding the chat logs here - we are running with packages on Debian too, but haven't experienced problems like that so far
20:25 archer121 We have it integrated to RFID hardware on our own, but in a safe manner
20:26 archer121 we have memcached and plack running
20:27 cait1 hm maybe you should add all that information to the mailing list threaad
20:27 cait1 together with the exact version of koha you are using
20:28 archer121 It;s been just 2 months since we migrated to koha, and the issue was there since the beginning. In this two months, we have upgraded twice, and once purged and reinstalled koha.
20:28 archer121 yes I will.
20:29 cait1 hm sounds not like fun :(
20:30 rangi hmm i haven't run into plack crashing, on any of the 50 or so sites i look after
20:30 rangi is it OOMing?
20:30 archer121 restarting apache won't help, but what the technicians do about it is to restart the entire system.
20:30 archer121 nope
20:30 rangi yeah restarting apache won't do anything to plack
20:30 archer121 4.2G free
20:30 rangi i dont think it's anything to do with apache
20:30 archer121 why do you guys think it is plack?
20:31 rangi because that is what the error is telling you
20:31 archer121 why do you guys think it has something to do with plack?
20:31 rangi HTTP: attempt to connect to Unix domain socket /var/run/koha/nitc/plack.sock (localhost) failed
20:31 rangi next time it happens run
20:32 archer121 I currently have it in the crashed state
20:32 rangi sudo koha-plack --restart instancename
20:32 rangi if that doesnt work
20:32 rangi try --stop
20:32 rangi then --start
20:34 archer121 alright, so plack was not running, so I could not restart or stop it (as it was not running)
20:34 rangi but starting it worked?
20:35 archer121 half worked.
20:36 archer121 the 503 goes away, but it is asif the zebra indexing is not done, you know, like all searches on opac returning empty
20:36 rangi restart zebra too then
20:37 rangi sudo koha-restart-zebra instancename
20:38 archer121 yeah, that worked.
20:38 rangi it really does feel like that your machine OOMed at some point in the past and killed zebra and plack but that is just a guess you'd have to go back through syslogs looking
20:38 rangi really odd for both plack and zebra to have died
20:38 archer121 I don;t think so, because OOMs should come in the dmesg
20:38 rangi so yeah, i think you are going to need to do some forensics to find out what is killing those 2 things
20:39 rangi bad ram, ooming, something else
20:39 rangi id set up some monitoring with monit, or icinga2 or something
20:39 rangi to monitor zebra and plack, see if you can pinpoint when it happens
20:40 rangi but, at least restarting works, which is a zillion times better than rebooting
20:40 archer121 I just found out when it happens from the plack-error.log: 2016/09/04-07:37:06 Server closing!
20:41 archer121 And at that time the library is closed.
20:41 kathryn joined #koha
20:41 rangi yeah, so you will want to track down what is doing it, see if you can find out when zebra was turned off/crashed too
20:50 archer121 at the same time: zebra-error.log: 20160904 07:37:06 nitc-koha-zebra: client (pid 19241) killed by signal 15, stopping
20:53 rangi yeah
20:53 rangi something did that
20:54 rangi thats not a crash
20:54 rangi so you need to find what was running at 7.30ish
20:54 rangi maybe logrotate
20:59 archer121 ah, I found something sweet in syslog!
20:59 archer121 cron.daily was executed at the same time.
20:59 rangi yeah, so its most likely logrotate
21:00 rangi that stops stuff, rotates the logs, and supposed to restart it
21:00 archer121 here, take a look:
21:02 rangi look at the syslog before that
21:02 rangi because syslog gets restarted as part of the logrotate too
21:03 cait1 archer121: i tihnk what rangi is trying to tell you is that it doesn't crash
21:03 cait1 it's shut down intentionally
21:03 cait1 to do some system taks - but it doesn't come back like it should
21:05 archer121 i see.
21:06 archer121 here is the syslog before the logrotate:
21:07 archer121 I do not see anything useful in it, but my eyes are not that trained,
21:07 archer121 I now think that this crash occurs every sunday.
21:07 archer121 and sunday is a calendar holiday
21:09 rangi yeah thats logrotate running
21:16 Francesca joined #koha
21:17 archer121 at this point the only thing that I can think of doing is to manually run all the commands in koha's cron.daily and see if it fails.
21:19 rangi i guarantee it is logrotate doing it
21:19 archer121 okey, so now I am stuck. what should I do?
21:20 rangi what version are you running ?
21:20 rangi (of koha)
21:21 archer121 3.22.10
21:21 archer121 but the crash was there since 3.22.08, which was our first version
21:21 rangi it's not a crash
21:22 archer121 Why is this not occuring daily if logrotate is doing it?
21:22 rangi it will be a race condition
21:22 rangi it'll be trying to start it again, while it is still stopping
21:23 rangi so the start will fail, and it will continue stopping, and be stopped
21:24 rangi if you look in /etc/logrotate.d/
21:24 rangi there is a file koha-common
21:24 rangi that is what tells it what to do
21:26 archer121 and this happens weekly!
21:27 rangi right so logrotate probably tells it to rotate weekly
21:27 rangi weekly
21:27 rangi yep
21:27 archer121 and the fix?
21:27 wahanui the fix is
21:28 archer121 is that a bot?
21:28 wizzyrea yes
21:28 wizzyrea also hi
21:28 archer121 hi
21:28 rangi[…]-July/045823.html
21:28 rangi maybe try that
21:28 rangi then check again next time it runs
21:29 rangi we now know when that is going to be
21:29 rangi 7.30am sunday morning
21:30 archer121 great! thanks a lot!
21:30 archer121 but will this change get overwritten on every update of koha?
21:31 rangi yep, but if it works, you can file a bug and say that is the fix, then it will go into koha
21:32 archer121 Will do that
21:37 archer122_ joined #koha
21:37 archer122_ hey, I dot disconnected for a moment
21:37 archer122_ got*
21:38 cait1 nothing happened
21:38 archer122_ great
21:38 Francesca joined #koha
21:39 archer122_ I am planning to confirm it this is the issue and if so file the bug report by manually triggering a logrotate right now. is that okey, rangi?
21:41 rangi yeah you probably didnt want to do that
21:42 archer122_ oops, why? I already did that. is it going to create any problems?
21:43 archer122_ and koha is functioning properly even after I did the logrotate
21:44 archer122_ (without any sleep)
21:47 archer122_ okey, so maybe zebra needs to do something in the background if this issue is to be reporduced.
21:49 rangi yeah, it won't be a real test unless its running for real
21:50 archer122_ okey, gotta hit bed. I have to attend classes tomorrow!
21:50 archer122_ thanks you again for helping.
21:57 Francesca joined #koha
22:09 eythian[…]for-overdue-books
22:15 rangi yeah, what a horrible idea
22:32 Archer121 joined #koha
22:33 Archer121 rangi: I got this mail from radek siman that he solved the same issue by replacing anacrom with cron.
22:34 rangi for some definition of the word solved
22:35 rangi :)
22:36 Archer121 What do you mean?
22:36 rangi it's not really something we can tell all users of koha to do
22:36 rangi fixing the actual logrotate job is a better way to actually fix it
22:37 Archer121 I see. So how does changing from amacron to cron fix it?
22:37 rangi who knows
22:37 Archer121 :-) bye!
22:37 rangi thats why i call it not an actual fix
22:40 mtj joined #koha
22:43 eythian It would be interesting to see if having anacron installed causes the problem, but in theory it should be identical to just cron on a server.
23:11 irma joined #koha
23:13 nage joined #koha
23:53 dcook joined #koha
23:54 * dcook waves
23:59 JoshB joined #koha
23:59 Francesca joined #koha
23:59 jamesb joined #koha

| Channels | #koha index | Today | | Search | Google Search | Plain-Text | plain, newest first | summary