Time  Nick     Message
16:02 imp      s/contain/container/
15:58 imp      thd: a friend just told me, that orcale and db2 are able to start a new contain for the tables and continue working (but he has now idea, too, if mysql can du such things)
12:26 Ropuch   Good morning #koha
07:49 imp      (like, no problem with 150gb in one file on ext3, anyway, it's still far below the limit for the maximal filesize)
07:48 imp      but i never had any problems with linux storing complete hdd images on any filesystem i used
07:46 imp      it's late here ;)
07:46 imp      dunno
07:30 thd      Do terabyte ext2-4 files require building the filesystem with large file option?
07:26 imp      mysql has a filesizelimit -> http://dev.mysql.com/doc/refman/5.5/en/full-table.html
07:25 imp      i don't know
07:25 imp      another good question :). i would maybe get a random (yes, random) set of lines from the old file, open a new one and perform place the new lines into the new file (so you'll gain some space to update the lines in the old one). on a search, i would take a look into both ones parallel
07:24 thd      Would the database perform such a split automatically?
07:21 thd      How would you implement a split?
07:20 imp      it's up to the database how it'll handle such a situation (never had this problem), maybe it'll split the db into two files? dunno
07:19 thd      Very good point
07:19 imp      lets asume, both operations would take the same amount of (cpu) time. you'll still need an index to store the rest of the metadata (which would be quite stupid to implement that stuff as well in a flat file (-> reimplementing a db)), so you need a db anyway
07:18 thd      ?
07:18 thd      Ok. another question about scalability: What happens when the size of the DB table storage exceeds the maximum file size for a single file on the file system.
07:16 imp      (for each one)
07:16 imp      a database is just one open file (or at least not many), using single files, your system has to create a new filedecriptor, write the stuff and close the file handle
07:15 thd      Libraries tend to be writing few records in real time.  Most activity is reading how would the issue differ if most activity would be writing?
07:13 thd      What would be the effect for DB vs. filesystem performance where thousands of bibliographic records were being automatically captured and written within a few minutes?
07:13 imp      a database may enforce writing back the stuff it stores directly
07:12 imp      and you maybe have a new file, which is not written onto the disk if it's created (write buffer), if the host fails, it might be lost
07:11 imp      thd: you'll get some other problems if you store stuff in small files, the mysql performs smarter caching then a filesystem cache will do (if free memory is needed, the os drops the fs cache first)
07:11 thd      In Koha we are not doing many simultaneous writes of bibliographic records in real time.
07:08 chris    back
06:59 chris    brb putting kids to bed
06:59 chris    (blanket statements like the one above annoy me)
06:59 chris    and it depends entirely on the application, the filesystem and the OS
06:58 chris    i would agree it could be less efficient .. but could and less ... not definitely and much
06:57 thd      I am actually not asking necessarily in relation to Koha
06:57 chris    the other thing is
06:56 chris    ie, that is the least of our worries
06:56 chris    the upshot is, there are about 10 million things that i would work on to speed up koha before i even bothered looking at that one
06:55 chris    and not often into userspace ram
06:55 chris    not all
06:55 chris    some do
06:55 thd      yet filesystems also do caching
06:54 chris    thus outperforming most filesystems
06:54 chris    thd: in a sufficiently ram endowed system, mysql would cache the most used blobs in ram
06:54 thd      chris: what would be the difference in performance between storing the blobs in the DB or on the filesystem?
06:52 chris    and the marcxml is actually text anyway and thats were what little db interaction we do is done
06:51 chris    we dont ever search the blobs
06:51 chris    but definitely wrong for the way we use them
06:51 chris    thd that may be true for some actions
06:24 thd      I built such a system myself at the same time as Koha was being created but my system was certainly not efficient for other reasons
06:23 thd      each record as its own file
06:23 thd      yes imp
06:02 imp      happy new year moodaepo :)
06:02 imp      you'll have much filesystem overhead and maybe get some trouble with your filesystem (max amount of files per fs / dir)
06:02 moodaepo Happy New Year!
06:01 imp      thd: each record as own file?
05:17 thd      chris: I have bn speaking to someone who claims that storing blobs in the database as we do for bibliographic records is much less efficient than storing them as individual files on the filesystem
01:11 imp      happy new year :)