Time Nick Message 16:02 imp s/contain/container/ 15:58 imp thd: a friend just told me, that orcale and db2 are able to start a new contain for the tables and continue working (but he has now idea, too, if mysql can du such things) 12:26 Ropuch Good morning #koha 07:49 imp (like, no problem with 150gb in one file on ext3, anyway, it's still far below the limit for the maximal filesize) 07:48 imp but i never had any problems with linux storing complete hdd images on any filesystem i used 07:46 imp it's late here ;) 07:46 imp dunno 07:30 thd Do terabyte ext2-4 files require building the filesystem with large file option? 07:26 imp mysql has a filesizelimit -> http://dev.mysql.com/doc/refman/5.5/en/full-table.html 07:25 imp i don't know 07:25 imp another good question :). i would maybe get a random (yes, random) set of lines from the old file, open a new one and perform place the new lines into the new file (so you'll gain some space to update the lines in the old one). on a search, i would take a look into both ones parallel 07:24 thd Would the database perform such a split automatically? 07:21 thd How would you implement a split? 07:20 imp it's up to the database how it'll handle such a situation (never had this problem), maybe it'll split the db into two files? dunno 07:19 thd Very good point 07:19 imp lets asume, both operations would take the same amount of (cpu) time. you'll still need an index to store the rest of the metadata (which would be quite stupid to implement that stuff as well in a flat file (-> reimplementing a db)), so you need a db anyway 07:18 thd ? 07:18 thd Ok. another question about scalability: What happens when the size of the DB table storage exceeds the maximum file size for a single file on the file system. 07:16 imp (for each one) 07:16 imp a database is just one open file (or at least not many), using single files, your system has to create a new filedecriptor, write the stuff and close the file handle 07:15 thd Libraries tend to be writing few records in real time. Most activity is reading how would the issue differ if most activity would be writing? 07:13 thd What would be the effect for DB vs. filesystem performance where thousands of bibliographic records were being automatically captured and written within a few minutes? 07:13 imp a database may enforce writing back the stuff it stores directly 07:12 imp and you maybe have a new file, which is not written onto the disk if it's created (write buffer), if the host fails, it might be lost 07:11 imp thd: you'll get some other problems if you store stuff in small files, the mysql performs smarter caching then a filesystem cache will do (if free memory is needed, the os drops the fs cache first) 07:11 thd In Koha we are not doing many simultaneous writes of bibliographic records in real time. 07:08 chris back 06:59 chris brb putting kids to bed 06:59 chris (blanket statements like the one above annoy me) 06:59 chris and it depends entirely on the application, the filesystem and the OS 06:58 chris i would agree it could be less efficient .. but could and less ... not definitely and much 06:57 thd I am actually not asking necessarily in relation to Koha 06:57 chris the other thing is 06:56 chris ie, that is the least of our worries 06:56 chris the upshot is, there are about 10 million things that i would work on to speed up koha before i even bothered looking at that one 06:55 chris and not often into userspace ram 06:55 chris not all 06:55 chris some do 06:55 thd yet filesystems also do caching 06:54 chris thus outperforming most filesystems 06:54 chris thd: in a sufficiently ram endowed system, mysql would cache the most used blobs in ram 06:54 thd chris: what would be the difference in performance between storing the blobs in the DB or on the filesystem? 06:52 chris and the marcxml is actually text anyway and thats were what little db interaction we do is done 06:51 chris we dont ever search the blobs 06:51 chris but definitely wrong for the way we use them 06:51 chris thd that may be true for some actions 06:24 thd I built such a system myself at the same time as Koha was being created but my system was certainly not efficient for other reasons 06:23 thd each record as its own file 06:23 thd yes imp 06:02 imp happy new year moodaepo :) 06:02 imp you'll have much filesystem overhead and maybe get some trouble with your filesystem (max amount of files per fs / dir) 06:02 moodaepo Happy New Year! 06:01 imp thd: each record as own file? 05:17 thd chris: I have bn speaking to someone who claims that storing blobs in the database as we do for bibliographic records is much less efficient than storing them as individual files on the filesystem 01:11 imp happy new year :)