| Time |
S |
Nick |
Message |
| 08:37 |
|
wajasu |
i've been working on koha-testing-docker. i wanted to be able to ktd up, and then ctrl-c then ktd up so i worked on trying to get the files/run.sh more idempotent. so one can restart. |
| 08:38 |
|
wajasu |
i might have fixed a couple of places, and though the first ktd up pulls the transator files about 2.7Gb, the restart is quicker. |
| 08:40 |
|
wajasu |
i wondered if we want the database to be left in place, on the subsequent ktd up runs. OR should it reset-all (i.e. drop the db, and repopluate) |
| 08:43 |
|
wajasu |
we could try doing the git clone of the translation po stuff as part of the Dockerfile build image for koha/koha-testing:master then in the run.sh it could just git pull the differences |
| 08:45 |
|
wajasu |
that way whoever builds that base image for koha/koha-testing:master would do that every few months or year. then that would be int the docker cache. |
| 08:45 |
|
wajasu |
that way we would only need to get the changes with git pull |
| 08:45 |
|
wajasu |
just an idea |
| 08:45 |
|
wajasu |
to speed up ktd |
| 08:47 |
|
wajasu |
that whole ktd is a work of art. i guess it's used for CD/CI builds too. |
| 17:40 |
|
wajasu |
hurray! i got my tweaked ktd docker-compose-light.yml starting, stopping, starting on my podman machine as rootless. |
| 17:54 |
|
wajasu |
well, docker pause koha_koha_1 and docker unpause koha_koha_1 works. not that we need that :0 |
| 20:34 |
|
tuxayo |
wajasu: hi :) |
| 20:34 |
|
tuxayo |
«i wanted to be able to ktd up, and then ctrl-c then ktd up so i worked on trying to get the files/run.sh more idempotent. so one can restart.» |
| 20:34 |
|
tuxayo |
I'm running `ktd up ; kd` so when I ctrl-c, it autocleans and I can restart without additional step. |
| 20:35 |
|
tuxayo |
> i might have fixed a couple of places |
| 20:35 |
|
tuxayo |
Work on KTD is very welcome :D |
| 20:55 |
|
tuxayo |
wajasu: about persistence there this: https://gitlab.com/koha-commun[…]erge_requests/410 |
| 20:55 |
|
tuxayo |
> i wondered if we want the database to be left in place, on the subsequent ktd up runs. OR should it reset-all (i.e. drop the db, and repopluate) |
| 20:55 |
|
tuxayo |
It might be a historical bias due to the CI usage. If persistence works well, it could be a default and people would just reset_all if needed |
| 20:56 |
|
tuxayo |
But maybe it's hard to account for the changes in the KTD that need a reset to get to a working state. I don't know how often that would break |
| 20:57 |
|
tuxayo |
And how subtlety it would break. |
| 20:57 |
|
tuxayo |
> we could try doing the git clone of the translation po stuff as part of the Dockerfile build image for koha/koha-testing:master then in the run.sh it could just git pull the differences |
| 20:57 |
|
tuxayo |
The big downside is even larger images when updating |
| 20:57 |
|
tuxayo |
It's already a lot |
| 20:58 |
|
tuxayo |
The current solution has it persisting so it's great |
| 20:58 |
|
tuxayo |
it: translations |
| 21:00 |
|
tuxayo |
instead of downloading the translation history once on first start, it would be downloaded on every image update. Even though download via git and via docker images isn't comparable, there is one done repeatedly so that takes over any difference |
| 21:02 |
|
tuxayo |
> that way whoever builds that base image for koha/koha-testing:master would do that every few months or year. then that would be int the docker cache. |
| 21:02 |
|
tuxayo |
It needs much more frequent updates due to changes in perl dependencies. |
| 21:02 |
|
tuxayo |
And changes in KTD itself. The image build is done automatically on a GitLab CI/CD worker. |
| 21:03 |
|
tuxayo |
> that way we would only need to get the changes with git pull |
| 21:03 |
|
tuxayo |
> to speed up ktd |
| 21:03 |
|
tuxayo |
I'm not sure I get how that would speed up KTD, the first run you mean? To avoid downloading via git clone the whole translation history? And have it in the image which would be faster to DL. |
| 21:04 |
|
wajasu |
yes download at first. then just changes afterward |
| 21:04 |
|
wajasu |
the docker will keep a layer of the original clone. |
| 21:05 |
|
tuxayo |
A lot is discarded on each image update |
| 21:05 |
|
tuxayo |
Maybe the images could be done in a different way |
| 21:06 |
|
wajasu |
how often does koha/koha_testing get rebuilt? monthly |
| 21:07 |
|
tuxayo |
More often |
| 21:07 |
|
tuxayo |
In the past it was each time there was a commit on main/master on Koha code, to be sure to have the perl deps |
| 21:07 |
|
tuxayo |
Now it's smarter |
| 21:07 |
|
tuxayo |
https://hub.docker.com/layers/[…]6?context=explore |
| 21:07 |
|
wajasu |
well, i can build it once in 15min and use my local one all day. work bugs, etc. |
| 21:08 |
|
tuxayo |
Every rebuild if the base Debian or Ubuntu is updated that invalidate all the layers bellow, so 100% |
| 21:09 |
|
wajasu |
i am going to try multi-sage builds |
| 21:10 |
|
tuxayo |
At least the image is rebuilt every KTD change on main/master to account for the changes in KTD itself |
| 21:10 |
|
tuxayo |
https://gitlab.com/koha-commun[…]r/?ref_type=HEADS |
| 21:10 |
|
tuxayo |
At least in depency change in Koha's cpanfile |
| 21:10 |
|
tuxayo |
https://git.koha-community.org[…]h/master/cpanfile |
| 21:12 |
|
wajasu |
maybe a FROM scratch RUN git clone ...koha. then next stage is FROM debian:bullseye so i can keep the koha repo and switch OSes with same koha repo with cloning |
| 21:12 |
|
tuxayo |
And maybe when our packing manager update perl dependencies in Koha deb repository. |
| 21:12 |
|
tuxayo |
In the end, I think here is the overall build history https://gitlab.com/koha-commun[…]ocker/-/pipelines |
| 21:13 |
|
wajasu |
i also got docker compose watch working so any change in the host koha that i clone from gets synced into the container one-way |
| 21:13 |
|
tuxayo |
> i am going to try multi-sage builds |
| 21:13 |
|
tuxayo |
Is that a way to avoid throwing away layers when the OS is updated for example? |
| 21:13 |
|
wajasu |
i haven't been araound for a year so i have to get up to dat onthe yarn gulp js stuff. |
| 21:14 |
|
tuxayo |
> maybe a FROM scratch RUN git clone ...koha |
| 21:14 |
|
tuxayo |
If it's about Koha's code, it still needs to be cloned manually before starting KTD the 1st time. |
| 21:15 |
|
wajasu |
does the current KTD watch and somehow rebuild the yarn or js stuff if new packages are added on the fly without a docker down/up? |
| 21:15 |
|
tuxayo |
> does the current KTD watch and somehow rebuild the yarn or js stuff |
| 21:15 |
|
tuxayo |
Not, that isn't in the images IIUC, it's done on start and on reset_all |
| 21:16 |
|
wajasu |
ok. |
| 21:16 |
|
tuxayo |
That would require to download like 1G every time there is a commit touching SCSS and typescript pushed on master |
| 21:17 |
|
wajasu |
because that docker watch has an action: rebuild that can rebuild the yarn stuff without user intervention, just like the sync |
| 21:17 |
|
tuxayo |
> i also got docker compose watch working so any change in the host koha that i clone from gets synced into the container one-way |
| 21:17 |
|
tuxayo |
What is the host Koha? |
| 21:17 |
|
tuxayo |
The repo of the code? |
| 21:18 |
|
tuxayo |
Isn't your Koha code mounted as a volume in your KTD, so always in sync |
| 21:18 |
|
wajasu |
yes the manual koha git pull. |
| 21:18 |
|
tuxayo |
*? |
| 21:19 |
|
wajasu |
my Dockerfile is COPY --chown=1000:1000 ./koha /kohadevbox/koha (4G) |
| 21:20 |
|
wajasu |
the koha git repo is colocated relative to my Dockerfile and docker-compose-light.yml |
| 21:20 |
|
wajasu |
i've been baby stepping this stuff |
| 21:21 |
|
tuxayo |
Is that for some sort of production/preproduction? |
| 21:21 |
|
tuxayo |
How do you work on the code if it's not differently synced? Like needed to restart KTD on each change. Maybe i'm totally missunderstanding |
| 21:21 |
|
wajasu |
i couldn't use docker watch without an initial copy into the container |
| 21:21 |
|
tuxayo |
the workflow here |
| 21:21 |
|
tuxayo |
*differently => directly |
| 21:21 |
|
tuxayo |
typo-- |
| 21:22 |
|
wajasu |
i added watch support the the compose file. https://docs.docker.com/compose/file-watch/ |
| 21:22 |
|
wajasu |
it came out in october2023 |
| 21:24 |
|
tuxayo |
Ok, so you indeed do dev work on that setup. |
| 21:24 |
|
tuxayo |
So each time you make a change there is some big process to have it deployed, right? That's why the startup time is even more critical for you. |
| 21:24 |
|
wajasu |
is added the watch lines to the existing docker-compose-light.yml and using my koha pre-copied into the koha-testing image, it |
| 21:25 |
|
wajasu |
yes i can do dev work from the host OS, and it shows up on the fly in the container. |
| 21:26 |
|
tuxayo |
on the fly after something like a rebuild and a restart of the container, is that how it works? |
| 21:26 |
|
wajasu |
the goal is to be able to change tt or javascript/packages, and see results on the fly. |
| 21:27 |
|
wajasu |
i am trying to get it so i don't have to stop/start the container. i just code, and then refresh the browser. |
| 21:29 |
|
wajasu |
it seems to work now where i can build my local koha-testing image, then i can work onthe files/run.sh and not have to wait to pull the translations pos. |
| 21:29 |
|
wajasu |
that saves time. |
| 21:29 |
|
tuxayo |
> i am trying to get it so i don't have to stop/start the container. i just code, and then refresh the browser. |
| 21:29 |
|
tuxayo |
The default KTD workflow allows that almost, just need to setup a watch for yarn. |
| 21:29 |
|
tuxayo |
Otherwise the default workflow is just to code, run `yarn build` or `yarn build --view opac` and refresh the browser. |
| 21:31 |
|
wajasu |
yes! if we add those yarn statements to a docker watch section action, it would happen on each change i think. |
| 21:33 |
|
tuxayo |
Why is there a need to have the code not mounted on a volume? Because that's what require rebuilding image or a least a light version and there restarting it. |
| 21:33 |
|
tuxayo |
* light version of a rebuild and there |
| 21:33 |
|
tuxayo |
*and then |
| 21:34 |
|
tuxayo |
Now having the code mounted from the host to the container implies a much heavier process to manage code changes. |
| 21:35 |
|
tuxayo |
*Not having |
| 21:35 |
|
tuxayo |
Sorry for the typos >_< |
| 21:35 |
|
wajasu |
i originally was trying to run on podman with a rootless container. no root. but the chown -R that happens in a docker volume whether directly or indirectly runs for way over 40 mins if it ever completes. |
| 21:36 |
|
wajasu |
so i move to my docker machine, an found out about docker compose watch. |
| 21:36 |
|
wajasu |
podman doesn't have watch, i think. |
| 21:37 |
|
tuxayo |
> i originally was trying to run on podman |
| 21:37 |
|
tuxayo |
cool |
| 21:37 |
|
tuxayo |
If it's a drop-in replacement for Docker, isn't that a bug with the volumes? |
| 21:37 |
|
wajasu |
docker doesn't do rootless |
| 21:38 |
|
wajasu |
with this setup, today, i actually ran the ktd rootless (no root) |
| 21:38 |
|
wajasu |
there were two places in the files/run.sh that complained about not having root |
| 21:39 |
|
wajasu |
hostname kohadevbox was one |
| 21:39 |
|
wajasu |
apachectl enable was another |
| 21:39 |
|
tuxayo |
Because it seems it really messes things up to have the volumes not working and having to circumvent that case of volumes being really really fitting for that. (code that one changes directly) |
| 21:39 |
|
tuxayo |
> i actually ran the ktd rootless (no root) |
| 21:39 |
|
tuxayo |
nice |
| 21:39 |
|
wajasu |
but maybe we could configure the koha dev user with sudo and allow those |
| 21:40 |
|
tuxayo |
Isn't there still a bug that prevents you from using volumes? That doesn't look like something inherently incompatible with a rootless setup |
| 21:40 |
|
tuxayo |
worst case, a network mount |
| 21:41 |
|
tuxayo |
to circumvent the volume issue |
| 21:41 |
|
wajasu |
the curent volume is a bind mount. |
| 21:41 |
|
wajasu |
i am goinf to try a standalone or compose created mount |
| 21:41 |
|
wajasu |
just for the koha git repo |
| 21:42 |
|
wajasu |
then we could configure the koha_db to mount it and populate itself upon restart. |
| 21:43 |
|
wajasu |
if that was useful |
| 21:43 |
|
tuxayo |
> the curent volume is a bind mount. |
| 21:43 |
|
tuxayo |
Isn't it this? ↓↓ |
| 21:43 |
|
tuxayo |
https://gitlab.com/koha-commun[…]ef_type=heads#L36 |
| 21:44 |
|
tuxayo |
> i am goinf to try a standalone or compose created mount |
| 21:44 |
|
tuxayo |
hope it works :) |
| 21:45 |
|
tuxayo |
> then we could configure the koha_db to mount it and populate itself upon restart. |
| 21:45 |
|
tuxayo |
Maybe there is interesting stuff for you in the place that use the USE_EXISTING_DB_FLAG , that's the current way to do persistence. |
| 21:45 |
|
tuxayo |
*places |
| 21:47 |
|
wajasu |
I think that SYNC_REPO is the colocated koha directory, i may be wrong |
| 21:48 |
|
tuxayo |
> use the USE_EXISTING_DB_FLAG |
| 21:48 |
|
tuxayo |
wait, that's the new thing introduced my the pending merge request for persistence, so you would find it in main/master |
| 21:49 |
|
tuxayo |
> I think that SYNC_REPO is the colocated koha directory, i may be wrong |
| 21:49 |
|
tuxayo |
Yes, so it's a volume, not a bind mount, I don't know if that changes something with rootless usage. |
| 21:50 |
|
wajasu |
i was debugging the files/run.sh and since i was trying to run it multiple times, not just on container rebuilds, the function append_if_absent or such might have been broke. so i fixed it. so I dont get multiple apache listner lines, and /etc/hosts entries |
| 21:51 |
|
wajasu |
i strted just doing docker build . with the dists/bullseye/Dockerfile, thenmoved on to the compose-light file. |
| 21:53 |
|
wajasu |
if the compose SYNC_REPO is a directory, its bind mount. if its a named_volume its a docker manaed volume. i believe |
| 21:53 |
|
wajasu |
i just got back into this stuff a couple of days ago. |
| 21:54 |
|
wajasu |
i was also experimenting with tmpfs volume, but gave up. i think i know what i did wrong. |
| 21:55 |
|
wajasu |
but if i could pull into memory based volumes, i might speed up things rebuilding. my new box has 64G RAM |
| 22:00 |
|
wajasu |
now i also copied the .env into /kohadevbox/dotenv in the Dockerfile and have this available if running scripts. |
| 22:01 |
|
wajasu |
i just tried switching USER 0 and USER 1000 in the Dockerfile so that files are chown'ed accordingly in the image. that kept me from doing that for the translations git pull. |
| 22:02 |
|
wajasu |
that recursive chown in the run.sh is slow if doing so in a docker volume for some reason. |
| 22:02 |
|
wajasu |
of course our koha git repo is huge. |
| 22:04 |
|
wajasu |
i was playing with bullseye-slim and it saved 100MB in the testing image. it just pulled more when the apt-get packages ran. |
| 22:05 |
|
wajasu |
perl:5.38.2-bullseye was an option to lock down the exact version of perl outside of the OS version. it might be work looking into. |
| 22:07 |
|
tuxayo |
> if the compose SYNC_REPO is a directory, its bind mount. if its a named_volume its a docker manaed volume. i believe |
| 22:07 |
|
tuxayo |
ah, ok, inspection confirmed it's a bind mount |
| 22:09 |
|
tuxayo |
> if its a named_volume its a docker manaed volume. |
| 22:09 |
|
tuxayo |
Would trying that way (named volumes) have more changed to get you to a seamless workflow that having rebuilt the images just to get code changes. |
| 22:10 |
|
tuxayo |
*chances |
| 22:11 |
|
wajasu |
having a named volume is more portable i think. it would get mounted the same way. |
| 22:11 |
|
tuxayo |
Still a bind mount? |
| 22:11 |
|
wajasu |
then we just git pull against it all day |
| 22:12 |
|
wajasu |
https://stackoverflow.com/ques[…]pe-bind-vs-volume |
| 22:13 |
|
wajasu |
the volume with grow over time, plus we can mount is and its sharable across containers. |
| 22:14 |
|
wajasu |
we could make the koha_db mount it readonly and execute the sql to populate. |
| 22:14 |
|
wajasu |
i might be able to stop/rm and rebuild the koha_db while keeping the other containers up. |
| 22:15 |
|
wajasu |
if the koha database reconnect logic is on place, we would be fine. |
| 22:15 |
|
wajasu |
then we could switch out postgres on the fly. |
| 22:16 |
|
tuxayo |
> https://blog.christophersmart.[…]-rootless-podman/ |
| 22:16 |
|
tuxayo |
thanks |
| 22:17 |
|
tuxayo |
oops, mixed the links |
| 22:17 |
|
wajasu |
i don;t know how the CD/CI pipelne is setup, but if ots using docker and we can share docker cache layers across different OS builds, it would be less resource intesnse |
| 22:17 |
|
tuxayo |
> https://stackoverflow.com/ques[…]pe-bind-vs-volume |
| 22:17 |
|
tuxayo |
thanks |
| 22:17 |
|
wajasu |
you will see there are tmpfs memory based volumes too. |
| 22:18 |
|
wajasu |
i wonder if i can build all in memory and speed up things |
| 22:18 |
|
wajasu |
of course git push to a repo. |
| 22:19 |
|
wajasu |
but if the CD/CI had lots of RAM it might prevent alot of DISK IO |
| 22:19 |
|
tuxayo |
> i wonder if i can build all in memory and speed up things |
| 22:19 |
|
tuxayo |
Maybe there is still hope to avoid having to rebuild a lot of stuff on code change: |
| 22:19 |
|
tuxayo |
https://blog.christophersmart.[…]-rootless-podman/ |
| 22:19 |
|
tuxayo |
https://www.tutorialworks.com/[…]rootless-volumes/ |
| 22:19 |
|
tuxayo |
I don't know enough about container to know if that fits your case. |
| 22:20 |
|
wajasu |
docker has "include" so docker fragments could be included and "export" so test coverage reports could be exported to disk |
| 22:21 |
|
wajasu |
the ktd scripts are awesome, and still the way to go for most folks. |
| 22:22 |
|
tuxayo |
But there must be a way for podman rootless users to have their app code being immediately available. I still can't believe one must rebuild the container for that. |
| 22:22 |
|
wajasu |
it would be nice to provide an koha cloud-int package, so folks could just run in linode or other cloud providers, |
| 22:23 |
|
wajasu |
podman containers are one thing. VM's are another. |
| 22:23 |
|
wajasu |
podman does run rootful as well. |
| 22:23 |
|
tuxayo |
Is that how everyone uses it? |
| 22:24 |
|
wajasu |
i can develop lightweight container services, running as a regular user in the container. |
| 22:24 |
|
tuxayo |
And that rootless is doomed no be able to just share code between host and container? |
| 22:25 |
|
wajasu |
when i need to map the the host, with a bind mount, the user inthe container is a different process id fromthe regular user on the host. |
| 22:25 |
|
wajasu |
so root can read/write anywhere so that works |
| 22:26 |
|
tuxayo |
If on the host, we chmod 777 the koha code, wouldn't it work? |
| 22:27 |
|
wajasu |
i hit the problem where the koha git repo bind mounted was user 101000 and the host was uid 1000. the ktd would try to write the po files or other files and have permission problems |
| 22:28 |
|
wajasu |
i could 777 the koha code, but podman is strict, the processes inthe container are not root(0) or the host user(1000), they get automatically assigned from a range. |
| 22:29 |
|
tuxayo |
> they get automatically assigned from a range |
| 22:29 |
|
tuxayo |
Wouldn't that still work with 777? |
| 22:31 |
|
wajasu |
so when the files/run.sh is run at koha_koha1 container start, it may try to write to the koha git repo ( a host uid, coud be root, cold be joe), but the uid in the container is kohadev(101000) and the host OS and linux kernel cgroups wont let you write to a shard filesystem. otherwise the the container would be able to write file as root, and that would be bad |
| 22:37 |
|
wajasu |
i used to buikld everything from scratch years ago, pulling allthe perl packages was a nightmare. i had a script do it, but slooow. |
| 22:37 |
|
tuxayo |
> and the host OS and linux kernel cgroups wont let you write to a shard filesystem |
| 22:37 |
|
tuxayo |
Ho really T_T |
| 22:37 |
|
wajasu |
thank goodnes for koha-common |
| 22:37 |
|
tuxayo |
^^ |
| 22:38 |
|
tuxayo |
> Ho really T_T |
| 22:38 |
|
tuxayo |
I though at worse having a umask at 000 would do with the issue for when the container needs to create files in the koha code. |
| 22:38 |
|
wajasu |
i think if we can get koha to run in a container rootless, more people will develop with it in the future. |
| 22:38 |
|
wajasu |
plus with my podman, we kould make kubernetes pods. :) |
| 22:39 |
|
tuxayo |
> i think if we can get koha to run in a container rootless, more people will develop with it in the future. |
| 22:39 |
|
tuxayo |
What barrier to development would that remove? |
| 22:39 |
|
wajasu |
it would be nice if ktd had two databases supporting two branches. centerville, and fairfield, then we could debug interlibrary loans and such |
| 22:40 |
|
wajasu |
just like having elasticsearch, we could startup the "other branch" |
| 22:40 |
|
tuxayo |
If any change to code needs an image rebuild **that** will be a barrier to developement. |
| 22:40 |
|
tuxayo |
On my 2012 PC, restart_all takes 30 seconds, I can't imagine if I have to rebuild images. 💀 |
| 22:41 |
|
wajasu |
tcohen or whoever would rebuild the koha testing image |
| 22:42 |
|
wajasu |
but i would expect could only be done monthly, for testers. |
| 22:44 |
|
wajasu |
i didn't mention that if we put a git pull/fetch in the run.sh, just the new changes will be pulled into the container volume. |
| 22:44 |
|
wajasu |
for the koha repo |
| 22:45 |
|
tuxayo |
> it would be nice if ktd had two databases supporting two branches. centerville, and fairfield, then we could debug interlibrary loans and such... (full message at <https://matrix.org/_matrix/med[…]tJUROpUMCtLUWIjyh>) |
| 22:45 |
|
tuxayo |
- make it easy to run two KTD side by side |
| 22:45 |
|
tuxayo |
- make it easy in KTD to spawn a second koha in the same container (the debian commands already do most of the work to have two instance, then some container command to expose more port should do it) |
| 22:45 |
|
tuxayo |
> or whoever would rebuild the koha testing image |
| 22:45 |
|
tuxayo |
rebuild the image for what? |
| 22:45 |
|
wajasu |
so if the image is built in january, then you refresh that, and the daily/hourly git pulls will be minimal |
| 22:46 |
|
wajasu |
the koha-testing image that is currently mentioned in docker-compose-light.yml |
| 22:47 |
|
wajasu |
i guess its being pulled from docker.io or some koha-community resource |
| 22:47 |
|
tuxayo |
> i didn't mention that if we put a git pull/fetch in the run.sh, just the new changes will be pulled into the container volume. |
| 22:47 |
|
tuxayo |
> for the koha repo |
| 22:47 |
|
tuxayo |
You mean having the whole Koha code in the image? 😱 |
| 22:48 |
|
wajasu |
yes. i have the whole koha code in the image now in my setup |
| 22:48 |
|
tuxayo |
How would local branches persist if the Koha code on the host? |
| 22:49 |
|
tuxayo |
*is not on the host |
| 22:49 |
|
wajasu |
the docker compose can rebuild a local koha git repo. but either way the image has the 4G koha repo in the volume. the translations inthe image too (2.7G) |
| 22:49 |
|
tuxayo |
> yes. i have the whole koha code in the image now in my setup |
| 22:49 |
|
tuxayo |
not really because it's copied from your host , it depend on having it on the host the rebuilding it |
| 22:50 |
|
wajasu |
yes, i have the option to rebuild locally, so i can work on it. |
| 22:51 |
|
wajasu |
but if this gets into the mainline, the one that people pull down from docker.io or such would start out that way. |
| 22:51 |
|
wajasu |
then git pulls from master would be small |
| 22:52 |
|
tuxayo |
> the docker compose can rebuild a local koha git repo. but either way the image has the 4G koha repo in the volume. the translations inthe image too (2.7G) |
| 22:52 |
|
tuxayo |
Isn't that way too large to have full koha code and i18n git in the images? |
| 22:52 |
|
tuxayo |
So better having none of that in the images. |
| 22:52 |
|
wajasu |
my koha testing image is 9G |
| 22:53 |
|
wajasu |
either you pull it down once, and docker caches it. OR youe keep pulling the 2.7G translation files on every ktd down/up cycle. |
| 22:54 |
|
wajasu |
right now the the run.sh will pull just the changes for the po files. |
| 22:54 |
|
tuxayo |
Which is fine |
| 22:54 |
|
wajasu |
but you just mentioned earlier that there are many po changes |
| 22:55 |
|
wajasu |
maybe its the nature of the web translation software. |
| 22:55 |
|
wajasu |
i assumed that an occasional file hear or there. |
| 22:56 |
|
tuxayo |
At worse, let's make disableable the auto update of the .po files for people that start ktd a lot and prefers to manually pull the i18n from time to time |
| 22:56 |
|
wajasu |
but there is a SKIP_L10N or such that could be turned on to skip that now. I was going to get that into the run.sh container environment |
| 22:57 |
|
wajasu |
but we might need to put a commandline ENV parameter withthe ktd up to skip pulling down. |
| 22:57 |
|
tuxayo |
> my koha testing image is 9G |
| 22:57 |
|
tuxayo |
I think that will literally take me 20 min to update the images with that size. And when trying stuff between multiple Debian or ubuntu version to troubleshoot a bug or try something on 23.11, 23.05, 22.11 etc, that will be big hiderance |
| 22:58 |
|
wajasu |
we don't want that. |
| 22:59 |
|
wajasu |
i suppose the cached 23.11, ... versions would get cached in the docker cache and subsequent builds when you switch would be quick. |
| 23:00 |
|
tuxayo |
> maybe its the nature of the web translation software. |
| 23:00 |
|
tuxayo |
At least that's of approach of weblate, make on commit for each translation session of a given contributor |
| 23:00 |
|
tuxayo |
To ensure attribution and traceability I think. |
| 23:01 |
|
wajasu |
i'm going to explore the resources needed to sustain all those builds versions. |
| 23:01 |
|
wajasu |
if you guys are using jenkins, and its building what we see in the dashboard, that wold be a goog test. |
| 23:02 |
|
wajasu |
when the make commit happens, does it rebuild all the po's or just the ones that changed? |
| 23:02 |
|
tuxayo |
> but there is a SKIP_L10N or such that could be turned on to skip that now. I was going to get that into the run.sh container environment |
| 23:02 |
|
tuxayo |
Great, so no need to include l10n or koha code in the image, then? |
| 23:02 |
|
wajasu |
i haven't used the weblate, so ... |
| 23:02 |
|
tuxayo |
> but we might need to put a commandline ENV parameter withthe ktd up to skip pulling down. |
| 23:02 |
|
tuxayo |
Isn't that SKIP_L10N ? .env and ENV are shared |
| 23:03 |
|
wajasu |
the compose file uses the .env |
| 23:03 |
|
tuxayo |
and ENV |
| 23:03 |
|
wajasu |
the Dockerfile doesn't |
| 23:03 |
|
wajasu |
i believe |
| 23:03 |
|
tuxayo |
ah maybe |
| 23:04 |
|
wajasu |
i put a COPY of the .env in the Dockerfile at /kohadevbox/dotenv and then is sourced it at the top of the run.sh |
| 23:05 |
|
wajasu |
i was going to put the skip var intheh .envbut now i realize that the run.sh wont get .env changes |
| 23:05 |
|
tuxayo |
> i suppose the cached 23.11, ... versions would get cached in the docker cache and subsequent builds when you switch would be quick. |
| 23:05 |
|
tuxayo |
There still would be a need to download, right? Even if not building locally. If I would need to build a 9G that more than 20 min I would need T_T |
| 23:05 |
|
wajasu |
so the Dockerfile idea is no good |
| 23:06 |
|
wajasu |
i haven;t checked. |
| 23:06 |
|
wajasu |
i am about to change the base image. to say bullseye-slim, and see. |
| 23:07 |
|
wajasu |
will it reuse the latter cached layers, or? wait |
| 23:07 |
|
tuxayo |
almost sure not |
| 23:07 |
|
tuxayo |
Because the base OS is first, so it can't keep the rest IIUC |
| 23:09 |
|
tuxayo |
Unless some special stuff would be done. But we surely don't do that. At the time when every commit to master would rebuild an image, even if I had yesterday's images, all layers would be downloaded |
| 23:11 |
|
wajasu |
i believe you are right, but i have an idea |
| 23:11 |
|
tuxayo |
:D |
| 23:11 |
|
tuxayo |
> i'm going to explore the resources needed to sustain all those builds versions. |
| 23:11 |
|
tuxayo |
If you aim for having stuff uptreamed to ktd, then try emailing the koha-devel about your ideas. There are a few people that use a lot docker and might have good ideas to help you. |
| 23:13 |
|
wajasu |
if i put the koho repo and translations before the OS, then maybe we can replace the OS, and keep those big pulls, then after latter, git pull/fetch the new changes |
| 23:13 |
|
tuxayo |
And also podman chats and forums maybe, there are likely people that solved similar problems as you face without having even larger images or having to rebuild the image to have the changes deployed |
| 23:13 |
|
wajasu |
that way we resue the first two repo layers |
| 23:14 |
|
tuxayo |
Ah that might work |
| 23:14 |
|
tuxayo |
How do local branches work in such a setup? |
| 23:15 |
|
tuxayo |
If I have my branch with a WIP how does it fare with KTD having the repo in the image? |
| 23:16 |
|
wajasu |
ok. you are correct. the latter layers are not using any CACHED layers. |
| 23:16 |
|
wajasu |
i expect the git brnach change command would work fine. i haven't tried it yet. |
| 23:17 |
|
tuxayo |
Where would it be stored and persisted? |
| 23:17 |
|
tuxayo |
If there isn't anymore the repo on the host |
| 23:17 |
|
wajasu |
if i have docker compose watch, it would see the git branch change (i hope) and sync the files. now we would be working another bug or feature. |
| 23:18 |
|
tuxayo |
Where would the git repo for the code be? |
| 23:18 |
|
tuxayo |
Only in the container? |
| 23:19 |
|
wajasu |
the compose watch rule woould automatically rebuild the yarn stuff, and the translations would stay the same as they would be gitignored |
| 23:19 |
|
wajasu |
i haven't got there yet. |
| 23:19 |
|
wajasu |
my bullseye-slim is building. lets see if it works. |
| 23:20 |
|
wajasu |
after that i'll try moving the repos to the front of the Docker |
| 23:20 |
|
tuxayo |
ok, hopefully there isn't the need to download twice the koha code ^^ Once on the host and one in the images |
| 23:20 |
|
tuxayo |
> i think if we can get koha to run in a container rootless, more people will develop with it in the future. |
| 23:20 |
|
tuxayo |
What was the idea behind that again? I don't think I heard of that being an barrier to contribution. |
| 23:22 |
|
wajasu |
well, i am in archlinux, and i am not sure podman can be installed with docker (coexist) i just built this new dev box |
| 23:23 |
|
wajasu |
people run docker in podman containers, but i need speed. |
| 23:25 |
|
wajasu |
well, bullseye-slim worked. the STAFF client is up. |
| 23:25 |
|
wajasu |
now i will explore putting the repos in front. |
| 23:30 |
|
wajasu |
i figure tuxayo is koha's AI that I've been talking too |
| 23:31 |
|
tuxayo |
koha's AI ?? |
| 23:32 |
|
tuxayo |
> well, i am in archlinux, and i am not sure podman can be installed with docker (coexist) i just built this new dev box |
| 23:32 |
|
tuxayo |
There isn't a sign of that https://wiki.archlinux.org/title/Podman |
| 23:32 |
|
tuxayo |
(of that being an issue) |
| 23:33 |
|
tuxayo |
> well, bullseye-slim worked. the STAFF client is up. |
| 23:33 |
|
tuxayo |
nice |
| 23:33 |
|
wajasu |
great. i'll put docker in as well if possible. |
| 23:37 |
|
wajasu |
i need to take a break. before i try to change the layer order, or use multi stage. i need to read up. |
| 23:37 |
|
wajasu |
grabbing dinner. |
| 23:37 |
|
wajasu |
i might be back in tonight, but maybe not. |
| 23:38 |
|
tuxayo |
good luck with your wizardry ^^ |
