| *** eltricos <eltricos!~eltricos@ip68-8-181-201.sd.sd.cox.net> has quit IRC (Ping timeout: 240 seconds) | 01:20 | |
| *** eltricos <eltricos!~eltricos@ip68-8-181-201.sd.sd.cox.net> has joined #oe | 03:16 | |
| *** eltricos <eltricos!~eltricos@ip68-8-181-201.sd.sd.cox.net> has quit IRC (Ping timeout: 268 seconds) | 03:37 | |
| *** alimon <alimon!~alimon@fixed-189-203-87-105.totalplay.net> has quit IRC (Ping timeout: 252 seconds) | 04:17 | |
| *** alimon <alimon!~alimon@189.172.174.124> has joined #oe | 04:19 | |
| *** tgamblin_ <tgamblin_!~tgamblin@d24-150-219-207.home.cgocable.net> has joined #oe | 05:41 | |
| *** tgamblin_ <tgamblin_!~tgamblin@d24-150-219-207.home.cgocable.net> has quit IRC (Remote host closed the connection) | 05:41 | |
| *** tgamblin <tgamblin!~tgamblin@d24-150-219-207.home.cgocable.net> has quit IRC (Ping timeout: 276 seconds) | 05:42 | |
| *** tgamblin_ <tgamblin_!~tgamblin@d24-150-219-207.home.cgocable.net> has joined #oe | 05:42 | |
| *** tgamblin_ <tgamblin_!~tgamblin@d24-150-219-207.home.cgocable.net> has quit IRC (Ping timeout: 245 seconds) | 05:47 | |
| *** Saur_Home32 <Saur_Home32!~Saur_Home@94-137-113-31.customers.ownit.se> has quit IRC (Quit: Client closed) | 05:47 | |
| *** Saur_Home32 <Saur_Home32!~Saur_Home@94-137-113-31.customers.ownit.se> has joined #oe | 05:48 | |
| *** alperak <alperak!~alperak@176.88.106.122> has joined #oe | 06:29 | |
| *** alimon <alimon!~alimon@189.172.174.124> has quit IRC (Ping timeout: 255 seconds) | 06:36 | |
| *** rob_w <rob_w!~rob@2001:a61:6072:1f01:b0e8:841b:3ebc:e380> has joined #oe | 06:42 | |
| *** alimon <alimon!~alimon@189.172.202.196> has joined #oe | 06:49 | |
| *** alimon <alimon!~alimon@189.172.202.196> has quit IRC (Ping timeout: 252 seconds) | 07:07 | |
| *** linfax <linfax!~linfax@eumail.topcon.com> has joined #oe | 07:17 | |
| *** alimon <alimon!~alimon@fixed-189-203-87-105.totalplay.net> has joined #oe | 07:22 | |
| *** Saur_Home32 <Saur_Home32!~Saur_Home@94-137-113-31.customers.ownit.se> has quit IRC (Quit: Client closed) | 07:27 | |
| *** Saur_Home32 <Saur_Home32!~Saur_Home@94-137-113-31.customers.ownit.se> has joined #oe | 07:27 | |
| *** mckoan|away is now known as mckoan | 07:36 | |
| mckoan | good morning | 07:36 |
|---|---|---|
| *** tgamblin <tgamblin!~tgamblin@d24-150-219-207.home.cgocable.net> has joined #oe | 08:01 | |
| *** Kubu_work <Kubu_work!~kubu@2a01cb05945b7e009bdc688723a24f31.ipv6.abo.wanadoo.fr> has joined #oe | 08:03 | |
| *** mvlad <mvlad!~mvlad@2a02:2f08:ea03:5200:f1e9:3a6b:537f:88e> has joined #oe | 08:42 | |
| *** leon-anavi <leon-anavi!~Leon@46.55.231.62> has joined #oe | 08:50 | |
| *** radanter <radanter!~rad@host81-141-251-211.range81-141.btcentralplus.com> has joined #oe | 08:59 | |
| *** JaMa <JaMa!~martin@ip-109-238-218-228.aim-net.cz> has quit IRC (Ping timeout: 268 seconds) | 09:57 | |
| *** JaMa <JaMa!~martin@ip-109-238-218-228.aim-net.cz> has joined #oe | 10:08 | |
| *** Saur_Home32 <Saur_Home32!~Saur_Home@94-137-113-31.customers.ownit.se> has quit IRC (Quit: Client closed) | 10:50 | |
| *** Saur_Home32 <Saur_Home32!~Saur_Home@94-137-113-31.customers.ownit.se> has joined #oe | 10:50 | |
| *** alperak <alperak!~alperak@176.88.106.122> has quit IRC (Quit: Client closed) | 11:04 | |
| *** alperak <alperak!~alperak@176.88.106.122> has joined #oe | 11:04 | |
| *** Saur_Home32 <Saur_Home32!~Saur_Home@94-137-113-31.customers.ownit.se> has quit IRC (Quit: Client closed) | 11:08 | |
| *** Saur_Home32 <Saur_Home32!~Saur_Home@94-137-113-31.customers.ownit.se> has joined #oe | 11:08 | |
| *** tnovotny <tnovotny!~tnovotny@ip4-83-240-26-162.cust.nbox.cz> has joined #oe | 11:46 | |
| *** LocutusOfBorg <LocutusOfBorg!~locutusof@151.58.174.15> has quit IRC (Ping timeout: 252 seconds) | 12:10 | |
| mort | are there any tricks for making the do_rootfs step faster? It ends up being by far the longest time when doing smaller incremental builds | 12:45 |
| mort | I assume most of the time goes to waiting for the package manager (opkg in my case) to decompress some gzip stream, if that could've been done with pigz instead of on a single core that would've done a lot I think | 12:46 |
| rburton | mort: opkg uses xz | 13:33 |
| rburton | i'd profile it, you never know what is actually taking the time | 13:34 |
| mort | TIL, but the same applies there with pixz rather than single core xz | 13:34 |
| rburton | sure | 13:34 |
| rburton | maybe opkg is single-thread extracting, maybe something else is taking forever | 13:34 |
| rburton | perf record a rootfs run and see :) | 13:34 |
| mort | hmm maybe I will | 13:34 |
| rburton | you might discover that its spending two thirds of the time running a job in qemu-user | 13:35 |
| mort | I did see a bunch of qemu in htop too yeah | 13:35 |
| rburton | i relatively recently fixed up a postinst that was running tens of times to run once | 13:35 |
| mort | what I really want, I think, is a multi core package manager | 13:35 |
| rburton | https://git.openembedded.org/openembedded-core/commit/?h=fbd8a57aa307bfda70a08cb78af3c97f05c39a3a fwiw | 13:36 |
| mort | nice work | 13:36 |
| rburton | mort: the package manager does almost nothing. you want fast unpacking which should be trivial, and fast I/O | 13:36 |
| mort | these images don't have man pages so I don't think it affects me but it would affect a lot of other people I assume | 13:36 |
| mort | it's the unpacking part of the package manager I want to be mulit core ;) | 13:37 |
| mort | and the postinstall script running | 13:37 |
| mort | I know all the good reasons why running postinstall scripts in parallel is dangerous but with enough tricks it should be possible to keep track of which filesystem paths the different scripts mess with and error if they mess with the same ones | 13:38 |
| mort | hey with a COW filesystem you could let all the scripts run in parallel against their own little COW world and then you have a merge and conflict detection operation at the end | 13:41 |
| *** alperak <alperak!~alperak@176.88.106.122> has quit IRC (Quit: Client closed) | 13:41 | |
| *** LocutusOfBorg <LocutusOfBorg!~locutusof@151.58.174.15> has joined #oe | 13:41 | |
| *** alperak <alperak!~alperak@176.88.106.122> has joined #oe | 13:42 | |
| rburton | you could easily do unpack in parallel, it's just a matter of running down the list respecting dependencies | 13:43 |
| rburton | like bitbake does builds in parallel | 13:43 |
| mort | yeah | 13:43 |
| rburton | patches welcome :) | 13:43 |
| mort | the problem is, from what I understand, bitbake just runs a giant `dpkg/opkg/rpm install <giant list of packages>` command to build the rootfs | 13:44 |
| rburton | but postinsts are effectively very fast, population is just writing files so IO bound. its the unpack which is slow. | 13:44 |
| rburton | yes, but those package managers know the dependency trees | 13:45 |
| rburton | talking about performance without benchmarking is pointless. we might accidentally be running ldconfig 10k times. :) | 13:45 |
| mort | which means the patch would have to be against those package managers, not against bitbake, and a giant hack to run unpacking in parallel might make sense for OE but not for debian | 13:45 |
| mort | certainly | 13:46 |
| rburton | i'm sure any of the package managers would take a patch to unpack in parallel if it meant install was many times faster | 13:46 |
| rburton | I don't think oe would take a 'giant hack' against a package manager that wasn't upstreamable | 13:46 |
| mort | no, carrying a huge downstream patch against the package managers would be a terrible idea | 13:48 |
| mort | another thing, a fair amount of time is spent essentially number crunching in python code. Has attempts been made at using pypy instead of cpython? | 14:17 |
| rburton | not afaik | 14:25 |
| rburton | the worker spawn will need fixing because right now it hardcodes what binary to use | 14:25 |
| rburton | but i had a patch for that somewhere and its trivial | 14:25 |
| rburton | bitbake shouldn't be doing _that_ much work though | 14:26 |
| mort | adding a folder with a `python3` file which just executes pypy3 should work as a hack | 14:26 |
| mort | aw pypy3's os module doesn't have a sched_getaffinity() | 14:27 |
| rburton | if pypy was faster for an actual image build then i'd worry that the algorithms in bitbake are very suboptimal and there's a basic optimisation that needed to be done, as bitbake is mostly orchestrating other jobs | 14:30 |
| mort | well, it has a bunch of work to do right | 14:32 |
| mort | parsing a whole ton of .bb and .bbappend files, resolving absolutely gigantic dependency trees | 14:32 |
| mort | I often see python processes eating 100% cpu for some time | 14:32 |
| mort | of course most of the time is taken by actually compiling stuff | 14:34 |
| mort | end even the time taken by the python3 process might be in things like a call to sort() or other stuff that's implemented in C, as you say talking about performance without benchmarking is a bit pointless | 14:37 |
| *** Xagen <Xagen!~Xagen@99-135-179-142.lightspeed.austtx.sbcglobal.net> has joined #oe | 15:12 | |
| *** Saur_Home32 <Saur_Home32!~Saur_Home@94-137-113-31.customers.ownit.se> has quit IRC (Quit: Client closed) | 15:43 | |
| *** Saur_Home32 <Saur_Home32!~Saur_Home@94-137-113-31.customers.ownit.se> has joined #oe | 15:43 | |
| *** eltricos <eltricos!~eltricos@ip68-8-181-201.sd.sd.cox.net> has joined #oe | 15:53 | |
| *** sakoman <sakoman!~steve@dhcp-72-234-108-41.hawaiiantel.net> has quit IRC (Ping timeout: 276 seconds) | 15:56 | |
| *** sakoman <sakoman!~steve@dhcp-72-234-108-41.hawaiiantel.net> has joined #oe | 16:10 | |
| *** rob_w <rob_w!~rob@2001:a61:6072:1f01:b0e8:841b:3ebc:e380> has quit IRC (Read error: Connection reset by peer) | 16:13 | |
| *** Kubu_work <Kubu_work!~kubu@2a01cb05945b7e009bdc688723a24f31.ipv6.abo.wanadoo.fr> has quit IRC (Quit: Leaving.) | 16:14 | |
| *** tnovotny <tnovotny!~tnovotny@ip4-83-240-26-162.cust.nbox.cz> has quit IRC (Quit: Leaving) | 16:16 | |
| *** zhmylove <zhmylove!~zhmylove@80.254.50.127> has joined #oe | 16:43 | |
| zhmylove | Hey Gods! Does anyone know what did change in fetcher since Nanbield? "Checking sstate mirror" on remote cache is now taking way much time than in Mickledore | 16:44 |
| rburton | that would be hashserv | 16:50 |
| zhmylove | rburton: you mean that old version of bb-hashserv slows down this process, right? We likely did not update it on our servers | 17:05 |
| *** sakoman <sakoman!~steve@dhcp-72-234-108-41.hawaiiantel.net> has quit IRC (Ping timeout: 276 seconds) | 17:08 | |
| *** mckoan is now known as mckoan|away | 17:19 | |
| zhmylove | (I mean that we use the same one with mickledore & nanbield) | 17:19 |
| *** sakoman <sakoman!~steve@dhcp-72-234-108-41.hawaiiantel.net> has joined #oe | 17:22 | |
| *** alperak <alperak!~alperak@176.88.106.122> has quit IRC (Quit: Client closed) | 17:25 | |
| *** leon-anavi <leon-anavi!~Leon@46.55.231.62> has quit IRC (Quit: Leaving) | 17:38 | |
| *** linfax <linfax!~linfax@eumail.topcon.com> has quit IRC (Ping timeout: 256 seconds) | 17:43 | |
| *** sakoman <sakoman!~steve@dhcp-72-234-108-41.hawaiiantel.net> has quit IRC (Ping timeout: 276 seconds) | 17:47 | |
| *** Saur_Home32 <Saur_Home32!~Saur_Home@94-137-113-31.customers.ownit.se> has quit IRC (Quit: Client closed) | 17:53 | |
| *** Saur_Home32 <Saur_Home32!~Saur_Home@94-137-113-31.customers.ownit.se> has joined #oe | 17:53 | |
| *** sakoman <sakoman!~steve@dhcp-72-234-108-41.hawaiiantel.net> has joined #oe | 18:01 | |
| khem | mort: fwiw in yoe we use zstd as default compression with opkg see https://github.com/YoeDistro/yoe-distro/blob/master/sources/meta-yoe/conf/distro/yoe.inc#L183 | 18:11 |
| khem | this helps quite a bit with rootfs time | 18:11 |
| *** radanter <radanter!~rad@host81-141-251-211.range81-141.btcentralplus.com> has quit IRC (Remote host closed the connection) | 19:37 | |
| *** zhmylove <zhmylove!~zhmylove@80.254.50.127> has quit IRC (Remote host closed the connection) | 19:47 | |
| rburton | khem: i'd lean towards speed over raw compression power as the default tbh | 20:17 |
| *** Leatherface <Leatherface!leatherfac@helvetet.eu> has quit IRC (Ping timeout: 256 seconds) | 20:31 | |
| *** Leatherface <Leatherface!leatherfac@helvetet.eu> has joined #oe | 20:41 | |
| khem | Yeah me too unless their is a genuine case to save space | 21:25 |
| *** mvlad <mvlad!~mvlad@2a02:2f08:ea03:5200:f1e9:3a6b:537f:88e> has quit IRC (Remote host closed the connection) | 22:35 | |
| *** eltricos <eltricos!~eltricos@ip68-8-181-201.sd.sd.cox.net> has quit IRC (Ping timeout: 245 seconds) | 22:38 | |
| *** eltricos <eltricos!~eltricos@ip68-8-181-201.sd.sd.cox.net> has joined #oe | 23:05 | |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!