| *** elruk_ <elruk_!~elruk@natasha2.familycom.org> has quit IRC | 00:18 | |
| *** learningc <learningc!~learningc@121.122.92.78> has quit IRC | 00:19 | |
| *** tgraydon <tgraydon!tgraydon@nat/intel/x-skpphsfafofltkvt> has quit IRC | 00:22 | |
| *** fatalhalt <fatalhalt!~fatalhalt@c-67-163-60-93.hsd1.il.comcast.net> has joined #yocto | 00:31 | |
| *** FailDev <FailDev!18d83107@24.216.49.7> has quit IRC | 00:36 | |
| *** learningc <learningc!~learningc@mti-37-145.tm.net.my> has joined #yocto | 00:36 | |
| *** learningc <learningc!~learningc@mti-37-145.tm.net.my> has quit IRC | 00:43 | |
| *** learningc <learningc!~learningc@mti-37-145.tm.net.my> has joined #yocto | 00:44 | |
| *** mrpelotazo <mrpelotazo!~mrpelotaz@HSI-KBW-085-216-027-117.hsi.kabelbw.de> has quit IRC | 01:31 | |
| *** mrpelotazo <mrpelotazo!~mrpelotaz@HSI-KBW-085-216-027-117.hsi.kabelbw.de> has joined #yocto | 01:33 | |
| *** chinhuat8 <chinhuat8!~chinhuat@192.198.146.173> has joined #yocto | 02:05 | |
| *** chinhuat <chinhuat!~chinhuat@192.198.146.173> has quit IRC | 02:06 | |
| *** georgem_home <georgem_home!uid210681@gateway/web/irccloud.com/x-ubcroiigjwhpfzrj> has quit IRC | 02:26 | |
| *** armpit <armpit!~armpit@2601:202:4180:c33:7c98:5faa:262d:a3af> has quit IRC | 03:13 | |
| *** chinhuat8 is now known as chinhuat | 03:15 | |
| *** armpit <armpit!~armpit@2601:202:4180:c33:b816:6e4b:1727:7819> has joined #yocto | 03:24 | |
| *** fatalhalt <fatalhalt!~fatalhalt@c-67-163-60-93.hsd1.il.comcast.net> has quit IRC | 03:46 | |
| *** camus <camus!~Instantbi@183.128.238.14> has joined #yocto | 03:47 | |
| *** kaspter <kaspter!~Instantbi@183.128.238.14> has quit IRC | 03:47 | |
| *** camus is now known as kaspter | 03:47 | |
| *** AndersD <AndersD!~AndersD@h-98-128-162-82.NA.cust.bahnhof.se> has joined #yocto | 05:18 | |
| *** milloni <milloni!~milloni@preemptable.org> has quit IRC | 05:52 | |
| *** milloni <milloni!~milloni@preemptable.org> has joined #yocto | 05:52 | |
| *** AndersD <AndersD!~AndersD@h-98-128-162-82.NA.cust.bahnhof.se> has quit IRC | 05:54 | |
| *** jeanba <jeanba!~jbl@77.243.63.34> has joined #yocto | 06:10 | |
| *** TobSnyder <TobSnyder!~schneider@ip5f5aa32f.dynamic.kabel-deutschland.de> has joined #yocto | 06:22 | |
| *** andycooper <andycooper!uid246432@gateway/web/irccloud.com/x-qoqwxyfafyjpsjfh> has quit IRC | 06:31 | |
| *** tprrt <tprrt!~tprrt@upc31-1-78-208-110-13.fbx.proxad.net> has joined #yocto | 06:41 | |
| *** kroon <kroon!~kroon@213.185.29.22> has joined #yocto | 06:53 | |
| *** jeanba <jeanba!~jbl@77.243.63.34> has left #yocto | 06:54 | |
| *** agust <agust!~agust@p54833DBB.dip0.t-ipconnect.de> has joined #yocto | 06:55 | |
| *** florian <florian!~florian_k@Maemo/community/contributor/florian> has joined #yocto | 07:09 | |
| *** BobPungartnik <BobPungartnik!~BobPungar@187.113.136.3> has joined #yocto | 07:21 | |
| *** pung_ <pung_!~BobPungar@187.113.136.3> has quit IRC | 07:22 | |
| silviof | How you are testing your images without hardware? via `-c testimage`? There exists some other tools which are good to know to test images? | 07:32 |
|---|---|---|
| *** asabil <asabil!~asabil@81.167.213.26.static.lyse.net> has joined #yocto | 07:35 | |
| *** Bunio_FH <Bunio_FH!~bunio@81-18-201-214.static.chello.pl> has joined #yocto | 07:37 | |
| *** pung_ <pung_!~BobPungar@187.113.136.3> has joined #yocto | 07:46 | |
| *** BobPungartnik <BobPungartnik!~BobPungar@187.113.136.3> has quit IRC | 07:48 | |
| *** asabil <asabil!~asabil@81.167.213.26.static.lyse.net> has quit IRC | 07:52 | |
| *** asabil <asabil!~asabil@81.167.213.26.static.lyse.net> has joined #yocto | 07:53 | |
| *** T_UNIX <T_UNIX!uid218288@gateway/web/irccloud.com/x-yllotgaqgvjxwmhu> has joined #yocto | 07:55 | |
| *** leitao <leitao!~leitao@2620:10d:c092:200::1:73da> has joined #yocto | 07:58 | |
| *** leitao <leitao!~leitao@2620:10d:c092:200::1:73da> has quit IRC | 08:10 | |
| *** leitao <leitao!~leitao@2620:10d:c092:200::1:73da> has joined #yocto | 08:11 | |
| *** pung_ <pung_!~BobPungar@187.113.136.3> has quit IRC | 08:17 | |
| *** pung_ <pung_!~BobPungar@187.113.136.3> has joined #yocto | 08:17 | |
| JaMa | RP: and with latest bitbake a5e084a266f63c2fd370122327615e49beaeb94e http://paste.ubuntu.com/p/YhMCHPmg56/ | 08:26 |
| *** kaspter <kaspter!~Instantbi@183.128.238.14> has quit IRC | 08:35 | |
| *** kaspter <kaspter!~Instantbi@183.128.238.14> has joined #yocto | 08:36 | |
| *** bluelightning_ <bluelightning_!~paul@pdpc/supporter/professional/bluelightning> has joined #yocto | 08:48 | |
| RP | JaMa: thanks | 08:51 |
| *** bluelightning <bluelightning!~paul@pdpc/supporter/professional/bluelightning> has quit IRC | 08:53 | |
| JaMa | RP: now I'm running it with even older 18d4a31fdcec1f0e5d2199d6142f0ce833fca1a7 | 08:54 |
| RP | JaMa: http://paste.ubuntu.com/p/RMGbm9DyZz/ is quite interesting, the sorted() shows up badly | 08:54 |
| RP | and pformat | 08:55 |
| JaMa | 18d4a31fdcec1f0e5d2199d6142f0ce833fca1a7 seems to be fine, it's already executing tasks | 08:56 |
| RP | JaMa: just to confirm, http://paste.ubuntu.com/p/RMGbm9DyZz/ was without the logger.debug removal? | 08:57 |
| RP | I think that sorted comes from part of pformat, going to need to be careful with that | 08:57 |
| JaMa | yes, http://paste.ubuntu.com/p/RMGbm9DyZz/ was 1f630fdf0260db08541d3ca9f25f852931c19905 without any modifications, it was supposed to be the "good old" baseline based on the times with old image, but it seems that this revision already regressed badly | 09:00 |
| JaMa | so it's something in 6 commits between 18d4a31 and 1f630fdf | 09:02 |
| JaMa | the later has the line logger.debug(2, "Holding off tasks %s" % pprint.pformat(self.holdoff_tasks)) | 09:04 |
| *** cquast <cquast!~cquast@90.85.130.193> has joined #yocto | 09:08 | |
| RP | JaMa: right, that line in particular performs terribly! :) | 09:10 |
| *** bluelightning_ <bluelightning_!~paul@pdpc/supporter/professional/bluelightning> has quit IRC | 09:10 | |
| *** BCMM <BCMM!~BCMM@unaffiliated/bcmm> has joined #yocto | 09:15 | |
| JaMa | RP: yes, I just thought that this was only in master-next at that time, not in master, will send an e-mail once last run with even older bitbake revision is uploaded to make it a bit clearer what is what | 09:20 |
| JaMa | RP: latest world build last night with latest bitbake shown some unexpected failures with missing native deps when building e.g. some node recipes, but maybe it wasn't caused by bitbake, will re-test | 09:21 |
| RP | JaMa: I'm not aware of deps going missing like that but there were various fixes made, this is partly why I've merged everything to master, so we all have a common test target with the known issues fixed | 09:25 |
| *** learningc <learningc!~learningc@mti-37-145.tm.net.my> has quit IRC | 09:27 | |
| *** learningc <learningc!~learningc@mti-37-145.tm.net.my> has joined #yocto | 09:27 | |
| *** learningc <learningc!~learningc@mti-37-145.tm.net.my> has quit IRC | 09:29 | |
| *** bluelightning_ <bluelightning_!~paul@pdpc/supporter/professional/bluelightning> has joined #yocto | 09:47 | |
| *** goliath <goliath!~goliath@clnet-p04-043.ikbnet.co.at> has joined #yocto | 09:49 | |
| mcfrisk | hmm, python3 doesn't seem to build without readline (GPLv3) due to do_install check. Any hints how to make the test compatible with non-default PACKAGECONFIG which removes readline support? | 09:59 |
| *** asabil <asabil!~asabil@81.167.213.26.static.lyse.net> has quit IRC | 10:15 | |
| *** asabil <asabil!~asabil@81.167.213.26.static.lyse.net> has joined #yocto | 10:16 | |
| *** pung_ <pung_!~BobPungar@187.113.136.3> has quit IRC | 10:22 | |
| *** kaspter <kaspter!~Instantbi@183.128.238.14> has quit IRC | 10:25 | |
| *** kaspter <kaspter!~Instantbi@183.128.238.14> has joined #yocto | 10:27 | |
| *** BobPungartnik <BobPungartnik!~BobPungar@187.113.136.3> has joined #yocto | 10:27 | |
| *** kaspter <kaspter!~Instantbi@183.128.238.14> has quit IRC | 10:34 | |
| mcfrisk | seems I really need to patch python3 recipe to remove check_build_completeness.py if I build without readline support | 10:48 |
| Saur | RP: I've sent a change to the bitbake list that adds a progress meter for the part between "Initialising tasks" and "Checking sstate mirror object availability" where runqueue currently hangs for a while without any output. | 10:51 |
| Saur | RP: So unless you can't make the delay introduced by that part of the code go away, this will at least make it obvious that something is happening. | 10:52 |
| *** kaspter <kaspter!~Instantbi@115.230.121.109> has joined #yocto | 10:53 | |
| *** wooosaiii <wooosaiii!~prix@89-212-21-243.static.t-2.net> has quit IRC | 10:56 | |
| *** erbo <erbo!~erik@linode.unixshell.se> has quit IRC | 11:02 | |
| *** BobPungartnik <BobPungartnik!~BobPungar@187.113.136.3> has quit IRC | 11:06 | |
| *** BobPungartnik <BobPungartnik!~BobPungar@187.113.136.3> has joined #yocto | 11:06 | |
| *** BCMM <BCMM!~BCMM@unaffiliated/bcmm> has quit IRC | 11:32 | |
| RP | Saur: ok, but wrapping everything in more progress bars isn't really the answer... | 11:39 |
| Saur | RP: Well, if the delay is unavoidable then the progressbar is needed to show that something is happening. But if you can find ways to make the delay go away, then that is of course even better. ;) | 11:42 |
| RP | Saur: I just hate progress bars | 11:42 |
| Saur | RP: Well, I hate long periods of no output even more. | 11:43 |
| *** berton <berton!~berton@181.220.83.67> has joined #yocto | 11:46 | |
| Crofton|work | Saur, welcome to my life | 11:47 |
| Crofton|work | I made a fake progress bar once, always fun when underlying thing crashed and kept showing progress | 11:48 |
| Saur | Crofton|work: That sounds like the worst of both worlds... | 11:48 |
| mcfrisk | I follow progress with strace if things are taking too long. should have known about bitbake -P earlier and started debugging the sometimes long delays.. | 11:48 |
| RP | Saur: does the progress bar even increment properly with the way you've written that patch? | 11:53 |
| Saur | RP: Yes it does. | 11:53 |
| *** falstaff <falstaff!~quassel@37.17.234.113> has quit IRC | 11:53 | |
| RP | Saur: it looks to me like the increments are around the fast part of the code | 11:53 |
| Saur | RP: "Properly" in the sense that it updates once per task. | 11:54 |
| RP | Saur: so it doesn't get quickly to say 95% then sit there? | 11:54 |
| Saur | No. | 11:54 |
| RP | then there is something very wrong :( | 11:54 |
| Saur | RP: Well, one thing I noticed was that nothing is ever added to tocheck (with the current state I have: some updated recipes and stuff). | 11:55 |
| Saur | Because I first had the update at the end before I noticed the continue statements, and then the update event never fired. | 11:56 |
| RP | Saur: a clean build or an existing completed build? | 11:56 |
| Saur | RP: A completed build that then have had a couple of layer updates that have not built as I've been using bitbake -n. | 11:57 |
| RP | Saur: tocheck should be added to for things it needs to check the sstate cache for | 11:58 |
| Crofton|work | Saur, I was told to do it :) So I did | 12:00 |
| Saur | RP: Ok, it seems tocheck isn't completely empty. There are 24 entries in it for my current build. | 12:00 |
| RP | Saur: try it on an empty build and it will be much higher | 12:01 |
| Saur | RP: I guess those 24 updates to the progress were so few compared to the 5600 tasks in total that the loop checks that i never saw the progress meter move. | 12:01 |
| RP | Saur: we need to fix the underlying code not to suck, this should be fast | 12:02 |
| RP | This is why I wanted to get correctness before looking at performance | 12:03 |
| Saur | RP: I'll probably leave that part to you. ;) | 12:03 |
| RP | Now I'm trying to optimise code before its even working properly :( | 12:03 |
| Saur | RP: Now you at least know where one of the delays are... | 12:03 |
| Saur | RP: Btw, if I remove the tmp directory, that delay goes away. | 12:05 |
| RP | Saur: right, no stamps so that code does less | 12:05 |
| RP | Saur: http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222&id=9ab56f41b8785fb8e3ede91bbad9d094f4806e27 (top patch) might help | 12:11 |
| RP | Saur: if that doesn't make it fast enough there is another change we can try. Each one of these moves the goalposts on the other issues I'm trying to fix of course :/ | 12:14 |
| Saur | RP: That change took the time for that part from 36 to 25 seconds for me. | 12:21 |
| Saur | RP: And no, I do not expect you to focus on the performance at this time. As long as you are aware of the performance implications, make it work first. | 12:23 |
| Saur | RP: Hmm, I no longer seem to have push rights to poky-contrib. Have something changed regarding it? | 12:25 |
| Saur | (Haven't tried pushing to it for a couple of months so it is not necessarily recent.) | 12:26 |
| RP | Saur: ssh git@git.yoctoproject.org will show the perms | 12:26 |
| RP | Saur: don't think anything changed | 12:26 |
| Saur | git@git.yoctoproject.org: Permission denied (publickey). | 12:26 |
| RP | Saur: sounds like the key isn't right :/ | 12:27 |
| Saur | It does, doesn't it... | 12:27 |
| RP | Saur: you can try adding http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222&id=1b5162b94c0b61317e547a3f881f569350aac0fb on top, see if that brings the time down further | 12:32 |
| RP | Saur: less sure about what that last one might break | 12:33 |
| RP | selftest says its ok | 12:35 |
| Saur | RP: That brought the time down to 3 seconds, so yes if that still does what it is supposed to, then I guess that progress meter is no longer needed. :) | 12:37 |
| RP | Saur: it moves work around | 12:38 |
| Saur | RP: Well, the lost 22 seconds didn't show up anywhere else at least as the total time decreased correspondingly. :) | 12:40 |
| RP | Saur: Its more a question of whether anything accesses those data structures while the data is out of date | 12:42 |
| RP | Looking at the updated profile, those two changes look to optimise most of the recent changes out the profile data | 12:47 |
| *** kroon <kroon!~kroon@213.185.29.22> has quit IRC | 12:52 | |
| *** BCMM <BCMM!~BCMM@unaffiliated/bcmm> has joined #yocto | 12:56 | |
| *** radsquirrel <radsquirrel!~bradleyb@mail.fuzziesquirrel.com> has quit IRC | 13:04 | |
| *** radsquirrel <radsquirrel!~bradleyb@mail.fuzziesquirrel.com> has joined #yocto | 13:05 | |
| jwessel | RP: Here is the latest update. bitbake is now out of the picture. | 13:05 |
| jwessel | I duplicated the problem outside of bitbake. | 13:05 |
| jwessel | /glibc-locale/2.28-r0/locale-tree% ls -d usr/lib64/locale/*.ISO* |xargs rm -rf ; make -j128 | 13:06 |
| *** kroon <kroon!~kroon@213.185.29.22> has joined #yocto | 13:06 | |
| jwessel | What ever the cross-localedef is doing is not safe to run in a massively parallel context. | 13:06 |
| jwessel | Because it is moving files from a .tmp -> real and or hard link, stuff is disappearing that is actually needed. | 13:07 |
| jwessel | I can probably run it without pseudo at all to rule pseudo out. | 13:08 |
| jwessel | It will require quite a bit of further analysis, but at least we know conclusively all the bitbake/oe and such is out of the picture. | 13:08 |
| *** BobPungartnik <BobPungartnik!~BobPungar@187.113.136.3> has quit IRC | 13:12 | |
| zeddii | these 5.2 libc-headers are going to make me insane | 13:13 |
| * zeddii goes for coffee | 13:13 | |
| *** BobPungartnik <BobPungartnik!~BobPungar@187.113.136.3> has joined #yocto | 13:13 | |
| *** JPEW_ <JPEW_!~JPEW@2605:a601:21d1:5200:620a:c230:ada8:3135> has joined #yocto | 13:16 | |
| *** kaspter <kaspter!~Instantbi@115.230.121.109> has quit IRC | 13:27 | |
| *** kaspter <kaspter!~Instantbi@115.230.121.109> has joined #yocto | 13:28 | |
| zeddii | coffee worked. bluez5 is building again. | 13:28 |
| *** kroon <kroon!~kroon@213.185.29.22> has quit IRC | 13:30 | |
| *** Bagira <Bagira!Phanes@surro/founder/phanes> has quit IRC | 13:33 | |
| *** bluelightning_ <bluelightning_!~paul@pdpc/supporter/professional/bluelightning> has quit IRC | 13:35 | |
| *** aidanh <aidanh!~aidanh@unaffiliated/aidanh> has quit IRC | 13:35 | |
| *** aidanh <aidanh!~aidanh@unaffiliated/aidanh> has joined #yocto | 13:36 | |
| *** Phanes <Phanes!~Phanes@surro/founder/phanes> has joined #yocto | 13:36 | |
| *** litb <litb!~js@pd907fca9.dip0.t-ipconnect.de> has joined #yocto | 13:50 | |
| litb | hello folk | 13:50 |
| litb | is it OK to query the value of DEBUG_BUILD in a recipe, and pass some package Makefile specific flag depending on it? | 13:51 |
| litb | I.e DEBUG_BUILD ?= "0" MyDebugFlag = "${DEBUG_BUILD}" ?. In the docs, it'ss said that DEBUG_BUILD is used to specify that yocto should build packages with debug information | 13:51 |
| RP | jwessel: cool, always good to narrow down the issue! :) | 13:53 |
| jwessel | heh... I already found the offending code in the cross-localedef | 13:54 |
| jwessel | I don't exactly know how to fix it yet, but I see why it is clearly not parallel safe. | 13:55 |
| jwessel | The code is written to assume hard links might not work. | 13:55 |
| jwessel | And it will unlink a file, which could be in the process of being tested and used by one of the other parallel treads. | 13:55 |
| jwessel | If you were to run this serially with just a single cross-localedef there is no way to hit that problem. | 13:56 |
| jwessel | It replaces the file with a hard link when they test out to be the same. | 13:56 |
| fray | the original use of cross-localedef was serial... | 13:57 |
| jwessel | So if you go back to that, the problem is solved. | 13:57 |
| jwessel | Or change cross-localedef to not be stupid and allow --force-hard links and write out a .tmp file to perform the test. | 13:58 |
| *** Phanes <Phanes!~Phanes@surro/founder/phanes> has quit IRC | 13:58 | |
| RP | serial is very very slow so hacking localedef would be better | 13:58 |
| jwessel | Testing against a file that might go-away for a fraction of a second will not end well. | 13:58 |
| RP | jwessel: no, it explains alot | 13:58 |
| fray | ya, that's why it moved to parallel.. but ya, definitely a race to check and then act.. | 13:59 |
| jwessel | I have no reason to believe that pseudo is the issue here. | 13:59 |
| jwessel | I'd say at this point we have the root cause. | 13:59 |
| fray | ya.. I agree, sounds more like fix the issue first.. :) | 14:00 |
| *** Phanes <Phanes!Phanes@surro/founder/phanes> has joined #yocto | 14:00 | |
| fray | other systems BTW do the hard linking in two steps.. one step to generate all of the files, then a second to hash them and set them as parallel | 14:00 |
| jwessel | It was mighty interesting to have a log though in pseudo for a place in the code that should never happen. | 14:00 |
| jwessel | We might consider keeping that change. | 14:01 |
| fray | 'er.. set them as hardlinks I mean | 14:01 |
| jwessel | Because it tells you... "You really screwed up somewhere else..." | 14:01 |
| jwessel | fray: You might have to do just that. | 14:02 |
| fray | if I remember right (like 18 years ago right) that is what was being done.. then it was merged into a single program.. | 14:03 |
| fray | the origin of the cross-localedef is either MontaVista or CodeSourcery.. it's -old- | 14:03 |
| fray | Looks liek CodeSourcery | 14:05 |
| fray | + Cross-Compiling GLIBC | 14:05 |
| fray | + Jim Blandy <jimb@codesourcery.com> | 14:05 |
| jwessel | Perform generation, and then coalesce | 14:05 |
| fray | ya | 14:05 |
| jwessel | Does the generation even support skipping the combine step? | 14:06 |
| fray | I knew it was old though, far pre-YP | 14:06 |
| jwessel | Because there are other programs that can combine stuff. | 14:06 |
| fray | Ya, looking at Khem's repo.. 2006 or before.. :) | 14:06 |
| fray | I know the first version was before.. :) | 14:06 |
| fray | the original version of the code didnt do the hardlink stuff.. | 14:07 |
| fray | we (WR) paid CodeSourcery to do that back in 2007/2008 timeframe? | 14:07 |
| fray | (not saying they wouldn't have done it on their own or someone else didn't pay them as well...) | 14:07 |
| fray | I just remember having that discussion with them as a system optimization | 14:07 |
| fray | (probably was before 2007 now that I think of it.. probably closer to 2005/2006 | 14:09 |
| jwessel | /* Compare the file with the locale data files for the same category in | 14:09 |
| jwessel | other locales, and see if we can reuse it, to save disk space. */ | 14:09 |
| jwessel | other_paths = siblings (output_path); | 14:09 |
| jwessel | FYI that is what kills you in the parallel case. | 14:10 |
| jwessel | Those other ones may in the act of being processed. | 14:10 |
| *** khem <khem!~khem@unaffiliated/khem> has quit IRC | 14:10 | |
| fray | ya.. I think 'easiest' to implement fix is break into two steps.. generate (& hash?) and (hash &) hardlink (not sure where the hashing should be done, but it can be done in parallel) | 14:11 |
| jwessel | Which is 100% the reason it could be unlinked and replaced with a hard link while someone else is executing. | 14:11 |
| fray | so maybe three steps? | 14:11 |
| jwessel | You don't necessarily have to. | 14:11 |
| jwessel | But it would be the best way to do it. | 14:11 |
| jwessel | The problem even with the current code is that since all these are running in parallel you really won't get the right number of hard links ever. | 14:13 |
| fray | there is the '--no-hard-links' option in locale generation.. enable that and do the hardlinks later? | 14:15 |
| jwessel | I was advocating adding one. | 14:15 |
| fray | Ohh ok | 14:15 |
| jwessel | It won't work though due to the problem I just mentioned. | 14:15 |
| jwessel | If you are generating these all in parallel some will simply be generating at the same time. | 14:15 |
| fray | the no-hard-links commit mentions: | 14:16 |
| jwessel | You can't check the existence of something that isn't there yet. | 14:16 |
| fray | Add --no-hard-links option to localedef (bug 23923) | 14:16 |
| yocti | Bug https://bugzilla.yoctoproject.org/show_bug.cgi?id=23923 was not found. | 14:16 |
| fray | 14:16 | |
| fray | Downstream distributions need consistent sets of hardlinks in | 14:16 |
| fray | order for rpm to operate effectively. This means that even if | 14:16 |
| fray | locales are built with a high level of parallelism that the | 14:16 |
| fray | resulting files need to have consistent hardlink counts. The only | 14:16 |
| fray | way to achieve this is with a post-install hardlink pass using a | 14:16 |
| fray | program like 'hardlink' (shipped in Fedora). | 14:16 |
| jwessel | Correct. | 14:16 |
| fray | so it might even be helpful to break it in two for determinism sake | 14:16 |
| jwessel | But why bother doing the summing and checking in cross-localedef? | 14:17 |
| jwessel | You can do the hardlink pass outside of pseudo if you want. | 14:17 |
| jwessel | psuedo will be happy so long as the inode had created was preserved. | 14:17 |
| fray | Has to be done under control of pseudo or the DB will get messed up.. | 14:17 |
| jwessel | For at least one file. | 14:18 |
| jwessel | Not really. Not the way it works... Because I read the code :-) | 14:18 |
| fray | that wasn't always true.. at one point it was a combo of inode and filename | 14:18 |
| fray | I suspect fakeroot still works based on filename.. (at least it also used to) | 14:18 |
| jwessel | fakeroot yes. pseudo is fine. | 14:18 |
| jwessel | I don't know what the timepenalty is for the hardlink | 14:19 |
| fray | I'd still be cautious and do the hard linking under pseudo.. | 14:19 |
| fray | We used 'hardlink' in the past for something, and I remember it being fast.. but I don't completely remember how it worked.. | 14:19 |
| jwessel | locale-tree% hardlink -n usr/lib64/locale | 14:22 |
| jwessel | Mode: dry-run | 14:22 |
| jwessel | Files: 4919 | 14:22 |
| jwessel | Linked: 389 files | 14:22 |
| jwessel | So we missed 389 files in my sample run. | 14:22 |
| jwessel | Just to prove my point :-) | 14:22 |
| *** khem <khem!~khem@unaffiliated/khem> has joined #yocto | 14:22 | |
| jwessel | I'd say we just add a fast exit on the checks in localedef either via cmdline arg or environment variable. | 14:23 |
| fray | looks like hardlink might be part of util-linux now (or at least is on Fedora) | 14:23 |
| fray | that's what the --no-hard-links is supposed to do.. fast exit | 14:23 |
| fray | or am I missing something? | 14:23 |
| jwessel | lets look... I didn't see that option. | 14:24 |
| fray | commit 8cebd4ffe67bf94508809ea0caa02a4f1d52e8b1 | 14:24 |
| fray | (in glibc of course) | 14:24 |
| fray | + if (hard_links) | 14:24 |
| fray | + other_paths = siblings (output_path); | 14:24 |
| fray | otherwise it skips that whole section of code | 14:25 |
| jwessel | That is not in the version I have. | 14:25 |
| jwessel | That is what I was considering adding. | 14:25 |
| fray | Date: Mon Nov 26 09:51:51 2018 -0500 | 14:25 |
| fray | I'm looking at master | 14:25 |
| fray | glibc 2.30 | 14:25 |
| fray | commit 8cebd4ffe67bf94508809ea0caa02a4f1d52e8b1 | 14:25 |
| fray | Author: Carlos O'Donell <carlos@redhat.com> | 14:25 |
| fray | Date: Mon Nov 26 09:51:51 2018 -0500 | 14:25 |
| fray | Add --no-hard-links option to localedef (bug 23923) | 14:25 |
| yocti | Bug https://bugzilla.yoctoproject.org/show_bug.cgi?id=23923 was not found. | 14:25 |
| jwessel | I am on thud glibc-2.28 | 14:25 |
| jwessel | Which is why I don't have it. | 14:25 |
| fray | ya, too old.. you'd need to backport it.. but the commit is -simple- | 14:26 |
| fray | it's not YP bugzilla this is upstream glibc | 14:26 |
| jwessel | All you need to do, to make this problem go away is use that option + the hardlink program. | 14:26 |
| fray | look at the glibc bugzilla | 14:26 |
| fray | I just don't know if 'hardlink' is now standard or not | 14:26 |
| jwessel | Then your mystery localedef issue is fixed :-) | 14:26 |
| fray | (on non-Fedora/RH based systems) | 14:26 |
| jwessel | It is on ubuntu, if you install it. | 14:26 |
| fray | Ubuntu 18.04 does NOT have hardlink | 14:27 |
| jwessel | It will have to go into host-native. | 14:27 |
| fray | ok.. so it's optional there.. | 14:27 |
| jwessel | % cat /etc/issue | 14:27 |
| jwessel | Ubuntu 18.04.2 LTS \n \l | 14:27 |
| fray | ya.. sounds like a host-native... | 14:27 |
| jwessel | It is optional. | 14:27 |
| jwessel | I shudder to think how far we have to back port, but I assume we need to at least fix thud. | 14:27 |
| fray | Fedora claims it's part of util-linux, but I don't know if it's 'real' util-linux or Fedora's version | 14:27 |
| fray | yes.. we need to at least fix thud.. | 14:28 |
| fray | I'd say fix master first, make that work and backport to warrior and thud | 14:28 |
| fray | since we now know the problem is common | 14:28 |
| fray | (and at least master has a partial fix already implemented from upstream, warrior probably does?) | 14:28 |
| jwessel | The generation should be faster with the checking gutted too. | 14:28 |
| *** TobSnyder <TobSnyder!~schneider@ip5f5aa32f.dynamic.kabel-deutschland.de> has quit IRC | 14:29 | |
| jwessel | That code was _really_ not efficient, or parallel safe. | 14:29 |
| fray | so the 'hard part' is getting 'hardlink' (or finding something equivalent) | 14:30 |
| jwessel | http://jak-linux.org/projects/hardlink/ | 14:31 |
| jwessel | I would just use what works. | 14:31 |
| fray | If that works | 14:31 |
| fray | I was looking at the Fedora version.. and it went 'away' | 14:31 |
| jwessel | Ubuntu used 0.2.0 the 14.04 release. | 14:32 |
| fray | https://src.fedoraproject.org/rpms/hardlink/c/6f80279847fc132377e1110b57360358c2920b07?branch=master | 14:32 |
| jwessel | 0.3 is the latest. | 14:32 |
| RP | jwessel: nice to finally have some answers, nice work! | 14:32 |
| fray | Fedora's version was 1.3 | 14:32 |
| jwessel | That site states fedora has a different version. | 14:32 |
| jwessel | It is different code. | 14:33 |
| fray | removed in Feb of this year I think.. | 14:33 |
| fray | ok | 14:33 |
| jwessel | I suspect there are other programs that do the same thing. | 14:33 |
| jwessel | hardlink from the jak linux just works. | 14:33 |
| fray | https://pagure.io/hardlink/blob/master/f/hardlink.c | 14:34 |
| fray | that appears to be the latest version | 14:34 |
| jwessel | Makes no difference to me where we get it from. | 14:35 |
| jwessel | I am just happy we don't have to fix pseudo. | 14:35 |
| fray | I did just verify.. util-linux-2.34 includes 'hardlink' | 14:35 |
| *** yates <yates!~user@rrcs-96-10-234-158.midsouth.biz.rr.com> has joined #yocto | 14:36 | |
| fray | version included looks like it's the Red Hat version.. (or at least a refactored one) | 14:36 |
| yates | https://paste.fedoraproject.org/paste/rnsbsvEHQbH9iVrc0A-fOg | 14:36 |
| yates | this just started showing up when building this image. i have no idea why | 14:36 |
| yates | clues would be appreciated | 14:36 |
| yates | the image was building fine a week or two ago.. | 14:37 |
| fray | not sure.. pyro is pretty old so it might justbe a simple bug | 14:37 |
| fray | I'd simply start with rebuilding (clean and build) the locale's | 14:38 |
| yates | fray: are you talking to me? | 14:38 |
| fray | yes | 14:39 |
| RP | fray, jwessel: please keep in mind that adding a util-linux-native depends to glibc-locale would be pretty bad for build performance | 14:39 |
| RP | If there is a python script we can use it would be much more efficient | 14:40 |
| * RP finally figures out the one line fix bitbake needs :/ | 14:40 | |
| fray | I'm more thinking we add just 'hardlink' until a time comes when all of the regular hosts have updated to more modern util-linux | 14:40 |
| fray | hardlink itself is very small.. 430 something lines | 14:40 |
| RP | fray: right, fine by me | 14:41 |
| fray | but with that said, re-implementing those 430 lines in python should be possible, just don't know the performance impact | 14:41 |
| fray | this isn't rocket science.. it basically parses the filesystem and checksums everything into a linked list.. with the items inode, device, checksum and filename.. | 14:42 |
| fray | it searches that list for the device and checksum, and if they are the same then hardlinks | 14:42 |
| fray | if item is unknown it adds to the data structure and moves on to the next file | 14:42 |
| RP | fray: we already do similar kinds of things elsewhere, its trivial in python | 14:43 |
| fray | looks like the hardest thing in the code is there is something that if the file is in-use it renames it and removes it when it hardlinks to keep the old file alive.. | 14:44 |
| fray | that shouldn't be a problem | 14:44 |
| RP | http://git.yoctoproject.org/cgit.cgi/poky-contrib/patch/?id=6e19aa62ad8d0d1ded272690d21186d95743bb00 - so simple yet problematic | 14:44 |
| fray | RP is there exisitng oe code that does checksum and hardlink behavior or would this be a new lib/oe function? | 14:45 |
| fray | (I know we have copy or hardlink functions) | 14:46 |
| jwessel | Dude... Don't do it python. | 14:47 |
| jwessel | Use the C one. It is faster than python is ever going to be. | 14:48 |
| fray | but it' one more thing to compile and support that's my only concern | 14:48 |
| jwessel | How is it any better than having to compile cross-localedef | 14:49 |
| fray | and for the python it would run in the native bitbake context, not execute 'another thing'.. (conceptually) | 14:49 |
| jwessel | If anything add it in there. | 14:49 |
| fray | cause that is existing | 14:49 |
| fray | either we have to add a new native recipe for hardlink itself (which may someday go away) or we implement it as a library function | 14:49 |
| fray | either way will work -- but it moves around support items | 14:49 |
| RP | jwessel: I suspect there isn't that much difference C to python for this... | 14:50 |
| fray | builing native-util-linux is too big and has been avoided for a long time | 14:50 |
| jwessel | https://code.google.com/archive/p/hardlinkpy/ | 14:50 |
| jwessel | That is the code you are looking for btw. | 14:50 |
| RP | jwessel: I say this as someone who thought bitbake in C might be faster, once. | 14:51 |
| *** berton <berton!~berton@181.220.83.67> has quit IRC | 14:51 | |
| jwessel | I guess that older version is not python 3 compliant. | 14:53 |
| jwessel | It says to run 2to3 on it... | 14:53 |
| RP | JPEW: I had to drop the extra stats code, it was causing the autobuilder to lock up :( | 14:53 |
| fray | that python one looks like it doesn't hash once and store, but instead compares each time | 14:53 |
| fray | that's going to be less efficient then hash and store | 14:53 |
| RP | yes, but that is just bad design | 14:53 |
| RP | (not python per se) | 14:53 |
| jwessel | I'd be inclined to just add the c file to the cross-localedef package. | 14:54 |
| fray | the modern 'hardlink' have two hash points.. one based on size, owner, group, mode.. and then one based on content.. | 14:54 |
| jwessel | The c file does exactly what is needed, and you have to compile the cross-localedef anyway. | 14:55 |
| fray | since if the first hash doesn't match, you can skip the second (kind of)... | 14:55 |
| jwessel | Just name it cross-localedef-hardlink. | 14:55 |
| fray | then if you find a match in the first, you have to hash them both to compare -- but then youc an cache those for the future.. | 14:55 |
| jwessel | The modern one is much more effecient. | 14:55 |
| fray | yes.. adding it to cross-localdef is fine | 14:55 |
| fray | (least in my opinion) key is to be careful that it's efficient in what it's doing | 14:55 |
| jwessel | Take the path of least resistance, call it a day. | 14:55 |
| fray | I'd take the util-linux (modern) hardlink, and put it into cross-localdef for now.. it'd be the easiest to maintain.. (I think) | 14:56 |
| RP | right | 14:56 |
| jwessel | You'll spend time debugging and porting it to python otherwise. I am saying you can't make it portable etc... but why bother. The hardlink compiles on everything. | 14:57 |
| jwessel | oops. I meant you can make it portable with python, but I don't believe the effort is worth it. | 14:57 |
| RP | I was working on the assumption there is a sane python version around which there isn't | 14:57 |
| RP | if we can get a C version and use that, fine | 14:58 |
| jwessel | You might as well just stick it in cross-localedef. No new package etc... Don't know if the license is compatible. | 14:59 |
| jwessel | Didn't look at that bit. | 14:59 |
| fray | it's GPLv2 so should be fine.. | 14:59 |
| fray | (latest util-linux version) | 14:59 |
| fray | https://git.kernel.org/pub/scm/utils/util-linux/util-linux.git/tree/misc-utils/hardlink.1 | 15:00 |
| fray | https://git.kernel.org/pub/scm/utils/util-linux/util-linux.git/tree/misc-utils/hardlink.c | 15:00 |
| fray | 'er.. ya .c not .1 | 15:00 |
| fray | #include "c.h" | 15:01 |
| fray | #include "xalloc.h" | 15:01 |
| fray | #include "nls.h" | 15:01 |
| fray | #include "closestream.h" | 15:01 |
| fray | that bit might need to be ported or hacked aorund.. but otherwise it looks entirely standard | 15:01 |
| fray | (and I don't see any reason to build it with the PCRE support which is newer then the old standalone version) | 15:02 |
| JaMa | khem: I've sent one more RDEPENDS fix which might look like duplicate, but it's for another package in scsirastools | 15:02 |
| yates | removing tmp and rebuilding got rid of that | 15:02 |
| JaMa | khem: feel free to squash them if you want, It was detected in follow-up builds, that's the only reason why I haven't put both in first commit | 15:03 |
| *** Bunio_FH <Bunio_FH!~bunio@81-18-201-214.static.chello.pl> has quit IRC | 15:08 | |
| *** nabokov <nabokov!~armand@67.218.223.154> has joined #yocto | 15:11 | |
| *** JPEW_ <JPEW_!~JPEW@2605:a601:21d1:5200:620a:c230:ada8:3135> has quit IRC | 15:14 | |
| *** kroon <kroon!~kroon@37-247-29-68.customers.ownit.se> has joined #yocto | 15:15 | |
| *** asabil <asabil!~asabil@81.167.213.26.static.lyse.net> has quit IRC | 15:30 | |
| khem | JaMa:yes queued up | 15:30 |
| khem | JaMa: the gcc problem you are seeing, whats best way to reproduce it, does it only happen with multilib builds | 15:31 |
| *** asabil <asabil!~asabil@81.167.213.26.static.lyse.net> has joined #yocto | 15:32 | |
| khem | JaMa:https://errors.yoctoproject.org/Errors/Build/86953/ was the full build where these qa errors are listed | 15:33 |
| khem | hoping your series will fix most if not all of them | 15:33 |
| JaMa | khem: I've fixed all which I was seeing in my builds | 15:39 |
| khem | ok | 15:39 |
| khem | didn't you realize I was hinting you to fix mine too :) | 15:40 |
| JaMa | e.g. hplip shown in ours is blacklisted in all our builds, because needs lp group which we don't have | 15:40 |
| JaMa | wasn't alex going to fix them? :) | 15:40 |
| khem | I see | 15:40 |
| JaMa | and mime-construct burn-boot libextutils-parsexs-perl webmin recipes aren't in my world | 15:42 |
| JaMa | will check the gomp issue on some non-multilib config soon | 15:43 |
| *** kroon <kroon!~kroon@37-247-29-68.customers.ownit.se> has quit IRC | 15:57 | |
| khem | ok | 15:58 |
| *** kroon <kroon!~kroon@37-247-29-68.customers.ownit.se> has joined #yocto | 15:58 | |
| khem | I will report the results and see if Alex helps | 15:58 |
| *** moto-timo <moto-timo!~ttorling@fsf/member/moto-timo> has joined #yocto | 15:59 | |
| *** berton <berton!~berton@181.220.83.67> has joined #yocto | 16:01 | |
| *** WillMiles <WillMiles!~Will@static-209-87-231-80.storm.ca> has joined #yocto | 16:01 | |
| *** florian <florian!~florian_k@Maemo/community/contributor/florian> has quit IRC | 16:27 | |
| *** BobPungartnik <BobPungartnik!~BobPungar@187.113.136.3> has quit IRC | 16:30 | |
| *** BobPungartnik <BobPungartnik!~BobPungar@187.113.136.3> has joined #yocto | 16:30 | |
| JaMa | RP: maybe I'm reading the profile badly, but to me it looks like worker thread was waiting in select most of the time and then finished executing tasks relatively quickly | 16:31 |
| JaMa | e.g. latest builds shows 4082.736 seconds in main process, 3977.939 seconds in worker and | 16:31 |
| JaMa | 877733 3294.319 0.004 3294.319 0.004 {built-in method select.select} | 16:32 |
| JaMa | 58394 366.063 0.006 366.063 0.006 {built-in method posix.fork} | 16:32 |
| JaMa | 58394 6.290 0.000 493.595 0.008 bitbake/bin/bitbake-worker:431(handle_runtask) | 16:32 |
| JaMa | if it was handling tasks for 8 minutes from 68, then it seems reasonable | 16:33 |
| JaMa | khem: the omp.h issue is reproducible without multilib as well, should I try it without 9.2.0 upgrade or is it definitely the gcc-runtime change? (I belive it is) | 16:36 |
| *** litb <litb!~js@pd907fca9.dip0.t-ipconnect.de> has quit IRC | 16:42 | |
| RP | JaMa: you read that correctly, it does spend most of its time waiting | 16:43 |
| RP | JaMa: turnaround time for tasks is important too though | 16:44 |
| Crofton|work | Rewrite bitbake in go | 16:45 |
| JaMa | rewrite go in bitbake | 16:45 |
| *** asabil <asabil!~asabil@81.167.213.26.static.lyse.net> has quit IRC | 16:48 | |
| Crofton|work | even better | 16:48 |
| *** pung_ <pung_!~BobPungar@187.113.136.3> has joined #yocto | 16:52 | |
| *** BobPungartnik <BobPungartnik!~BobPungar@187.113.136.3> has quit IRC | 16:55 | |
| RP | we just add the new sentience module to bitbake and make it self hosting... | 16:57 |
| *** kroon <kroon!~kroon@37-247-29-68.customers.ownit.se> has quit IRC | 17:00 | |
| jwessel | The hardlink program takes 2.3 seconds in pseudo context, and .7 seconds outside of pseudo. | 17:01 |
| jwessel | That is still way faster than what it takes for the cross-localedef | 17:02 |
| RP | jwessel: that is a nice win! | 17:03 |
| *** vineela <vineela!~vtummala@134.134.139.77> has joined #yocto | 17:03 | |
| * RP -> away (most of the weekend to) | 17:04 | |
| *** pung_ <pung_!~BobPungar@187.113.136.3> has quit IRC | 17:06 | |
| *** pung_ <pung_!~BobPungar@187.113.136.3> has joined #yocto | 17:06 | |
| Crofton|work | Would more machine learning help? | 17:06 |
| *** falstaff <falstaff!~quassel@37.17.234.113> has joined #yocto | 17:12 | |
| jwessel | I wonder what the bugzilla id is for the glibc-locale issue. | 17:17 |
| JaMa | jwessel: https://bugzilla.yoctoproject.org/show_bug.cgi?id=12434#c48 | 17:17 |
| yocti | Bug 12434: normal, Medium+, 2.8 M3, randy.macleod, ACCEPTED , pseudo: Incorrect UID/GID in packaged files | 17:17 |
| jwessel | Thanks. | 17:17 |
| jwessel | I am testing a fix for it. | 17:18 |
| jwessel | I'll send it out as an RFC, since it is kind of ugly looking. | 17:18 |
| JaMa | cool, let me know if you want me to test it as well | 17:18 |
| jwessel | For sure, we'll need it tested. | 17:18 |
| JaMa | pseudo standard :) | 17:18 |
| jwessel | I'll have to forward port it when I am done because I did all the work on thud, where I am stuck at the moment. | 17:19 |
| *** falstaff <falstaff!~quassel@37.17.234.113> has quit IRC | 17:24 | |
| *** Bunio_FH <Bunio_FH!~bunio@clj-165.netdrive.pl> has joined #yocto | 17:26 | |
| khem | JaMa:I have a fix that I am testing | 17:28 |
| *** leitao <leitao!~leitao@2620:10d:c092:200::1:73da> has quit IRC | 17:29 | |
| JaMa | khem: that was quick! thanks will include it in next world build | 17:41 |
| JaMa | khem: btw for those gcc patches it would be useful to use "git format-patch --no-signature --no-numbered" when refreshing them for new version (like devtool does now) | 17:44 |
| *** JPEW <JPEW!cc4da337@204.77.163.55> has quit IRC | 17:45 | |
| JaMa | anyone hiding bullet3 or netpbm recipes from layerindex? | 17:46 |
| khem | JaMa: thats a good idea, it has bothered me a bit but not enough | 17:49 |
| khem | I keep a local fork for these patches | 17:50 |
| khem | and devtool workflow does not fit in there | 17:50 |
| *** JPEW_ <JPEW_!~JPEW@2605:a601:21d1:5200:620a:c230:ada8:3135> has joined #yocto | 17:57 | |
| JaMa | khem: understood, I use the same e.g. with qt* patches, I've noticed only because when replacing 9.2.0 v1 with v2 there were much more changes than expected just because many patches were re-numbered after removal of one and then the include path change changed the git signature as well :) | 18:07 |
| *** pung_ <pung_!~BobPungar@187.113.136.3> has quit IRC | 18:19 | |
| *** pung_ <pung_!~BobPungar@187.113.136.3> has joined #yocto | 18:19 | |
| *** JPEW_ is now known as JPEW | 18:25 | |
| JPEW | RP: Strange that the stat code caused lockups. | 18:26 |
| *** T_UNIX <T_UNIX!uid218288@gateway/web/irccloud.com/x-yllotgaqgvjxwmhu> has quit IRC | 18:51 | |
| *** pung_ <pung_!~BobPungar@187.113.136.3> has quit IRC | 18:52 | |
| *** pung_ <pung_!~BobPungar@187.113.136.3> has joined #yocto | 18:52 | |
| *** asabil <asabil!~asabil@2a01:79d:7375:2ca4:6410:7de1:3184:de99> has joined #yocto | 19:05 | |
| mischief | can i put INITRAMFS_IMAGE somewhere other than build/conf/local.conf? | 19:06 |
| *** asabil <asabil!~asabil@2a01:79d:7375:2ca4:6410:7de1:3184:de99> has quit IRC | 19:08 | |
| *** vmeson <vmeson!~rmacleod@24-52-238-240.cable.teksavvy.com> has quit IRC | 19:14 | |
| *** BCMM <BCMM!~BCMM@unaffiliated/bcmm> has quit IRC | 19:15 | |
| robbawebba | mischief: yup! I have INITRAMFS_IMAGE defined in a distro conf file in my layer. I'm guessing you could define INITRAMFS_IMAGE in an image recipe as well, but I've never tried it. I would definitely appreciate another perspective on that possibility. | 19:16 |
| mischief | we're starting to use yocto with pretty much 0 experience, so i'm sure i've done something wrong | 19:17 |
| *** JPEW <JPEW!~JPEW@2605:a601:21d1:5200:620a:c230:ada8:3135> has quit IRC | 19:17 | |
| mischief | i made a directory for TEMPLATECONF and that is checked into git but build/conf/local.conf is not in git, where INITRAMFS_IMAGE is currently defined | 19:17 |
| mischief | robbawebba: do you have something i can read about making distro layers/confs? | 19:18 |
| robbawebba | mischief: sure! give me a few minutes to find the right links | 19:18 |
| mischief | thank you.. i've asked a few questions here previously but you are the first to answer any of them :-) | 19:19 |
| robbawebba | mischief: This entire section, section 7.1, contains tons of information on how to create and manage layers. Please let me know if you have any other questions! | 19:22 |
| *** asabil <asabil!~asabil@2a01:79d:7375:2ca4:6410:7de1:3184:de99> has joined #yocto | 19:22 | |
| robbawebba | https://www.yoctoproject.org/docs/current/mega-manual/mega-manual.html#understanding-and-creating-layers | 19:22 |
| robbawebba | I'll try and find info on Distros next, but you'll need a layer before you create a custom distro :) so the layer stuff should keep you busy in the meantime | 19:23 |
| *** falstaff <falstaff!~quassel@37.17.234.113> has joined #yocto | 19:24 | |
| mischief | right now we have a git repo with meta-$COMPANY with basically just our kernel recipe, and poky as a submodule. not sure if this is right but i've been able to make at least a kernel build. | 19:24 |
| mischief | so i guess we have one layer. | 19:26 |
| robbawebba | mischief: nice! That's essentially a layer. Did you add that meta-$COMPANY to your build/conf/bblayers.conf file, either manually or using the bitbake-layers tool? | 19:26 |
| robbawebba | also, here's the distro documentation: https://www.yoctoproject.org/docs/current/mega-manual/mega-manual.html#creating-your-own-distribution | 19:26 |
| mischief | yes | 19:26 |
| robbawebba | cool! Then you have a layer :) | 19:26 |
| mischief | i put that in meta-$COMPANY/conf/bblayers.conf.sample | 19:26 |
| mischief | so unfortunately if i modify it in build/conf then it is not tracked in git.. but if i modify bblayers.conf.sample it would require a user to re-initialize build/conf, correct? | 19:27 |
| robbawebba | yup! | 19:28 |
| mischief | is there no better way to track build/conf changes in revision control? | 19:29 |
| mischief | (i.e, for INITRAMFS_IMAGE) | 19:29 |
| robbawebba | and as long as the command `bitbake-layers show-layers` command shows your layer, then your're good! Plus, you mentioned that you're able to build your kernel recipe that's in your layer, so that's another sign that bitbake can find your layer | 19:29 |
| robbawebba | and yes, there are a couple of options, the TEMPLATECONF mechanism is one that you already mentioned | 19:30 |
| mischief | would it be a Bad Idea (tm) to just put build/conf/*.conf in the git repo? | 19:31 |
| robbawebba | yup, I believe it is not a good practice | 19:32 |
| Crofton|work | I'm pretty sure that would be awful :) | 19:32 |
| robbawebba | mischief: are you aware that you can put local.conf.sample in your layer, just like bblayers.conf.sample? | 19:33 |
| mischief | robbawebba: that is currently what i am doing | 19:33 |
| Crofton|work | https://github.com/balister/meta-sdr/tree/master/conf | 19:33 |
| mischief | but my concern is that if i change a variable and commit that to git, other developers would not see it unless they re-initialize build/conf from TEMPLATECONF. | 19:33 |
| Crofton|work | good concern | 19:34 |
| Crofton|work | auto.conf and site.conf are also searched | 19:34 |
| Crofton|work | in the BBPATH | 19:34 |
| Crofton|work | https://github.com/balister/meta-sdr/blob/master/conf/bblayers.conf.sample#L5 | 19:34 |
| Crofton|work | I use that so all build on one machine check .oe in my home dir | 19:35 |
| Crofton|work | and I can insert something like site.conf where I might point to a local source mirros | 19:35 |
| mischief | some of my coworkers are going to embedded linux conference and now i am somewhat regretting not attending ._. | 19:37 |
| Crofton|work | as well you should | 19:37 |
| Crofton|work | well I am not going either, but I will be at ELCE | 19:37 |
| robbawebba | Crofton|work: nice! So mischief can just add site.conf to his layer's conf directory and check those configuration options into version control? | 19:38 |
| Crofton|work | with careful tweaking of BBPATH | 19:38 |
| mischief | my BBPATH is just $TOPDIR | 19:39 |
| Crofton|work | and use custom distro.conf also | 19:39 |
| Crofton|work | https://github.com/balister/sdr-build | 19:39 |
| Crofton|work | do something like this and have actual content with conf files also | 19:39 |
| Crofton|work | you want to split things that imapct product from things that are truly local | 19:40 |
| Crofton|work | like local source irrors, DL_DIR etc | 19:40 |
| Crofton|work | yes I ahve screwed up having important stuff in local.conf | 19:40 |
| mischief | Crofton|work: currently we have just one git repo containing meta-$COMPANY/{conf,recipes-*} and poky/ as a submodule | 19:41 |
| *** BCMM <BCMM!~BCMM@unaffiliated/bcmm> has joined #yocto | 19:41 | |
| *** Bunio_FH <Bunio_FH!~bunio@clj-165.netdrive.pl> has quit IRC | 19:41 | |
| Crofton|work | personally, I do what I do and drop the meta-poky stuff and use your own distro.conf | 19:43 |
| Crofton|work | and keep the settings there | 19:43 |
| Crofton|work | depending on what they are of course | 19:43 |
| Crofton|work | and we are sorry this is really well spelt out for people | 19:43 |
| mischief | i'm not sure my skill level with this stuff is high enough to jump the boat on poky just yet :-) | 19:44 |
| Crofton|work | you are asking the questions that suggest you are there | 19:45 |
| Crofton|work | You are basically changing stuff set in poky.conf? | 19:45 |
| * Crofton|work has never used poky :) | 19:45 | |
| kergoth | creating a distro .conf is not difficult. cp poky.conf to your own, go | 19:47 |
| mischief | i see, you are suggesting removing meta-poky layer from layers.conf? i thought you meant remove the poky/ submodule from my git repo :s | 19:47 |
| *** Bunio_FH <Bunio_FH!~bunio@clj-165.netdrive.pl> has joined #yocto | 19:49 | |
| *** kaspter <kaspter!~Instantbi@115.230.121.109> has quit IRC | 19:53 | |
| *** kaspter <kaspter!~Instantbi@115.230.121.109> has joined #yocto | 19:54 | |
| *** asabil <asabil!~asabil@2a01:79d:7375:2ca4:6410:7de1:3184:de99> has quit IRC | 19:56 | |
| yocti | New news from stackoverflow: Yacto/BitBake new dir does not show up <https://stackoverflow.com/questions/57530178/yacto-bitbake-new-dir-does-not-show-up> | 20:02 |
| *** vmeson <vmeson!~rmacleod@24-52-238-240.cable.teksavvy.com> has joined #yocto | 20:06 | |
| *** asabil <asabil!~asabil@2a01:79d:7375:2ca4:6410:7de1:3184:de99> has joined #yocto | 20:11 | |
| *** asabil <asabil!~asabil@2a01:79d:7375:2ca4:6410:7de1:3184:de99> has quit IRC | 20:14 | |
| *** WillMiles <WillMiles!~Will@static-209-87-231-80.storm.ca> has quit IRC | 20:24 | |
| *** imm_ <imm_!~imm_@unaffiliated/imm/x-7821412> has joined #yocto | 20:25 | |
| *** florian <florian!~florian_k@Maemo/community/contributor/florian> has joined #yocto | 20:27 | |
| *** JaMa <JaMa!~martin@ip-217-030-068-212.aim-net.cz> has quit IRC | 20:29 | |
| *** falstaff <falstaff!~quassel@37.17.234.113> has quit IRC | 20:32 | |
| *** JPEW <JPEW!~JPEW@2605:a601:21d1:5200:620a:c230:ada8:3135> has joined #yocto | 20:48 | |
| __angelo | in a machine.conf, is the field "#@SOC:" used for any special purpose or it's just a comment ? | 20:53 |
| khem | mischief: in conf/ dir you can create a site.conf file and put your customizations in that file | 20:59 |
| khem | __angelo:I think linux yocto uses it | 20:59 |
| khem | i mean linux-yocto kernel recipe | 21:00 |
| __angelo | khem, ok. mmm i have a linux-mystuff recipe | 21:00 |
| khem | then it wont bother you | 21:00 |
| *** nagio <nagio!~andrea@host158-71-dynamic.4-87-r.retail.telecomitalia.it> has quit IRC | 21:01 | |
| *** agust <agust!~agust@p54833DBB.dip0.t-ipconnect.de> has quit IRC | 21:02 | |
| __angelo | khem, thanks !รน | 21:02 |
| *** agust <agust!~agust@p548339E7.dip0.t-ipconnect.de> has joined #yocto | 21:02 | |
| khem | Crofton|work and mischief take a look here https://github.com/YoeDistro/yoe-distro | 21:03 |
| khem | thats one way to solve it | 21:05 |
| jwessel | I put the RFC patch set out there to solve the glibc-locale issue. | 21:07 |
| *** leitao <leitao!~leitao@2a02:c7f:a63:f000:83b:ce0f:f5ae:3633> has joined #yocto | 21:08 | |
| *** nagio <nagio!~andrea@host158-71-dynamic.4-87-r.retail.telecomitalia.it> has joined #yocto | 21:09 | |
| jwessel | We could certainly benefit from some testing on it for those who were most affected by this issue. | 21:09 |
| jwessel | I only saw it once, myself. But our automated build cluster would have a number of failures each week. | 21:09 |
| jwessel | There could be other corner cases, but they will certainly not be the same as the one the patch set corrects. | 21:09 |
| RP | JPEW: I don't understand it. We also have a small race with the hash reporting and the generation of the sstate package. I think the latter is lagging the former and leading to a race :/ | 21:10 |
| khem | jwessel:cool.. pointer ? | 21:11 |
| JPEW | RP: That... shouldn't be possible? | 21:11 |
| jwessel | [OE-core] [RFC PATCH 1/2] cross-localedef-native: Add hardlink resolver from util-linux | 21:11 |
| jwessel | It is a two patch set with some explanation about the problem, but not about how it was tracked down. | 21:12 |
| JPEW | RP: Unless we are thinking of different things | 21:12 |
| jwessel | I had to use a modified version of pseudo, which I'd likely use in the future, if we have other such problems. | 21:12 |
| RP | jwessel: I think this may be related to all of: https://bugzilla.yoctoproject.org/show_bug.cgi?id=11551 https://bugzilla.yoctoproject.org/show_bug.cgi?id=11299 https://bugzilla.yoctoproject.org/show_bug.cgi?id=12664 https://bugzilla.yoctoproject.org/show_bug.cgi?id=12434 | 21:13 |
| yocti | Bug 11551: normal, Medium+, 2.99, ross.burton, NEW , inconsistent dependencies in glibc-locale packages | 21:13 |
| yocti | Bug 11299: normal, Medium+, 3.1, Qi.Chen, NEW , glibc-locale warning [host-user-contaminated] | 21:13 |
| yocti | Bug 12664: normal, Medium+, 2.8 M4, ross.burton, NEW , glibc-binary-localedata packages may contain .tmp files | 21:13 |
| yocti | Bug 12434: normal, Medium+, 2.8 M3, randy.macleod, ACCEPTED , pseudo: Incorrect UID/GID in packaged files | 21:13 |
| RP | JPEW: https://autobuilder.yoctoproject.org/typhoon/#/builders/97/builds/282/steps/8/logs/step1b is interesting. There is obviously a bitbake problem with the rebuild of encodings but why wasn't it in sstate? | 21:14 |
| jwessel | I don't doubt there are a number of cases to reference for this particular problem. | 21:14 |
| RP | jwessel: the .tmp thing in particular in hindsight is a smoking gun | 21:14 |
| jwessel | I don't think I ever saw a build with .tmp files though. | 21:15 |
| jwessel | Something really had to go wrong to have that happen. | 21:15 |
| jwessel | But given the nature of the race. It is probably possible. | 21:15 |
| RP | jwessel: there were references to X.tmp files in that log you shared weren't there? | 21:15 |
| jwessel | Sure but that is the mechanics of how the linkage works. | 21:16 |
| RP | jwessel: I suspect somehow that leaks. Perhaps this doesn't solve it, not sure | 21:16 |
| jwessel | Oh it solves it. | 21:16 |
| jwessel | There are no more .tmp files at all with this change. | 21:16 |
| jwessel | I can see how a tmp file could have leaked. | 21:17 |
| RP | jwessel: right. We've only seen that .tmp thing on very few occasions | 21:17 |
| jwessel | The timing window on that one was way smaller for failure, than promoting the link to a file. | 21:17 |
| jwessel | Which was happening in ever single build. | 21:17 |
| jwessel | We just didn't know it. | 21:17 |
| jwessel | If I put some sleeps right after the unlink. You can make it explode into flames easily. | 21:18 |
| JPEW | RP: How many AB nodes are hitting the hash server at the same time? | 21:19 |
| jwessel | But there is no point anymore. This is a concise type of failure, with a number of different symptoms. | 21:19 |
| JPEW | RP: ^ I'm trying to see if I can get the timeouts to reproduce locally | 21:19 |
| jwessel | I do think we should consider patching pseudo a bit to have the special logs. But that is a back burner task. | 21:20 |
| jwessel | I don't know any other way to track down a problem like this, even when it is not pseudo that is the root cause. | 21:20 |
| jwessel | Only an strace log could have told the story. And those are even harder to read. :-/ | 21:20 |
| RP | JPEW: ~40 builds | 21:22 |
| RP | jwessel: totally agreed. I'm just trying to think ahead about which bugs this covers | 21:22 |
| RP | jwessel: indeed, they're also huge | 21:22 |
| jwessel | Probably a number of them with glibc-locale | 21:23 |
| khem | jwessel: do we have to include hardlink files into cross-locale-def, or can it call the utility from util-linux-native and then we depend on util-linux-native that gives us maintained version of hardlink utility | 21:25 |
| RP | khem: I just replied on list about that | 21:25 |
| RP | khem: its a build performance nightmare if we add that dependency | 21:25 |
| *** nabokov <nabokov!~armand@67.218.223.154> has quit IRC | 21:26 | |
| RP | khem: I'd kind of preferred a simple python script but that didn't appear so simple | 21:27 |
| *** berton <berton!~berton@181.220.83.67> has quit IRC | 21:28 | |
| JPEW | RP: I've forgotten; does bitbake check for an sstate tarball before adding the _setscene task to the runqueue? Perhaps the encoding unihash changed after it was added but before the task actually ran? | 21:28 |
| JPEW | im;\' | 21:29 |
| jwessel | I would have preferred to just depend on util-linux-native... RP said it was too much of a problem. | 21:29 |
| RP | I'm just the person who's spent a lot of time trying to speed the builds up, what do I know :( | 21:30 |
| jwessel | I would have thought the host tools simply don't recompile too often. RP, you have more data points than I. | 21:30 |
| RP | JPEW: I think you don't quite comprehend the runqueue as it isn't an actual list of tasks we're running, its a list of all tasks we could possibly run | 21:31 |
| khem | I think if its adding some significant bit then I agree but there is maintainance part as well | 21:31 |
| jwessel | So I implemented what was asked. | 21:31 |
| jwessel | To make it easier, should it need to be upgraded for some reason. I did separate the patch to the raw version. | 21:31 |
| khem | it would not matter much for endusers since they hardly rebuild things so deep so often | 21:31 |
| RP | My instinct says it will hurt things. Its based on us having done this in places before and knowing what the changes on performance in glibc-locale do to the build | 21:31 |
| RP | JPEW: in that sense if you're asking when we mark it as buildable, we'd check the sstate cache after something is marked as having a new unihash. | 21:33 |
| jwessel | I wouldn't worry to much about needing to replace it down the road. The reasons for the patches over the years were all to add features. Any kind of bug fixes were few and far between. | 21:33 |
| jwessel | There are only so many ways to make a checksum and a series of calls to link() unlink() | 21:33 |
| RP | JPEW: so you're right, we should have already checked this was valid in sstate so its strange it suddenly wasn't | 21:33 |
| jwessel | That being said, we can just use the host version 3-4 years from now, when util-linux is upreved in the distros. | 21:34 |
| jwessel | hardlink was a very recent addition from RedHat. | 21:34 |
| RP | khem: it would hurt anyone building from scratch and people do that a fair bit and criticise the project for it | 21:34 |
| JPEW | RP: Ya, I think I'm confusing some terminology somewhere... need to read more code :) | 21:34 |
| khem | RP: I understand that but I dont know how much, | 21:35 |
| RP | khem: based on experience it will hurt a fair bit, particularly on people with fewer cores | 21:36 |
| khem | and util-linux appears in many recipes as dependency | 21:36 |
| khem | e.g. python3-native needs util-linux-native | 21:37 |
| khem | do we build glibc before that ? | 21:37 |
| khem | so my caution is to not create technical debt for ourselves unless absolutely required | 21:39 |
| RP | khem: I suspect that is a recently introduced performance regression then :( | 21:39 |
| RP | khem: If we don't care about build times, sure but I have done one heck of a lot of work to try and keep the bootstrap process clear of this kind of painpoint | 21:40 |
| * RP notes systemd also has the dependency. On the plus side most of the rest of the system does not | 21:40 | |
| RP | we patch it out of quilt-native for example | 21:40 |
| RP | khem: glibc is kind of special as its the choke point on the task graph. The locales connect to it not as a direct blocker but influencing packaging | 21:42 |
| khem | RP:we do care, but if python3-native is required early enough and its building util-linux-native for us, then we can certainly use it in cross-localedef isnt it or am I missing something | 21:42 |
| RP | khem: IMO we should fix python3-native, not use it as a justification for slowing down the build further | 21:43 |
| khem | its not justifying it, its calling out what is is | 21:47 |
| khem | I looked a bit of forever technical debt that we will carry and it does come at some cost | 21:48 |
| RP | khem: its not forever, its until we can depend on it from the host systems | 21:50 |
| RP | this is why I also preferred the python option but what do I know | 21:51 |
| fray | at one point util-linux-native was also a 'provided' thing.. stuff could specify it, but it never actually built | 21:53 |
| khem | perhaps uninative should capture all native dependencies and we should have none of those during build maybe | 21:55 |
| khem | so which hosts dont yet provide it | 21:56 |
| RP | khem: that isn't the point of uninative but was the point of buildtools-tarball | 21:56 |
| fray | Ubuntu | 21:56 |
| fray | I assume current Debian doesn't as well, but not sure | 21:56 |
| fray | Fedora and RHEL/CentOS have for a very long time | 21:57 |
| khem | RP: uninative is the right answer a collection of packages which are distro independendent and are downloaded automatically during build there is no third party involved | 21:57 |
| RP | khem: it is not the problem uninative is designed to solve | 21:58 |
| khem | current debian/buster is going to live for next 5 years :( I was hoping it was some old stinky centos | 21:58 |
| khem | RP: I understand that | 21:58 |
| RP | khem: it could be extended to solve that problem sure but its a very different one | 21:58 |
| khem | I am asking for widening its scope | 21:59 |
| RP | and we do already have a different solution there called buildtools-tarball, its just not downloaded automatically | 21:59 |
| fray | this would be a candidate for buildtools-tarball, but reality is that we'd then need to make hardlink a sanity checked item... | 21:59 |
| fray | and the out-of-box-experience for this one binary is better this way (for now)... | 22:00 |
| RP | khem: after all the complaints about naming we need to try and be really clear with names. uninative has a specific purpose and I'd prefer not to break that... | 22:00 |
| khem | I think its something we should do, otherwise we are spreading ourselves into supporing host distros | 22:00 |
| RP | khem: we already do that | 22:01 |
| fray | that's why (in this case) the tooling goes into the recipe and isn't distributed elsewhere.. it's local to limit support | 22:01 |
| khem | RP:question is should we keep doing that or have some solution to abstract on host distro | 22:01 |
| fray | we (WR) use the buildtools-tarball to bring old OSes up to date to be able to build.. but since hardlink is -so- new, even 'current' versions don't have the missing functionality to we're stuck for a bit | 22:02 |
| RP | khem: good question. I'd just like to know who is going to do the work in developing it and making it work and handling the transition and its relative priority to everything else | 22:03 |
| RP | meanwhile it would be nice to fix the locale corruption issue | 22:03 |
| khem | fray: thats understood, instead of making tools tarball an optional thing it should be made made mandatory and downloaded automatically during build | 22:04 |
| *** agust <agust!~agust@p548339E7.dip0.t-ipconnect.de> has quit IRC | 22:05 | |
| khem | currently we build same number of native packages irrespective of buildtools tarball is my understanding, it just brings those arcane machine to something that can build modern OE | 22:06 |
| RP | khem: people hate prebuilt binaries to the point I can't make uninative the default for OE-Core | 22:06 |
| *** JPEW <JPEW!~JPEW@2605:a601:21d1:5200:620a:c230:ada8:3135> has quit IRC | 22:07 | |
| khem | you mentioned build times earliers thats why I brought this up, its the time we spend building python-native and other large time consuming enabling packages that irritates the user | 22:07 |
| RP | khem: but buildtools-tarball doesn't help us with python-native or perl-native :( | 22:08 |
| RP | those recipes are so tightly coupled you can't do much but have identical versions -native and target | 22:09 |
| khem | We need to have option to rebuild the releases tarballs, otherwise shared state and buildtools tarballs are same, people should hate them equally | 22:09 |
| RP | python-nativesdk which is what would be in the buildtools-tarball isn't good enough | 22:09 |
| RP | khem: I do understand the attraction but I've been told binaries like that aren't acceptable | 22:10 |
| khem | and same folks would be using containers eh ? | 22:10 |
| RP | khem: no idea. Perhaps people have come around to the idea of uninative now | 22:11 |
| *** florian <florian!~florian_k@Maemo/community/contributor/florian> has quit IRC | 22:11 | |
| RP | or nobody uses poky and therefore they don't see it, I don't know | 22:11 |
| khem | anyway, I would rest my case | 22:14 |
| *** leitao <leitao!~leitao@2a02:c7f:a63:f000:83b:ce0f:f5ae:3633> has quit IRC | 22:15 | |
| fray | we use uninative at WR, but in the same way that poky does.. | 22:16 |
| fray | we use buildtools to augment old (broken) OSes.. | 22:16 |
| fray | the combo works really well for us an our customers.. | 22:16 |
| khem | fray: do you use ASSUME_PROVIDED then to not build bunch of native packages becasue they now are in buildtools ? | 22:18 |
| fray | Only the same ones that Poky has | 22:18 |
| fray | python3, make, tar were the major things we've needed on older RHEL | 22:18 |
| fray | (RHEL / CentOS 7 is still REALLY common..) | 22:19 |
| *** tprrt <tprrt!~tprrt@upc31-1-78-208-110-13.fbx.proxad.net> has quit IRC | 22:20 | |
| *** falstaff <falstaff!~quassel@37.17.234.113> has joined #yocto | 22:20 | |
| khem | yes they are | 22:22 |
| khem | oh btw. glibc also needs python3 now during build, fun fun | 22:22 |
| *** falstaff <falstaff!~quassel@37.17.234.113> has quit IRC | 22:24 | |
| *** leitao <leitao!~leitao@2a02:c7f:a63:f000:83b:ce0f:f5ae:3633> has joined #yocto | 22:25 | |
| *** leitao <leitao!~leitao@2a02:c7f:a63:f000:83b:ce0f:f5ae:3633> has quit IRC | 22:30 | |
| *** falstaff <falstaff!~quassel@37.17.234.113> has joined #yocto | 22:32 | |
| *** falstaff <falstaff!~quassel@37.17.234.113> has quit IRC | 22:33 | |
| *** falstaff <falstaff!~quassel@37.17.234.113> has joined #yocto | 22:39 | |
| *** falstaff <falstaff!~quassel@37.17.234.113> has quit IRC | 22:40 | |
| *** georgem_home <georgem_home!uid210681@gateway/web/irccloud.com/x-bunorvvikoqicjld> has joined #yocto | 22:44 | |
| RP | khem: can that not come from the host system though? | 22:46 |
| RP | python3-native is really for the python builds and python modules, not general python3 | 22:47 |
| RP | khem: I guess a key question is any version reqs? | 22:52 |
| mischief | i have a kernel module.. and it has RPROVIDES_${PN} += "kernel-module-${PN}" inside of it. if i add kernel-module-$foo to PACKAGE_INSTALL, i get an error saying 'no installation candidate' even though it also says it is a virtual package provided by ${PN}. why is that? | 22:54 |
| mischief | if i just use the kernel module ${PN} for PACKAGE_INSTALL it seems to work | 22:55 |
| *** khem <khem!~khem@unaffiliated/khem> has quit IRC | 23:09 | |
| *** elvispre <elvispre!~elvispre@2001:8b0:e0:884d:99db:5cdc:4b13:fabc> has quit IRC | 23:13 | |
| *** khem <khem!~khem@unaffiliated/khem> has joined #yocto | 23:16 | |
| *** BCMM <BCMM!~BCMM@unaffiliated/bcmm> has quit IRC | 23:31 | |
| *** vineela <vineela!~vtummala@134.134.139.77> has quit IRC | 23:42 | |
| *** awe00 <awe00!~awe00@unaffiliated/awe00> has joined #yocto | 23:59 | |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!