Friday, 2019-08-16

*** elruk_ <elruk_!> has quit IRC00:18
*** learningc <learningc!~learningc@> has quit IRC00:19
*** tgraydon <tgraydon!tgraydon@nat/intel/x-skpphsfafofltkvt> has quit IRC00:22
*** fatalhalt <fatalhalt!> has joined #yocto00:31
*** FailDev <FailDev!18d83107@> has quit IRC00:36
*** learningc <learningc!> has joined #yocto00:36
*** learningc <learningc!> has quit IRC00:43
*** learningc <learningc!> has joined #yocto00:44
*** mrpelotazo <mrpelotazo!> has quit IRC01:31
*** mrpelotazo <mrpelotazo!> has joined #yocto01:33
*** chinhuat8 <chinhuat8!~chinhuat@> has joined #yocto02:05
*** chinhuat <chinhuat!~chinhuat@> has quit IRC02:06
*** georgem_home <georgem_home!uid210681@gateway/web/> has quit IRC02:26
*** armpit <armpit!~armpit@2601:202:4180:c33:7c98:5faa:262d:a3af> has quit IRC03:13
*** chinhuat8 is now known as chinhuat03:15
*** armpit <armpit!~armpit@2601:202:4180:c33:b816:6e4b:1727:7819> has joined #yocto03:24
*** fatalhalt <fatalhalt!> has quit IRC03:46
*** camus <camus!~Instantbi@> has joined #yocto03:47
*** kaspter <kaspter!~Instantbi@> has quit IRC03:47
*** camus is now known as kaspter03:47
*** AndersD <AndersD!> has joined #yocto05:18
*** milloni <milloni!> has quit IRC05:52
*** milloni <milloni!> has joined #yocto05:52
*** AndersD <AndersD!> has quit IRC05:54
*** jeanba <jeanba!~jbl@> has joined #yocto06:10
*** TobSnyder <TobSnyder!> has joined #yocto06:22
*** andycooper <andycooper!uid246432@gateway/web/> has quit IRC06:31
*** tprrt <tprrt!> has joined #yocto06:41
*** kroon <kroon!~kroon@> has joined #yocto06:53
*** jeanba <jeanba!~jbl@> has left #yocto06:54
*** agust <agust!> has joined #yocto06:55
*** florian <florian!~florian_k@Maemo/community/contributor/florian> has joined #yocto07:09
*** BobPungartnik <BobPungartnik!~BobPungar@> has joined #yocto07:21
*** pung_ <pung_!~BobPungar@> has quit IRC07:22
silviofHow you are testing your images without hardware? via `-c testimage`? There exists some other tools which are good to know to test images?07:32
*** asabil <asabil!> has joined #yocto07:35
*** Bunio_FH <Bunio_FH!> has joined #yocto07:37
*** pung_ <pung_!~BobPungar@> has joined #yocto07:46
*** BobPungartnik <BobPungartnik!~BobPungar@> has quit IRC07:48
*** asabil <asabil!> has quit IRC07:52
*** asabil <asabil!> has joined #yocto07:53
*** T_UNIX <T_UNIX!uid218288@gateway/web/> has joined #yocto07:55
*** leitao <leitao!~leitao@2620:10d:c092:200::1:73da> has joined #yocto07:58
*** leitao <leitao!~leitao@2620:10d:c092:200::1:73da> has quit IRC08:10
*** leitao <leitao!~leitao@2620:10d:c092:200::1:73da> has joined #yocto08:11
*** pung_ <pung_!~BobPungar@> has quit IRC08:17
*** pung_ <pung_!~BobPungar@> has joined #yocto08:17
JaMaRP: and with latest bitbake a5e084a266f63c2fd370122327615e49beaeb94e
*** kaspter <kaspter!~Instantbi@> has quit IRC08:35
*** kaspter <kaspter!~Instantbi@> has joined #yocto08:36
*** bluelightning_ <bluelightning_!~paul@pdpc/supporter/professional/bluelightning> has joined #yocto08:48
RPJaMa: thanks08:51
*** bluelightning <bluelightning!~paul@pdpc/supporter/professional/bluelightning> has quit IRC08:53
JaMaRP: now I'm running it with even older 18d4a31fdcec1f0e5d2199d6142f0ce833fca1a708:54
RPJaMa: is quite interesting, the sorted() shows up badly08:54
RPand pformat08:55
JaMa18d4a31fdcec1f0e5d2199d6142f0ce833fca1a7 seems to be fine, it's already executing tasks08:56
RPJaMa: just to confirm, was without the logger.debug removal?08:57
RPI think that sorted comes from part of pformat, going to need to be careful with that08:57
JaMayes, was 1f630fdf0260db08541d3ca9f25f852931c19905 without any modifications, it was supposed to be the "good old" baseline based on the times with old image, but it seems that this revision already regressed badly09:00
JaMaso it's something in 6 commits between 18d4a31 and 1f630fdf09:02
JaMathe later has the line logger.debug(2, "Holding off tasks %s" % pprint.pformat(self.holdoff_tasks))09:04
*** cquast <cquast!~cquast@> has joined #yocto09:08
RPJaMa: right, that line in particular performs terribly! :)09:10
*** bluelightning_ <bluelightning_!~paul@pdpc/supporter/professional/bluelightning> has quit IRC09:10
*** BCMM <BCMM!~BCMM@unaffiliated/bcmm> has joined #yocto09:15
JaMaRP: yes, I just thought that this was only in master-next at that time, not in master, will send an e-mail once last run with even older bitbake revision is uploaded to make it a bit clearer what is what09:20
JaMaRP: latest world build last night with latest bitbake shown some unexpected failures with missing native deps when building e.g. some node recipes, but maybe it wasn't caused by bitbake, will re-test09:21
RPJaMa: I'm not aware of deps going missing like that but there were various fixes made, this is partly why I've merged everything to master, so we all have a common test target with the known issues fixed09:25
*** learningc <learningc!> has quit IRC09:27
*** learningc <learningc!> has joined #yocto09:27
*** learningc <learningc!> has quit IRC09:29
*** bluelightning_ <bluelightning_!~paul@pdpc/supporter/professional/bluelightning> has joined #yocto09:47
*** goliath <goliath!> has joined #yocto09:49
mcfriskhmm, python3 doesn't seem to build without readline (GPLv3) due to do_install check. Any hints how to make the test compatible with non-default PACKAGECONFIG which removes readline support?09:59
*** asabil <asabil!> has quit IRC10:15
*** asabil <asabil!> has joined #yocto10:16
*** pung_ <pung_!~BobPungar@> has quit IRC10:22
*** kaspter <kaspter!~Instantbi@> has quit IRC10:25
*** kaspter <kaspter!~Instantbi@> has joined #yocto10:27
*** BobPungartnik <BobPungartnik!~BobPungar@> has joined #yocto10:27
*** kaspter <kaspter!~Instantbi@> has quit IRC10:34
mcfriskseems I really need to patch python3 recipe to remove if I build without readline support10:48
SaurRP: I've sent a change to the bitbake list that adds a progress meter for the part between "Initialising tasks" and "Checking sstate mirror object availability" where runqueue currently hangs for a while without any output.10:51
SaurRP: So unless you can't make the delay introduced by that part of the code go away, this will at least make it obvious that something is happening.10:52
*** kaspter <kaspter!~Instantbi@> has joined #yocto10:53
*** wooosaiii <wooosaiii!> has quit IRC10:56
*** erbo <erbo!> has quit IRC11:02
*** BobPungartnik <BobPungartnik!~BobPungar@> has quit IRC11:06
*** BobPungartnik <BobPungartnik!~BobPungar@> has joined #yocto11:06
*** BCMM <BCMM!~BCMM@unaffiliated/bcmm> has quit IRC11:32
RPSaur: ok, but wrapping everything in more progress bars isn't really the answer...11:39
SaurRP: Well, if the delay is unavoidable then the progressbar is needed to show that something is happening. But if you can find ways to make the delay go away, then that is of course even better. ;)11:42
RPSaur: I just hate progress bars11:42
SaurRP: Well, I hate long periods of no output even more.11:43
*** berton <berton!~berton@> has joined #yocto11:46
Crofton|workSaur, welcome to my life11:47
Crofton|workI made a fake progress bar once, always fun when underlying thing crashed and kept showing progress11:48
SaurCrofton|work: That sounds like the worst of both worlds...11:48
mcfriskI follow progress with strace if things are taking too long. should have known about bitbake -P earlier and started debugging the sometimes long delays..11:48
RPSaur: does the progress bar even increment properly with the way you've written that patch?11:53
SaurRP: Yes it does.11:53
*** falstaff <falstaff!~quassel@> has quit IRC11:53
RPSaur: it looks to me like the increments are around the fast part of the code11:53
SaurRP: "Properly" in the sense that it updates once per task.11:54
RPSaur: so it doesn't get quickly to say  95% then sit there?11:54
RPthen there is something very wrong :(11:54
SaurRP: Well, one thing I noticed was that nothing is ever added to tocheck (with the current state I have: some updated recipes and stuff).11:55
SaurBecause I first had the update at the end before I noticed the continue statements, and then the update event never fired.11:56
RPSaur:  a clean build or an existing completed build?11:56
SaurRP: A completed build that then have had a couple of layer updates that have not built as I've been using bitbake -n.11:57
RPSaur: tocheck should be added to for things it needs to check the sstate cache for11:58
Crofton|workSaur, I was told to do it :) So I did12:00
SaurRP: Ok, it seems tocheck isn't completely empty. There are 24 entries in it for my current build.12:00
RPSaur: try it on an empty build and it will be much higher12:01
SaurRP: I guess those 24 updates to the progress were so few compared to the 5600 tasks in total that the loop checks that i never saw the progress meter move.12:01
RPSaur: we need to fix the underlying code not to suck, this should be fast12:02
RPThis is why I wanted to get correctness before looking at performance12:03
SaurRP: I'll probably leave that part to you. ;)12:03
RPNow I'm trying to optimise code before its even working properly :(12:03
SaurRP: Now you at least know where one of the delays are...12:03
SaurRP: Btw, if I remove the tmp directory, that delay goes away.12:05
RPSaur: right, no stamps so that code does less12:05
RPSaur: (top patch) might help12:11
RPSaur: if that doesn't make it fast enough there is another change we can try. Each one of these moves the goalposts on the other issues I'm trying to fix of course :/12:14
SaurRP: That change took the time for that part from 36 to 25 seconds for me.12:21
SaurRP: And no, I do not expect you to focus on the performance at this time. As long as you are aware of the performance implications, make it work first.12:23
SaurRP: Hmm, I no longer seem to have push rights to poky-contrib. Have something changed regarding it?12:25
Saur(Haven't tried pushing to it for a couple of months so it is not necessarily recent.)12:26
RPSaur: ssh will show the perms12:26
RPSaur: don't think anything changed12:26 Permission denied (publickey).12:26
RPSaur: sounds like the key isn't right  :/12:27
SaurIt does, doesn't it...12:27
RPSaur: you can try adding on top, see if that brings the time down further12:32
RPSaur: less sure about what that last one might break12:33
RPselftest says its ok12:35
SaurRP: That brought the time down to 3 seconds, so yes if that still does what it is supposed to, then I guess  that progress meter is no longer needed. :)12:37
RPSaur: it moves work around12:38
SaurRP: Well, the lost 22 seconds didn't show up anywhere else at least as the total time decreased correspondingly. :)12:40
RPSaur: Its more a question of whether anything accesses those data structures while the data is out of date12:42
RPLooking at the updated profile, those two changes look to optimise most of the recent changes out the profile data12:47
*** kroon <kroon!~kroon@> has quit IRC12:52
*** BCMM <BCMM!~BCMM@unaffiliated/bcmm> has joined #yocto12:56
*** radsquirrel <radsquirrel!> has quit IRC13:04
*** radsquirrel <radsquirrel!> has joined #yocto13:05
jwesselRP: Here is the latest update.   bitbake is now out of the picture.13:05
jwesselI duplicated the problem outside of bitbake.13:05
jwessel/glibc-locale/2.28-r0/locale-tree% ls -d usr/lib64/locale/*.ISO* |xargs rm -rf  ; make -j12813:06
*** kroon <kroon!~kroon@> has joined #yocto13:06
jwesselWhat ever the cross-localedef is doing is not safe to run in a massively parallel context.13:06
jwesselBecause it is moving files from a .tmp -> real and or hard link, stuff is disappearing that is actually needed.13:07
jwesselI can probably run it without pseudo at all to rule pseudo out.13:08
jwesselIt will require quite a bit of further analysis, but at least we know conclusively all the bitbake/oe and such is out of the picture.13:08
*** BobPungartnik <BobPungartnik!~BobPungar@> has quit IRC13:12
zeddiithese 5.2 libc-headers are going to make me insane13:13
* zeddii goes for coffee13:13
*** BobPungartnik <BobPungartnik!~BobPungar@> has joined #yocto13:13
*** JPEW_ <JPEW_!~JPEW@2605:a601:21d1:5200:620a:c230:ada8:3135> has joined #yocto13:16
*** kaspter <kaspter!~Instantbi@> has quit IRC13:27
*** kaspter <kaspter!~Instantbi@> has joined #yocto13:28
zeddiicoffee worked. bluez5 is building again.13:28
*** kroon <kroon!~kroon@> has quit IRC13:30
*** Bagira <Bagira!Phanes@surro/founder/phanes> has quit IRC13:33
*** bluelightning_ <bluelightning_!~paul@pdpc/supporter/professional/bluelightning> has quit IRC13:35
*** aidanh <aidanh!~aidanh@unaffiliated/aidanh> has quit IRC13:35
*** aidanh <aidanh!~aidanh@unaffiliated/aidanh> has joined #yocto13:36
*** Phanes <Phanes!~Phanes@surro/founder/phanes> has joined #yocto13:36
*** litb <litb!> has joined #yocto13:50
litbhello folk13:50
litbis it OK to query the value of DEBUG_BUILD in a recipe, and pass some package Makefile specific flag depending on it?13:51
litbI.e  DEBUG_BUILD ?= "0"   MyDebugFlag = "${DEBUG_BUILD}"   ?. In the docs, it'ss said that DEBUG_BUILD is used to specify that yocto should build packages with debug information13:51
RPjwessel: cool, always good to narrow down the issue! :)13:53
jwesselheh... I already found the offending code in the cross-localedef13:54
jwesselI don't exactly know how to fix it yet, but I see why it is clearly not parallel safe.13:55
jwesselThe code is written to assume hard links might not work.13:55
jwesselAnd it will unlink a file, which could be in the process of being tested and used by one of the other parallel treads.13:55
jwesselIf you were to run this serially with just a single cross-localedef there is no way to hit that problem.13:56
jwesselIt replaces the file with a hard link when they test out to be the same.13:56
fraythe original use of cross-localedef was serial...13:57
jwesselSo if you go back to that, the problem is solved.13:57
jwesselOr change cross-localedef to not be stupid and allow --force-hard links and write out a .tmp file to perform the test.13:58
*** Phanes <Phanes!~Phanes@surro/founder/phanes> has quit IRC13:58
RPserial is very very slow so hacking localedef would be better13:58
jwesselTesting against a file that might go-away for a fraction of a second will not end well.13:58
RPjwessel: no, it explains alot13:58
frayya, that's why it moved to parallel..  but ya, definitely a race to check and then act..13:59
jwesselI have no reason to believe that pseudo is the issue here.13:59
jwesselI'd say at this point we have the root cause.13:59
frayya.. I agree, sounds more like fix the issue first.. :)14:00
*** Phanes <Phanes!Phanes@surro/founder/phanes> has joined #yocto14:00
frayother systems BTW do the hard linking in two steps.. one step to generate all of the files, then a second to hash them and set them as parallel14:00
jwesselIt was mighty interesting to have a log though in pseudo for a place in the code that should never happen.14:00
jwesselWe might consider keeping that change.14:01
fray'er.. set them as hardlinks I mean14:01
jwesselBecause it tells you... "You really screwed up somewhere else..."14:01
jwesselfray: You might have to do just that.14:02
frayif I remember right (like 18 years ago right) that is what was being done.. then it was merged into a single program..14:03
fraythe origin of the cross-localedef is either MontaVista or CodeSourcery.. it's -old-14:03
frayLooks liek CodeSourcery14:05
fray+                        Cross-Compiling GLIBC14:05
fray+                  Jim Blandy <>14:05
jwesselPerform generation, and then coalesce14:05
jwesselDoes the generation even support skipping the combine step?14:06
frayI knew it was old though, far pre-YP14:06
jwesselBecause there are other programs that can combine stuff.14:06
frayYa, looking at Khem's repo..  2006 or before.. :)14:06
frayI know the first version was before.. :)14:06
fraythe original version of the code didnt do the hardlink stuff..14:07
fraywe (WR) paid CodeSourcery to do that back in 2007/2008 timeframe?14:07
fray(not saying they wouldn't have done it on their own or someone else didn't pay them as well...)14:07
frayI just remember having that discussion with them as a system optimization14:07
fray(probably was before 2007 now that I think of it.. probably closer to 2005/200614:09
jwessel  /* Compare the file with the locale data files for the same category in14:09
jwessel     other locales, and see if we can reuse it, to save disk space.  */14:09
jwessel  other_paths = siblings (output_path);14:09
jwesselFYI that is what kills you in the parallel case.14:10
jwesselThose other ones may in the act of being processed.14:10
*** khem <khem!~khem@unaffiliated/khem> has quit IRC14:10
frayya.. I think 'easiest' to implement fix is break into two steps.. generate (& hash?) and (hash &) hardlink   (not sure where the hashing should be done, but it can be done in parallel)14:11
jwesselWhich is 100% the reason it could be unlinked and replaced with a hard link while someone else is executing.14:11
frayso maybe three steps?14:11
jwesselYou don't necessarily have to.14:11
jwesselBut it would be the best way to do it.14:11
jwesselThe problem even with the current code is that since all these are running in parallel you really won't get the right number of hard links ever.14:13
fraythere is the '--no-hard-links' option in locale generation.. enable that and do the hardlinks later?14:15
jwesselI was advocating adding one.14:15
frayOhh ok14:15
jwesselIt won't work though due to the problem I just mentioned.14:15
jwesselIf you are generating these all in parallel some will simply be generating at the same time.14:15
fraythe no-hard-links commit mentions:14:16
jwesselYou can't check the existence of something that isn't there yet.14:16
frayAdd --no-hard-links option to localedef (bug 23923)14:16
yoctiBug was not found.14:16
fray    Downstream distributions need consistent sets of hardlinks in14:16
fray    order for rpm to operate effectively. This means that even if14:16
fray    locales are built with a high level of parallelism that the14:16
fray    resulting files need to have consistent hardlink counts. The only14:16
fray    way to achieve this is with a post-install hardlink pass using a14:16
fray    program like 'hardlink' (shipped in Fedora).14:16
frayso it might even be helpful to break it in two for determinism sake14:16
jwesselBut why bother doing the summing and checking in cross-localedef?14:17
jwesselYou can do the hardlink pass outside of pseudo if you want.14:17
jwesselpsuedo will be happy so long as the inode had created was preserved.14:17
frayHas to be done under control of pseudo or the DB will get messed up..14:17
jwesselFor at least one file.14:18
jwesselNot really.   Not the way it works... Because I read the code :-)14:18
fraythat wasn't always true..  at one point it was a combo of inode and filename14:18
frayI suspect fakeroot still works based on filename.. (at least it also used to)14:18
jwesselfakeroot yes.  pseudo is fine.14:18
jwesselI don't know what the timepenalty is for the hardlink14:19
frayI'd still be cautious and do the hard linking under pseudo..14:19
frayWe used 'hardlink' in the past for something, and I remember it being fast.. but I don't completely remember how it worked..14:19
jwessellocale-tree% hardlink -n usr/lib64/locale14:22
jwesselMode:     dry-run14:22
jwesselFiles:    491914:22
jwesselLinked:   389 files14:22
jwesselSo we missed 389 files in my sample run.14:22
jwesselJust to prove my point :-)14:22
*** khem <khem!~khem@unaffiliated/khem> has joined #yocto14:22
jwesselI'd say we just add a fast exit on the checks in localedef either via cmdline arg or environment variable.14:23
fraylooks like hardlink might be part of util-linux now (or at least is on Fedora)14:23
fraythat's what the --no-hard-links is supposed to do.. fast exit14:23
frayor am I missing something?14:23
jwessellets look...  I didn't see that option.14:24
fraycommit 8cebd4ffe67bf94508809ea0caa02a4f1d52e8b114:24
fray(in glibc of course)14:24
fray+  if (hard_links)14:24
fray+    other_paths = siblings (output_path);14:24
frayotherwise it skips that whole section of code14:25
jwesselThat is not in the version I have.14:25
jwesselThat is what I was considering adding.14:25
frayDate:   Mon Nov 26 09:51:51 2018 -050014:25
frayI'm looking at master14:25
frayglibc 2.3014:25
fraycommit 8cebd4ffe67bf94508809ea0caa02a4f1d52e8b114:25
frayAuthor: Carlos O'Donell <>14:25
frayDate:   Mon Nov 26 09:51:51 2018 -050014:25
fray    Add --no-hard-links option to localedef (bug 23923)14:25
yoctiBug was not found.14:25
jwesselI am on thud glibc-2.2814:25
jwesselWhich is why I don't have it.14:25
frayya, too old.. you'd need to backport it.. but the commit is -simple-14:26
frayit's not YP bugzilla this is upstream glibc14:26
jwesselAll you need to do, to make this problem go away is use that option + the hardlink program.14:26
fraylook at the glibc bugzilla14:26
frayI just don't know if 'hardlink' is now standard or not14:26
jwesselThen your mystery localedef issue is fixed :-)14:26
fray(on non-Fedora/RH based systems)14:26
jwesselIt is on ubuntu, if you install it.14:26
frayUbuntu 18.04 does NOT have hardlink14:27
jwesselIt will have to go into host-native.14:27
frayok.. so it's optional there..14:27
jwessel% cat /etc/issue14:27
jwesselUbuntu 18.04.2 LTS \n \l14:27
frayya.. sounds like a host-native...14:27
jwesselIt is optional.14:27
jwesselI shudder to think how far we have to back port, but I assume we need to at least fix thud.14:27
frayFedora claims it's part of util-linux, but I don't know if it's 'real' util-linux or Fedora's version14:27
frayyes.. we need to at least fix thud..14:28
frayI'd say fix master first, make that work and backport to warrior and thud14:28
fraysince we now know the problem is common14:28
fray(and at least master has a partial fix already implemented from upstream, warrior probably does?)14:28
jwesselThe generation should be faster with the checking gutted too.14:28
*** TobSnyder <TobSnyder!> has quit IRC14:29
jwesselThat code was _really_ not efficient, or parallel safe.14:29
frayso the 'hard part' is getting 'hardlink' (or finding something equivalent)14:30
jwesselI would just use what works.14:31
frayIf that works14:31
frayI was looking at the Fedora version.. and it went 'away'14:31
jwesselUbuntu used 0.2.0 the 14.04 release.14:32
jwessel0.3 is the latest.14:32
RPjwessel: nice to finally have some answers, nice work!14:32
frayFedora's version was 1.314:32
jwesselThat site states fedora has a different version.14:32
jwesselIt is different code.14:33
frayremoved in Feb of this year I think..14:33
jwesselI suspect there are other programs that do the same thing.14:33
jwesselhardlink from the jak linux just works.14:33
fraythat appears to be the latest version14:34
jwesselMakes no difference to me where we get it from.14:35
jwesselI am just happy we don't have to fix pseudo.14:35
frayI did just verify.. util-linux-2.34 includes 'hardlink'14:35
*** yates <yates!> has joined #yocto14:36
frayversion included looks like it's the Red Hat version.. (or at least a refactored one)14:36
yatesthis just started showing up when building this image. i have no idea why14:36
yatesclues would be appreciated14:36
yatesthe image was building fine a week or two ago..14:37
fraynot sure.. pyro is pretty old so it might justbe a simple bug14:37
frayI'd simply start with rebuilding (clean and build) the locale's14:38
yatesfray: are you talking to me?14:38
RPfray, jwessel: please keep in mind that adding a util-linux-native depends to glibc-locale would be pretty bad for build performance14:39
RPIf there is a python script we can use it would be much more efficient14:40
* RP finally figures out the one line fix bitbake needs :/14:40
frayI'm more thinking we add just 'hardlink' until a time comes when all of the regular hosts have updated to more modern util-linux14:40
frayhardlink itself is very small.. 430 something lines14:40
RPfray: right, fine by me14:41
fraybut with that said, re-implementing those 430 lines in python should be possible, just don't know the performance impact14:41
fraythis isn't rocket science.. it basically parses the filesystem and checksums everything into a linked list.. with the items inode, device, checksum and filename..14:42
frayit searches that list for the device and checksum, and if they are the same then hardlinks14:42
frayif item is unknown it adds to the data structure and moves on to the next file14:42
RPfray: we already do similar kinds of things elsewhere, its trivial in python14:43
fraylooks like the hardest thing in the code is there is something that if the file is in-use it renames it and removes it when it hardlinks to keep the old file alive..14:44
fraythat shouldn't be a problem14:44
RP - so simple yet problematic14:44
frayRP is there exisitng oe code that does checksum and hardlink behavior or would this be a new lib/oe function?14:45
fray(I know we have copy or hardlink functions)14:46
jwesselDude... Don't do it python.14:47
jwesselUse the C one.  It is faster than python is ever going to be.14:48
fraybut it' one more thing to compile and support that's my only concern14:48
jwesselHow is it any better than having to compile cross-localedef14:49
frayand for the python it would run in the native bitbake context, not execute 'another thing'.. (conceptually)14:49
jwesselIf anything add it in there.14:49
fraycause that is existing14:49
frayeither we have to add a new native recipe for hardlink itself (which may someday go away) or we implement it as a library function14:49
frayeither way will work -- but it moves around support items14:49
RPjwessel: I suspect there isn't that much difference C to python for this...14:50
fraybuiling native-util-linux is too big and has been avoided for a long time14:50
jwesselThat is the code you are looking for btw.14:50
RPjwessel: I say this as someone who thought bitbake in C might be faster, once.14:51
*** berton <berton!~berton@> has quit IRC14:51
jwesselI guess that older version is not python 3 compliant.14:53
jwesselIt says to run 2to3 on it...14:53
RPJPEW: I had to drop the extra stats code, it was causing the autobuilder to lock up :(14:53
fraythat python one looks like it doesn't hash once and store, but instead compares each time14:53
fraythat's going to be less efficient then hash and store14:53
RPyes, but that is just bad design14:53
RP(not python per se)14:53
jwesselI'd be inclined to just add the c file to the cross-localedef package.14:54
fraythe modern 'hardlink' have two hash points.. one based on size, owner, group, mode.. and then one based on content..14:54
jwesselThe c file does exactly what is needed, and you have to compile the cross-localedef anyway.14:55
fraysince if the first hash doesn't match, you can skip the second (kind of)...14:55
jwesselJust name it cross-localedef-hardlink.14:55
fraythen if you find a match in the first, you have to hash them both to compare -- but then youc an cache those for the future..14:55
jwesselThe modern one is much more effecient.14:55
frayyes.. adding it to cross-localdef is fine14:55
fray(least in my opinion) key is to be careful that it's efficient in what it's doing14:55
jwesselTake the path of least resistance, call it a day.14:55
frayI'd take the util-linux (modern) hardlink, and put it into cross-localdef for now.. it'd be the easiest to maintain.. (I think)14:56
jwesselYou'll spend time debugging and porting it to python otherwise.  I am saying you can't make it portable etc... but why bother.   The hardlink compiles on everything.14:57
jwesseloops.  I meant you can make it portable with python, but I don't believe the effort is worth it.14:57
RPI was working on the assumption there is a sane python version around which there isn't14:57
RPif we can get a C version and use that, fine14:58
jwesselYou might as well just stick it in cross-localedef.   No new package etc...  Don't know if the license is compatible.14:59
jwesselDidn't look at that bit.14:59
frayit's GPLv2 so should be fine..14:59
fray(latest util-linux version)14:59
fray'er.. ya .c not .115:00
fray#include "c.h"15:01
fray#include "xalloc.h"15:01
fray#include "nls.h"15:01
fray#include "closestream.h"15:01
fraythat bit might need to be ported or hacked aorund.. but otherwise it looks entirely standard15:01
fray(and I don't see any reason to build it with the PCRE support which is newer then the old standalone version)15:02
JaMakhem: I've sent one more RDEPENDS fix which might look like duplicate, but it's for another package in scsirastools15:02
yatesremoving tmp and rebuilding got rid of that15:02
JaMakhem: feel free to squash them if you want, It was detected in follow-up builds, that's the only reason why I haven't put both in first commit15:03
*** Bunio_FH <Bunio_FH!> has quit IRC15:08
*** nabokov <nabokov!~armand@> has joined #yocto15:11
*** JPEW_ <JPEW_!~JPEW@2605:a601:21d1:5200:620a:c230:ada8:3135> has quit IRC15:14
*** kroon <kroon!> has joined #yocto15:15
*** asabil <asabil!> has quit IRC15:30
khemJaMa:yes queued up15:30
khemJaMa: the gcc problem you are seeing, whats best way to reproduce it, does it only happen with multilib builds15:31
*** asabil <asabil!> has joined #yocto15:32
khemJaMa: was the full build where these qa errors are listed15:33
khemhoping your series will fix most if not all of them15:33
JaMakhem: I've fixed all which I was seeing in my builds15:39
khemdidn't you realize I was hinting you to fix mine too :)15:40
JaMae.g. hplip shown in ours is blacklisted in all our builds, because needs lp group which we don't have15:40
JaMawasn't alex going to fix them? :)15:40
khemI see15:40
JaMaand mime-construct burn-boot libextutils-parsexs-perl webmin recipes aren't in my world15:42
JaMawill check the gomp issue on some non-multilib config soon15:43
*** kroon <kroon!> has quit IRC15:57
*** kroon <kroon!> has joined #yocto15:58
khemI will report the results and see if Alex helps15:58
*** moto-timo <moto-timo!~ttorling@fsf/member/moto-timo> has joined #yocto15:59
*** berton <berton!~berton@> has joined #yocto16:01
*** WillMiles <WillMiles!> has joined #yocto16:01
*** florian <florian!~florian_k@Maemo/community/contributor/florian> has quit IRC16:27
*** BobPungartnik <BobPungartnik!~BobPungar@> has quit IRC16:30
*** BobPungartnik <BobPungartnik!~BobPungar@> has joined #yocto16:30
JaMaRP: maybe I'm reading the profile badly, but to me it looks like worker thread was waiting in select most of the time and then finished executing tasks relatively quickly16:31
JaMae.g. latest builds shows 4082.736 seconds in main process, 3977.939 seconds in worker and16:31
JaMa  877733 3294.319    0.004 3294.319    0.004 {built-in method}16:32
JaMa   58394  366.063    0.006  366.063    0.006 {built-in method posix.fork}16:32
JaMa    58394    6.290    0.000  493.595    0.008 bitbake/bin/bitbake-worker:431(handle_runtask)16:32
JaMaif it was handling tasks for 8 minutes from 68, then it seems reasonable16:33
JaMakhem: the omp.h issue is reproducible without multilib as well, should I try it without 9.2.0 upgrade or is it definitely the gcc-runtime change? (I belive it is)16:36
*** litb <litb!> has quit IRC16:42
RPJaMa: you read that correctly, it does spend most of its time waiting16:43
RPJaMa: turnaround time for tasks is important too though16:44
Crofton|workRewrite bitbake in go16:45
JaMarewrite go in bitbake16:45
*** asabil <asabil!> has quit IRC16:48
Crofton|workeven better16:48
*** pung_ <pung_!~BobPungar@> has joined #yocto16:52
*** BobPungartnik <BobPungartnik!~BobPungar@> has quit IRC16:55
RPwe just add the new sentience module to bitbake and make it self hosting...16:57
*** kroon <kroon!> has quit IRC17:00
jwesselThe hardlink program takes 2.3 seconds in pseudo context, and .7 seconds outside of pseudo.17:01
jwesselThat is still way faster than what it takes for the cross-localedef17:02
RPjwessel: that is a nice win!17:03
*** vineela <vineela!~vtummala@> has joined #yocto17:03
* RP -> away (most of the weekend to)17:04
*** pung_ <pung_!~BobPungar@> has quit IRC17:06
*** pung_ <pung_!~BobPungar@> has joined #yocto17:06
Crofton|workWould more machine learning help?17:06
*** falstaff <falstaff!~quassel@> has joined #yocto17:12
jwesselI wonder what the bugzilla id is for the glibc-locale issue.17:17
yoctiBug 12434: normal, Medium+, 2.8 M3, randy.macleod, ACCEPTED , pseudo: Incorrect UID/GID in packaged files17:17
jwesselI am testing a fix for it.17:18
jwesselI'll send it out as an RFC, since it is kind of ugly looking.17:18
JaMacool, let me know if you want me to test it as well17:18
jwesselFor sure, we'll need it tested.17:18
JaMapseudo standard :)17:18
jwesselI'll have to forward port it when I am done because I did all the work on thud, where I am stuck at the moment.17:19
*** falstaff <falstaff!~quassel@> has quit IRC17:24
*** Bunio_FH <Bunio_FH!> has joined #yocto17:26
khemJaMa:I have a fix that I am testing17:28
*** leitao <leitao!~leitao@2620:10d:c092:200::1:73da> has quit IRC17:29
JaMakhem: that was quick! thanks will include it in next world build17:41
JaMakhem: btw for those gcc patches it would be useful to use "git format-patch --no-signature --no-numbered" when refreshing them for new version (like devtool does now)17:44
*** JPEW <JPEW!cc4da337@> has quit IRC17:45
JaMaanyone hiding bullet3 or netpbm recipes from layerindex?17:46
khemJaMa: thats a good idea, it has bothered me a bit but not enough17:49
khemI keep a local fork for these patches17:50
khemand devtool workflow does not fit in there17:50
*** JPEW_ <JPEW_!~JPEW@2605:a601:21d1:5200:620a:c230:ada8:3135> has joined #yocto17:57
JaMakhem: understood, I use the same e.g. with qt* patches, I've noticed only because when replacing 9.2.0 v1 with v2 there were much more changes than expected just because many patches were re-numbered after removal of one and then the include path change changed the git signature as well :)18:07
*** pung_ <pung_!~BobPungar@> has quit IRC18:19
*** pung_ <pung_!~BobPungar@> has joined #yocto18:19
*** JPEW_ is now known as JPEW18:25
JPEWRP: Strange that the stat code caused lockups.18:26
*** T_UNIX <T_UNIX!uid218288@gateway/web/> has quit IRC18:51
*** pung_ <pung_!~BobPungar@> has quit IRC18:52
*** pung_ <pung_!~BobPungar@> has joined #yocto18:52
*** asabil <asabil!~asabil@2a01:79d:7375:2ca4:6410:7de1:3184:de99> has joined #yocto19:05
mischiefcan i put INITRAMFS_IMAGE somewhere other than build/conf/local.conf?19:06
*** asabil <asabil!~asabil@2a01:79d:7375:2ca4:6410:7de1:3184:de99> has quit IRC19:08
*** vmeson <vmeson!> has quit IRC19:14
*** BCMM <BCMM!~BCMM@unaffiliated/bcmm> has quit IRC19:15
robbawebbamischief: yup! I have INITRAMFS_IMAGE defined in a distro conf file in my layer. I'm guessing you could define INITRAMFS_IMAGE in an image recipe as well, but I've never tried it. I would definitely appreciate another perspective on that possibility.19:16
mischiefwe're starting to use yocto with pretty much 0 experience, so i'm sure i've done something wrong19:17
*** JPEW <JPEW!~JPEW@2605:a601:21d1:5200:620a:c230:ada8:3135> has quit IRC19:17
mischiefi made a directory for TEMPLATECONF and that is checked into git but build/conf/local.conf is not in git, where INITRAMFS_IMAGE is currently defined19:17
mischiefrobbawebba: do you have something i can read about making distro layers/confs?19:18
robbawebbamischief: sure! give me a few minutes to find the right links19:18
mischiefthank you.. i've asked a few questions here previously but you are the first to answer any of them :-)19:19
robbawebbamischief: This entire section, section 7.1, contains tons of information on how to create and manage layers. Please let me know if you have any other questions!19:22
*** asabil <asabil!~asabil@2a01:79d:7375:2ca4:6410:7de1:3184:de99> has joined #yocto19:22
robbawebbaI'll try and find info on Distros next, but you'll need a layer before you create a custom distro :) so the layer stuff should keep you busy in the meantime19:23
*** falstaff <falstaff!~quassel@> has joined #yocto19:24
mischiefright now we have a git repo with meta-$COMPANY with basically just our kernel recipe, and poky as a submodule. not sure if this is right but i've been able to make at least a kernel build.19:24
mischiefso i guess we have one layer.19:26
robbawebbamischief: nice! That's essentially a layer. Did you add that meta-$COMPANY to your build/conf/bblayers.conf file, either manually or using the bitbake-layers tool?19:26
robbawebbaalso, here's the distro documentation:
robbawebbacool! Then you have a layer :)19:26
mischiefi put that in meta-$COMPANY/conf/bblayers.conf.sample19:26
mischiefso unfortunately if i modify it in build/conf then it is not tracked in git.. but if i modify bblayers.conf.sample it would require a user to re-initialize build/conf, correct?19:27
mischiefis there no better way to track build/conf changes in revision control?19:29
mischief(i.e, for INITRAMFS_IMAGE)19:29
robbawebbaand as long as the command `bitbake-layers show-layers` command shows your layer, then your're good! Plus, you mentioned that you're able to build your kernel recipe that's in your layer, so that's another sign that bitbake can find your layer19:29
robbawebbaand yes, there are a couple of options, the TEMPLATECONF mechanism is one that you already mentioned19:30
mischiefwould it be a Bad Idea (tm) to just put build/conf/*.conf in the git repo?19:31
robbawebbayup, I believe it is not a good practice19:32
Crofton|workI'm pretty sure that would be awful :)19:32
robbawebbamischief: are you aware that you can put local.conf.sample in your layer, just like bblayers.conf.sample?19:33
mischiefrobbawebba: that is currently what i am doing19:33
mischiefbut my concern is that if i change a variable and commit that to git, other developers would not see it unless they re-initialize build/conf from TEMPLATECONF.19:33
Crofton|workgood concern19:34
Crofton|workauto.conf and site.conf are also searched19:34
Crofton|workin the BBPATH19:34
Crofton|workI use that so all build on one machine check .oe in my home dir19:35
Crofton|workand I can insert something like site.conf where I might point to a local source mirros19:35
mischiefsome of my coworkers are going to embedded linux conference and now i am somewhat regretting not attending ._.19:37
Crofton|workas well you should19:37
Crofton|workwell I am not going either, but I will be at ELCE19:37
robbawebbaCrofton|work: nice! So mischief can just add site.conf to his layer's conf directory and check those configuration options into version control?19:38
Crofton|workwith careful tweaking of BBPATH19:38
mischiefmy BBPATH is just $TOPDIR19:39
Crofton|workand use custom distro.conf also19:39
Crofton|workdo something like this and have actual content with conf files also19:39
Crofton|workyou want to split things that imapct product from things that are truly local19:40
Crofton|worklike local source irrors, DL_DIR etc19:40
Crofton|workyes I ahve screwed up having important stuff in local.conf19:40
mischiefCrofton|work: currently we have just one git repo containing meta-$COMPANY/{conf,recipes-*} and poky/ as a submodule19:41
*** BCMM <BCMM!~BCMM@unaffiliated/bcmm> has joined #yocto19:41
*** Bunio_FH <Bunio_FH!> has quit IRC19:41
Crofton|workpersonally, I do what I do and drop the meta-poky stuff and use your own distro.conf19:43
Crofton|workand keep the settings there19:43
Crofton|workdepending on what they are of course19:43
Crofton|workand we are sorry this is really well spelt out for people19:43
mischiefi'm not sure my skill level with this stuff is high enough to jump the boat on poky just yet :-)19:44
Crofton|workyou are asking the questions that suggest you are there19:45
Crofton|workYou are basically changing stuff set in poky.conf?19:45
* Crofton|work has never used poky :)19:45
kergothcreating a distro .conf is not difficult. cp poky.conf to your own, go19:47
mischiefi see, you are suggesting removing meta-poky layer from layers.conf? i thought you meant remove the poky/ submodule from my git repo :s19:47
*** Bunio_FH <Bunio_FH!> has joined #yocto19:49
*** kaspter <kaspter!~Instantbi@> has quit IRC19:53
*** kaspter <kaspter!~Instantbi@> has joined #yocto19:54
*** asabil <asabil!~asabil@2a01:79d:7375:2ca4:6410:7de1:3184:de99> has quit IRC19:56
yoctiNew news from stackoverflow: Yacto/BitBake new dir does not show up <>20:02
*** vmeson <vmeson!> has joined #yocto20:06
*** asabil <asabil!~asabil@2a01:79d:7375:2ca4:6410:7de1:3184:de99> has joined #yocto20:11
*** asabil <asabil!~asabil@2a01:79d:7375:2ca4:6410:7de1:3184:de99> has quit IRC20:14
*** WillMiles <WillMiles!> has quit IRC20:24
*** imm_ <imm_!~imm_@unaffiliated/imm/x-7821412> has joined #yocto20:25
*** florian <florian!~florian_k@Maemo/community/contributor/florian> has joined #yocto20:27
*** JaMa <JaMa!> has quit IRC20:29
*** falstaff <falstaff!~quassel@> has quit IRC20:32
*** JPEW <JPEW!~JPEW@2605:a601:21d1:5200:620a:c230:ada8:3135> has joined #yocto20:48
__angeloin a machine.conf, is the field "#@SOC:" used for any special purpose or it's just a comment ?20:53
khemmischief: in conf/ dir you can create a site.conf file and put your customizations in that file20:59
khem__angelo:I think linux yocto uses it20:59
khemi mean linux-yocto kernel recipe21:00
__angelokhem, ok. mmm i have a linux-mystuff recipe21:00
khemthen it wont bother you21:00
*** nagio <nagio!> has quit IRC21:01
*** agust <agust!> has quit IRC21:02
__angelokhem, thanks !รน21:02
*** agust <agust!> has joined #yocto21:02
khemCrofton|work and mischief take a look here
khemthats one way to solve it21:05
jwesselI put the RFC patch set out there to solve the glibc-locale issue.21:07
*** leitao <leitao!~leitao@2a02:c7f:a63:f000:83b:ce0f:f5ae:3633> has joined #yocto21:08
*** nagio <nagio!> has joined #yocto21:09
jwesselWe could certainly benefit from some testing on it for those who were most affected by this issue.21:09
jwesselI only saw it once, myself.  But our automated build cluster would have a number of failures each week.21:09
jwesselThere could be other corner cases, but they will certainly not be the same as the one the patch set corrects.21:09
RPJPEW: I don't understand it. We also have a small race with the hash reporting and the generation of the sstate package. I think the latter is lagging the former and leading to a race :/21:10
khemjwessel:cool.. pointer ?21:11
JPEWRP: That... shouldn't be possible?21:11
jwessel[OE-core] [RFC PATCH 1/2] cross-localedef-native: Add hardlink resolver from util-linux21:11
jwesselIt is a two patch set with some explanation about the problem, but not about how it was tracked down.21:12
JPEWRP: Unless we are thinking of different things21:12
jwesselI had to use a modified version of pseudo, which I'd likely use in the future, if we have other such problems.21:12
RPjwessel: I think this may be related to all of:
yoctiBug 11551: normal, Medium+, 2.99, ross.burton, NEW , inconsistent dependencies in glibc-locale packages21:13
yoctiBug 11299: normal, Medium+, 3.1, Qi.Chen, NEW , glibc-locale warning [host-user-contaminated]21:13
yoctiBug 12664: normal, Medium+, 2.8 M4, ross.burton, NEW , glibc-binary-localedata packages may contain .tmp files21:13
yoctiBug 12434: normal, Medium+, 2.8 M3, randy.macleod, ACCEPTED , pseudo: Incorrect UID/GID in packaged files21:13
RPJPEW: is interesting. There is obviously a bitbake problem with the rebuild of encodings but why wasn't it in sstate?21:14
jwesselI don't doubt there are a number of cases to reference for this particular problem.21:14
RPjwessel: the .tmp thing in particular in hindsight is a smoking gun21:14
jwesselI don't think I ever saw a build with .tmp files though.21:15
jwesselSomething really had to go wrong to have that happen.21:15
jwesselBut given the nature of the race.  It is probably possible.21:15
RPjwessel: there were references to X.tmp files in that log you shared weren't there?21:15
jwesselSure but that is the mechanics of how the linkage works.21:16
RPjwessel: I suspect somehow that leaks. Perhaps this doesn't solve it, not sure21:16
jwesselOh it solves it.21:16
jwesselThere are no more .tmp files at all with this change.21:16
jwesselI can see how a tmp file could have leaked.21:17
RPjwessel: right. We've only seen that .tmp thing on very few occasions21:17
jwesselThe timing window on that one was way smaller for failure, than promoting the link to a file.21:17
jwesselWhich was happening in ever single build.21:17
jwesselWe just didn't know it.21:17
jwesselIf I put some sleeps right after the unlink.  You can make it explode into flames easily.21:18
JPEWRP: How many AB nodes are hitting the hash server at the same time?21:19
jwesselBut there is no point anymore.  This is a concise type of failure, with a number of different symptoms.21:19
JPEWRP: ^ I'm trying to see if I can get the timeouts to reproduce locally21:19
jwesselI do think we should consider patching pseudo a bit to have the special logs.  But that is a back burner task.21:20
jwesselI don't know any other way to track down a problem like this, even when it is not pseudo that is the root cause.21:20
jwesselOnly an strace log could have told the story.   And those are even harder to read. :-/21:20
RPJPEW: ~40 builds21:22
RPjwessel: totally agreed. I'm just trying to think ahead about which bugs this covers21:22
RPjwessel: indeed, they're also huge21:22
jwesselProbably a number of them with glibc-locale21:23
khemjwessel: do we have to include hardlink files into cross-locale-def, or can it call the utility from util-linux-native and then we depend on util-linux-native that gives us maintained version of hardlink utility21:25
RPkhem: I just replied on list about that21:25
RPkhem: its a build performance nightmare if we add that dependency21:25
*** nabokov <nabokov!~armand@> has quit IRC21:26
RPkhem: I'd kind of preferred a simple python script but that didn't appear so simple21:27
*** berton <berton!~berton@> has quit IRC21:28
JPEWRP: I've forgotten; does bitbake check for an sstate tarball before adding the _setscene task to the runqueue? Perhaps the encoding unihash changed after it was added but before the task actually ran?21:28
jwesselI would have preferred to just depend on util-linux-native...  RP said it was too much of a problem.21:29
RPI'm just the person who's spent a lot of time trying to speed the builds up, what do I know :(21:30
jwesselI would have thought the host tools simply don't recompile too often.  RP, you have more data points than I.21:30
RPJPEW: I think you don't quite comprehend the runqueue as it isn't an actual list of tasks we're running, its a list of all tasks we could possibly run21:31
khemI think if its adding some significant bit then I agree but there is maintainance part as well21:31
jwesselSo I implemented what was asked.21:31
jwesselTo make it easier, should it need to be upgraded for some reason.  I did separate the patch to the raw version.21:31
khemit would not matter much for endusers since they hardly rebuild things so deep so often21:31
RPMy instinct says it will hurt things. Its based on us having done this in places before and knowing what the changes on performance in glibc-locale do to the build21:31
RPJPEW: in that sense if you're asking when we mark it as buildable, we'd check the sstate cache after something is marked as having a new unihash.21:33
jwesselI wouldn't worry to much about needing to replace it down the road.   The reasons for the patches over the years were all to add features.   Any kind of bug fixes were few and far between.21:33
jwesselThere are only so many ways to make a checksum and a series of calls to link() unlink()21:33
RPJPEW: so you're right, we should have already checked this was valid in sstate so its strange it suddenly wasn't21:33
jwesselThat being said, we can just use the host version 3-4 years from now, when util-linux is upreved in the distros.21:34
jwesselhardlink was a very recent addition from RedHat.21:34
RPkhem: it would hurt anyone building from scratch and people do that a fair bit and criticise the project for it21:34
JPEWRP: Ya, I think I'm confusing some terminology somewhere... need to read more code :)21:34
khemRP: I understand that but I dont know how much,21:35
RPkhem: based on experience it will hurt a fair bit, particularly on people with fewer cores21:36
khemand util-linux appears in many recipes as dependency21:36
kheme.g. python3-native needs util-linux-native21:37
khemdo we build glibc before that ?21:37
khemso my caution is to not create technical debt for ourselves unless absolutely required21:39
RPkhem: I suspect that is a recently introduced performance regression then :(21:39
RPkhem: If we don't care about build times, sure but I have done one heck of a lot of work to try and keep the bootstrap process clear of this kind of painpoint21:40
* RP notes systemd also has the dependency. On the plus side most of the rest of the system does not21:40
RPwe patch it out of quilt-native for example21:40
RPkhem: glibc is kind of special as its the choke point on the task graph. The locales connect to it not as a direct blocker but influencing packaging21:42
khemRP:we do care, but if python3-native is required early enough and its building util-linux-native for us, then we can certainly use it in cross-localedef isnt it or am I missing something21:42
RPkhem: IMO we should fix python3-native, not use it as a justification for slowing down the build further21:43
khemits not justifying it, its calling out what is is21:47
khemI looked a bit of forever technical debt that we will carry and it does come at some cost21:48
RPkhem: its not forever, its until we can depend on it from the host systems21:50
RPthis is why I also preferred the python option but what do I know21:51
frayat one point util-linux-native was also a 'provided' thing.. stuff could specify it, but it never actually built21:53
khemperhaps uninative should capture all native dependencies and we should have none of those during build maybe21:55
khemso which hosts dont yet provide it21:56
RPkhem: that isn't the point of uninative but was the point of buildtools-tarball21:56
frayI assume current Debian doesn't as well, but not sure21:56
frayFedora and RHEL/CentOS have for a very long time21:57
khemRP: uninative is the right answer a collection of packages which are distro independendent and are downloaded automatically during build there is no third party involved21:57
RPkhem: it is not the problem uninative is designed to solve21:58
khemcurrent debian/buster is going to live for next 5 years :( I was hoping it was some old stinky centos21:58
khemRP: I understand that21:58
RPkhem: it could be extended to solve that problem sure but its a very different one21:58
khemI am asking for widening its scope21:59
RPand we do already have a different solution there called buildtools-tarball, its just not downloaded automatically21:59
fraythis would be a candidate for buildtools-tarball, but reality is that we'd then need to make hardlink a sanity checked item...21:59
frayand the out-of-box-experience for this one binary is better this way (for now)...22:00
RPkhem: after all the complaints about naming we need to try and be really clear with names. uninative has a specific purpose and I'd prefer not to break that...22:00
khemI think its something we should do, otherwise we are spreading ourselves into supporing host distros22:00
RPkhem: we already do that22:01
fraythat's why (in this case) the tooling goes into the recipe and isn't distributed elsewhere.. it's local to limit support22:01
khemRP:question is should we keep doing that or have some solution to abstract on host distro22:01
fraywe (WR) use the buildtools-tarball to bring old OSes up to date to be able to build..  but since hardlink is -so- new, even 'current' versions don't have the missing functionality to we're stuck for a bit22:02
RPkhem: good question. I'd just like to know who is going to do the work in developing it and making it work and handling the transition and its relative priority to everything else22:03
RPmeanwhile it would be nice to fix the locale corruption issue22:03
khemfray: thats understood, instead of making tools tarball an optional thing it should be made made mandatory and downloaded automatically during build22:04
*** agust <agust!> has quit IRC22:05
khemcurrently we build same number of native packages irrespective of buildtools tarball is my understanding, it just brings those arcane machine to something that can build modern OE22:06
RPkhem: people hate prebuilt binaries to the point I can't make uninative the default for OE-Core22:06
*** JPEW <JPEW!~JPEW@2605:a601:21d1:5200:620a:c230:ada8:3135> has quit IRC22:07
khemyou mentioned build times earliers thats why I brought this up, its the time we spend building python-native and other large time consuming enabling packages that irritates the user22:07
RPkhem: but buildtools-tarball doesn't help us with python-native or perl-native :(22:08
RPthose recipes are so tightly coupled you can't do much but have identical versions -native and target22:09
khemWe need to have option to rebuild the releases tarballs, otherwise shared state and buildtools tarballs are same, people should hate them equally22:09
RPpython-nativesdk which is what would be in the buildtools-tarball isn't good enough22:09
RPkhem: I do understand the attraction but I've been told binaries like that aren't acceptable22:10
khemand same folks would be using containers eh ?22:10
RPkhem: no idea. Perhaps people have come around to the idea of uninative now22:11
*** florian <florian!~florian_k@Maemo/community/contributor/florian> has quit IRC22:11
RPor nobody uses poky and therefore they don't see it, I don't know22:11
khemanyway, I would rest my case22:14
*** leitao <leitao!~leitao@2a02:c7f:a63:f000:83b:ce0f:f5ae:3633> has quit IRC22:15
fraywe use uninative at WR, but in the same way that poky does..22:16
fraywe use buildtools to augment old (broken) OSes..22:16
fraythe combo works really well for us an our customers..22:16
khemfray: do you use ASSUME_PROVIDED then to not build bunch of native packages becasue they now are in buildtools ?22:18
frayOnly the same ones that Poky has22:18
fraypython3, make, tar were the major things we've needed on older RHEL22:18
fray(RHEL / CentOS 7 is still REALLY common..)22:19
*** tprrt <tprrt!> has quit IRC22:20
*** falstaff <falstaff!~quassel@> has joined #yocto22:20
khemyes they are22:22
khemoh btw. glibc also needs python3 now during build, fun fun22:22
*** falstaff <falstaff!~quassel@> has quit IRC22:24
*** leitao <leitao!~leitao@2a02:c7f:a63:f000:83b:ce0f:f5ae:3633> has joined #yocto22:25
*** leitao <leitao!~leitao@2a02:c7f:a63:f000:83b:ce0f:f5ae:3633> has quit IRC22:30
*** falstaff <falstaff!~quassel@> has joined #yocto22:32
*** falstaff <falstaff!~quassel@> has quit IRC22:33
*** falstaff <falstaff!~quassel@> has joined #yocto22:39
*** falstaff <falstaff!~quassel@> has quit IRC22:40
*** georgem_home <georgem_home!uid210681@gateway/web/> has joined #yocto22:44
RPkhem: can that not come from the host system though?22:46
RPpython3-native is really for the python builds and python modules, not general python322:47
RPkhem: I guess a key question is any version reqs?22:52
mischiefi have a kernel module.. and it has RPROVIDES_${PN} += "kernel-module-${PN}" inside of it. if i add kernel-module-$foo to PACKAGE_INSTALL, i get an error saying 'no installation candidate' even though it also says it is a virtual package provided by ${PN}. why is that?22:54
mischiefif i just use the kernel module ${PN} for PACKAGE_INSTALL it seems to work22:55
*** khem <khem!~khem@unaffiliated/khem> has quit IRC23:09
*** elvispre <elvispre!~elvispre@2001:8b0:e0:884d:99db:5cdc:4b13:fabc> has quit IRC23:13
*** khem <khem!~khem@unaffiliated/khem> has joined #yocto23:16
*** BCMM <BCMM!~BCMM@unaffiliated/bcmm> has quit IRC23:31
*** vineela <vineela!~vtummala@> has quit IRC23:42
*** awe00 <awe00!~awe00@unaffiliated/awe00> has joined #yocto23:59

Generated by 2.11.0 by Marius Gedminas - find it at!