18:02:30 <bmwiedemann> #startmeeting
18:02:30 <bugbot> Meeting started Mon Dec 12 18:02:30 2011 UTC.  The chair is bmwiedemann. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:02:30 <bugbot> Useful Commands: #action #agreed #help #info #idea #link #topic.
18:02:32 <Holgi> hi bernhard
18:02:38 <Holgi> hi larry
18:02:46 <bmwiedemann> Hello everyone
18:02:46 <bmwiedemann> As always, today's agenda is on http://en.opensuse.org/openSUSE:Testing_meeting
18:02:46 <bmwiedemann> You can still add to the agenda on the wiki while the meeting is in progress
18:03:31 <bmwiedemann> #topic 12.1 testing experience - what worked, what didn't
18:03:45 <bmwiedemann> shall I start?
18:04:22 <LWFinger> Yes
18:04:57 <bmwiedemann> openQA has caught some issues that would very likely have remained unnoticed.
18:04:57 <bmwiedemann> a) last-minute bugs
18:04:57 <bmwiedemann> b) rarely tested things (e.g. zdup from 11.3)
18:05:24 <bmwiedemann> but I was still unsatisfied with 12.1-Beta and RC1 which suffered from major bugs
18:05:24 <bmwiedemann> https://bugzilla.novell.com/show_bug.cgi?id=720571 that remained open for 3 weeks, 9 dups
18:05:24 <bmwiedemann> https://bugzilla.novell.com/show_bug.cgi?id=725722 that was introduced on last day, 9 dups
18:05:24 <bmwiedemann> but overall, we have seen worse with earlier releases (such as several Milestones of 11.3 not booting)
18:05:28 <bugbot> openSUSE bug 720571 in openSUSE 12.1 (X.Org) "X11 evdev segfault" [Critical,Resolved: duplicate]
18:05:31 <bugbot> openSUSE bug 725722 in openSUSE 12.1 (Installation) "Package Manager during installation crashes yast" [Critical,Resolved: fixed]
18:06:31 <bmwiedemann> all this time spent on filing dups could have been spent to test/file other, new & interesting bugs
18:07:38 <bmwiedemann> at least for such last-minute-bugs - shouldn't we rather release some days later?
18:07:50 <LWFinger> I find b.n.o searches for bugs to be frustrating - even when I know the bug has been filed.
18:07:52 <bmwiedemann> at least for Beta&RC
18:08:22 <bmwiedemann> LWFinger: yep. I filed a bug on "not finding DUPLICATEs"
18:09:10 <bmwiedemann> often enough, you can go to Advanced and tell it to include RESOLVED bugs
18:10:04 <LWFinger> I also think the timing between releases in the final stages is too short.
18:10:58 <LWFinger> Are we trying too much for an 8 month GM cycle?
18:11:06 <Holgi> did you talk to coolo ?
18:11:20 <LWFinger> Holgi: No.
18:11:36 <Holgi> Bernhard ?
18:11:41 <bmwiedemann> coolo said, that after RC1, only bugfixes are included
18:12:25 <LWFinger> That is OK, but what about the ones introduced in RC1 and RC2. They are the killers.
18:13:06 <bmwiedemann> I think, the original planning had more time at the end, but some MS slipped and then things were rescheduled and squeezed together.
18:14:21 <Holgi> bmwiedemann: I do not understand what "only bugfixes are included"
18:14:30 <bmwiedemann> and as I understood it, coolo does not want the full freeze phase takes too long
18:14:31 <Holgi> bmwiedemann: should mean
18:15:08 <Holgi> bmwiedemann: infact bug reports do report problems
18:15:25 <Holgi> bmwiedemann: if fixes creates new problems
18:15:39 <Holgi> bmwiedemann: they may not be fixed then?
18:16:06 <bmwiedemann> yep. but it is supposed to remain more stable than in full factory mode when every new (beta)version gets in
18:16:47 <Holgi> bmwiedemann: I agree that we do not put in new features at that time
18:16:54 <Holgi> bmwiedemann: but I think nobody requested
18:17:19 <Holgi> bmwiedemann: what we are worrying about is that bugs are not fixed or fixes causes new problems
18:17:28 <bmwiedemann> I once calced that the shortest useful release-test-report-fix-cycle is three weeks.
18:19:09 <Holgi> I also have issues (intel graphics) that does not get fixed - as always, developer orders his bugs, renames it but do not fix anything
18:19:28 <LWFinger> The RC timing is a problem for the many testers that do not even start with a new release until RC1.
18:19:39 <bmwiedemann> we had 3 weeks between Beta and RC1, which was good, but between RC1-RC2 and RC2-GM was very little time
18:19:48 <Holgi> bnc#732308
18:20:12 <Holgi> bug 732308
18:20:25 <LWFinger> Holgi: Despite your issues with i915, the KMS video drivers in 12.1 are much improved.
18:20:26 <Holgi> bugbot does not like me
18:20:55 <Holgi> also I have random crashed when waking up
18:21:00 <Holgi> not report yet
18:21:58 <bmwiedemann> most of those could be upstream issues, and we don't always have enough developers on that... or maybe the bugs just don't get to the right people
18:22:22 <LWFinger> At least it seems to me that there are fewer complaints in the forums about graphics problems. For me, i915 (1 system) and nouveau (2 systems) both work.
18:23:36 <bmwiedemann> what do you guys think, what we should do again or do better for 12.2? so more time between RCs would be #1
18:24:48 <Holgi> to be honest - I have no idea
18:25:10 <Holgi> maybe we should change the pizza event: instead pizza for testing we offer pizza for bug fixing
18:25:44 <LWFinger> I think 12.1 was better than 11.4, but that was mostly openQA, and the rest of us can only bask in the glory.
18:25:45 <bmwiedemann> I think, the fixing did take place, because more developers actually ran 12.1 and saw bugs
18:25:53 <Holgi> regarding upstream: I would expect that such a bug report would go to solution "UPSTREAM" then
18:26:54 <bmwiedemann> but then people don't find the RESOLVED bug again
18:27:30 <Holgi> but what helps an open bug where nobody is working on, just collecting all the duplicates?
18:27:39 <bmwiedemann> how hard is it to setup a better bugtracker for openSUSE?
18:28:02 <bmwiedemann> yes, better open bugs, so that people see that it is known and still needs to be fixed.
18:28:11 <LWFinger> One thing that would help is to add the upstream developer to the bug report. For me, this happens with both Fedora and Ubuntu.
18:28:40 <bmwiedemann> if those have bnc accounts...
18:29:17 <LWFinger> Why cannot we add a non-user E-mail address?
18:29:50 <bmwiedemann> don't think we can.
18:29:52 <LWFinger> If the developer gets reports, they can always join later.
18:30:14 <LWFinger> Should we change that policy?
18:31:10 <bmwiedemann> problem is, we have little control on bnc. => own bugtracker?
18:31:40 <LWFinger> Is it controlled by Novell now?
18:32:42 <Holgi> it is controlled by Novell still
18:33:36 <Holgi> but I don't think this is the problem - I think you can make the developer make himself an account by opensing a bug in upstream bug tracker and just add URL of our bug
18:34:08 <Holgi> most will be curious enough ;-)
18:34:41 <Holgi> but opensuse developer needs to do this, bug reporter usually don't know
18:34:58 <Holgi> if it's an opensuse problem or upstream problem
18:35:04 <LWFinger> Unfortunately, with bugzilla.kernel.org still down, there is no upstream bug tracker for kernel issues. Still, one can post to LKML.
18:35:34 <Holgi> for me it's hard to understand that this is taking so long
18:35:48 <Holgi> we have same problems with LSB test suite
18:35:53 <Holgi> server still down
18:36:42 <LWFinger> They are being very careful to check every piece so that the penetration point is not reopened.
18:37:31 <LWFinger> I even lost my account at kernel.org because I live far away from any other kernel dev and have no way to get my key signed.
18:38:02 <Holgi> where do you live?
18:38:15 <bmwiedemann> Greg K-H offered to sign keys on conferences
18:38:17 <LWFinger> Kansas City, Missouri, USA.
18:38:41 <Holgi> I see
18:39:14 <LWFinger> As I no longer have an employer to pay, I no longer travel. If a conference comes to Kansas City or St. Louis, I'll go.
18:39:50 <Holgi> any plans for vacation already? ;-)
18:40:32 <LWFinger> If I tried to plan a vacation "to a conference", then it would cost me alimony.
18:40:47 <Holgi> LOL
18:41:42 <bmwiedemann> can we get back on topic?
18:41:52 <Holgi> sure
18:42:18 <Holgi> regarding keys, I had issues with enigmail accessing key servers
18:42:35 <Holgi> but this is fixed (but not released yet)
18:42:55 <Holgi> https://bugzilla.novell.com/show_bug.cgi?id=733002
18:42:57 <bmwiedemann> so opening bugs in upstream bugtrackers for upstream bugs, it certainly a good thing to do.
18:42:59 <bugbot> openSUSE bug 733002 in openSUSE 12.1 (Firefox) "Thunderbird/Enigmail is endless accessing PGP Keyserver" [Normal,Assigned]
18:43:31 <bmwiedemann> or just linking our and their bug together
18:43:48 <LWFinger> Is the issue with Kmail fixed?
18:44:42 <Holgi> which exactly?
18:44:51 <bmwiedemann> not sure. heard of a collegue having problems with migration to kmail2
18:45:43 <LWFinger> It seemed to cause a lot of traffic on the forum - I think it had to do with migrating old mail.
18:46:02 <Holgi> bugs to both problems are still open
18:46:26 <Holgi> e.a. https://bugzilla.novell.com/show_bug.cgi?id=728755
18:46:30 <bugbot> openSUSE bug 728755 in openSUSE 12.1 (KDE4 Applications) "KMail migration failure with 'Unable to fetch the resource collection'" [Critical,Assigned]
18:46:53 <bmwiedemann> there are two workarounds on http://en.opensuse.org/openSUSE:Most_annoying_bugs_12.1
18:46:53 <Holgi> https://bugzilla.novell.com/show_bug.cgi?id=731558
18:46:57 <bugbot> openSUSE bug 731558 in openSUSE 12.1 (KDE4 Applications) "new kmail (and its other processes) use a lot of memory and cpu and runs pretty instable" [Critical,Needinfo]
18:47:41 <LWFinger> Another bug is https://bugzilla.novell.com/show_bug.cgi?id=732692
18:47:45 <bugbot> openSUSE bug 732692 in openSUSE 12.1 (KDE4 Applications) "KMail migration (moving data to mysql databases) stops quietly: "Synchronizing email folder inbox" never finishes, messages never show up." [Critical,New]
18:47:59 * tigerfoot back
18:48:36 <tigerfoot> I've some report of user trying to boot livecd or even netinstall on i3+optimus =  Initializing gfx code directly, they never see the boot menu
18:49:09 <Holgi> hi tigerfoot
18:49:11 <tigerfoot> don't know if there's a workaround for those of them that can't desactivate the optimus in bios
18:50:07 * tigerfoot has read backlog ... bmwiedemann the idea to get a more longuer period between BETAS/RC sound really reasonable to me, but we need to convince coolo & end-users
18:51:03 <tigerfoot> bugzilla start to fill only after the release ... and now most of people are fixing 12.1 (yeah SP2 for sles too)  and it seems pretty much nobody care about factory
18:52:04 <bmwiedemann> factory will only get interesting next year when MS1 gets closer
18:53:18 <tigerfoot> at least for me, I think the most stable version we have actually in 11.4, this is bad in a sense ... We always act as fixing in released, so in a sense we are never preparing the future
18:53:25 <LWFinger> I don't know how to get end-users to start testing earlier - I did not make a lot of friends on the forum by asking "What is the bug number".
18:53:31 * tigerfoot opinion
18:53:47 <tigerfoot> LWFinger: :D
18:54:39 <tigerfoot> we need to market the fact that only feature that end-user have tested and reported have a chance to work. but that would also less last minutes changes
18:54:56 <tigerfoot> too much things has changed between B1 and RC2
18:55:16 <tigerfoot> was exactly the same with 11.3 & 11.4
18:55:48 * Holgi agrees
18:55:59 <Holgi> just think about systemd
18:56:15 <LWFinger> And the same will be true for all future releases.
18:56:28 <tigerfoot> Holgi: systemd was working in B1, then in release apache2 get not started, nor postfix etc ...
18:56:52 <tigerfoot> LWFinger: in that case, we just loose the battle ...
18:56:52 <Holgi> tigerfoot: but it was developed within beta cycle
18:57:16 <Holgi> as a solution we need a hard feature freeze deadline
18:57:36 <tigerfoot> Holgi: I really would like that.
18:57:50 <LWFinger> I also had problems with systemd - at least there was a workaround with SystemV.
18:58:33 <Holgi> but to me it looks like, during milestone period things are happening that should usually happen during factory time
18:58:41 <tigerfoot> LWFinger: I've followed all the systemd cycle, fight a lot during Mx, then get it working nicely in B1, then start to be junked after ...
18:58:55 <Holgi> during beta cycle things are happening that should happen during milestone time
18:59:10 <Holgi> and during rc period some things are happening from beta time
18:59:42 <tigerfoot> I don't know really if there's a solution ...
19:00:19 <tigerfoot> actually it also a bit a mess in obs, like rules changing all the time (once we have to build in -devel, now it's directly factory etc ... )
19:00:22 <LWFinger> Were those cases the result of bugs in fixes that should have been in B or RC?
19:00:42 <tigerfoot> some strict rules and schema of how we work has to be settle I think ...
19:01:16 <tigerfoot> LWFinger: apache2 not starting was fixed between me & fcrozat in August
19:01:42 <tigerfoot> now people have apache2 not starting by default if it is set on ...
19:02:08 <LWFinger> Why?
19:05:47 <LWFinger> Systemd has problems with NFS mounts when using wireless. After changing them to automount, all is good.
19:06:19 <tigerfoot> LWFinger: when I try to run automount in yast2, it ask me to setup ldap :-) why I don't have
19:06:24 <tigerfoot> which
19:07:07 <LWFinger> I admit, I did it from the command line. I did not know that YaST could do it.
19:08:15 <bmwiedemann> but how to make it better? I expect systemd to mature more during 12.2 cycle, so we will likely not see these problems again (but grub2 is coming ;)
19:09:45 <tigerfoot> bmwiedemann: for 12.2 what we really need is grub2 (or able to use efi, hdd >3.5TB etc) + drakcut/plymouth so we can drop lot of shit that will goes wrong with systemd
19:10:03 <tigerfoot> I found systemd quite usefull, working great once setup correctly ...
19:10:31 <LWFinger> Yes, and I expect problems for those that dual-boot with Windows. Fortunately, not a problem for me.
19:10:51 <bmwiedemann> can't grub2 chainload anymore?
19:10:54 <tigerfoot> There's a lot of report about crypt+systemd, that need to be fixed too (also how to make encryption with btrfs :-)
19:11:15 <LWFinger> I don't know the details, just what I have read.
19:12:18 <LWFinger> tigerfoot: There is a new systemd that is supposed to fix that.
19:12:54 <LWFinger> Any ideas when btrfs will get a proper fsck?
19:14:35 <bmwiedemann> not this year
19:15:00 <LWFinger> I think that the btrfs option on the installation page should have a warning. The noobs are the ones modt likely to trash their fs.
19:15:22 <tigerfoot> :D
19:16:24 <bmwiedemann> yep. it's dangerous
19:17:12 <LWFinger> I had one response to my newsletter item about this meeting. That user noted that the final graphics are usually one of the last things.
19:17:55 <LWFinger> He usually has many installs, and would like a change in graphics beginning with MS1.
19:18:21 <LWFinger> Would a color change, or something else that is easy be possible?
19:18:22 <bmwiedemann> we want that in 12.2 MS1. I already have a great draft for fallback artwork :D
19:18:42 <bmwiedemann> http://www.zq1.de/~bernhard/temp/Kroete1-122MS1v4.png
19:19:23 <LWFinger> That would certainly catch ones eye.
19:20:56 <bmwiedemann> so hopefully, all 12.2 releases will look different than 12.1
19:22:03 <LWFinger> It is not a problem for me as my early testing is with VB, and I keep the VM machine labels up to date.
19:22:51 <bmwiedemann> I use kvm a lot. and I guess, this will help avoid confusion in many places.
19:26:28 <bmwiedemann> any other proposals what to improve or what to repeat?
19:28:49 <bmwiedemann> then, we shall move to our next topic
19:29:01 <LWFinger> tigerfoot: It appears that the only support that YaST has for automount is with LDAP. That needs to be addressed. I use "direct mapping" for my mounts.
19:29:20 <bmwiedemann> #topic next meeting
19:29:56 <bmwiedemann> how about 2012-01-16 ?
19:30:06 <bmwiedemann> should still be before MS1
19:30:26 <bmwiedemann> or we meet afterwards, to have more to discuss
19:31:18 <LWFinger> One thing I would like to discuss is the makeup and the future of the TCT, but not today.
19:32:04 <bmwiedemann> yes, I suppose, there are some things that could change
19:32:08 <Holgi> bmwiedemann: when in M1 ?
19:32:12 <Holgi> bmwiedemann: when is M1 ?
19:32:33 <bmwiedemann> Roadmap is not out yet, but my guess is end of January
19:33:35 <Holgi> ok
19:33:45 <LWFinger> January 16 is OK with me.
19:34:19 <Holgi> for me as well
19:34:19 <tigerfoot> should work for me too
19:34:33 <bmwiedemann> OK. so we shall meet on 2012-01-16 18:00 UTC again
19:35:00 <bmwiedemann> #topic open discussion
19:36:19 <bmwiedemann> #info will push to have different artwork in 12.2-MS1
19:36:48 <LWFinger> I have been note taking on things to put in newsletter. At this point, I have (1)suggestions on how to get testers started earlier,
19:36:59 <bmwiedemann> #info want to have more time between RCs (3 weeks)
19:37:26 <LWFinger> How to get them to file bug reports, and (2) distinctive artworl early in release. Anything else?
19:38:56 <bmwiedemann> sometimes, I had wished for a place, where I could simply drop some issue I stumbled upon when I didn't have time to file a bug... would need some people to monitor that and maybe try to reproduce & file
19:39:22 <bmwiedemann> firefox betas had an nice feedback button integrated for that
19:40:09 <LWFinger> bmwiedemann: The forum for beta would be a good place for that kind of thing.
19:41:05 <bmwiedemann> sometimes the forum was acting strange on me, though
19:41:43 <LWFinger> How so? I usually read it with NNTP, and rarely soo the web interface.
19:41:52 <LWFinger> s/soo/see/
19:42:14 <bmwiedemann> I only read it in browser - and sometimes it wanted me to login and then again and again
19:42:39 <LWFinger> I think that is fixed now.
19:43:01 <bmwiedemann> good :)
19:46:42 <bmwiedemann> what ideas did you have to attract more people to test and file?
19:48:10 <LWFinger> I don't really have any. We need a good statement of what to do - most people don't file bugs, and too many start late in the cycle.
19:48:45 <LWFinger> It seems that changing to a Beta did not help.
19:49:06 <bmwiedemann> so we should tell them more about what happens if too few people test
19:49:23 <LWFinger> Yes. :)
19:49:50 <LWFinger> One thing I did note - When Linus found a bug, it got fixed quickly.
19:50:03 <Holgi> LOL
19:50:09 <bmwiedemann> for one he supplied a patch. that is easy to apply
19:50:36 <bmwiedemann> but most reporters don't include a patch
19:51:15 <LWFinger> Which is why I always try to supply a patch for Linus's bugs, but I'm not usually successful.
19:51:37 <LWFinger> But then, I don't get that many chances.
19:54:28 <bmwiedemann> fixing bugs quickly would be a good way to motivate testers (but it is not always possible)
19:55:16 <LWFinger> Having them fixed before GM should also motivate them.
19:56:49 <bmwiedemann> yes. but fast feedback also feels nice. e.g. a "thanks for reporting this - I can reproduce it and will work on it as soon as possible" message would be nice sometimes
19:57:40 <LWFinger> Why does triage take so long?
19:58:51 <bmwiedemann> a) because many bugs first go to screening team
19:59:03 <bmwiedemann> b) because developers don't have comparable setup
19:59:15 <bmwiedemann> or are busy at that time
20:01:20 <bmwiedemann> or we organize more community members to watch the opensuse-bugs@opensuse.org ML
20:02:45 <bmwiedemann> a CGI I once made could come in handy for those - it finds who should feel responsible for a package - e.g. VBox: http://aw.lsmod.de/cgi-bin/public/opensusemaintainer/virtualbox
20:04:48 <bmwiedemann> once bugs get to the right person, they often move faster.
20:05:01 <LWFinger> I filed https://bugzilla.novell.com/show_bug.cgi?id=735695 on December 8. I got a reference to a fix on Gnome's bugzilla, but no action from oS yet.
20:05:05 <bugbot> openSUSE bug 735695 in openSUSE 12.1 (X11 Applications) "Totem crashes on start" [Normal,New]
20:07:08 <LWFinger> How do I tell who handles libcogl.so?
20:07:23 <bmwiedemann> you could check the package name with rpm -qf
20:09:30 <bmwiedemann> then see rpm -qi libcogl2 to find that the source was named cogl and then see on http://aw.lsmod.de/cgi-bin/public/opensusemaintainer/cogl that it doesn't have a maintainer, but gnome-maintainers could maybe help
20:10:34 <Hubba> https://bugzilla.novell.com/show_bug.cgi?id=733368 - filed 29/11 and does not look like anybody looked at it yet  - I am not too worried anymore as SLES11 SP2 is unaffected but does not really encourage logging of bugs
20:10:38 <bugbot> openSUSE bug 733368 in openSUSE 12.1 (Other) "on btrfs: running 'tar --sparse' corrupts ASCII files and turns them into binary format with all bits set to zero" [Major,New]
20:11:49 <bmwiedemann> Hubba: could be upstream tar or btrfs bugs?
20:14:21 <Hubba> not sure - have not had the time to compare 12.1 and SLES11 SP2 versions of involved components - just thought I mention it anyway to highlight earlier comment about slow feedback on bugzilla. these days
20:17:09 <bmwiedemann> yes, some more feedback would be good.
20:17:47 <bmwiedemann> it appears, Jeff M. (one of our kernel guys) had trouble reproducing it on his 12.1 machine
20:19:12 <bmwiedemann> I could probably add a test for tar --sparse to openQA.
20:20:45 <bmwiedemann> anyone else has ideas to improve testing of 12.2?
20:22:52 <bmwiedemann> #info devs/screening/triagers should provide faster and more feedback to bugreporters
20:23:04 <Hubba> I have just reprouced the tar issue on another 12.1 box and on a factory build
20:24:42 <LWFinger> Do you have any other type of fs on which to test?
20:25:45 <Hubba> LWFinger: Have not been able to reproduce this on reiser
20:26:14 <LWFinger> So the problem is btrfs, and not tar.
20:27:54 <Hubba> LWFinger: I think it is the interaction of tar with the filesystem when the --sparse option is used but I am not an expert on how that works. - It only cropped up as I was doing testing of ReaR with btrfs which uses this option
20:30:18 <bmwiedemann> what command do you use for decompression?
20:30:38 <LWFinger> The only thing I can suggest is that you send the link to the bug report to the maintainer of btrfs and to the appropriate ML. Chris Mason <chris.mason@oracle.com>, and linux-btrfs@vger.kernrl.org.
20:31:09 <Hubba> I did not use compression - just tar
20:32:15 <bmwiedemann> it has a --gzip flag for the tar creation ;)
20:32:51 <bmwiedemann> but what command do you use to "restore" the files from the tar?
20:33:03 <Hubba> just tar -xvf
20:34:59 <LWFinger> Is the --sparse needed for untarring?
20:36:17 <bmwiedemann> the problem does not occur, when tarring from ext4 and untarring on btrfs. but happens when tarring on btrfs and untarring on ext - so the tar is already bad.
20:36:24 <Hubba> not that I know off and it does not make a difference - just tried it - I would not expect so as you never would know if somebody else has used that option in the first place
20:37:06 <Hubba> yes - I think you can also check  that the tar is bad with the --diff option
20:37:27 <bmwiedemann> would be good to make a minimal reproducer from that
20:37:37 <bmwiedemann> a two-line tar call is not so nice
20:38:09 <Hubba> I know but that is basically the line that ReaR uses internally
20:40:47 <LWFinger> Does it fail with tar --sparse zcvf /test.tar.gz   /etc
20:41:23 <bmwiedemann> yes
20:41:53 <bmwiedemann> even with tar --sparse -cf /test.tar /etc
20:42:21 <LWFinger> So, compression is not involved..
20:42:57 <LWFinger> That definitely should be passed on to the btrfs developers.
20:43:14 <bmwiedemann> yep
20:44:02 <LWFinger> One more reason to not use btrfs. :)
20:44:57 <bmwiedemann> anyway. let us end our meeting. we can still trace these bugs after it
20:45:03 <bmwiedemann> #endmeeting