MySQL 5.1.33, now with 4 secret bugs
Some unsettling things happened in MySQL in the past week or so.
New storage engine not mentioned in the changelog
There’s a bit of a storm brewing over at the MySQL Performance Blog, where Vadim reports discovering a new storage engine added without mention in the 5.1.33 changelog. This is in defiance of the policy of not making changes in a production release. And it certainly belongs in the changelog — but there is no sign that anyone will remedy this problem.
Arjen Lentz, who is ex-MySQL and was Employee #25, reported a bug on the licensing of this storage engine. To my eyes, the engine’s license does not look right to include in a GPL database. Arjen agrees.
But the bug report he entered is now marked secret. This is a great way to draw attention to it. Now I’m wondering: does this have something to do with the MySQL 5.1 community/enterprise split that was announced last year but hasn’t been implemented yet? Maybe they’re going to unwrap something at the conference this year, like they did last year? I hope not. That was unpleasant and should not be repeated.
Three private bug reports in the changelog
I noticed one private bug report in the 5.1.33 changelog itself. I wrote to the author of the 5.1.33 release announcement about it 3 days ago, but have received no response.
So after seeing that Arjen’s report was private, I just clicked through all the bugs mentioned in the 5.1.33 changelog and found two more that I’m barred from viewing. Here are all three:
* The MD5 algorithm now uses the Xfree implementation. (Bug#42434: http://bugs.mysql.com/42434) * Use of USE INDEX hints could cause EXPLAIN EXTENDED to crash. (Bug#43354: http://bugs.mysql.com/43354) * MySQL 5.1 crashed with index merge algorithm and merge tables. A query in the MyISAM merge table caused a crash if the index merge algorithm was being used. (Bug#40675: http://bugs.mysql.com/40675)
What reason can there be to make those private? I would like to encourage MySQL to discontinue this practice except when vitally necessary to protect client data. In my opinion, when someone submits a private bug to a software project that wants to be open-source, there should be a strong push-back. Private client data can go into a private comment; the whole bug report doesn’t have to be sealed off. Consider the harm caused by private bug reports: it’s now much harder for me to see what changeset fixed the bug. I can’t see any of the discussion about it. I can’t make any decisions about whether it affects me or clients.
I am concerned about the lack of openness and transparency in all of these issues, and others have told me that they are too.
http://lists.mysql.com/commits/68968?f=plain
Sven Deutz
9 Apr 09 at 9:28 am
I won’t comment on these bugs in particular, but one reason for making a bug private is if it exposes a security flaw. I’ve seen several bugs in the past that were made private until they were fixed, and then made public again once there was a patch available to protect the system against malicious users.
Scott Noyes
9 Apr 09 at 9:47 am
A split would be a bad bad bad idea!
Even for companies (like us) that have Enterprise-Support :-/
erkules
9 Apr 09 at 10:06 am
@Scott:
That undermines the whole point of open-source. It turns the product into a secretive closed-source monster. The product is open source. Security flaws, should thus be open and certainly not covered up. People should know about the flaws so that they can deal with them.
Now if MySQL was truly open, then the community could be called to help fix security flaws in the product. Not that I am trying to imply that Sun’s MySQL developers can’t do the job – I’m simply saying there’s amazingly talented people in the community that would gladly step-up if MySQL allowed commit access (without selling one’s soul).
Frankly, I don’t think ANY bug should be marked private. If the issue is customer sensitive information, make THAT private and not the bug itself.
I know Sun/MySQL keeps saying that they are trying to make MySQL a more truly open product, but things like this makes me think that they are shouting empty promises in an effort to keep the community at bay.
Tim Soderstrom
9 Apr 09 at 11:07 am
It almost seems worthwhile to reports a bug about the secret bug reports…
Rob Smith
9 Apr 09 at 11:13 am
It actually takes extra effort to be truly open about things. The path of least resistance is some kind of middle-of-the-road approach. I know parts of the process of opening things up more must meet with a lot of friction, and that’s understandable. Some people inside Sun/MySQL have been “trying to open things up” for a long time, and after a while it might just seem like talk, but part of me understands that it is a fight.
Heck, I still can’t get Peter Zaitsev to stop sending Maatkit bug reports and feature requests to my personal email ;-) These are the things that make life better for the masses, at the cost of a little extra work for a few people.
I also see a need for everyone who wants the openness to keep being loud about it.
Xaprb
9 Apr 09 at 12:04 pm
Ah, but most things that are worthwhile in life are difficult :)
Tim Soderstrom
9 Apr 09 at 12:33 pm
@Tim, old friend:
Seriously on this count you don’t know what you’re talking about. There are many many times in many open sourced software projects where exploits are not publicized until after the fix has been made available. This is common practice. It may simply be a case where the “hidden” status of some of these bugs had not been reverted after the release was made.
Regarding user sensitive information, If the initial description of the bug includes sensitive information that whole bug may need to be made private.
Matthew Montgomery
9 Apr 09 at 12:38 pm
> . This is in defiance of the policy of not making changes in a production release.
Wait, there is a policy to not make changes in a production release? I wonder how they manage to fix bugs. Also, what is the impact of adding a new storage engine? Does it affect users using the InnoDB storage engine?
Ashton Moore
9 Apr 09 at 12:47 pm
Though I don’t necessarily agree in principle, I see your point. But, I still have to go back to my original comment – if you make the bugs public, and allow the greater public to fix them, everyone wins (and by everyone I’m including Sun/MySQL, Enterprise customers, and the community at large).
Regards to sensitive user information, if the description of the bug contains sensitive information, simply private that comment and strip the public version of said information. No need to mark the whole thing private IMO.
Let’s not forget that Sun/MySQL is in a precarious place right now and SunMySQL needs to stop talking about trying to do the right thing and to simply just do it.
Tim Soderstrom
9 Apr 09 at 12:50 pm
A google search can do wonders:
Bug#43354: http://lists.mysql.com/commits/68867
Bug#42434: http://lists.mysql.com/commits/68922
Bug#40675: http://lists.mysql.com/commits/66021
If the patches with detailed explanations are public, the bug reports are probably marked private due confidential information or something like. Otherwise, doesn’t make much sense.
Ashton Moore
9 Apr 09 at 12:58 pm
IBM DB2 plug-in belongs in the changelog; probably just got missed in the hurry.
Don’t really understand making the entire bug report regarding the license private, and it seems that’s been corrected.
“This is in defiance of the policy of not making changes in a production release.” might be overstating – this is a plug-in, not compiled-in storage engine. The main reason to not make significant changes in a production release is stability and continuity of user experience – a pluggable storage engine, available only on System i, which must be explicitly enabled by the user doesn’t really meet those criteria, IMHO. I think you’re making more of this aspect than is justified.
Too bad that not much is made of the fact that this useful addition was made available to Community (before Enterprise, even), or that it takes nothing away from Community at all. I understand there may be other features which you want added to GA releases, and this makes you wonder why you can’t have them, but I think using the DB2 for i5 storage engine plug-in as an argument for this is pretty flimsy, really. Wish you could be happy for the System i users that will surely benefit.
Todd
9 Apr 09 at 2:36 pm
Todd,
Let me ask you here also
Can be PBXT storage engine (even theoretically ?) be included in 5.1 tree ?
It is plugin, it does not affect any other stuff, it will make many users happier.
VadimTk
9 Apr 09 at 3:05 pm
Not just PBXT, what about InnoDB 1.0.3? Or XtraDB?
Mark Callaghan
9 Apr 09 at 3:09 pm
Theoretically, it makes sense to me (although I’m not the one that makes such decisions).
That said, IBM and Sun have developed a formal relationship that makes inclusion in both Community and Enterprise possible. This isn’t just some random storage engine thrown in – there’s been a lot of investment in this effort by both IBM and Sun staff to ensure the end result benefits the community, customers, IBM and Sun.
Todd
9 Apr 09 at 3:19 pm
@Mark, Yes it would be great to include any of those in the main 5.1 source tree, if they were not still in beta, or Sun actually did any of the development work for those engines.
Matthew Montgomery
9 Apr 09 at 4:35 pm
Why hasn’t the community been invited to participate in the IBM engine’s development? What benefit is the community going to get out of this? I’m not against an IBM engine in principle, but I don’t understand how it’ll benefit the people you mention. But I do know that people generally care about fairness and following the rules. Could that effort have been invested more productively to benefit the mentioned parties? Invested in actual MySQL development, instead of partnership development?
Just a few minutes ago, someone asked Percona to fix yet another simple but serious bug that’s gone nowhere for many years. The final comment on the bug? “We can’t change a stable release. This won’t happen till 5.2.” That was a long time ago, of course.
Xaprb
9 Apr 09 at 5:26 pm
Do you really want to do systems programming for an iSeries (as/400) server?
Mark Callaghan
9 Apr 09 at 5:37 pm
@Baron:
“Why hasn’t the community been invited to participate in the IBM engine’s development?” In what way do you have in mind? It has been available and advertised for several iterations (http://solutions.mysql.com/engines/ibm_db2_storage_engine.html). We certainly want Community involvement in beta testing; that’s part of the reason it’s now in Community rather than a separate download – it becomes much easier.
“What benefit is the community going to get out of this?” The community that has DB2 data on i/5 and wants to expose it to the web using MySQL’s robust support (PHP, Java, Ruby, whatever).
“Could that effort have been invested more productively to benefit the mentioned parties? Invested in actual MySQL development, instead of partnership development?” That’s a silly question – it implies that somebody chose to ignore InnoDB features and focus on the DB2 storage engine, or something similar. It’s frankly ridiculous to suggest that this was done at the expense of other work, really. The DB2 storage engine for i/5 surely has a very targeted audience, but it makes MySQL and i/5 deployments work together in ways that are very significant to those users.
“Just a few minutes ago, someone asked Percona to fix yet another simple but serious bug that’s gone nowhere for many years.” Not really any useful context to comment on here, other than to say that this is wholly unrelated to the inclusion of the DB2 storage engine, despite suggestions to the contrary.
Todd
9 Apr 09 at 5:49 pm
@Mark:
“Do you really want to do systems programming for an iSeries (as/400) server?”
Me? No. ;)
But there are a lot of iSeries users that have DB2 data that want to expose it in ways that may be difficult to do natively, yet are essentially trivial using MySQL.
And Sun/MySQL and IBM have been great partners in making this happen.
It’s fascinating to watch this be twisted into something that is bad for the Community, or done at the expense of the Community. I recognize that there are other ongoing arguments, but this is a poor illustration of any of the points that Baron seems to be trying to make, IMHO.
Todd
9 Apr 09 at 5:56 pm
Todd,
So you are saying that ibmdbt2i engine, first time appearing in tree, is already in the stable stage, not even in beta ?
And you are saying it never will crash, or if by some reason it crashes – it will not affect other engines ? you said this is not significant change, so I guess any instability in this engine (crash, memory leak, etc) will be invisible for InnoDB tables ?
Is my understanding correct ?
VadimTk
9 Apr 09 at 6:07 pm
@Vadim:
“So you are saying that ibmdbt2i engine, first time appearing in tree, is already in the stable stage, not even in beta ?”
No, I said:
“It has been available and advertised for several iterations (http://solutions.mysql.com/engines/ibm_db2_storage_engine.html). We certainly want Community involvement in beta testing; that’s part of the reason it’s now in Community rather than a separate download – it becomes much easier.”
The referenced URL, as well as my comments on your blog, make it even more clear that the storage engine is beta/RC.
“And you are saying it never will crash, or if by some reason it crashes – it will not affect other engines ? you said this is not significant change, so I guess any instability in this engine (crash, memory leak, etc) will be invisible for InnoDB tables ?”
That’s a ridiculous claim to make – for this storage engine, for XtraDB, or any storage engine – I’m fairly confident that you could not find any storage engine developer that can make such an assertion. I think it’s safe to say that we are aware of no crashing bugs affecting the DB2 storage engine, based on a bugs.mysql.com search.
“Is my understanding correct ?”
I’m going to have to go with “no” here. ;)
Todd
9 Apr 09 at 6:27 pm
Hi,
3 of the 4 bug reports you mentioned are no longer private. I’m not sure why 42434 is private, but I’ve asked for its status to be reviewed, because I don’t see why it should be private, either — it appears not to contain any customer information or to concern a security issue that could be detrimental to our users if widely publicised prior to being fixed. Given that the licensing issue is now fixed, I don’t think we’re exposing ourselves or any users to any legal liability.
(In addition to what MatthewM mentions in his remarks above, it should also be noted that making a security issue public before making a fix available to Enterprise customers could quite possibly leave us open to legal action.)
And complaining about something missing from the changelog is a bit odd coming from someone who’s demonstrated many, many times that he knows what Documentation bugs are, and how to file them. :)
Have you yet filed such a bug?
Jon
9 Apr 09 at 6:45 pm
Todd,
Ok, I am trying to keep all things together, let me summarize what I see.
1. You saying ibmdbt2i is in beta stage and included in community release to be better tested.
2. In comment above Matthew Montgomery says XtraDB can’t be included in 5.1 yet, because it is on beta stage.
I am feeling cognitive dissonance.
VadimTk
9 Apr 09 at 8:24 pm
@Vadim, sure. I don’t think that the “beta” label is what prevents XtraDB from being distributed in 5.1. I think I’ve already made clear what I feel are the compelling reasons to include the ibmdbt2i SE.
That said, it is beta/RC, and has been distributed as a beta outside of “Community” previously.
The partnership between IBM and Sun that produced this storage engine and it’s availability to Community (and soon, Enterprise) users is great news for the community, customers, IBM and Sun – despite efforts to see it as anything less than that. It’s too bad that there are those who fail to recognize that.
Todd
9 Apr 09 at 8:58 pm
Jon, this isn’t a documentation bug like others I’ve filed. Look at the chain of events as they were revealed: 1) Vadim finds it and talks about its undocumented status — but there’s silence from Sun. 2) Arjen files a bug about the license, which gets abruptly closed off from public view. I think it’s really reasonable to say that this is unsettling, which is how I opened up the post.
I am even more disturbed about this now that it’s acknowledged this storage engine is in beta.
Todd, your points are strongly and clearly stated. There is a point of view, from where Sun stands, that this is a win for everyone. But I don’t see you also being willing to look at it from the view of Vadim and others, who have very valid stances too IMO. Among these are an objection to a glaring double standard. Sun has pushed something into the GA release that, according to the page you linked to, “should not be installed on production level systems or systems with critical data.”
Resources put into ANY project are resources that could have been used for another one. There’s a lot more to work on in the server than InnoDB, and I wasn’t suggesting InnoDB should have gotten the resources invested in this project. It’s absolutely relevant that Percona gets requests to fix simple bugs — these are the low-hanging fruit that users can’t get. It’s symptomatic. As Arjen has said — what is really best for the users? Shipping PBXT with the server is not a bad idea!
About community involvement: very few people in the community are likely to look at a nondescript page buried in the “MySQL Partners and Solutions” section of the site. From the front page of that section, I can’t even find how to navigate to this page, and as a community member that section of the site is frankly not interesting.
No one is making “efforts” to twist this into a bad thing. It would be good to hear someone from Sun say “I see your point of view” instead of discounting it as a conspiracy theory or an effort to make Sun look bad. There’s a missed opportunity here, and I think that’s the real shame.
Xaprb
9 Apr 09 at 11:31 pm
@Baron:
“It would be good to hear someone from Sun say “I see your point of view†instead of discounting it as a conspiracy theory or an effort to make Sun look bad.”
I validated your point on the missing changelog entry:
“IBM DB2 plug-in belongs in the changelog; probably just got missed in the hurry.”
I validated your point regarding the private nature of the bug report:
“Don’t really understand making the entire bug report regarding the license private, and it seems that’s been corrected.”
I said that I agree that PBXT would be possible (in theory).
There’s more to it than simply deciding to ship a storage engine, though. MySQL will support the DB2 storage engine, and IBM will provide assistance through that partnership. That’s pretty critical for customers, and I’m not saying that such a partnership with PBXT cannot be reached, but it did take 2 years for the IBM/Sun partnership to bear the fruit it has borne.
I don’t think that’s a double-standard at all. It’s not just about bundling a storage engine; it’s about support, indemnification, testing, monitoring – all the things that go into MySQL Enterprise. If there’s a will and a way to make that happen for PBXT, or some other storage engine, I’m entirely supportive.
Regarding whether this is being twisted to be a bad thing, it’s pretty hard to follow the logic.
First, it’s bad because it wasn’t in the changelogs, but you didn’t submit a bug report – you wanted to “be loud” (your words) about it, instead.
Then the problem was that the license was incompatible, according to Arjen, so he filed a bug report. That, too, apparently has been resolved to Arjen’s satisfaction.
Then the problem was that the bug report was marked private. That was acknowledged and fixed.
Then the problem was that it was destabilizing a GA release, but it’s a plug-in, has to be explicitly enabled by users, and specific for a platform you’ll never use.
Then the problem was that it didn’t meet some fantastic “promise to never crash, and if it does, do nothing bad” criteria that XtraDB doesn’t claim (http://www.percona.com/docs/wiki/percona-xtradb:info:faq).
Then the problem was that the community wasn’t invited to participate in IBM’s development of the storage engine, completely neglecting the fact that they *have* and this release makes it far more likely that they will be able to participate in the beta.
Then the problem was a disagreement with Sun’s business decisions to invest any time in this effort when there are bugs to be fixed somewhere else in the server, despite the fact the two are wholly unrelated.
Then the problem was that the information about the DB2 beta wasn’t public enough.
Then the problem was that this storage engine project, which has been the effort of a two-year partnership between IBM and Sun, was added to 5.1 while PBXT (without any such partnership) was not.
Really, that’s a whole long litany of throwing things at the wall and seeing what sticks.
I agree with your points about the release notes and the private nature of the bug report. I’m not a lawyer, but those I know say the license is compatible. I agree that PBXT would be awesome to see in 5.1, if there’s a way to form a partnership as constructive and mutually-beneficial as the one IBM and Sun have developed.
I completely disagree with the argument that the DB2 storage engine should not be in 5.1. I also believe you are really reaching to make an argument that something was done wrong here, and I think that’s sad.
Todd
10 Apr 09 at 12:38 am
Todd,
I think other commenters have made much stronger arguments than I have about wrongdoing. I mainly expressed concern about transparency (refer to the post title!) and then about fairness. I’m not ascribing to malice what I can adequately explain by oversight. And I’m very happy that the bug reports are not private anymore.
I’ll say it again — I don’t see anyone making *efforts* to twist this against Sun. I just see people drawing natural conclusions and making reasonable objections. I don’t share all the opinions posted here, so please don’t think I’m ganging up against you. I don’t want to see XtraDB in 5.1, for example.
Perhaps we should declare a truce? I think many points of view have been pretty well articulated.
Xaprb
10 Apr 09 at 1:22 am
I just keep commenting because it’s free publicity for the new DB2 storage engine and the Sun/IBM partnership. ;)
Did I mention how awesome I think it is that this is available to the community?
Todd
10 Apr 09 at 1:30 am
Todd,
I wonder why this blog is only source about new DB2 storage engine. Is MySQL so shy of this so can’t put info on mysql.com site ?
VadimTk
10 Apr 09 at 1:43 am
@VadimTk
There are been pages on this engine available on MySQL site since February.
http://solutions.mysql.com/engines/ibm_db2_storage_engine.html
http://solutions.mysql.com/engines/ibm_db2faq.html
Can we stop the conspiracy theory now?
Giuseppe
Giuseppe Maxia
10 Apr 09 at 8:21 am
Not only have there been MySQL pages and information available for some time, but this has been publicized heavily by IBM in the iSeries user space. This information isn’t of interest to most casual MySQL users – or perhaps even most existing MySQL users, period – but it’s of great interest to iSeries users.
I don’t think there is an expectation that suddenly a bunch of MySQL users will go out and purchase iSeries machines (although I’m sure IBM wouldn’t mind that), but rather that existing iSeries users see a compelling new value proposition in leveraging MySQL on their platform of choice. It just makes sense to me that the bulk of communication be focused on iSeries users (which I presume you are not), rather than blasting the general MySQL user base.
Todd
10 Apr 09 at 10:48 am
Giuseppe,
Nope we can’t, it’s fun :)
VadimTk
10 Apr 09 at 12:23 pm
[...] bugs (or while they’re on us), let’s look at some MySQL ones. Baron Schwartz found, in MySQL 5.1.33, 4 secret bugs. This follows an item on MySQL and IBM by Baron’s colleague Vadim, in which the latter [...]
Log Buffer #142: a Carnival of the Vanities for DBAs | Pythian Group Blog
10 Apr 09 at 12:43 pm
I filed a documentation bug as suggested.
http://bugs.mysql.com/44217
Xaprb
11 Apr 09 at 11:48 am
[...] the feature preview is based on MySQL 5.1, which has recently seemed to be getting some significant changes even though it’s a GA release. Does this signal a change to MySQL’s release [...]
MySQL replication breaks single-threaded limitation? at Xaprb
17 Apr 09 at 1:00 pm