Fish Magic

Query cache = useless?

2009-09-25T07:24:50Z

Following up on Mark Callaghan's blog post on how useful query cache is nowadays.

One has to say it aloud at last: let's kill it. Say, in 5.4.
It's useless on modern hardware anyway, and if you look at re-designing it, it turns out there is need for multiple solutions, since no single one performs well in all caching workloads.
What would it give our users? One less gotcha-like feature in the server that they need to learn, learn to forget. It will open up a niche for third-party projects to contribute. And the core server development team will have one less thing to worry about.

Bug#8523, MySQL users

2009-05-30T21:41:19Z

If you're in airline business, what you do has a lot of impact. Airports are crossroads of emotions, dramas, life-changing events. What you do can give you a lot of satisfaction, as well as bring a lot of satisfaction to passengers.

Doing databases, or, at least, working on MySQL, to me feels the same. I feel personally responsible for success or failure of the entire organization, in other words, whether our, MySQL, "flight" is on time. Emotionally, this is very hard. Hard to objectively accept and tackle mistakes, hard to stay motivated when you feel that your contribution is not making a difference.
But it's what it makes a dream job, too, at least for me.

Bugs like Bug#8523, and before that, Bug#989 is what makes MySQL an airline company.
So if you're on the other side of it, be sure, your comments do make a difference.

Love Slashdot

2009-05-14T22:21:44Z

Slashdot news post from today: Monty is doing MySQL refactoring and doing MySQL alliance.
Oh come on... MySQL is a trademark by Sun, to start with. Monty's product is MariaDB. MySQL started refactoring half a year ago. Does a MySQL engineer have to explain this kind of things to a journalist?

Working in Hyatt Regency Santa Clara

2009-04-19T17:56:21Z

I'm in Santa Clara since yesterday, at MySQL Users Conference.

This will be my fifth conference: I missed only one since 2003. I was supposed to come as a guest, but my friend and colleague Dmitri didn't get a US visa, so I am speaking in his stead. Me and Peter Gulutzan will present new all-engine foreign keys. Oh.

So I'm working from Hyatt. It's a great sunny day, the air here is clear and fresh, and everybody is enjoying the weekend.
Well, as mentioned, except me, and maybe Drizzle developers, who already started Drizzle developer conference :)

Musing about this all, I took a step back, and thought: so typical of me:

Next Moscow MySQL User Group meeting.

2008-12-07T21:07:02Z

It looks like we're going to meet on Monday, 22nd of December. I'll give a talk on subqueries in 4.1 and 6.0.
I'll post additional information here when I know more :)

Kaj in Moscow, December 1st

2008-11-20T11:45:20Z

Kaj Arnö is coming to Moscow on 1-3 of December.

There will be a Moscow MySQL User Group meeting with Kaj in High School of Economics:

Date: Monday, 1st of December
Time: 19-00
Topic: Sun and MySQL: what's happening
Location: Myasnitskaya 20, room 124

Everybody is welcome to join. To come, you need to register (there is security at the entrance
to the building, and it needs a list of names, usual stuff), there is no url for a registration
form yet ;), you could just leave a comment on this entry, I'll add you to the list.

Moscow MySQL User Group

Visit this group

What is getting fixed

2008-06-25T20:52:54Z

krow's post on prepared statements inspired me to write an update on what is getting fixed there.

First, a fix for Bug#27430 is in 5.1 tree.
Overall, this makes prepared statements significantly more stable. I know of only two crashes left ;) -- one is when you run out of memory, which is a design gotcha. The other is Bug#32124. Speaking of this last one, we now have stupid rules saying that if a bug doesn't affect too many people, even if it's a crash, it's not that important. This is a difficult bug, and the rules give an excuse to keep it unfixed.
The few important "feature" bugs that remain are Bug#11638 and Bug#2812, both scheduled, one to 6.0 and another to 6.2.
After having these two done, we can look into adding a global cache, automatic memory management, support for all SQL commands, allowing placeholders in a great many new places in the grammar, and even supporting piecewise blob operations. Yes. Sometime ;).

kostja_osipov @ 2008-03-12T01:25:00

2008-03-11T22:31:34Z

Since about 20 months after start, a solution for Bug#12713 finally reached the main tree.
If you're not using stored functions this may have a very little effect on you, since other than fixing the problem at hand it was no more than a cleanup of the execution flow and the server/storage engine interaction. Anyway, it was a difficult one, one out of a dozen of design gotchas we added to the server with 5.0. The all-time record for that sort of difficult bugs
is Bug#989, reported back in 2003 and planned for a fix in 6.0 only.

SHOW CREATE TRIGGGER, Bug#26141, Bug#24989

2007-07-10T22:44:08Z

SHOW CREATE TRIGGER was added two weeks ago into 5.1 branch - it turned out there is no way to do mysqlbackup without this statement when the trigger is defined in a character set that is not UTF-8.

And while we are on triggers, some most painful locking bugs with triggers are being fixed too (in 5.0) - Bug#26141 mixing table types in trigger causes full table lock on innodb table and Bug#24989 'Explicit or implicit commit' error/server crash with concurrent transactions. With these two fixes in, triggers should actually become usable with InnoDB.

CREATE TABLE t1 may fire an AFTER DELETE trigger on table t2.

2007-07-10T07:41:47Z

Nothing is impossible.
Here is the trick:

drop table if exists t1, t1_op_log;

create table t1 (id int primary key auto_increment, operation varchar(255));
-- not that one

create table t1_op_log(operation varchar(255));

create trigger trg_t1_ad after delete on t1
for each row 
  insert into t1_op_log (operation)  values (concat("After DELETE, old=", old.operation));

insert into t1 (operation) values ("INSERT");

set @id=last_insert_id();

create table if not exists t1 replace
select @id, "CREATE TABLE ... REPLACE SELECT, deleting a duplicate key";
-- voila

select * from t1_op_log;

Update: this works if t1 is a view.

Query cache is slow to the point of being unusable - what is being done about that.

2007-06-29T15:38:17Z

We spent a lot of time this month trying to fix Bug#21074 "Large query_cache freezes mysql server sporadically under heavy load".

In a nutshell, invalidation of a table can be dead slow (seconds) when there are tens of thousands of cached queries associated with this table, and, moreover, invalidation
freezes the entire server when it happens.
It's so funny, this thing happens under two singleton mutexes (one instance of the mutex exists in the entire server) both of which are required for every single query that the server gets.

Invalidation is indeed somewhat slow, but making it a bit faster will only shift the threshold when the query cache becomes unusable from tens of thousands of cached results, to, say, hundreds of thousands. So we thought it'll only change the depth of the hole in which people will discover they've shoot themselves in the foot.
Besides, any change of that sort requires quite an overhaul of internal data structures in the cache - not something one would do at a beta stage (the work is being done for 5.1).

So, instead, we're trying to make the whole thing more concurrent. Eek, perhaps these two singleton mutexes are not needed after all?

LOCK_open was taken in mysql_rm_table_part2. This thing does the actual job of DROP TABLE.
That place we fixed to take an exclusive name lock on the table instead of keeping the entire
system frozen (good for other things as well). Thanks to some prior patch that was done by Dmitri earlier this year in 5.1, exclusive name locks on table metadata are now possible.
The problem with structure_guard_mutex was harder to crack. The thing is, invalidation of one table may potentially trigger invalidation of every single result in the query cache,
and that's not known in advance, and there is no way to lock a sub-part of the cache and then
shift the lock to some other part.
And implementing this would, again, mean, changing the whole thing quite a bit.

So we now simply disable the cache during invalidation.
Now, I'm writing this to get beaten. If it's wrong and will break your application, please tell. If, on the contrary, you like the idea and want to test it, stay tuned. The patch is not yet in, but soon will be.

My top 5 MySQL wishes

2007-06-29T14:53:13Z

There has been a trend in the blogs for top5 MySQL wishes.
Many people, including Ronald Bradford, Alan Kasindorf, Jim Winstead, Jonathon Coombes, Jeremy Cole, Jay Pipes, Antony Curtis, Stewart Smith coined in.

Here're my 5:

1. Remove excessive fuss. OK, we know what needs to be done, just give us time to get things in order. Anything related to MySQL gets huge visibility, and we tend to overestimate the importance of "opinion of community". If you have a cool new feature for the server or a cool new wish, let it cook for a while. If you have or want to write a patch, first get in touch with developers, discuss things on internals@ or commits@, then post your patch on the Forge and see if it takes off. Be sure it'll get in if it's a really good idea. But not next week. Perhaps in a year or two. Want your code to be accepted faster? Write a good fix for a bug. We have 1100 of them. Still want it in faster? Fix a crash.

2. Open the development process. This is the other side of the previous wish. It takes a year or two to introduce a new engineer into all the existing conventions. So an external engineer just can't write a good patch for the server. Besides, we mostly in need of contributions that improve and simplify the existing code base, not add new code. But it is increasingly hard to change the existing code -- implicit dependencies, no good tests, almost non-existent unit test coverage. This is a barrier for penetration, and it's a catch, catch-22.

3. Get to a normal release schedule. Have XXX.1, XXX.2, XXX.3 releases once or twice a year. There is a lot in this wish that makes it very hard to satisfy -- planning, compatibility, healthy life cycle, high quality support. But we do need to shift the load of patches pushed to the period when a version is in alpha, not when it's in GA, and we do need to release more often.

4. Establish productive relationship with the majority of users. Something tells me that visibility does not equate to quality of input, and we get most of our input from early adopters, not from the large majority. We need this for the next wish:

5. Find a way to do incompatible changes with minimal pain for users. If we don't deprecate the 'cool non-standard features most needed now' that were added during brave times of 3.23 and 4.0, the software is a dead fish.

On infninite usefulness of DELETE FROM t1 ORDER BY 1

2007-06-25T11:15:55Z

Did you know that MySQL supports:

DELETE FROM t1 ORDER BY a;
?

MySQL manual says:
If the ORDER BY clause is specified, the rows are deleted in the order that is specified.

You can use stored functions and subselects there too if you like.

I should start using this feature immediately.

Hey, Stupid!

2007-05-24T09:10:54Z

Today I tried to add this little function to our impekkable List template:

template
inline
List::List(const List &rhs, MEM_ROOT *mem_root)
:base_list(rhs, mem_root)
{
/* Make a deep copy of each element */
List_iterator it(*this);
T *el;
while ((el= it++))
it.replace(new (mem_root) T(*el));
}

That is, for the referecne, the default copy constructor for List template is shallow.
So, I needed a _normal_ copy constructor. I don't want to rewrite the existing one -- it is used eveywhere (and don't ask me how one can make any sane use of a shallow copy constructor of a container - because all the usages are insane).
And here we go - this won't compile:

sql_list.h: In constructor ‘List::List(const List&, MEM_ROOT*) [with T = Item]’:
item.cc:6917: instantiated from here
sql_list.h:581: error: cannot allocate an object of abstract type ‘Item’
item.h:446: note: because the following virtual functions are pure within ‘Item’:
item.h:541: note: virtual Item::Type Item::type() const

item.cc:6917 is gorgeous:

#ifdef HAVE_EXPLICIT_TEMPLATE_INSTANTIATION
template class List;
template class List_iterator;
template class List_iterator_fast;
template class List_iterator_fast;
template class List;
#endif

And explicit instantiation is the default. And class Item is of course an abstract base.
So, boy, have you had fun with meta-programming? Alright, now do some real stuff - write
mysql_copy_list(void *from, void *to);

Speaking of conferences

2007-04-30T07:08:33Z

MySQL Users Conference was a great success.
Oh, screw that.
I don't actually think it was. I just keep hearing this sentence in announcements and press-releases year after year and my structure-seeking brain classified the sentence as a set phrase. But the conference was okay. Good tracks, decent food, a lot of new engineers from our new and numerous storage engine building partners. I don't know if it was good value for the money, since I didn't have pay a dime - I was a speaker, and the conference organizers paid the admission fee for me, while the company covered the expenses. But it certainly was a good use of time.

LinuxFest NorthWest

2007-04-30T06:49:40Z

I've had a very good day today.
I don't stop and think and say that to myself very often.

Despite all the driving, sitting, bad spicy food and me voluntarily suffering while pretending I can be a vegetarian, I feel that it was a day worth to live.

In the morning we had to drive over 90 miles to Linuxfest Northwest 2007 and despite long distance and apparent unimportance of the conference it was just the right kind of American experience I like to have.

People were coming there with their kids to listen to talks about Zope and Python and find out more about basics of the General Public License. Presenters whose profile seemed to be too high for this conference, like BrianA and BradFitz seemed to be at home and at ease. Seeing these two kinds together is part of the American society I've always been pleased with.

Nothing particularly special about the talks, so I was just sitting there and trying to kill time coding a native C++ connector for MySQL. Another piece of software that will likely go nowhere.

Then we drove back, had a nice walk around Capitol Hill and a dinner.
C. from Subversion project with his friend, and C. from Amazon joined us for dinner.
Who joined whom, that I don't know.
C. turned out to be very passionate about software patents and copyright law, and we spent the evening chatting about the topic. This was so unusual. My general feeling is that in the US an intelligent conversation about any matter not directly related to business can take place only on a very rare occasion.
Tonight we had a very nice walk and conversation.

A couple of phrases of the day, a hypocritical one that I heard very often and thus regard as language clutter - "pretty awesome" - and an interesting one that I heard just once - "a pure matter of contemporary fact".

On another note: prepared statements and the query cache.

2007-03-09T23:43:15Z

Thanks to Guilhem Bichot, we are adding some support for Query Cache to Prepared Statements. Should become public in one of the future 5.1 versions. It took only 3 years of procrastination and a few days of diligence to do it.

Features added in 4.1, 5.0, 5.1

2007-03-09T21:52:13Z

While reading "Crossing the Chasm" by Jeffrey Moore, I discovered that I never actually questioned the features we added in the recent releases.
I came to MySQL with a background similar to Peter Zaitsev's, we worked together in Spylog. MySQL was used there as storage for an in-house scale-out application that served TBytes of data per month and hundreds of thousands of requests per day.
The thing is, almost none of the features implemented in the recent years I would use for that sort of appliance.
Top things on my list would be:
- integrated inverse indexes/full-text search, both InnoDB and MyISAM. And these indexes would need to scale well to give low response times for at least 500 MB tables.
- scaling to multi-core CPUs and high amount of memory (64GB and more)
- better and faster networking that would not spawn off a thread for every connection, so that the server can actually handle a connection per user.

So what would I use? Prepared statements, maybe, only if they were noticeably faster. Mixed mode replication, that's for sure. Extended character set support? Hardly, I'd probably run the system in pure Unicode.

It seems we were trying to add all these stored procedures and triggers to appeal to some ERP systems and to become a viable replacement for some extinguishing behemoths like Sybase/whatever.
The thing is, we're not there yet, and I don't know when we will be.
I wonder if any of the modern Web 2.0 system is using this stuff.

Huh.

Why open source is better than contraception

2007-02-26T19:58:21Z

Tonight in the Moscow English Communication Club I enjoyed an
interesting conversation with a member of http://www.freestateproject.org.
We were arguing whether Universal Ethics exist or not, and if it does exist,
whether it has to be self-evident or not.
Stephen was aware and very supportive of the Open Source movement.

This got me thinking. Having been on board for almost 4 years, I've had a
chance to read a lot of introduction letters, talk to people from
whatever-the-current-number-is different countries, origins, beliefs. We
have everyone in the company, vegans and catholics, atheists and Jehovah's
witnesses. Everyone inside and outside share the same belief - that open
source is a good thing.

This sounds like abandoning slavery - the idea is in the set of core values
of practically everyone.

No matter what the business model is, whether we as a company cross the
chasm or not, there will be a product and organization that will tip.

This is encouraging :)

Back in Moscow

2007-02-11T22:45:04Z

-13 feels here almost the same as -5 in Stockholm

In Stockholm

2007-02-05T22:21:07Z

OK, I am in Stockholm, at a development meeting. Stockholm is a nice city. It has soul, and it is smart. Probably the best European capital I've been to.

This is my first impression - so far I've seen only a few quarters and the road from the airport. But something tells me it'll stay that way. Maybe I need not see the city and an image is ready made. I like the roads, the houses with walls that are 4-5 bricks thick. And it's the first airport with oaken parquetry floors I've been to.

Tomorrow is a day to revisit some prehistoric server problems, such as Bug#989. Need to get back to preparing my lot.

PS I can't make my P990 work with Linux. So no photographs until I get back home.

MySQL User Conference 2007

2007-01-25T23:27:25Z

It's been a while since I posted anything to this blog. Now there is a cause to rev up - my talk about the client library internals was accepted.
The puzzle how to deliver anything more than superficial in scope of a 45 minute presentation remains to be solved, and perhaps not without help of the blogosphere.

QotD

2007-01-18T18:28:48Z

20:48 < andrey> and please ask knielsen to modify pushbuild to run the unstable
suite
20:48 < kostja> andrey: and what is that for?-> (run it)
20:48 < andrey> ? I also fear that nobody will run the unstable suite :(
20:49 < marc_alff> honestly, I sure won't
20:49 * andrey gazes at marc

Get A Life

2006-12-29T23:48:41Z

A lot of thinking recently has been devoted to getting a life.
Currently I can not have a life and also get my work done the way I want.
I've been watching people who seemed to be more effective, bugging them with silly questions, making observations about my own biorhythms and here is what I found:

- I'm a night owl. My peak hours are in the evening. I can work late into the night easily
- I need one hour just to think over the events of the day before I can fall asleep.
- my performance depends on my mood
- I have a sort of attention deficit disorder - I naturally switch to thinking about something else when encounter a predicament with the current task.

People who manage to actually have a life and also get things done are:
- early birds
- priorities driven. I can't believe they are actually not dependent on the mood: rather, it seems, they have deeply internalized the priorities and their mood is a function of the priorities.
- it seems they can actually focus a lot better than I do.
- they are not more physically capable than I am. On the contrary, they spend more time to take care of their physical health than I do. I don't listen to my body carefully enough and thus can easily drive myself to exhaustion.

Written at 2:47 am.

A quote of the day

2006-12-16T23:07:38Z

A people hire A+ people. B people hire C people.
This came from our new VP Development, and here's the unabriged version:

"I only ever hire 'A' players for me and my companies. If I hire 'A' players, they will hire 'A+' people below them. If I hire 'B' players, they will hire 'C' players below them. 'A' players don't get threatened by better people below them; 'B' players do."