Majarkeun

Lose the Teaching Work is Learning

A year back, when I decided to go with the idea of ThoughtWorks University V2, I was seriously influenced by Jay Crosss idea of a workscape. We already had a very successful graduate induction program in the form of ThoughtWorks University, and to change it meant flouting rules like "If it aint broke, dont fix it." But then, I think a bit like my friend Aaron Silvers - "We should be fixing things all the time". I wanted to move from a purely academic approach towards learning to an approach that we drove through real work. I wanted to rethink the reason we ran the program and if instructional soundness really translated into workplace effectiveness. Our team took a gamble towards workscaping the program. I can say safely today that weve seen results that make me proud to be part of the program and I believe were making a dent in the universe like never before. Last Thursday, the team of graduates at ThoughtWorks University enabled a local charity Sukrupa by launching their website complete with a donations system. This apart, theyve made significant progress with a student records system and an ecommerce project for the same client. In todays blogpost, I want to reflect on our experience of the last six weeks and share with you how real work has made ThoughtWorks University a better program for the company and the world at large.

You Cant Train People on Values
"Profit, smarts, and growth is essential, but the meaning of work-life must extend beyond the bottom line." - Jonathan Wolter
At ThoughtWorks, our values and mission are at the heart of how we work. My colleague Jonathan Wolter, has written eloquently about our mission to advocate social justice. While our mission and values are at the core of our DNA, I dont believe we can train anyone on this. We can do great presentations and elearning courses and convince ourselves that we did a great job, but frankly you believe in a mission by pursuing it. You cherish values by living them. Teaching of any kind lacks the realism to drive home a passion for missions and values. At ThoughtWorks Univeristy, we set our graduates the task of enabling Sukrupa - a local school thats on a mission to educate children from Bangalores slums. Its amazing how a challenge with a real world impact can galvanise people. In solving this problem our grads not only pursued software excellence and helped change the world in their own little way, but they also lived ThoughtWorks values as a means to the end. If you read about our values youll soon realise how a small, real world problem can easily help them live each of our cultural traits.

Experience is the Biggest Teacher
"Experience is what you get when you dont get what you wanted" - Randy Pausch
Often it takes seeing what doesnt work, to realise what could actually work. We took a risk by starting the project with little upfront analysis. In the initial weeks of the project we were literally playing it by ear, taking our clients requirements as they came. While we took this chance knowing the potential benefit for Sukrupa and for the graduates experience, the uncertainty and the lack of coordination on the team was disconcerting. There was even an occasion when our otherwise affable client was put off by our amateurish approach. When you put a group of talented, passionate people together though, problems are only an opportunity for greater responsibility. A quick shuffle of team roles and our analysts had nominated a testing lead, a UX analyst, a project manager and backlog cop. From that point on we never looked back. The graduates have forged such a strong relationship with our client, that itll drive most experienced account managers to shame. Technically, and process wise - our graduates struggled; but their passion kept them resilient through ups and downs of the six weeks. The role of the trainers was just to be coaches - guides on the side, who would chip in with their experience and help the project along in the right direction. At the end of the day, the trainers have learned heaps about leadership and the students have learned heaps about client management and software delivery. We didnt teach them anything - they learned through experience.

Nothing Succeeds like Success

"You know that youve done well, if your customer breaks out in tears during a showcase." - Patric Fornasier
You can teach people all you know, but theyll never know what its really like until they achieve real success. In the past six weeks our graduates wrote real tests and real code, configured one-click deployment for the project, elicited and delivered real user stories, worked in a true Agile environment, and managed a real client relationship. As I sat through graduation on Friday, I looked at each of our 20 new consultants -- their confidence and energy was evident. As I later wrote to these consultants they arent rookies anymore - they know what it takes to release software. They can now stand alongside their more experienced team mates and still be confident given their success on our project with Sukrupa. I hope its heaps easier for them to pull their weight on a project, knowing that theyve seen real success.

---

This ThoughtWorks University brought for me, my happiest moments as a ThoughtWorker. Its been an absolute privilege working alongside each of these 20 young people - and its an experience thatll take some beating. Its also provided evidence to my hypothesis that challenges and failure can be great catalysts to learning. Ive seen that smart, passionate people, given a worthy task, will learn and rise to challenge. I hope our story can inspire you to make real work an integral part of your learning programs. Do chime in with your thoughts in the comments section - Im very keen to hear what you think. And by the way, please donate unabashedly to Sukrupa - a small donation from your side could go a really long way for this really special school.

Available link for download

Making A Dark Wall Color Work in a Small Apartment

 A & B prefer to work with Behr paints, so for the dark grey accent wall we chose the dusky Evening Hush, and then the beautiful soft chalky white Swan Wing for the rest of the walls.




A & B inquired about which accent colours to add here. A dark grey and white palette, paired with warm woods, naturally lend themselves to greyed pastel or jewelled tone accents.

My choice here would be to add a little femininity through some amethyst accents, look at the difference in feel created by simply adding a few throw pillows…





As a side note, as shown on the sofa, I found an Etsy seller, ArtPillow whose textiles are printed with her original artworks, offering up some pretty spectacular swirling explosions of colour available as shower curtains, pillows, and duvet covers…



Colorlicious, right? Art for your sofa. Dying.


Okay, I am easily distracted by art and colour, but getting back on track :)) every consult has one main driving issue, and this one was how to use a dark colour in a smaller space with one window..

There were four things to take note of here:

1. Using a similar wall colour as the sofa, the largest piece of furniture in the space, groups them together as one, visually, and keeps the room feeling more open.

2. Large, primarily white art fronted by glass bounces light into the space.

3. Painting out the other walls in a soft white, and using a floor-to-ceiling white window treatment, allows for maximum light and minimizes visual clutter on the other walls, allowing the room to "open up".

4. One element should bridge the gap between colors, here the rug fills that need.


My clients were very happy with the end results, which is the main focus around here…

What do you think? Would this design work for you?

am xo

Available link for download

Making TBTF Policy Work Prompt Corrective Action

In my earlier post on TBTF policy, I started to discuss why "prompt corrective action" (PCA) is the key to making TBTF policy work, and I want to expand on that here. Done right, I really do think PCA could be the glue that holds TBTF policy together. The resolution authority is clearly the most important aspect of TBTF policy, since "too big to fail" just means "too systematically important to put into Chapter 11." Give us a resolution authority that can wind down systematically important nonbank financial firms without causing a financial crisis, and suddenly no firm would be TBTF anymore — the TBTF problem would instantly vanish. The trick is obviously to design such a resolution authority. Easier said than done. One of the biggest problems is that there are no trial runs — we wont know whether the resolution authority we end up with actually works (i.e., allows regulators to wind down systematically important financial firms without causing a financial crisis) until we use it during a crisis, which policymakers will be loath to do the first time because itll still be untested. But thats where PCA should come in. PCA can act as "foam on the runway" for the resolution authority (to borrow a phrase from Geithner). In the earlier post, I argued that one of the PCA triggers should be contingent on the tenor of a financial institutions overall liabilities—that is, the Fed should be required to take prompt corrective action once a large financial institution allows the tenor of, say, 20% of its overall liabilities, or 50% of its daily funding requirements, to drop below one week (again, I just pulled those numbers out of the air). In other words, the new PCA regime should be focused more on a large firms liquidity ratios than on its capital ratios. What sort of "corrective action" should a financial firm that runs afoul of this or other similar PCA triggers be forced to take? Broadly, Id say that the PCA regime needs to force a firm to secure a certain amount of medium/long-term financing for its capital markets activities (e.g., $X >30 days, $Y >100 days; "medium-term" and "long-term" are relative terms here, since were most talking about capital markets positions). The key is to make sure that a systematically important firms survival or failure is not contingent on its ability to obtain ultra short-term financing. As a firm relies more and more on overnight repos and rehypothecated collateral and prime brokerage accounts to fund its positions, its balance sheet gets exponentially more chaotic and confusing — securities are being pledged overnight all over the place, the firm is rehypothecating all the collateral it can (and some that it cant), etc., etc. If the firm ends up failing, this makes a smooth resolution 100 times harder, in part because of the inevitable delay required to figure out where everything is. Lehman was still transferring assets in between its various European and North American branches at a furious pace right up until its bankruptcy filing. As a result, a lot of hedge funds and other investors were very surprised to discover that their assets were not in segregated accounts in Lehmans North American unit, but in fact had been transferred to Lehman Brothers International (Europe) and then rehypothecated. This was a huge source of uncertainty in the days following Lehmans bankruptcy filing. It was also the main reason why hedge funds all started pulling their prime brokerage accounts from Morgan Stanley and Goldman, which both investment banks had relied on to some extent for short-term financing (the so-called "free credits"). The new PCA regime should aim to prevent this kind of thing from happening by making a large firms survival or failure contingent on its ability to obtain medium/long-term financing for its capital markets positions, or by giving the Fed the authority to restrict transactions between affiliates once a large firm passes a certain PCA trigger. If we can use PCA to ensure that a large firm doesnt rely more and more on ultra short-term financing as it edges closer to failure, then a successful resolution under a new resolution authority is very realistic. The point is that PCA needs to be a way to force both the financial institution and the regulators to start getting their ducks in a row in preparation for a resolution. If PCA can ensure the conditions necessary for a smooth (i.e., non-catastrophic) resolution of a large financial institution, then well have a legitimately "successful" resolution authority for large nonbank financial institutions, and the problem of TBTF will be no more. Of course, Ive been through more than enough rounds of financial regulatory reform to know that its highly unlikely itll actually happen that way. But getting a framework in place that could get us to that point isnt out of the question, so long as the fairweather populists can be kept in check.

Available link for download

Maintenance Work Yahoo Messengers System Down

Ada apa dengan Yahoo ! Messenger..


Berikut penuturan dari Sarah Bacon :

Kami akan melakukan pemeliharaan pada sistem kami. Selama ini, beberapa versi dari Yahoo! Messenger dan fitur - fiturnya, seperti Pingbox mungkin untuk sementara tidak tersedia. Kami menghargai kesabaran Anda dan mohon maaf atas ketidaknyamanan. Kami berharap ini terakhir pemeliharaan untuk beberapa jam.

Sarah Bacon | Product Manager

Available link for download

Loneliness Hard Work and Giving Thanks

I am inundated with thoughts as I sit down to write this blog post. I am unsure how title it correctly; I want to write about my loneliness, finally working hard, and giving thanks for the new and surprising blessings in my life. It is always so easy to focus on the negative going on in our lives. I find myself doing that sometimes and I shouldnt. That is not how I operate. That is not how I live my life.

Available link for download

Making Feedback Work The Anti Patterns

A few weeks back, I spoke on the topic of "Making Feedback Work in your Teams", at the Agile 2010 conference. I felt like a really popular speaker looking at the full house I attracted to the talk and in hindsight I can imagine why people felt interested in the topic. Feedback is crucial to the learning process and supports the iterative nature in which people pick up their skills. You learn a bit, do a bit, get some feedback, learn a little more, do a little more and so on. Anyway, I dont need to harp on the value of feedback -- Im pretty sure you agree.

Something else has got me thinking by the way. When I was at Hampi this weekend, our guide Basava was really nice to let me ride his motorbike across the city. While I am a motorcyclist in Bangalore city, Im not the most skilled rider youll ever meet. So I had to go back into my bag of experiences to tell myself how I should ride the bike. Heres what I did:

1. I told myself that Basavas bike was generally quite similar to my bike - the same manufacturer, same engine capacity and the same number of gears. Good start -- I was making a bit of a generalisation.
2. I then looked at the gear-shifts and they were a bit different. After a bit of a struggle, I tried to convince my brain that this new bike, with a significantly more powerful engine was really just my bike with a slightly different gear shift. That wasnt true, but it was a way to represent information in my world. I was making a distortion of reality.
3. Finally, I started to ride the bike in top gear. That said, this wasnt going to be my bike forever, so I didnt need to know everything about it. I just ignored everything that was new on the bike - the LED dashboard, the new controls, the decals, etc. I couldnt let myself be bothered by details that made no sense to me. I was deleting information.

So, why am I telling you this story? Its because this is the way we generally build our perceptions. We view the world from a filter of past experiences and then create a new interpretation based on our generalisations, distortions and deletions. This is where feedback can be both valuable and meaningless. Feedback is valuable because it gives us valuable insight into the perceptions that our behaviour can often drive. On the other hand, when we dont see the correlation between reality and the resulting perception, feedback starts to lose its value. With that rather long introduction, I want to introduce a few anti-patterns which we should avoid when were trying to share feedback.

Perception minus Reality


Most often, its our perception that drives our feedback. For example we find some people really helpful, others to be really brilliant co-workers and others to be absolutely irritating. So, it isnt surprising that sometimes we jump straight to the perception instead of stating the reality that drove the perception. For example you may say, "Sumeet, youre a very committed co-worker." While it feels very nice to hear something like that, it gives me very little value, because I dont know about what behaviours I should repeat and why you really feel this way. OTOH, if you were to tell me what made you feel this way, it gives me some behaviour I can try and repeat. For example you could say, "Even when you werent at Bangalore, I could see that you were in touch with us on the discussion forum, IM and video-conference. You took up a lot of tasks for the team during that time and it makes me feel that youre really committed to your work." Youll notice that the new version of the same feedback has two parts to it -- the observation and then the perception. This gives the recipient a direct correlation between behaviour and the resulting perception. Also, the observation is reality which is undeniable. It quantifies your perception, which by virtue of being personal is completely undeniable as well!

Adjective Overload
Adjective overload sounds like this, "Weiwei is awesome to work with. He is very a understanding and effective partner. He is also very dedicated." Carrying on from the previous anti-pattern, adjective overload does precious little to strengthen any ones confidence or to improve their effectiveness. What makes Weiwei awesome? Why do you feel hes understanding and effective? What makes you believe that he is dedicated? Adjective loaded feedback leaves a lot of unanswered questions. OTOH, if the giver was to quantify this perception, its likely to help the recipient understand the feedback much better. The observation+impact model works perfectly here. For example, "Weiwei always hears out my ideas when were pairing. We draw out pictures on a whiteboard to reach common ground. This makes him an effective partner for me."

Unspecified Conclusions
On a certain team, I read the following notes "You work with everyone with no personal agenda." and "I think you should be a project manager soon." Going on from the last two anti-patterns, even this style of feedback gives the receiver nothing to work with. What does it take to be a project manager? What qualities does the individual already display? What knowledge, skills and aptitude does the individual need to pick up for the role? The unspecified conclusion gives none of this information. It takes the abstracted perception and doesnt justify it in anyway for the recipient -- as a consequence, its nothing but a short lived ego-booster.

Improve Effectiveness? No Chance!
A lot of written feedback Ive seen reads like this, "Im sorry, I cant think of anything you can improve on." People need to be damn perfect for us to not find any ideas to help them grow. My ex-colleague, and Gordon Pask award winner Liz Keogh suggests that if someone was really good at their current job/ role, then we should investigate the next challenge theyre moving onto or are longing for. That information can help us make suggestions thatll help them bridge the gap between what theyre doing today and what they need to do tomorrow. So even if youre giving feedback to the ultimate perfectionist, consider what will help them grow into the next assignment theyre moving onto.

In my Defense
Last but not the least, I want to make a case for not being defensive when receiving feedback. If youve known me for a while, you know that Im a big fan of Dr. Randy Pauschs last lecture. My favourite story is the one Dr Pausch told about his football coach, Jim Graham:


That story has stuck with me for the last three years or so. Feedback is a gift - people offer it to you with the best intentions. To argue against it or to offer justifications is equivalent to rejecting a gift. Granted all gifts dont come in the best wrappers and granted that all gifts arent useful. The best you can do when youre receiving a gift though is to say "Thank You!". Just as well often go back home and then think of the best way to use the gift, its a good idea to reflect on the feedback we get. Often well realise how we can grow from the feedback and equally often well realise that we cant do much with the feedback. Both possibilities are fine -- we just need to establish our openness to the gifts people offer us.

---

To be frank, there are several other anti-patterns I could discuss here and we could keep going on until the cows come home. The fact is that Ive got to close this post now and you must be tired of reading this weekly rant. If you like posts of this nature, Ill try touch upon other aspects of feedback in future posts. In the mean time please do let me know how you liked this article. Your comments help me tailor this blog to the way youd like it to be.

Available link for download

Longer sequence work

Today we are going to work on a bit of a longer sequence work. Lets not forget our ultimate goal with the Floreio practice is to be able to flow freely on the ground, using our body and the space around us to create this physical expression. On the way we get many physical and mental benefits, not unlike other flow practices like Yoga or some forms of movement or dance.
Keep practicing with this mindset. Try to pause as little as you can during the prescribed sets and reps and make sure not only the movements themselves are under control and flowing but also the transitions. Do not let your breath stop for long periods of time, usualy it is advisable to breath through the movements, and let the natural connection between breath and movement occur - if a movement expands your ribcage - breath in, (as in entering a back bridge) and when a movement compresses your upper body breath out. (as in entering a squat) This is just a general recommendation, when actualy the exact opposite should also be understood, practiced and made possible. When flow is the ultimate goal, changes in speed, movement patterns, breathing, etc.. is not only a possibility but a certainty, so do not limit yourself.

Corta Capim Rotations Into Bridge Rotations With QDR Slide Out



Start in a squat. Perform 3 Corta Capim Rotations and continue the last Corta Capim until the moving leg is behind you, pointing back in a straight line. While doing so, place the same side hand, with fingers pointing back at you in front of you on the floor. From this position, hop into both feet, (flat feet on the ground) and start the Rotation Into High Bridge With QDR Exit.
Simple, for the ones of you who already mastered the previous movements, if you havent, you have no buisness training this Floreio piece yet, dedicate your time to perfecting the basic skills, as I have outlined them here from the first Floreio Art workout.
Build up to it, and you wont get injured, frustrated with the practice and finaly.. leave it. Flowing is not easy, but it is well worth the effort to be able to move in space in such a graceful, dominant, strong and flexible way. Put in the time and work and I am sure you wont regret it.


Workout Beginner

Warm Up - Mobility
A1. 5 min Joint Mobility
A2. Wrists Routine X 1 set
A3. Shoulder ROM and Stabilization routine X 1 set
A4. Idos Squat Clinic Routine X 1 set
A5. Scapula Mobilization Routine X 1 set

Equilibre
B1. Handstand Body Line Wall Drill X 10-20 sec X 3 sets / 45-60 sec of rest

Floreio
C1. Head Bridge To QDR Work (choose the variation you are capable of) 10-20 reps / 3 sets / 60 sec
D1. Rotations into High Bridge X 10 reps / 3 sets / 60 sec rest
or
D1. High Bridge Static Hold 30 sec
D2. Rotations Into Arch Using The Wall X 10 reps / 3 sets / 60 sec rest after D2
E1. Corta Capim Rotations X 15 reps per side / 3 sets / 60 sec

Conditioning
F1. 90-90 Iso Pull Up Hang 10-20 sec
F2. Gatherings Beginner Variations 10-15 reps
perform 5 sets resting 90 sec after F2

Prehab
G1. Scapula Mobilization Routine X 2 sets / 60 sec rest


Workout Intermidiate

Warm Up - Mobility
A1. 5 min Joint Mobility
A2. Wrists Routine X 1 set
A3. Shoulder ROM and Stabilization routine X 1 set
A4. Idos Squat Clinic Routine X 1 set
A5. Scapula Mobilization Routine X 1 set

Equilibre
B1. Handstand Body Line Wall Drill X 20-30 sec X 3 sets / 45-60 sec of rest

Floreio
C1. Rotation Into High Bridge to Head to QDR Exit X 10 reps / 3 sets / 60 sec
D1. Corta Capim Rotations X 30 reps per side / 3 sets / 60 sec

Conditioning
F1. 90-90 Iso Pull Up Hang or rope assisted 90-90 One Arm Iso Chin Up Hang 10-30 sec
F2. Gatherings Intermidiate Variations 10-15 reps
F3. Gatherings Beginner Variations 10-15 reps
perform 5 sets resting 90 sec after F2

Prehab
G1. Scapula Mobilization Routine X 2 sets / 60 sec rest


Workout Advanced

Warm Up - Mobility
A1. 5 min Joint Mobility
A2. Wrists Routine X 1 set
A3. Shoulder ROM and Stabilization routine X 1 set
A4. Idos Squat Clinic Routine X 1 set
A5. Scapula Mobilization Routine X 1 set

Equilibre
B1. Handstand Body Line Wall Drill X 35 sec
B2. Handstand Straighten Into Line X 50 sec total work
Complete 2 cycles of B1-B2, resting 60 sec between exercises

Floreio
C1. Corta Capim Rotations Into Bridge Rotation With QDR Exit 10 reps / 3 sets / 90 sec

Conditioning
F1. 90-90 One Arm Iso Chin Up Hang rope assisted or one arm only 10-20 sec
F2. Gatherings Advanced 10 reps
perform 5 sets resting 90 sec after F2

Prehab
G1. Scapula Mobilization Routine X 2 sets / 60 sec rest


I have been recieving a lot of letters from people who are following the workouts and experiencing some tremendous gains in mobility, strength, stability and stamina. Things like the low squat position, QDR control, handstand work, bridge flexibility and overall mobility have been hammered often in the workouts. I also preach for alot of joint prep and prehab work, and I am hearing a lot of people are getting more stable, leaving their injuries behind and becomming... bullet proof. Keep up the good work, friends!
If you havent been training along side, but just reading the material and building your inspiration, I invite you to join us! Enjoy the hard but rewarding work of the Floreio practice, start here with the first workout today!

Questions? Forum!
Ido.

Available link for download

Looking at Work in Grayscale

Lee Kaloidis, whom Ive referenced a few times in this blog, has created a Flickr album of his Whale Songs series, showing it in grayscale as well as in color.  He says:

Im posting the Whale Songs in gray scale so that I have a record of them. But also because studying paintings in gray scale is fundamental to the way I work and why I do so much work in black and white, and anyone whos interested in my work should know that. Throughout every painting session Im studying gray scale photos of the piece as it develops. I know if a piece is articulate, balanced, well-constructed, legible, and engaging in gray scale it is fundamentally sound. Color is a strange thing and is a little like money in that it distorts clarity of vision and judgment, and so it is good, IMO, to be able to remove it from the equation in order to assess things at a distance that is free from the distortions of the sensational.

I would not say that a painting cant be successful if color if it is not successful in black and white, but it does seem like a good tool for looking at your work in progress.  Seeing your work minus the strong language of color allows you to see things you wont see otherwise.  This is similar to seeing your work from a distance, through a camera lens, or on a computer screen.  It gives you a different view, which can be revealing. 

I just did a few grayscale versions of some 8"x8" print/collages to see how they looked:

Available link for download

Make Stress Work for You

• Try not to think of setbacks as defeats.
• View stress as an energizer. Consider each new demand as a challenge, no matter how forbidding it may seem.
• Always ask yourself, “What’s the best that can happen?” rather than “What’s the worst possible outcome?”
• Take  charge. Although you can’t control other people’s actions, you can control your response to what comes your way. When it comes to managing your emotions, you’re the boss.
• Don’t try to please everyone – you can’t.
• Get the big picture. Think in terms of long-range goals, not just day-to-day problems.

Available link for download

Making Feedback work in your Teams Webinar Notes

Today I delivered a webinar for the Thought Leadership Webinars series from ThoughtWorks Studios. I thought it could be  useful to share stuff from the webinar on this blog, for the benefit of those who may be interested in this topic. My topic had nothing to do with the technical mumbo jumbo of creating software. While feedback is a key XP value, it usually manifests itself on Agile projects as a way to improve software.

I like to look at feedback as a way to improve people as well, because people that improve continuously are more likely to produce better software in teams. Anyways, the slides for talk are on Slideshare and you can download them from there. Ive tried to design my slides so theyre self-explanatory to some extent. Im happy to answer questions if you have any.

During the webinar, I also shared an elearning module on Feedback, which you can find here. Hopefully, this could be useful for any team to develop a healthy practice of sharing feedback. Let me know what you think.

This apart, chat transcripts and a webinar recording will soon be available at the ThoughtWorks Studios community, if they arent there already.

Available link for download

Making slave pre fetching work better with SSD

In the recent few weeks I have spent some time for creating yet another slave prefetching tool named "Replication Booster (for MySQL)", written in C/C++ and using Binlog API. Now Im happy to say that I have released an initial version at GitHub repository.

The objective of Replication Booster is same as mk-slave-prefetch: avoiding or reducing replication delay under disk i/o bound workloads. This is done by prefetching relay logs events, converting to SELECT, and executing SELECT before SQL Thread executes the events. Then SQL thread can be much faster because target blocks are already cached.

On my benchmarking environment Replication Booster works pretty well.

On HDD bound benchmarks(update by pk), SQL threads qps was:
- Normal Slave (without prefetch): 400 update/s
- With prefetch: 1,500 update/s

On SSD (SAS SSD) bound benchmarks(update by pk), SQL threads qps was:
- Normal Slave: 1,780 update/s
- With prefetch: 5,400 update/s

It is great that slave could handle many more updates per second without replication delay on disk i/o bound benchmarks. It works very well on SSD, too. The below is a graph that shows Seconds_Behind_Master is gradually decreasing by using Replication Booster.



In this benchmark I executed 4,000 updates per second on master. On the slave server, by default slave delayed continuously because the slave could handle only 1,779 updates per second. When starting using Replication Booster on the slave, the slave could execute 5,418 updates per second. This was higher than the masters qps so Seconds_Behind_Master gradually decreased. After the slave caught up with the master, the slave could execute as same volume of updates as the master (4,000 update/s), so no replication delay happened. This means on this environment we can raise maximum update traffics many more (1,780 update/s -> 4,000-5,400 update/s) without investing for new H/W.

I also tested on some of our production slaves (not used for services) and it showed good results, too. I could get 30-300% improvements, depending on cache hit ratio. If data was fully cached, of course I didnt get any benefit, but it didnt cause negative impacts either.

I hope this tool is interesting to you.

In this blog post, Ill explain backgrounds for developing this tool and basic design. I believe many more optimizations can be done in future. Your feedbacks are welcome.

Good concurrency, bad single threaded performance

I mentioned at Percona Live London that using SSD on slaves is a good practice to reduce replication delay, and SATA/SAS SSD is practical enough because unit price is much cheaper than PCI-E SSD, and SATA/SAS SSD shows not bad concurrency with many drives when using recent RAID controller(most applications actually do not need 30,000-50,000 read iops, even though running many MySQL instances on the same machine). It is certainly an advantage that many SATA/SAS drives (6-10) can be installed on 1U box.

The biggest thing Im concerned about using SATA/SAS SSD is single thread read iops. You can get only 2,000 read iops from SATA/SAS SSD with RAID controller. If you do not use RAID controller, it is not impossible to get 3,000+ read iops, but this is still much lower than using PCI-Express SSD. You can get 10,000 signle thread read iops from PCI-Express SSD.

When using SATA/SAS SSD, it is easy to predict that slave delays much earlier than using PCI-E SSD. Especially if running multiple MySQL instances per single server, innodb_buffer_pool_size has to be small (i.e. 4GB-12GB), so lots of disk reads will happen. By using 6-10 SATA/SAS drives, maximum throughput can be competitive enough against PCI-Express SSD, but single thread read iops is not improved. This is an issue.

"Slave prefetching" is a well known, great approach to make SQL Thread faster.

What is slave prefetching?

The concept of "slave prefetching" is (I think) well known, but I briefly describe here in case you dont know..

SQL Thread is single threaded. When SQL thread has to do lots of disk i/o by itself, replication is easily delayed. In almost all cases of slave lagging, I/O thread has received all binary log events (and saved as relay logs), but SQL thread delays execution due to massive random disk i/o. So there are many relay log events from SQL threads current position (Relay_Log_Pos) to the end of relay log (EOF of relay logs).

Random disk reads happen when target blocks(records/indexes) are not cached. If they are cached, random reads wont happen. If you can cache all entries before SQL Thread executes, SQL thread does not have to do random disk reads. Then SQL thread can be much faster.

How can you do that? Read relay logs before SQL Thread executes, covert DML statements (especially UPDATE) to SELECT, then execute SELECT on the slave in parallel.

I believe this concept was introduced to MySQL community by Paul Tackfield at YouTube 4-5 years ago. mk-slave-prefetch is an open source implementation.

Desire for C/C++ based, raw relay log event hanlding tool

At first I tested mk-slave-prefetch on my benchmarks. But as far as I tested, unfortunately it didnt work as I expected. I think the main reasons are as below:

* mk-slave-prefetch uses mysqlbinlog to parse relay logs. But mysqlbinlog is not as flexible and fast as reading raw relay log events. For example, mysqlbinlog output events have to go through the file to the main prefetching program. mysqlbinlog is an external command line tool, so the main prefetching program has to fork a new process to run mysqlbinlog, which opens and closes relay logs every time.

* mk-slave-prefetch is written in Perl. In general, a prefetching tool has to be fast enough to read, convert and execute SELECT statements before SQL thread executes. The tool has to be multi-threaded. The tool probably has to run on the same machine as MySQL slave, in order to minimize network overheads. The resource consumption (CPU and memory) should be small enough so that it doesnt hurt MySQL server performance.
I dont believe Perl is a good programming language for developing such a tool.


I believe C/C++ is the best for programming language for this purpose. And I believe handling raw relay log events is much more efficient than using mysqlbinlog.

Based on the above reasons, I decided to develop a new slave prefeching tool by myself. I had some experiences for parsing binary/relay logs when developing MHA, so at first I planned to create a simple relay log parser program. But immediately I changed my mind, and tried mysql-replication-listener (Binlog API). Binlog API is a newly released utility tool from Oracle MySQL team. Binlog API has a "file driver" interface, which enables to parse binary log or relay log file and handle events one by one. By using Binlog API, handling raw binlog events becomes much easier. For example, its easy to parse binlog events, get updated entries, store to external software such as Lucene/Hadoop, etc.

Oracle says Binlog API is pre-alpha. But as far as I have tested for slave prefetching purpose, it works very well. Its fast enough, and I didnt encounter any crashing or memory leak issues. So I decided to develop a new slave prefetching tool using Binlog API.

Introduction to Replication Booster for MySQL

I named the new slave prefetching tool as "Replication Booster". Keywords "slave" and "prefetch" were already used by mk-slave-prefetch, so I used different words.

The below figure is a basic architecture of Replication Booster.

Design notes

- Replication Booster is a separated tool (runs as a MySQL client). It works with normal MySQL 5.0/5.1/5.5/5.6. Starting/stopping Replication Booster is possible without doing anything on MySQL server side.

- Replication Booster is written in C/C++, and using boost::regex for converting UPDATE/DELETE to SELECT. Binlog API also uses boost.

- Using Binlog API to parse relay logs, not using mysqlbinlog
  - Using file driver, not tcp driver. file driver does not connect to MySQL server, and just reading relay log files. So even if file driver has bugs, impacts are rather limited (If it has memory leak, thats serious, but I havent encountered yet).

- Main thread parses relay log events, picking up query log events, passing to internal FIFO queues
  - Binlog API has an interface to get a binlog event header (event type, timestamp, server-id, etc) and an event body. So it is easy to pick up only query log events.
  - Parsing row based events is not supported yet. It should be worth implementing in the near future.

- Multiple worker threads pop query events from queues, and convert query events to SELECT statements

- A dedicated thread (monitoring thread) keeps track of current SQL Threads position (Relay_Log_Pos)

- Worker threads do not execute a SELECT statement if the querys position is behind current SQL Threads position. This is because its not needed (too late).

- Main thread stops reading relay log events if the events timestamp is N seconds (default 3) ahead of SQL Threads timestamp
  - This is for cache efficiency. If reading too many events than needed, it causes negative impacts. In the worst case cache entries that SQL thread needs now are wiped out by newly selected blocks.

- When slave is not delayed, Replication Booster should not cause negative impacts. It shouldnt use noticeable CPU/Disk/Memory resources. It shouldnt prevent MySQL server activities by holding locks, either. Of course, it shouldnt execute converted SELECT statements because they are not useful anymore. The last one is not easy to work on various kinds of environments (i.e. HDD/SSD/etc), but should be controllable by some external configuration parameters

- Bugs on Replication Booster should not result in MySQL server outage.

- Replication Booster works locally. It doesnt allow to connect to remote MySQL servers. This is for performance reasons. Executing tens of thousands of queries per second from this tool remotely will cause massive fcntl() contentions and use high network resources (both bandwidth and CPU time). I dont like that.

Configuration Parameters

--threads: Number of worker threads. Each worker thread converts binlog events and executes SELECT statements. Default is 10 (threads).

--offset-events: Number of binlog events that main thread (relay log reader thread) skips initially when reading relay logs. This number should be high when you have faster storage devices such as SSD. Default is 500 (events).

--seconds-prefetch: Main thread stops reading relay log events when the events timestamp is --seconds-prefetch seconds ahead of current SQL threads timestamp. After that the main thread starts reading relay logs from SQL threadss position again. If this value is too high, worker threads will execute many more SELECT statements than necessary. Default value is 3 (seconds).

--millis-sleep: If --seconds-prefetch condition is met, main thread sleeps --millis-sleep milliseconds before starting reading relay log. Default is 10 milliseconds.

- MySQL connection parameters: MySQL slave user, password, socket file or local ip/port

How to verify Replication Booster works on your environments

You may want to run Replication Booster where Seconds_Behind_Master is sometimes growing. If Replication Booster works as expected, you can get the following benefits.

  - Seconds_Behind_Master gets decreased, or growth rate of Seconds_Behind_Master has decreased
  - Update speed has improved (i.e. Com_update per second has increased) by this tool

Replication Booster has some statistics variables, and prints these statistics when terminating the script (Ctrl+C) like below. If slave delays but "Executed SELECT queries" is almost zero, something is wrong.

Running duration: 847.846 seconds
Statistics:
 Parsed binlog events: 60851473
 Skipped binlog events by offset: 8542280
 Unrelated binlog events: 17444340
 Queries discarded in front: 17431937
 Queries pushed to workers: 17431572
 Queries popped by workers: 5851025
 Old queries popped by workers: 3076
 Queries discarded by workers: 0
 Queries converted to select: 5847949
 Executed SELECT queries: 5847949
 Error SELECT queries: 0
 Number of times to read relay log limit: 1344
 Number of times to reach end of relay log: 261838

I havent spent so much time on this project yet (just started a few weeks ago). Current algorithm is simple. I believe many more optimizations can be done in future, but even so current benchmark numbers are pretty good. I hope we can use this tool on many places where we want to avoid replication delay but dont want to spend too much money for faster storage devices.

Available link for download