Saturday, December 29, 2012
Debugging Shufersal Targeted Promotions
My wife and I are frequent customers of Shufersal, a major Israeli food retailer, as there is a store near our house. We even have a membership card, and use it every time we make any purchases.
From time to time, Shufersal send us coupons by mail, that seem to be tailored to our needs.
7 months ago our baby was born, and so our consuming habits changed. Diapers and milk formula became part of our weekly shopping. But Shufersal prices for these items are relatively high, and so we barely ever bought them there, and instead always bought them in pharma stores (mainly Superpharm), or online, upon promotions. But we did shop for few baby-related items there, so Shufersal could know through the membership card usage that we now shop for a baby.
It seems that recently Shufersal decided to find out what are the exact brands that we are consuming (as you probably know, customers are usually very loyal to specific brands and products, when it comes to babies - "if it works, don't change it").
They have sent us very attractively priced coupons for every possible milk formula products, baby food products, diapers, bottles, etc... Naturally, we did use the few that were really relevant for us.
Now I wonder what will happen next. Will we receive more coupons for the products we purchased ? Will they be priced equally attractively ? Did we disclose our customer preference and willingness-to-pay limit through the coupons ? Will they try to affect our preference by suggesting low prices for substitute brands ?
Updates will follow...
Saturday, August 27, 2011
Decisions
I have recently read two articles on "Decisions Fatigue": one was published in NY Times, the other one - in this blog post. This got me thinking - do development managers and team leaders suffer from such decision fatigue as well ?
It seems that in many cases, the management level that directly manages software engineers takes much more decisions than necessary, going into much more details, and do not empower enough the individual engineers to take more decisions, and also be accountable for the results of these decisions.
I am sure that those engineers who are willing to step up, indeed do so and take many decisions themselves (such as described in the 2nd post), when properly empowered by their managers. But what about those that do not do so ? They seem to be in a very convenient position, where their manager does most of the decisions, and then also faces all the consequences in case of a failure, while they are at most the "innocent bystanders".
Now, in real life, the vast majority of these people are able to take a lot of very important decisions: organizing their wedding, buying a new apartment, taking care of their children and elders, etc. All these decisions seem to be much more difficult than some decisions on design, user experience, quality, priorities or any other decisions that a software engineer might need to take in the day-to-day work.
So what is different ? Well, in real life there is no manager to take the decision instead, the person needs to take these decisions and be accountable for the results.
Now, someone would ask: what do we need the managers for then ? Why do we, the developers, need to take all the responsibility, where is the responsibility of the manager ?
Well, the responsibility of the managers is to determine which decisions can be taken directly by the people they manage, help these people by teaching them and providing them the tools to make such decisions, and setting the standards and expectations for these decisions and their outcome. Eventually, the managers are accountable for any outcome of the work done by the people they manage (both for failures and for success), but the point here is that each and every individual should be equally accountable as well for their work.
Now, what does it all have to do with decision fatigue ? Same as discussed regarding the engineers' decision fatigue in the 2nd blog post, a manager also can process only a limited amount of input every day, and take a limited amount of decisions.
The results of such decision fatigue can be truly devastating for the product:
If more decisions were taken directly by the engineers, the managers would have more mental capacity for taking the more strategic decisions, that have more effect in the long-term, and in fact do more of the decisions that they were so far pushed up to their managers or aside to the product manager/owner or the product architects.
From my experience in implementing these principles so far, I can say that it worked better in some cases and worth in other. Here are some of the pitfalls to avoid:
It seems that in many cases, the management level that directly manages software engineers takes much more decisions than necessary, going into much more details, and do not empower enough the individual engineers to take more decisions, and also be accountable for the results of these decisions.
I am sure that those engineers who are willing to step up, indeed do so and take many decisions themselves (such as described in the 2nd post), when properly empowered by their managers. But what about those that do not do so ? They seem to be in a very convenient position, where their manager does most of the decisions, and then also faces all the consequences in case of a failure, while they are at most the "innocent bystanders".
Now, in real life, the vast majority of these people are able to take a lot of very important decisions: organizing their wedding, buying a new apartment, taking care of their children and elders, etc. All these decisions seem to be much more difficult than some decisions on design, user experience, quality, priorities or any other decisions that a software engineer might need to take in the day-to-day work.
So what is different ? Well, in real life there is no manager to take the decision instead, the person needs to take these decisions and be accountable for the results.
Now, someone would ask: what do we need the managers for then ? Why do we, the developers, need to take all the responsibility, where is the responsibility of the manager ?
Well, the responsibility of the managers is to determine which decisions can be taken directly by the people they manage, help these people by teaching them and providing them the tools to make such decisions, and setting the standards and expectations for these decisions and their outcome. Eventually, the managers are accountable for any outcome of the work done by the people they manage (both for failures and for success), but the point here is that each and every individual should be equally accountable as well for their work.
Now, what does it all have to do with decision fatigue ? Same as discussed regarding the engineers' decision fatigue in the 2nd blog post, a manager also can process only a limited amount of input every day, and take a limited amount of decisions.
The results of such decision fatigue can be truly devastating for the product:
- Setting incorrect priorities
- Settling for the "quick-and-dirty" solutions just so as to remove the problem from the pile of unresolved issues, instead of making sure that there is an adequate design
- Not creating a long-term strategy for the product
- Developing features that have bad user experience, only because we didn't put more effort into designing their usability better
- Taking technical decisions that result in bad performance and scale problems later on
- Taking decisions that do not address well enough some people's aspirations for personal development, etc...
If more decisions were taken directly by the engineers, the managers would have more mental capacity for taking the more strategic decisions, that have more effect in the long-term, and in fact do more of the decisions that they were so far pushed up to their managers or aside to the product manager/owner or the product architects.
From my experience in implementing these principles so far, I can say that it worked better in some cases and worth in other. Here are some of the pitfalls to avoid:
- The engineers are provided with too much decision freedom, and as a result decisions are taken in a counter-productive direction (focusing on things that are not important, etc...).
- Need to identify cases when the engineers do not have the tools to reach the decision on their own, and they do need extra guidance. Otherwise, the process can come to a hold due to no one taking a decision.
- In order to take decisions, the engineers need to have sufficient information. The managers must be an "information hub" in this sense - pushing the relevant information to the relevant individuals.
- Sometimes the outcome of the decision has a very meaningful impact and the engineers do not feel comfortable taking such a decision on their own.
Saturday, July 2, 2011
Processing feedback (a retrospective)
Wow - never thought that my recent post (Software In Pictures) would become so popular.
Almost 700 views during 2 weeks, and still counting !
I have also received a lot of comments, and the post contents were even discussed in a couple of Linkedin groups. As a junior blogger, I find this very encouraging.
However, naturally, not all feedback is good:
- Some people did not like my "good code" metaphor.
- Some of the other examples were put in other contexts, where they are actually welcome and functional, with good cost-efficiency (such as the door in the wall, that can be used to load goods to an upper floor).
- Some of my examples were seen as possibly offending to the less fortunate populations, that come to these solutions out of necessity, and without a better option.
What I learn from all that:
- The more people read your blog, the more feedback you will receive. And a certain portion of it will be criticism.
- People seem to be more encouraged commenting on topic upon which they disagree, than on topics they do agree upon.
- All feedback is good and welcome. The more feedback I will get for my posts, the more I will learn and the better my next posts will become.
- The more often I write, the shorter will these improvement cycles be.
When thinking of it, it's not so different from building a new product.
As you start, and produce your first prototypes, or even the 1.0 release, most of the feedback will be negative - as chances are that your initial product will suck in some areas.
By collecting the feedback, processing it, and learning from it, you will gradually improve and so will the product.
The negative feedback is indeed the preferred one, since it can help you understand where to focus your next efforts.
So, my bottom line for all the bloggers out there is: write, get feedback, process it, rinse and repeat.
And the same - for the all the product developers out these.
And the more frequently you write/release, the faster the improvement process will happen.
Almost 700 views during 2 weeks, and still counting !
I have also received a lot of comments, and the post contents were even discussed in a couple of Linkedin groups. As a junior blogger, I find this very encouraging.
However, naturally, not all feedback is good:
- Some people did not like my "good code" metaphor.
- Some of the other examples were put in other contexts, where they are actually welcome and functional, with good cost-efficiency (such as the door in the wall, that can be used to load goods to an upper floor).
- Some of my examples were seen as possibly offending to the less fortunate populations, that come to these solutions out of necessity, and without a better option.
What I learn from all that:
- The more people read your blog, the more feedback you will receive. And a certain portion of it will be criticism.
- People seem to be more encouraged commenting on topic upon which they disagree, than on topics they do agree upon.
- All feedback is good and welcome. The more feedback I will get for my posts, the more I will learn and the better my next posts will become.
- The more often I write, the shorter will these improvement cycles be.
When thinking of it, it's not so different from building a new product.
As you start, and produce your first prototypes, or even the 1.0 release, most of the feedback will be negative - as chances are that your initial product will suck in some areas.
By collecting the feedback, processing it, and learning from it, you will gradually improve and so will the product.
The negative feedback is indeed the preferred one, since it can help you understand where to focus your next efforts.
So, my bottom line for all the bloggers out there is: write, get feedback, process it, rinse and repeat.
And the same - for the all the product developers out these.
And the more frequently you write/release, the faster the improvement process will happen.
Saturday, June 25, 2011
Software in pictures
Did you ever try to explain what is "bad code" to someone who does not come from software background ? How do you do that ?
In this post, I would like to visualize some bad design and coding using some real life parallels.
But first a disclaimer: as bad as many of the following examples look like, sometimes compromises need to be made - to meet a short-term deadline, to save cost in the short-term, due to temporary lack of expertise, making some feature work for a customer on the spot, etc.
As long as you acknowledge in advance that what you did is not the right way to do it, and are willing to improve and proceed to apply the proper solution...
First, let's take a look at the equivalent of some messy code.
In this post, I would like to visualize some bad design and coding using some real life parallels.
But first a disclaimer: as bad as many of the following examples look like, sometimes compromises need to be made - to meet a short-term deadline, to save cost in the short-term, due to temporary lack of expertise, making some feature work for a customer on the spot, etc.
As long as you acknowledge in advance that what you did is not the right way to do it, and are willing to improve and proceed to apply the proper solution...
First, let's take a look at the equivalent of some messy code.
 |
| Source: http://bit.ly/jmL7RC |
 |
| Source: http://bit.ly/kiWmRV |
When one of the power lines dysfunctions, how long do you think it would take to fix it ? Or just to find it ?
Same is with bugs in code that looks like this...
Now, how about using a workaround solution rather than upgrading the infrastructure to handle it properly ?
 |
| Source: http://bit.ly/myfL5v |
Sometimes, just because you can make two pieces of software work together doesn't mean you should:
 |
| Source: http://bit.ly/lpx3vE |
Here is how it looks like when you apply a patch rather than rewriting a broken piece of code:
 |
| Source: http://bit.ly/iYIyAa |
And finally, this how is it looks like when you leave some useless code around:
 |
| Source: http://bit.ly/jfFKqE |
For contrast, here is the probable visualization of a good piece of software - fast, sleek, great design, and it gets you to where you want to go to:
 |
| Source: http://bit.ly/jpZAGU |
Do you have any more similar design or code visualizations ?
Friday, June 10, 2011
Your code - keep it covered !
Code coverage by unit-tests (or integration tests - we will not discuss the differences here) is the easiest to add - in many cases much simpler than adding complex end-to-end system tests, and provides good protection against many stupid and easy-to-detect bugs. Here I am only referring to tests that take just a few minutes to run, and can be run as part of the basic build process. No one is saying that it makes the code full-proof against bugs, but no one can argue that having ready tests reduces risks when having to fix a bug or make functionality changes in some code. Code coverage measurement tools are able to provide reports on the portions of code covered by these tests.
Naturally, some developers are adding much more tests on average than others. I don't know where does the lack of writing tests come from - is it out of pressure to meet a deadline, lack of skills or experience, laziness, just not caring, not presenting a sense of ownership ("it's someone else problem"), or any other reason... Also, sometimes, for the same reasons, people tend to remove or disable tests rather than fixing them - either at the same moment, or shortly after committing the code that broke these tests. This, inevitably, leads to having the coverage percentage going down.
Quite for some time, I have been thinking about a method that will hopefully continuously increase code coverage by tests. I am sure that I am not inventing anything new here, but so far I did not encounter any teams that are actually using such a method, and anyway - this is a great way to get feedback on ideas :)
Continuous integration tools, such as Jenkings (former Hudson), have appropriate plugins for displaying code coverage results over time (see a screenshot example) . Moreover, these tools can be configured to mark the build as failed, if the code coverage percentage for the current revision is lower than for the previous revision.
This way, if someone commits code with lower coverage percentage than the current percentage in the trunk (by not writing enough tests for the new/fixed code, or disabling existing tests), the build will be broken, just as if this person committed code that doesn't compile, or caused any tests to fail, and it will require a similar response (either provide a quick fix, or revert the commit).
For example, let's say we start the process at the current state, and we have 40% code coverage by such measurable automatic tests. The above process ensures that we will never go below this coverage percentage. But how do we go up ? Well, statistically, some of the developers will have much higher code coverage on their code, thus moving the overall average higher. This will automatically set a new, higher, goal from that moment on.
But what if adding more tests for some specific change is highly difficult and is not cost effective ?
Well, my suggestion is that in this case, since the developers do not want to break the build, they much compensate by adding tests for different (possibly even unrelated) code, merely so that the overall coverage won't go down. Over time, such an "escape route" will ensure that at least all the easy-to-test areas in code will be well covered.
Of course, reaching 100% coverage is virtually impossible, and also at most times non cost-effective. So we can set a lower percentage threshold (75-85%) for coverage, and once reached, just make sure that the coverage does not go below this threshold. In addition, at all times the code that still does not have code coverage, will be the code for which writing tests is the least cost-effective, or that is better to be covered by more complex system tests, that cannot be run as part of the continuous integration system measurement tools.
I am sure that such an approach is a bit controversial, so I would be happy to hear feedbacks and suggestions how to improve such a method.
My main concern for adoption of such a method is the social concern - how well will it be perceived by the developers, will the developers that indeed lowered the coverage percentage take the responsibility to fix it quickly, or will there be just a continuously "red" continuous integration system status, that no one cares about...
I must say that I still do not have any actual experience or results for this method. Hopefully we will be able to try it in our group in the next months.
I will update on any insights and conclusions as we go...
Naturally, some developers are adding much more tests on average than others. I don't know where does the lack of writing tests come from - is it out of pressure to meet a deadline, lack of skills or experience, laziness, just not caring, not presenting a sense of ownership ("it's someone else problem"), or any other reason... Also, sometimes, for the same reasons, people tend to remove or disable tests rather than fixing them - either at the same moment, or shortly after committing the code that broke these tests. This, inevitably, leads to having the coverage percentage going down.
Quite for some time, I have been thinking about a method that will hopefully continuously increase code coverage by tests. I am sure that I am not inventing anything new here, but so far I did not encounter any teams that are actually using such a method, and anyway - this is a great way to get feedback on ideas :)
Continuous integration tools, such as Jenkings (former Hudson), have appropriate plugins for displaying code coverage results over time (see a screenshot example) . Moreover, these tools can be configured to mark the build as failed, if the code coverage percentage for the current revision is lower than for the previous revision.
This way, if someone commits code with lower coverage percentage than the current percentage in the trunk (by not writing enough tests for the new/fixed code, or disabling existing tests), the build will be broken, just as if this person committed code that doesn't compile, or caused any tests to fail, and it will require a similar response (either provide a quick fix, or revert the commit).
For example, let's say we start the process at the current state, and we have 40% code coverage by such measurable automatic tests. The above process ensures that we will never go below this coverage percentage. But how do we go up ? Well, statistically, some of the developers will have much higher code coverage on their code, thus moving the overall average higher. This will automatically set a new, higher, goal from that moment on.
But what if adding more tests for some specific change is highly difficult and is not cost effective ?
Well, my suggestion is that in this case, since the developers do not want to break the build, they much compensate by adding tests for different (possibly even unrelated) code, merely so that the overall coverage won't go down. Over time, such an "escape route" will ensure that at least all the easy-to-test areas in code will be well covered.
Of course, reaching 100% coverage is virtually impossible, and also at most times non cost-effective. So we can set a lower percentage threshold (75-85%) for coverage, and once reached, just make sure that the coverage does not go below this threshold. In addition, at all times the code that still does not have code coverage, will be the code for which writing tests is the least cost-effective, or that is better to be covered by more complex system tests, that cannot be run as part of the continuous integration system measurement tools.
I am sure that such an approach is a bit controversial, so I would be happy to hear feedbacks and suggestions how to improve such a method.
My main concern for adoption of such a method is the social concern - how well will it be perceived by the developers, will the developers that indeed lowered the coverage percentage take the responsibility to fix it quickly, or will there be just a continuously "red" continuous integration system status, that no one cares about...
I must say that I still do not have any actual experience or results for this method. Hopefully we will be able to try it in our group in the next months.
I will update on any insights and conclusions as we go...
Thursday, April 21, 2011
Creativity where you don't expect it
Recently, I have encountered two seemingly unusual examples of businesses trying to provide additional value to customers in some rather surprising ways.
The first would be the funerals market (you might have already seen it on the Jimmy Kimmel show last week). What can possibly be innovated in this market ? Well, in Compton, California, the Robert L. Adams Mortuary are now offering a new product: a "drive-by mortuary" - you can drive-through near the open casket of the deceased and depart from the beloved one, all that without even having to leave the comfort of your car.
The second innovator would be McDonald's, that are now offering wedding packages in their branches in Hong-Kong. Apparently, this allows many Hong-Kong citizens the opportunity to get married at reasonable prices, as compared to their other options. McDonald's even offer to supply wedding gowns, made out of balloons, for the happy brides.
So next time you're eating a Big Mac in Hong-Kong, be careful with that ketchup - a bride in a white dress might be just behind you.
Both cases, while probably being rather amusing for a person coming from a different cultural background, are actually great examples of "thinking out of the box". Both are created in rather traditional businesses, they seem to create additional value for their customers and thus increase revenues, without any substantial changes or investments in their business core. So you have to admire the creativity of the marketing and product managers behind these offers.
Makes you think what kind of "additional value" can you create with your software products, with some very little additional investment in development...
The first would be the funerals market (you might have already seen it on the Jimmy Kimmel show last week). What can possibly be innovated in this market ? Well, in Compton, California, the Robert L. Adams Mortuary are now offering a new product: a "drive-by mortuary" - you can drive-through near the open casket of the deceased and depart from the beloved one, all that without even having to leave the comfort of your car.
The second innovator would be McDonald's, that are now offering wedding packages in their branches in Hong-Kong. Apparently, this allows many Hong-Kong citizens the opportunity to get married at reasonable prices, as compared to their other options. McDonald's even offer to supply wedding gowns, made out of balloons, for the happy brides.
So next time you're eating a Big Mac in Hong-Kong, be careful with that ketchup - a bride in a white dress might be just behind you.
Both cases, while probably being rather amusing for a person coming from a different cultural background, are actually great examples of "thinking out of the box". Both are created in rather traditional businesses, they seem to create additional value for their customers and thus increase revenues, without any substantial changes or investments in their business core. So you have to admire the creativity of the marketing and product managers behind these offers.
Makes you think what kind of "additional value" can you create with your software products, with some very little additional investment in development...
Friday, April 15, 2011
Agile Israel 2011 visit summary
On last Monday, April 12th 2011, I have visited the Agile Israel 2011 event.
As a rule, I tried to focus on lectures by guests, especially on guests coming from abroad. I have already received a 2-day Scrum training by AgileSparks representatives in the past, and so preferred to hear things from a fresh angle. I would also like to say "kudos" to AgileSparks for bringing such a large event together, year after year !
The short list of lectures I've attended:
- Keynote: Scrum & XP Beyond the Trenches - Henrik Kniberg
- Agile Software Craftmanship – Principles, Practices and Best Practices - Gal Marder
- Moving from Yearly Plans to Continuous Planning using Lean Product Management – Inbal Oren
- Lean Startup – Guy Nirpaz
- The Dolt’s Guide to Self-organization: How to Make Self-organization Work in an Agile Business – Jurgen Appelo
- Visualizing System Archetypes – Visualization and Systems Thinking – Karl Scotland
- Continuous Deployment – Outbrain Case Study – Itai Hochman
All presentations are available at http://agilesparks.com/AgileIsrael2011Agenda.
Following is a description of the lectures I have attended, followed by some of my notes – in case you are interested reading further into it.
Feel free to scroll through it and read about those that interest you.
The keynote was given by Henrik Kniberg from Sweden (see his blog here, including the slides from the Monday’s presentation) , the author of “Scrum and XP from the trenches”. Recently, with AgileSparks help, it was translated to Hebrew.
He gave a very interesting lecture, extending some of the common scrum ideas with some more advanced thoughts.
Some of the items I’ve collected from the lecture:
- Performing “bug bash” Tuesdays.
- Backlog workshop for the next sprint – perform once a week (for example – every Wednesday) – spreading it over the three weeks of the current sprint.
- Provide estimations only on stories and not on tasks. The tasks are small enough, and the estimations on them do not really help, as long as the whole thing gets done. Each task should be roughly the same size of a few hours or one day.
- Focus factor – estimated from previous sprints.
- Don’t fill the sprint – have it about 70% utilized, leave slack.
- Ask the scrum team about their top 3 impediments (during retrospective) – so as to focus on removing these for the next sprint.
- “King pattern” – one of the stories in the scrum is the top priority – “the king”, and everyone should help to get it done. Once it is done, the next important story becomes the “king”.
- Definition of done:
- Merged to trunk (mainline).
- Acceptance tested.
- No increased technical dept (the code base was not messed up as a result).
- Simple design
- Passes all tests
- No duplication
- Readable
- Minimal
- Story that is “ready for development”: define when something is clear enough and is ready for development. This requires to have the estimations and the acceptance tests ready before starting to develop.
- Tests automation backlog – part of the sprint content and gets some of the time within the sprint.
- Top 5 internal (technical) improvements – part of the sprint content and gets some of the time as well.
Gal is the CEO of Trainologic - a company that focuses on software teaching and consulting. I have attended a 6-day Java course that was passed by him to our group in VMware, and I was impressed already back then by his ideas and ability to present them, so I wanted to attend this one.
The presentation focused on the real day-to-day aspects of programmers who would like to consider themselves as craftsmen, and that being agile is an integrated part of it.
We heard some of the similar ideas from Uri Lavi from the "Software Craftsmanship in Israel" group, and I think it is good to being reminded once in a while what “software craftsmanship” means.
Here are some points I wrote down from this lecture:
- Agile architecture – has more abstract models, less little details.
- Have a high-level architecture modeling – whiteboard sketch is enough. See http://www.agilemodeling.com/ as a resource.
- Models (creating these should take no more than a few days, even for a large project):
- Technology diagram
- UI flow diagram
- Domain model diagram
- Change cases: how will you change the architecture when some assumptions change.
- How much initial modeling should be done ?
- Known technology – none or simple.
- Unknown technology – very little.
- Partially known – little.
- Clean code: code is a way of communication and documentation, therefore it needs to be clean and readable.
- Test-Driven-Development – provides an immediate feedback, serves as a safety net.
I have attended the first 15 minutes of this lecture, but was a bit disappointed - as no ideas were quite new for me, and I did not quite like the way the presentation was made.
One thing he mentioned was a recommended book by Marty Cagan (a former Ebay product manager) – “Inspired: How To Create Products Customers Love” .
So I have moved in the middle to the following lecture.
Lean Startup – Guy Nirpaz:
Guy, the CEO of Totango, was talking about the ways agile and lean startups should choose to increase feedback and value creation.
It is of course relevant not only for proper startups but also for emerging product groups within large companies, such as our APM group within VMware.
Guy is also a great speaker and it was an enjoyable lecture.
Some ideas from the presentation:
- Lean is about speed of execution, not the cost.
- MVP – minimum value product – what is the minimum we can do that will make the customer interested.
- Lean startup – driven by a compelling vision, rigorous about testing this vision.
Jurgen Appelo is from Nederlands, and is the author of the “Management 3.0” book.
His lecture focused on the power of self-organizing teams, and the role of team leaders and development managers in an agile environment, and about ways the manager can move the power to the team.
The manager should be:
- Managing the individuals – making sure the rules are fair.
- Managing the environment – to protect the self-organizing system.
- Tell – make decision as the manager.
- Sell – convince people about decision.
- Consult – get input from team before decision.
- Agree – make decision together with the team.
- Advice – influence decision made by the team.
- Inquire – ask feedback after a decision by the team.
- Delegate – no influence, let the team work it out.
Karl Scotland from the UK comes from Rally Software – one of the leading Agile Consulting and Implementation companies.
His lecture focused on understanding the limits and constraints in our processes and how can we remove or reduce them, and find the root cause and reduce it.
The lecture was on a theoretical level for the most of it (using some “feedback loops” models), but had some good examples.
One initial example he gave: You drink coffee when you’re tired. But the coffee just treats the symptoms, to really treat the tiredness you might want to do more sports, have a more healthy lifestyle, etc…
Some notes from the meeting:
- Put limit on tasks: don’t do more than X things at the same time. Adding more tasks will degrade performance.
For example: do not ever have more than 2 items “in progress”.
The limit needs to be found iteratively – add a task and see how the performance is affected, until the correct balance is found. - Shifting the burden: if you have a resource (a person, a hardware), that is a bottleneck, you need to see how to dissolve it (move skills to others, etc…).
- Limiting the total activity, putting activities into “swimlanes” – so that they won’t interfere one with another.
- Classes of “service” – do prioritization, have different classes of “service” for different issues (for example – different service level for bugs from Beta and bugs not from Beta). Always be able to do the most important stuff first.
Outbrain have mastered an excellent continuous deployment process – a developer commit starts running on production in just 30 minutes. Itai leads the engineering at this company.
Their business is to suggest additional related articles when reading an article/blog, so in some cases there are no severe outcomes from a bug.
On other hand, some of their customers are CNN and Ynet, and their servers receive around 100,000,000 hits a day. Their total deployment is over 100 servers.
Some points on their process and how did they achieved the above:
- They have a lot of very separated and single-responsibility services, having EVERYTHING automated (extremely high automatic tests coverage, automatic monitoring of all the systems, constant self checking of all services and modules).
- Their developers are the total quality owners – there are about 15-20 developers and just 3 QA engineers, who’s role is mostly to help the developers to plan the tests and to implement the automation for the feature, and are a very integrated part of the development process.
- Each developer owns a specific feature, and this reduces the interdependencies.
- Each bug or a problem is analyzed immediately, and for each problem they install measures to avoid it next time (automatic test, additional monitoring ability, etc…).
- All code reviews are posted to reviewboard and are done immediately, and the feedbacks are treated with high priority and are fixed as soon as possible.
- The deployment is done on a staging environment as a first phase, and a portion of the incoming traffic is directed to it.
- They get also immediate feedback on performance and SLA of the new code push (they use Nagios and another tool for SLA, didn’t get the name).
- They use TeamCity for their continuous integration.
- For new features, they use on-off switches (flags) and internal (“hidden”) URLs, so that the QA/PM/Support can play with the feature, while the customer is not exposed to these features. This happens for a short time, after which the flags are removed, and the feature becomes part of the main flow.
That’s it for the long summary.
Hope this was interesting and helped you.
Feel free to follow-up with any questions or suggestions.
Baruch.
Subscribe to:
Posts (Atom)