Massive scalability is a key component of elasticity that in turn is the key advantage of cloud computing. Handling massive amounts of data is far from easy whether you use cloud computing or not. To get the real benefits of the cloud there are a couple of limiting factors that needs to be considered – at least that is the way the official dogma goes.

We cannot “have-it-all” with big data

Many seasoned developers/archtects are used to working with, or even designing, databases that offer perfect consistency and very good availability. Sadly, this is a more challenging task with big data.

Spreading out the data on multiple machines is a good way to improve availability. That will enable your solution to serve more requests per time unit and also gives you the opportunity to implement automatic failover.

However, if we use that way to improve availability it will impact consistency. If we save data to machine A and that machine immediately fails, machine B will take over so that our availability remains high. The consequence is that the data that we just saved to machine A will not be reflected on machine B, i.e. our consistency will be less than perfect.

According to the CAP theorem (I will not explain that here) we have to prioritize between availability and consistency. This is a tough choice since we generally want both.

What kind of choices are available?

The choices that we do have is to forfeit consistency or to forfeit availability. This is not as dramatic as it sounds, since we would still have good consistency and good availability! However, sometimes consistency is of utmost importance; It is required that all data is completely consistent at all times in all machines. In such a scenario we have to accept reduced availability. On the other hand, if availability is too important to reduce we can choose to reduce the data consistency. Here are some terms that is important to understand before reading the rest of this blog post:

Eventual consistency is a term that was popularized by Werner Vogels. Data is saved and committed to some but not all machines (often called nodes) before your write request returns. The advantages include that only some of the machines must be available when data is saved and that some time may be saved due to contacting fewer machines before the saving operation returns. However, this leaves a window of inconsistency, meaning that during a period of time requests might be served using old data. Working on reducing the inconsistency window so that it is closed before the next expected request arrives is a recommended strategy.
Read your writes means that if a process writes data it will always have access to that updated data and will not have to deal with older values. (This is only interesting to discuss when dealing with eventual consistency.)
Monotonic reads means that if a process once accesses a data entry it will never be presented to an older version of that data on a later occasion. (This is only interesting to discuss when dealing with eventual consistency.)

The big cloud vendors Amazon, Microsoft and Google all offer data stores suitable for building cloud solutions with huge amounts of data. Amazon and Google offer eventually consistent reads and strongly consistent reads in their products, but Microsoft do not offer any such options. In Microsoft Azure consistent reads is the only option.

Do these choices make any difference?

A very interesting study on data consistency in the cloud that compare these cloud vendors was published earlier this year (2011) by a bunch of researchers based in Australia. Their findings are very interesting and is summarized in the table below.

Interestingly, the results show that during these tests only one option (Amazon’s SimpleDB with Eventually consistent read) gave rise to situations where the reader of the data saw any effects of eventual consistency. Another interesting finding is that SimpleDB using the Consistent reads option was slightly faster – contrary to some of the hoped for benefits of choosing eventual consistency.

As for the Google App Engine Data Store using the Eventual consistent read option the results presented in the table above are marked with an asterisk (*) and here’s why: 11 out of 3,311,081 read operations returned stale data when reader and writer were not running in the same application. This consistency level is very high for an eventual consistency option. The explanation for these results might be that data is fetched from a secondary replica only if the primary one is unavailable. Since stale values were only returned when reader and writer was running in different applications Read you writes consistency seems to be offered.

Some advice

Based on the findings in this research this is what I recommend you to do when you are working with massive amounts of data in the cloud:

1. Express requirements regarding availability and consistency in business terms
2. Carefully consider your availability and consistency options with your business needs and implementation costs in mind
3. Perform tests with realistic machine configurations, realistic amounts of traffic and realistic amounts & structure of data
4. Evaluate and choose you implementation strategy
5. Keep track of how your vendor changes their implementation and what that means for your solution

Expressing your consistency and availability needs in business terms is essential if you want to arrive at a decent solution. Evaluating without being able to compare the positive business outcomes of increased availability and possibly added development costs associated with eventual consistency, might lead you astray. Thinking about perfect consistency might also lead you astray. Although perfect consistency might be good for your business it has to be compared to what the business outcomes of reduced availability will be. Your own tests (and other tests e.g. benchmark tests) may also help you reason about how much performance and availability may be affected by your different options.

The last point (no 5) is very important to remember when you are using a cloud based solution since most vendors changes their hardware configuration, software configuration and software implementation from time to time. When cloud vendors make these changes you may be presented with new options, but some of these changes might just be carried out without giving you any new options; You simply get a “better service” – meaning that you may have to go back to testing again. At the end of the day, re-running your tests on a regular basis might be your best option. That way you do not have to worry about missing out on important vendor updates.

Photo by aftab

Agile is gaining more and more popularity compared to plan-driven software development approaches. There is a strong community push in favor of agile, and truth be said agile has quite a few advantages. However, a couple of questions needs to be asked. Should all projects use agile methods? To what degree should a particular project be agile?

In the book Balancing Agility and Discipline the authors Barry Boehm and Richard Turner take a look at the opposite ends of the agility spectrum. On the one side we have plan-driven methods that prescribe detailed up front analysis, a lot of planning, engineering and detailed documentation. Prominent examples of plan driven methods are Cleanroom and Team Software Process. On the other side of the spectrum we have agile methods that prescribe less up-front analysis and formal documentation, focusing instead on the creation of business value in smaller steps (called iterations). Popular methods for agile software development are XP and Scrum.

Using risk to balance strengths and weaknesses

The authors conclude that plan driven methods and agile methods both have their strenghts and weaknesses. As these methods may have complimentory advantages it may be difficult to select the right approach for your project. In most cases they see that projects would benefit from a hybrid between agile and plan-driven, meaning that you should

incorporate both agility and discipline in proportion to your project’s needs

In the book the authors recommend using risk to guide our choice concerning how much we should lean towards agile or plan-driven methods.The key question to answer in finding the right balance is:

How much [up-front] planning and architecting is enough?

I inserted “up-front” into the quote to firmly illustrate that agile doesn’t mean no planning and no architecture. Up-front should be interpreted here as “before actual development has taken place”.

Identified risks

Identifying the risks that we should take into consideration in order to make a balanced decision between agility and discipline is the next topic of the book. The authors have identified the following risks:

Size This refers primarily to number of personnel. The number of persons that works on a project very often correlate to the amount of code written. Therefore a related measurement of size is KLOC (how many thousand lines of code). This is a very rough measurement since one line of code could perform a lot of logic or very little logic, be complex or simple. Despite this shortcoming the KLOC measurement of size have been used in a study that concludes that large projects can reduce their length by up to 25% by spending more time up-front on architecture, analysis and design.

Criticality A projects criticality is a measure of the consequences when a project fails. The more that is lost when a project fails, the more critical the project is. Loss of lives are at the top of the scale and loss of comfort at the other end. The authors claim that more critical projects require more up-front planning.

For critical projects I personally think that testing is just as important (if not more important) than up-front analysis, design and planning. The agile way of handling a critical project would be to trust the team to arrive at a good solution, but also to verify that it is correct. Creating the right tests and verifying that these tests are correct does require some time, but in my experience testing is just as important in agile as in plan-driven methods.

An aside: I have a small problem with this way to measure criticality since it is negative. Isn’t a project that makes it possible to earn a lot of money critical? Or a project that makes it possible to save a lot of lives? The risk then becomes losing the opportunity to e.g. save a lot of lives.

Personnel This factor refers to the skill of the people involved in the project. To measure skill levels the authors use Alistair Cockburn‘s Level 1-3 (based on the three levels of understanding in Aikido: Shu-Ha-Ri). The authors argue that agile development requires more skillful team members than plan-driven methods. An important note here is that skillful team members are necessary in both cases, but that agile requires a higher percentage of the team members to be skilled.

Dynamism Dynamism refers to the number of changes in a project. If the percentage of requirement changes per month is hight, agile methods should be favored. Such change can come about for several reasons, e.g. that the business persons changes their minds about things, or due to requirements that are emergent rather than specifiable. I think this is a very good way to see it, but please also consider that agile methods often practice a freezing of requirements. So, when you use agile methods you would normally not allow changes to the requirements during an iteration.

Culture The culture of the company or organization is a very important aspect to consider. In this case the scale goes from thriving on chaos to thriving on order. The order or chaos we are talking about here is weather or not people in your company prefer clear priorities and mandates as opposed to many degrees of freedom to get the job done. The more a company thrives on chaos the more suitable an agile methodology is. I do however think that there is some middle ground here. Getting some degrees of freedom within limits seems to be a popular choice. In that case you do not necessarily need a lot of up-front project-specific planning, but would benefit from understanding strategic consideration and some kind of proactive governance.

Visualizing your project’s risk profile

Balancing Agility and Discipline (click for a larger image)

Other possible risks

I have been thinking about other risks that could provide important input when deciding between agile and plan-driven methods. The risks presented in the book may resonate with a wide audience and many projects, but I believe that can be said about other risks as well. Here is a short list of some other such risks.

People churn The percentage of people leaving. In some projects or companies people, and therefore knowledge, tend to stick around for ages. In other cases, we know for sure that people will not stick around. A common example of the latter is when your company hires consultants to do the job. More people churn would favor more documentation, meaning that it leans more towards the plan-driven approach. However, specification by example has become very popular in agile projects. It is an elegant way to document your system in a way that guarantees that your documentation is up to date.

Time to market Seriously, all companies want a short time to market – don’t they? Well, yes – but they want short time to market in different ways. In some cases they want the whole project to be completed ASAP, in other cases they may want a part of the project to be delivered early on and the rest at a later point in time. If you know for sure that you want to deliver a part of the project as early as possible an agile methodology may come in more handy.

Interconnectedness This refers to the number of external systems or APIs that your software must connect to. When it is connected to a lot of other systems you may have a lot of complexity to handle just in interacting with these other systems. Also, this is a sign that multiple teams may be working together in parallell or more likely on different time schedules. This requires a great deal of coordination and is in favor of a more plan-driven approach.

Innovativeness The more inventive or innovative a project is – to me this means it is very experimental – the more important it will be to be able to redo stuff. Planning a lot based on assumptions that may change altogether is not a very good idea. In this case agile development is your best friend. On the other hand, if something has been done multiple times before you may be able to plan in detail what to do – and you may also be able to stop bothering yourself and your company and just buy some COTS software instead of developing the software by yourself.

Conclusion

The kind of risk that you identify can help you determine how agile you should go. Just as important is the fact that the identified risks also can be used as guidance as to what kind of up-front architecture, analysis and planning you need to do. E.g. if you have multiple teams working together – it may be a good idéa to plan for coordinating the teams so that they can work relatively independently from one another. If you interact with several different external APIs you may need to plan for and test those interactions some more.

This way of working also supports another important principle: It is way better to build your method up than to tailor it down from a huge method monster. When you do so you can combine aspects from agile as well as from plan-driven methods. These things are also discussed in the same book.

About handling risks

Making all decisions at the time when ignorance is at it’s peak (at the start of the project) doesn’t seem like a good idea as it can add a tremendous amount of risk. On the other hand, refusing to make decisions early on when you actually do have some information to act on can also add a lot of risk. Ideally we would make all decisions with perfect timing; when we know a lot and have enough time to avoid trouble or take advantage of opportunities. We all know that this is very difficult and sometimes even impossible. A lot of the risks out there are unknown, the worst of those are called unknown unknowns – things we don’t even know that we don’t know about.

A very good risk handling technique is to reassess, that is improving your guess using whatever you have learned since you made your last guess. If you want to try this risk handling technique one key to success is to reduce the amount of work necessary in order to reassess. The more work you have done when you want to change things, the more work you have to do to actually get the change done.

When it comes to handling unknown unknowns agile shines!

Thinking Digital Conference is a wonderful mix of innovation, technology and great thinkers. The 2011 edition of the conference even caught the attention of The Guardian that wrote several articles about it. In this post I want to share some of the impressions that stuck with me after joining the conference and spending a little more than two days in the beautiful Newcastle Gateshead in the UK.

Day 1

The Future of Media

Gerd Leonhard was the first speaker on the first day and he started by discussing how fast information flows. Breaking news can be found on twitter long before it is found on other more traditional media. From time to time it even gets out a little too fast. The number of connected devices is expected to grow to around 50 billion around the year 2020. There is also a sharp increase in the amount of data that is produced, a trend seems to go on year by year. Considering these things the future will not be a fight for distribution, but rather a fight for attention. These forces will bring about changes in many ways, such as in the quality of ads. They will need to become better, so good that the ads themselves are percieved as high quality content. Another way to survive in the future is to think differently about who should pay for what. TDC broadband was presented as a very successful example of changing who pays. They developed TDC PLAY and offered free music to their customers. This contributed to an unprecedented 50% reduction in churn for TDC broadband customers.

About 30 seconds after Walter de Brouwer stepped on-stage I kew he was a true health geek. According to him the do it yourself healthcare movement is unstoppable. The public healthcare system is bypassed over and over again, simply because health is worth investing in. He observed that medical records consist of punctuated data with huge voids in between since we tend to measure health only when people are sick or dying. A number of cool devices exist to capture more information, and some people capture a lot of information (a health focused version of life caching). Using this information we are able to learn about what things that affect our health and act accordingly. Taking a pill is easier but what we really need is to change our behavior, so maybe the doctors of the future will prescribe apps instead of pills.

Walter continued by pointing out some important things for the health information industry. Health and privacy does not go hand in hand since private data cannot be used in research or other ways of learning. To improve knowledge of health he suggests we should share health-related information. Not only anonymously, but perhaps openly between friends. That way we could find answers to why some of our friends are feeling so good. Do they sleep more, exercise better or eat less? Another important aspect is that all this health related information must be simplified and presented in a way that is easy to understand. In Walters words “Health will become a number”. Also, gathering all this information must become easier. This could be achieved by incorporating sensors in everyday objects such as toothbrushes and clothes. His future vision for health is something similar to OnStar for humans. Constant monitoring in combination with convenient suggestions for measures should be able to improve health for any of us.

Thinking Digital

Bobby Paterson did a good job presenting Happiest – a social motivation network that facilitates happiness by focusing on 7 healthy habits: Gratitude, Eat healthy, Wisdom, Get active, Meditation, Mantras and keeping a happiness Journal. According to Herb Kim they’re kind of like TripAdvisor for happiness. In his talk Bobby referred to Chip Conleys TED talk Measuring what makes life worthwhile where he talks about GDP versus GNH . If we can find out what makes us happy that information can be useful to ourselves as well as others. He also thinks that businesses can evolve themselves a lot if they are mindful of their happiness footprint.

Erin McKean is an american lexicographer meaning that she is very much into words. Her word of the day was skeuomorph and she set that in context by discussing how design features that were important in the physical world sometimes are pointless or even counter-effective in the digital world. The wordnik site that is ”a place for all words” had to find new solutions to the fact that dicitionaries took a long time to develop. The main problem was that there were too few lexicographers. Their solution to the problem came about when they took a look at how people understand words. A definition doesn’t help us as much as a good example and understanding how a word relates to other words does. So, at wordnik anybody can help build the dictionary by providing such things. To make this possible they had to develop data storages that could hold a lot of relations between words. This has become a huge wordograph with a humongous number of words and connections between words. After a while they realized that this meant that wordnik had become something else than what they set out to create – a meaning discovery engine. It can answer a variety of text based questions. Today they look into commercializing this so that companies can answer questions that their clients have using information from their own websites.

Conrad Wolfram talked about how we can deal with information overload and where computation meets knowledge. Conrad stresses that we do not need search, but instead a good way to provide answers. In his own words computational knowledge is worth much more than dead information. He also feels that this should be the case in the public sector, research & development and the corporate world. He provided samples like financial reporting, natural disasters and even natural language searches where the user can modify guesses and preferences and immediately see the effects of those changes. To get to this new level of knowledge we need access to the right tools and better education. He feels that the math education should focus less on calculus and more on the application of math. In 2010 he delivered an excellent TED talk on that very topic.

Mind over matter

A mobile manspace

Sam Martin delivered an entertaining talk about Manspaces. It quickly lead to an intense twitterburst and a comment saying that manspace should be interpreted the same way mankind is. Manspaces can take various different, and sometimes very surprising, forms although most of them are rooms or small buildings. Manspaces is all about getting away from things in order to wind down and relax.

Resonance

Ewan McIntosh had an interesting story to tell about how he worked with the Scottish National Party during their historic win of this year’s election. He stressed that he isn’t politically active, but just likes to win. He told us a story about how they were able to crowdsource politics by using an online driven strategy. To succeed in this campaign they needed a strategy (e.g. to always be positive online), a clear message (e.g. to always deliver the same message) and learn how to listen (using social media). They spread good information and was able to find out who were for, against or unsure about specific and important political resolutions. The whole reason for the online effort was to make people do offline activities like discussing with people and of course voting. Ewan feels that they have changed how politicians in Scotland are listening to people.

Heather Knight did an inspiring presentation on robotics. Actually, she let her robot Data help her with part of her presentation – which he delivered in great style! Heather is interested in using robotics to help us communicate better with computers, and suggests that we should replace computers – not humans – with robots. Data even danced for us – or was it for Heather? Anyways, his dance was a much appreciated part of the conference that you too can see below.

Day 2

The Medium is the Message

Matthew Postgate talked about broadcasting in the information age. Obviously broadcasting has to evolve as the information age evolves. Matthew provided two interesting experiments initiatives coming from BBC. One of these initiatives was synchronizing interactive media with traditional TV programmes. The other was to me slightly more innovative and called sorround vision. Using special cameras they are able to project the surroundings of a (kind of) normal recording on the walls around the TV.

Caspar Berry delivered a very animated talk on uncertainty and risk in poker, business and life. He started off the presentation by explaining exactly how his presentation would end since on a previous occasion somebody had complained that they felt a bit uncertain as to where his presentation would go. A funny move from a man that knows very well that uncertainty is a part of our lives – whether we like it or not. The worst kind of uncertainties are the unknowns that are unknown, that is the things we don’t even know that we don’t know. Takes a minute or two to find out what that is, doesn’t it? He drew our attention to semingly unimportant details such as a small piece of metal lying on the runway when Concorde Flight 4590 was to take off, or exactly when (the exact second) our parents, grand-parents or grand-grand parents mated. Plane crashes and our very own existence is affected tremendously by details that we cannot control or even set in relation to important events – at least not before the events occur. This is hard to accept for mankind, since studies have shown that we need control. If we feel that we are out of control there may be serious impact on our health. Caspar illustrated this by telling us about several experiments involving electric shocks and taking away persons possibilities to affect their living conditions. Surprisingly, experiments with electric shocks distributed at random have been repeated on numerous occasions but the other experiment will probably not be repeated at all. The persons that were unable to affect their living died much earlier than the control groups. This need of being in control leads to a lot of behaviors, including that of trying to predict the future. Caspar recommends instead that we embrace uncertainties and adopt a language that acknowledges uncertainties. He ended his talk by stating that “it is only because tomorrow is uncertain, that today has any meaning for us at all” – if everything is certain there is no point in doing anything.

From data to wisdom

Chris Hatala and Wesley Burt from Massive Black made an impressive live demonstration creating original artwork and an animation on stage. We were also shown how a few interesting scenes of Lord of the ring, e.g. when Shelob wrapped Frodo (about 1:10 into the clip), came to life by the use of prototyping.

Dr. Vincent Li works on what could very well become a medical revolution. Normally our blood vessels grow very much when we are in the womb. Apart from that blood vessels grow during menstruation, pregnancy and when wounds are healing. Recent research shows that we need a good balance in how the vessels grow. A balance between growth and pruning. If that balance is upset it can lead to a variety of diseases. The only thing these diseases seem to have in common is their link to the growth of blood vessels – a.k.a. Angiogenesis. This knowledge has amongst other things led to new forms of cancer treatment. The theory behind it is that cutting off the blood supply to a tumor kills it. Since the blood vessels in the tumor is different from those in healthy tissue some drugs that are targeted to prune these blood vessels have been created, and in practice some amazing results have been shown. However, Vincent and many others with him would like to see better results. One big problem that greatly reduces the results is that cancer often is diagnosed in a late stage, remember Walters talk? Other interesting things have also been discovered about the drugs that affect angiogenesis. The active substances of the drugs can be found in our food, and experiments shows that combinations of substances can radically change their effect. So, it is entirely possible that foods can be more effective than drugs. This new knowledge has led to a new insight; Prevention of disease may be possible by eating the right foods. Green tea, dark chocolate and tomatoes are foods with specific effects that are being catalogued. Even the way foods are cooked may matter. An example is that tomatoes that have simmered in olive oil have a much better effect than raw or cooked tomatoes alone. It is in other words possible to eat to starve cancer in its infancy! Obesity is also a disease that is liked to angiogenesis. It is therefore possible to add, rather than to remove, foods to prevent obesity. There is a great and expanding catalogue of information around this topic to be found at eattodefeat.org, a site that is maintained by The Angionesis Foundation, a truly applaudable initiative! Vincents brother William did a similar talk on TED a while ago.

Jer Thorphe calls himself a digital artist and has worked a lot with data visualizations. In his talk he showed a graphical representation of twitter discussions concerning New York Times articles. These discussion threads are so complex and contain so much information that that graphical representation is necessary for humans to be able to analyze them. Using his tool he was able to pinpoint the tweets that lead to bursts in the conversations. He then went on to talk about Twin Tower Memorial. He was assigned with the task of devising an algorithm that took into account the relations between the victims of the 9-11 attack and then laid out the names in a suitable way. Names were carved into plates so that related names were close and all the names had a good position on the bars. Since there are a lot of aesthetics and judgment calls to be made in this kind of work, Jer developed an algorithm that delivered computer aided design instead of letting the computer do all the work. This project was only recently made public and you can read more about it in the New Yorker.

Cognitive surplus

Tan Le is without any doubt a world leader in the field of human-computer interfaces. Her team was faced with the task of finding an unobtrusive way of reading peoples mind, or rather reacting to different states in the brain. They considered several alternatives before settling for EEG. Although EEG was a promising technology the technical equipment that was available at the time was both expensive and difficult to operate. On top of that the resulting measurements were difficult to interpret correctly. After a period of intense research in the fields of hardware and software they have found a good solution. The have an open source headset with an accompanying software API Using this multiple companies have started to develop applications that Tan couldn’t have envisioned all by herself. There are games available where the gamer is empowered with telekinetic abilities but also applications for controlling wheelchairs. In fact there are already a number of products in the making in multiple fields. Tan even did a demonstration together with a brave guy from the audience that was fitted with the Emotiv Epoc headset. In just a short time he was able to use only his mind – hmm and the headset – to move a cube on the screen. Certainly interesting and inspiring. Around this time someone with a brain on hyperdrive tweeted that it would be cool to live in a world where we had robots like the one from Heather that used Wolfram Alpha to solve problems and that we could control by using the emotiv headset. To me that qualifies as a true vision, but maybe not a true prediction, for the future!

Aftermath

The conference was great, my compliments to the organizers, speakers and the talkative delegates. All these ideas and information will take some time to process. But I’m sure I will get a second crop out of it (apart from the nice experience of being here). It was definitely worth travelling through the ashes to get here. (But I do hope I’ll get back home…)

See the panorama of river Tyne in 3D @ Photosynth

[update 2011-05-29]

Eat to Defeat and WolframAlpha – a fruitful cooperation?

I just had an idéa. How about letting the WolframAlpha team work with the Angiogenesis Foundation to make the datasets of Eat to Defeat computable? Wouldn’t it be great to be able to learn about suitable foods using age, gender, height, weight and perhaps symptoms, problems, diagnoses (e.g. allergies) as input? Apart from a list of suitable foods suitable combinations of foods and even recipes (plus things that I cannot imagine right now) could be presented in the results. I would certainly like to be able to do that!

Photo by Saguaro Pictures

A metaphor is an analogy between ideas. We use metaphors to explain or understand something in terms of something else. Some metaphors are implied by everyday phrases, e.g. “the foot of the mountain”, “raining cats and dogs” and “time runs fast”. The latter one is particularly interesting. Time is a concept that can be hard to grasp, at least compared to mountains and rain. The expression “time runs fast” implies that we think of time as physical movement or a journey. This will make us think of not only time but our lives as a journey.

The tendency for us to think in terms of metaphors is so widespread that Jonathan Haidt, a professor of psychology at the University of Virginia, claims in The Happiness Hypothesis: Finding Modern Truth in Ancient Wisdom that humans need metaphors:

Human thinking depends on metaphor. We understand new or complex things in relation to things we already know.

Gareth Morgan, a research professor that specializes in management thinking and works at York University in Toronto, puts it slightly differently in his book Images of organizations:

all theory is metaphor

On the very same page of that book Gareth Morgan goes on to state a very important fact:

Metaphor stretches imagination in a way that can create powerful insights, but at the risk of distortion

Although metaphors can help us gain new insight, all metaphors are incomplete, biased and skewed. A metaphor helps us see certain things, but makes us blind to other things. Gareth’s advice is to use a number of metaphors collectively so that the different metaphors can complement each other. Allowing multiple metaphors to complement each other may not be an easy task but according to Gareth it is well worth our time.

So what metaphors can we find that are relevant for software development? I have gathered some of my favorite metaphors in this post. They complement each other in that they see the same phenomenon (software development of course) with very different eyes (remember that’s where the beauty is). Armed with these metaphors we can understand and focus on multiple different and important aspects of software development.

Building airplanes in the sky

This metaphor adds the insight that when we create software for a business that is up and running we cannot disturb their current business. We have to create and introduce software in such a way that we do not stop or disturb important business activities. This can add a lot of new considerations and therefore make our jobs more complicated. Being aware of this added complexity is a good thing. EDS made a commercial a few years back that illustrates this metaphor. The film is brilliant and only takes a minute to watch. Enjoy!

Herding cats

Herding cats is a figure of speech that is used to convey that a task is extremely difficult. We have an image of cats as being individualists that doesn’t cooperate very well. To bring cats together in a herd is therefore a very challenging task. In some respects creating software is similar to herding cats. Bringing a group of diverse stakeholders together and make them agree upon things can also be a daunting task. Making ideas, information and technology work well together is perhaps easier, but still a challenge. Awareness of these kinds of problems can help us prepare for the tasks and realize that we need to handle that kind of problems in order to create successful software. Again, EDS has made a very entertaining and illustrative commercial about herding cats that I really recommend viewing.

Running of the squirrels

The running of the squirrels is yet another idea signed EDS. This time the point is that for most companies the running of the bulls – that is avoiding the big and strong competitors – is not the biggest issue. Instead you need to worry about handling squirrels – the small, fast and nimble competitors. To do this we need software that is easy to change and adapt to new business requirements. Another one-minute commercial illustrates this.

Technical debt

The Technical Debt metaphor was coined by Ward Cunningham and is a financial analogy. Borrowing money is a quick way to get hold of new funds and indirectly other resources. As most of us know, when you have borrowed money you need to repay that money at a later time. You should also be prepared to pay interest rate. If you just keep borrowing money you will eventually have to pay so much interest rate that paying interest rates is all that you can afford. If you throw some software out there, quick and dirty, again and again, you will eventually end up in a situation where the code becomes impossible to work with unless you clean it up. The quick and dirty approach is analogous to borrowing money and you now have a sizeable technical debt. With such a debt on your hands you will be terrified to add new features to your code (since you do not really understand it), furthermore you will most likely have to spend a lot of time on fixing defects (a.k.a. bugs) to prevent your software from malfunctioning. The software development analogue to repayment is refactoring. By rewriting and cleaning up your code you make repayments so that you eventually can understand the code and work effectively with it again. In this interview Ward Cunningham explains by himself how he developed this particular metaphor.

A collaborative game

Alistair Cockburn came up with the idéa that software development is a cooperative game. In this analogy a game is to be thought of as “An activity providing entertainment or amusement” so that the competition aspects of games are reduced – but not eliminated completely. The team developing a piece of software can still be competing against other teams that develop similar software. The game is goal seeking (finishing the software) and therefore has a natural end. However, the joy of developing software is not reduced to reaching the goal. Much of the joy comes from the process of developing the software, much like rock-climbers enjoy the climb – not only reaching the peak. Also, the goal of finishing the software is a secondary goal. Secondary to things like gaining knowledge, growing the company, increasing ones influence and creating a career. Another secondary goal of the game is to create an advantageous position for the next game that consists of altering the system or creating a neighboring system. This is done by learning about the current system and creating markers or reminders of what was learned. Jeff Atwood concludes that the secret is:

learning how to play the game so that everyone is having fun

Design

The design field is described as focusing on producing specifications that can be used to build products that are intended to accomplish certain goals under certain restrictions. This metaphor emphasizes that the process of software development is the cultivation of detailed specifications or instructions that can then be used to build programs. In both product and software design it is important that the end product is functional and that the product is easy to use and maintain. In both disciplines it is highly recommended to use proven design concepts and base the designs on the work of others. A great example of the success of this metaphor is the popularity of software design patterns.

If I missed out on your favorite software development metaphor then please comment on this post and I will update it accordingly.

[update 2011-04-20]

Gardening

Andrew Hunt and David Thomas put forward the idea that programming is similar to gardening in their book The Pragmatic Programmer. They argue that engineering and construction is very orderly but also unrealistic, and therefore not very much like software development. In gardening you would instead expect a lot of less orderly things to happen, e.g. a plant gets too big or dies, actual colors of the plants doesn’t mix well and due to its organic nature a garden needs constant maintenance or tending. In an interesting interview Andy Hunt says that

You do plan. … You’ve got a great plan, a whole design. … [but the] garden doesn’t quite come up the way you drew the picture.

This metaphor can make us see software development as something that is not as neat as our orderly plans, and be prepared to constantly work on improving the resulting software. Unlike most buildings, a software program can be changed and molded quite a bit after its initial delivery.

Original photos by onesecbeforethedub and jurvetson

There has been a lot of debate around what is better; SOAP based Web Services or RESTful services. This debate is sometimes surprisingly heated with expressions like WS-Deathstar and RESTafarian tossed around all over the place. This is yet another interjection into that debate that specifically focuses on the public cloud.

To make the context clear, in this post I want to discuss services that are publically available and hosted in the cloud – SaaS. The services I have in mind are services that are meant to get a wide adoption across multiple countries, technologies and devices.

Multi-tenancy

According to the NIST-definition of cloud computing the ability to pool resources to serve multiple client organizations (a.k.a. tenants) is an essential characteristic of cloud computing. A multi-tenant service is a service that serves many. The opposite of multi-tenancy is having different services for different clients.

Supporting multiple clients

Supporting multiple clients with one service can be difficult as it involves designing for reuse. Neither Web Services nor REST services can guarantee that our services will be reusable or posess any of the other beneficial properties that we commonly associate with SOA. Despite that fact I still think that REST has an advantage over Web Services in this case – and the reason is purely about technology.

The dominant (or only) way to implement REST services today is to utilize HTTP. HTTP is supported by a huge amount of server and client technologies that are found in various different assets ranging from the biggest servers (even mainframes) to PCs and cellphones. This is a really good match for multi-tenancy as it is to be expected that different client organizations consume services using different technologies and different assets. HTTP was recently described as a credit card that is always accepted everywhere.

As for Web Services there is extensive support for that too, but way less than the support for HTTP.

Servicing multiple clients

When you are servicing multiple clients you need to worry about performance. A service with better performance will be able to handle more requests with the same amount of resources. It will help you provide a good enough service to your customers in a more cost-effective way. In a lot of publically available services, information is read more often that it is written – this is (hopefully) true for tweets, facebook status updates and blog posts. Again, the difference between Web Services and REST is purely about technology.

The semantics of HTTP allows us to define in a standardized way how data should or could be cached. Due to this reason a horde of web enabled products have built in support for HTTP caching. Better yet, this caching functionality can be utilized in multiple layers. It means that not only the web server that hosts your cloud solution will be able to cache your data. Caching can be used in web servers that the request will pass on its way to your client, or even in the client software. When data is cached, some of the read requests to your service will never even have to reach your server. It will make the client applications perform better, and enable your service to take care of more clients.

As for Web Services there is no way to define caching in a standardized way.

Real usage

Another way to investigate this question is to take a look at the real usage out there on the web. Statistics from the Programmable Web shows that REST has gone from a 60 % to a 74 % dominance on publicly available APIs. SOAP based Web Services on the other hand has plummeted from 25 % to 15 % during the same period. Back in 2003 (I haven’t been able to find anything newer than that) Amazon found that 85 % of customers preferred their RESTful API (they offer both REST and SOAP).

Conclusion

For services tageted for the public cloud, REST is a better option than SOAP based Web Services.

REST versus SOAP revisited

When people discuss REST in contrast to SOAP based Web Services they generally think of it as a dichotomy, when in reality we’re dealing with a whole range of options that is more akin to a continuum. Web Services with a lot of WS-* have a different level of complexity compared to Web Services with a little WS-*. Web Services without WS-* is even simpler than that. Of course the functionality that they can offer differ too. Using WS-* standards like WS-Addressing and WS-Security can offer detailed addressing features and a level of security that the simple Web Service cannot offer.

Original photos by onesecbeforethedub and jurvetson

REST also comes in different flavors. Some of the flavors below may not be considered to be RESTful by all REST proponents, but when we discuss REST versus SOAP based Web Services they all tend to end up in the RESTful corner.

The POX – Plain Old XML over HTTP – approach use HTTP as a transport protocol and transfer data using XML. It is similar to SOAP based Web Services, but doesn’t have a SOAP envelope or a WSDL. It is a way of doing RPC with data transferred in the body of the request.

The URI tunneling approach use HTTP as a transport protocol and is further based on URI templates. It uses the URI to transfer data, i.e. the data is transported inside the query string.

The CRUD – Create Read Update Delete – approach uses HTTP as an application protocol. Resources are identified with unique URIs and HTTP verbs (POST, GET, PUT and DELETE) are used to manipulate these resources. Data is transferred in the body and can be encoded using XML, JSON or other formats.

The HATEOAS – Hypermedia As The Engine Of Application State – approach uses HTTP as an application protocol and identify resources using unique URIs, just like CRUD. Furthermore it uses hypermedia, essentially links and forms, to guide the consumer through a number of interactions with the service. Popular standardized media types that allow for the use of hypermedia are ATOM and XHTML.

These different REST flavors are positioned at different levels of the Richardsons maturity model. To learn more about REST (as well as Richardsons model) please refer to the REST in Practice book.

The maturity of publically available REST APIs

It seems to me that far from all those public RESTful APIs on the web qualify as hypermedia-enabled services. Below follows a few randomly picked examples that reinforces that thought:

• Twitter: This search result (XML+Atom) offer some links like the next page of the search result. However, they do not offer links like
  • retweet
  • reply to
  • follow
• Facebook: This search result (JSON) offers no hypermedia.
• Linkedin: The Connections API call (XML) offers no hypermedia.
• Amazon Elastic Compute Cloud: The Start Instances API call (XML) offers no hypermedia.

Update: Cesare Pautasso suggested I also link to the paper that he, Olaf Zimmerman and Frank Leymann published back in 2008: RESTful Web Services vs. “Big” Web Services.

Original photo by worak

Today J.B. Rainsberger made a very commendable effort to simplify the Agile Manifesto. He was inspired by Simplified English and the reduction of the ten commandments into two simple rules. In his post he challenge us (everybody) to find any meaning in the Agile Manifesto that his simplified version do not cover. I urge you to take a look at it.

Inspired by J.B. I made my own attempt at simplifying his simplified version of the value statements (in his post he also simplified the principles of the Agile Manifesto).

• People working together over precise rules
• Doing what’s needed over keeping old promises

It is much shorter and (at least to me) easier to understand than the original value statements of the Agile Manifesto:

• Individuals and interactions over processes and tools
• Working software over comprehensive documentation
• Customer collaboration over contract negotiation
• Responding to change over following a plan

Photo by jworth

Cloud computing is the hot buzzword of the day and is currently at the very the top of the hype cycle. I do believe that there is more to the cloud than that hype, meaning that some companies could benefit greatly from cloud computing. In spite of the hype and those real opportunities that the cloud brings, some things may stop companies from actually adopting cloud computing.

Below I will briefly describe what I believe to be the three most common showstoppers, plus one that I stumbled upon and – quite frankly – was a little surprised to learn about.

Trusting the cloud provider

We (that is humans) have a tendency to trust ourselves more than we trust others. We like to hold the steering wheel by ourselves. We can be intimidated and scared when we are not in control. Consider flying in a commercial aeroplane: The flight captain makes a living out of flying. She has great education and training. Statistics tells us that flying is way safer than driving a car. We have a tencendy to feel that doing something by ourselves is safer, but in reality we can create a tremendous mess. When we think about how somebody else performs a task we also have a tendency to compare that to how we would be able to do it given an abundance of time, money and other resources – and that is unrealistic most of the time.

The fact that cloud provider is a relatively new service is bound to reduce the trust further. Many cloud platforms are immature, despite the fact that they are built upon more mature technologies and products. At most the cloud providers have been running their cloud based services for a few years. This will definitely be a warning sign for companies that shy away from adopting new technology.

One way to increase the trust is to have the provider sign a contract that would give you as a consumer a very large economic compensation should something go amiss. It may seem to be a good idea, but would make contracts with cloud providers much more difficult to agree upon. Also, with the prescence of such a contract and in the event of an incident, the provider may have to pay up large sums of money – maybe so large sums of money that they will be bancrupt. And then you will be without a cloud provider, even if the provider didn’t let you down. If you are able to negotiate this kind of agreement you can be sure that others have too.

Legal issues

If you live (like I do) outside of US where most of the big cloud vendors work, legal issues may be more difficult to handle when comparing cloud computing to regular hosting. This has to do with the fact that regular hosting can be confined within one country, whilst cloud computing is often international. Some kind of information comes with legal demands that it must be saved and stored say within the borders of EU, or even within the borders of one specific country. Even if you do not have to take such regulations in mind you may want to think about legal issues anyway.
Some laws in the hosting companys country may apply for the information that we store in their servers, even if the actual servers are located in our own country. One example is the Patriot act. The Patriot act demands US government access to information upon suspicion of terrorism, and may affect data that is stored in Europe by a European customer if the cloud provider is a US company. It is important to notice the word may here. To know for sure somebody needs to

1. Store data in the cloud using a cloud provider from another country or continent.
2. A law, e.g. the patriot act, must be waived.
3. There must be a dispute that is not resolved through negotiations.
4. The courts will decide what laws apply.

Until something like this happens we simply do not know. To make matters worse laws change. To keep up with how laws change in one country may be difficult. To keep track of and adapt to complex international law is not to be underestimated. Some initiatives, like the safe harbor framework, aim to reduce these insecurities.

Convoluted enterprise architecture

Companies that have little or no enterprise architecture (that is they do not have a fair amount of control over their information, processes and existing services and applications) will find the adoption of cloud computing to be very troublesome. In order to do that they need to literally rip out parts of their existing system landscape and / or open up parts of their existing landscape and then do complex integration efforts. Integration is no easy task, and with a convoluted enterprise archtecture the task is extremely difficult, time consuming and costly. Sometimes you may get away without complex integration, but security will stop you. A lot of companies have invested time and money in creating a Single-Sign-On solution. It may be a very big challenge to extend that Single-Sign-On into the cloud. Not extending it to the cloud would lead to extra efforts in managing access rights, with the added risk of making mistakes and thereby compromising security. Sadly, too few companies have a Federated Identity solution in place today and that will be a great showstopper. SOA may help companies be better prepared for the cloud, but most companies that I have heard about are very far away from the taget state of SOA.

The unexpected showstopper is OPEX

Elasticity can be described as the ability to rapidly scale out to unprecedented demand without big up-front investments, plus the ability to save money when scaling in. Elasticity and pay-per-use are the most important selling points for cloud computing. A consequence of elasticity is that expenses are moved from CAPEX (Capital Expenditure) to OPEX (Operational Expenditure). It should come as no surprise that some companies would prefer to spend some capital up front, (that is investing in hardware and software, to be able to save money in the long run) while others prefer to pay as they go and thereby reduce risks as well as the need for capital up front.

A conversation that I had revealed another way to think about this that I hadn’t thought of before. A CIO claimed that he didn’t want to be in a situation where his superiors could tell him to cut his operational costs by reducing the amount of cloud resources he consumed. To him the fact that his services and applications were hosted in the cloud actually meant that the risk of loosing those resources was much higher than if he had invested capital to buy his own servers and software. I guess he is right. OPEX isn’t automatically better or safer than CAPEX. It all depends on how you see things and where you come from.

I guess it is true then: The CFO, not the CIO, will drive cloud adoption…

What do you consider to be the biggest showstopper? Please let me know!

[Thanks to Thomas Rischbeck for commenting on a daft of this post]

A new and improved description of the Idempotent Capability pattern has just been published @ soapatterns.org. The pattern explains why idempotency is important as well as why and how a non-idempotent capability can be transformed into an idempotent capability.

Many thanks to my co-contributor Cesare Pautasso as well as rewievers Radovan Janecek, Ian Robinson and Nicolai Josuttis.

The third installment of the International SOA Symposium and the second installment of the International Cloud Symposium will be one huge co-located event in Europe from the 5th to the 6th of October 2010. The first time around it was arranged in the beautiful city of Amsterdam, the second time (and the first time for the cloud symposium) in the modern city of Rotterdam. This year it will come to the world city of Berlin.

I will be presenting at both the SOA and the Cloud symposiums alongside multiple highly esteemed SOA and Cloud computing experts. I’ve had the pleasure of working with several of them e.g. as a co-author in the Service-Oriented Computing book series and the SOA Manifesto. Some of them I have also met at one or both previous symposiums.

Also, do not miss the panel discussions where like-minded and sometimes wildly different thinkers (I guess that makes if more fun!) meet to discuss hot topics. This year I will participate in the panels “Revisiting the SOA Manifesto” and “SOA Design Patterns”.

Hope to see you there!

PS If you are a member of IASA or Cloud Sweden please check your Linkedin group to find information concerning discounts for these events!

Finally! The book is officially lauched and available at well-assorted bookshops.

Local pre-launches in Stockholm

The book was actually pre-launched in Stockholm on two occasions:

• SOA Summit, where we handed out and signed 20 copies
• Beakfast meeting with SOA School, where we handed out and signed 5 copies

These copies were printouts of a an earlier version of the manuscript (weeks earlier not months earlier) so those of you that won those books have a “special edition” that is close to the final version!

Official launch at TechEd US 2010

Yesterday the book was officially launched at TechEd US. I would have loved to be there to present together with my co-authors, but sadly I wasn’t able to make it. Some good news is that the book was on the bestseller list of the TechEd Bookstore both monday and tuesday! (Possibly also for the rest of the week?)

Read more about the book here: http://www.soabooks.com/net

My co-authors were (listed alphabetically by last name):
David Chou
John deVadoss
Thomas Erl
Nitin Gandhi
Hanu Kommalapati
Brian Loesgen
Christoph Schittko
Mickey Williams

I would also like to exted a special thank you to my friend and former colleague Arman Kurtagic that helped out a lot with the book.