Mozzilla Thinks Crashes are a GOOD Thing…Really?

My six-year-old can tie her own shoes. I honestly did not realize how big of a deal that was until her teacher told me a few months ago that she had, for a short time, become the designated shoe tier in her classroom. Apparently, thanks to the advent of Velcro closures for kids’ shoes, nobody else in her kindergarten class knew how to tie their shoes.
The problem with being a “star” of your kindergarten class, however, is that all the kids want their shoes tied by her. As a result, she was trying to tie shoes very fast – too fast, in fact – and started making mistakes, which got her frustrated when the knots don’t come out right.
Seeing this frustration, I calmly remind her that it is better to do something right than to do it fast. This is a lesson many software development teams also need to remember.
While you would think “getting it right” should be the first mantra of developers, though, we see more and more examples of teams finding ways to do things “faster” rather than focusing on quality. While it is true that in order to keep up with competition and demand the current market dictates shorter development cycles than decades past, that does not mean quality needs to be sacrificed or done “on the fly.”
Nevertheless, eschewing quality for speed seems like that’s exactly what’s going on over at Mozilla. Over on the appropriately named blog “It Will Never Work in Theory,” a section from a paper titled “Do Faster Releases Improve Software Quality? An Empirical Case Study of Mozilla Firefox” by Foutse Khomh, Tejinder Dhaliwal, Ying Zou, and Bram Adams is studied. The paper finds:

Users experience crashes earlier during the execution of versions developed following a rapid release model.
The Firefox rapid release model fixes bugs faster than using the traditional model, but fixes proportionally less bugs.
With a rapid release model, users adopt new versions faster compared to the traditional release model.

The post goes on to evaluate these findings, noting that the third point is good news, item two is kinda good news, but item one is a head scratcher.
I doubt anybody would find fault with the third finding from the Mozilla case study. Adopting new versions faster than traditional models is certainly a positive in business. However, there’s something missing here; are developers building these versions faster AND stable or are they just developing them faster without concern for application software quality? If those versions are just being done faster and are not being built well, they will require development teams to go back and constantly fix issues and could possibly lead to major malfunctions that interrupt business continuity. I don’t see how that is a good thing.
What is really confounding, though, is the first finding in the Mozilla case study, “Users experience crashes earlier during the execution of versions developed following a rapid release model.”
How exactly is that positive in any way, shape or form?
The last I checked, the earlier an application crashed the more poorly written, less reliable, more destructive and more useless it was. I don’t think there’s a single Marketing department in the world who could successfully promote its application software by saying, “We Crash Faster!”
I would have to guess the authors of the case study believe that if users experience crashes earlier in the execution of versions it means developers at Mozilla can start fixing those bugs sooner. That’s not a great way to run a business, though. Where’s the concern for software quality? Moreover, when did Mozilla start paying its users to serve as its software quality inspectors?
Maybe that’s why Mozilla Firefox is offered as a free application to its users…because users should know they get what they pay for!
Eventually my child’s teacher stopped sending students to her to have their shoes tied because the teacher was just having to retie them. I suspect as Mozilla users experience these earlier crashes, they, too, will look elsewhere to “have their shoes tied.”

Android Application Failures Still Try Our Souls

Happy Independence Day everybody! I only hope those of you reading this on your Android device have not turned it sideways or performed some other seemingly innocuous action that has made this application fail.
I say this because I recently read yet another blog about “workarounds” to compensate for application failures inherent in Android devices. These pieces have become almost ubiquitous over the past 18 months to the point where one would think Google would just go back and perform the structural quality analysis it needs to do to address the issues.
Their failure to do so reminds me on this day before Independence Day of the opening lines of Thomas Paine’s “Common Sense”:
These are the times that try men’s souls: The summer soldier and the sunshine patriot will, in this crisis, shrink from the service of his country; but he that stands by it now, deserves the love and thanks of man and woman.
As Google continues to “shrink” from its responsibility to provide application software that is of sound structural quality, they are certainly “trying men’s [and women’s] souls.”
I Have Not Yet Begun to Fight!
I continue to be amazed that Google appears more interested in what to call their next Android OS. As “enamored” (can you feel the sarcasm dripping from that word?) as I was last year with “Ice Cream,” I am even more captivated by their latest one – Jelly Bean. I am betting he name really fits the product  – looks solid on the outside, but if Android’s history is any indication it will most certainly be a piece of gelatinous mush on the inside.
Maybe Google continues to fall into the trap of believing its own press clippings – the positive ones, at least – because they seem more concerned with marketing than they are with software quality. Google’s mobile operating systems continue to feature one flaw after another with these flaws not being “discovered” until after the system has been rolled out and installed by the consumer. And these flaws are not just minor ones that inconvenience the user like the ones mentioned in the work arounds blog to which I referred above. They include battery-draining and security flaws that cost time and money for those using the devices..
Nevertheless, they continue to build one iteration after another atop mobile platforms they know to have flaws – or at least by now they should know – and they continue to fail to fix them.
We Find these Truths to be Self-Evident
It’s truly a shame Google won’t use the same methodology as Thomas Jefferson and the Continental Congress did in forging one of the world’s greatest documents – the Declaration of Independence. From the time Jefferson began working on the Declaration of Independence on June 11, 1776, he wrote and rewrote, edited and re-edited versions of the document for almost three weeks until he came to what he felt was a product of optimal quality. And yet even after the document with all those versions and all those edits was presented to the Continental Congress on June 28, 1776, for a vote, those men debated for another five days over the contents of the document and made another 33 changes to it!
Obviously there was no Marketing Department pushing Congress to get out the final product.
The Declaration of Independence – a document that truly did have urgency behind it as men were fighting and dying for the values it espoused – was edited and changed many dozen times before it was delivered. If that’s the case, why can’t the marketing people at Google allow their developers to perform a bit of automated analysis and measurement on Android software before they declare its “independence” from internal production? Were they to do this (harkening Paine again), they would “deserve the love and thanks of man and woman.”
These truly are the times that try our souls.

Foretelling Facebook’s IPO Failure

I’m not one who believes in fortune tellers or those who claim to be able to predict the future. Heck, I don’t even read my horoscope and cringe whenever someone attempts to force it upon me. Only when my wife has attempted to read me my horoscope have I offered even as much as a polite “hmm.” Nevertheless there are many out there who swear by those who claim to be able to predict the future, especially in the financial industry.
And while there were those who predicted a rocky road for Facebook’s IPO, it is doubtful that anybody could have foreseen NASDAQ’s technical melt down that surrounded the Facebook IPO. While the stock price predictions for Facebook may be coming true, surely the technical issues that NASDAQ experienced on Facebook’s IPO day could not have been predicted…or could they?
Not in the Cards
As Scott Sellers points out over on Computerworld, it seemed like NASDAQ understood the kind of volume it would be facing and had taken the necessary precautions. He notes, “Exchange officials claimed that they had tested for all circumstances and for thousands of hours. I believe them.”
I believe them, too, but like we’ve said here many times before, and it’s a point to which Sellers alludes in his post, testing isn’t enough. As Sellers puts it, there needs to be “a resilient underlying infrastructure.” Functionality does not always mean structural quality, yet functionality is all that is needed to ensure that applications pass muster when tested. The functionality issues that might be found in an application are merely the tip of the proverbial iceberg that can potentially sink an application after it sails.
This is what, in all likelihood, happened to the NASDAQ on Facebook IPO day and will probably happen again. Why? Because application failures have happened before on numerous occasions and yet NASDAQ did not take heed from those who had gone (down) before them. Last year alone the London Stock Exchange, Euronext, Borsa Italiana (bought by the LSE in 2007) and the Australian Stock Exchange all suffered outages due to technical flaws.
Obviously there’s a lot more to keeping an exchange running than the functional testing can detect and on this point Sellers adeptly points to the CRASH study on application software health released in December. He notes that:
Exchanges are complex, high-performance systems that can be difficult to build, upgrade and debug. According to CAST Software, “an average mission critical application has just under 400,000 lines of code, 5,000 components, 1000 database tables and just under 1000 stored procedures.”
He later adds that, “Having a robust – and well-reviewed architecture nearly always results in a clear competitive advantage.”
Applying the Crystal Ball
Truth is, software failures like the ones experienced by NASDAQ and the other exchanges have become all too commonplace in all industries. Unless it affects a company’s finances directly – as the NASDAQ failure may have done by holding up trading of the Facebook IPO – we treat news of software failures as though they were inevitable and almost expected.  In NASDAQ’s case, however, there are now calls for investigations and answers about what happened.
In my book, that’s a good thing. After all, when exactly did we decide that software failure was an unavoidable part of business and an acceptable excuse to leave us hanging and waiting?
NASDAQ, the London Stock Exchange, Euronext…in fact, all exchanges and financial companies need to do a better job of assessing the structural quality of software before it is deployed rather than merely depending on functional or load testing after it is deployment ready. There’s no crystal ball needed here, just automated analysis and measurement, which is now readily available in the marketplace on a SaaS basis. Not doing structural analysis throughout the build is like waiting for an application to fall on its face and fall it will…faster than the share price of Facebook stock.

Who’s Minding the Store?

Before I could enjoy my Father’s Day brunch this past weekend, I found myself with a list of things to do around the house – cleaning out the garage, vacuuming the car, replacing our mailbox which “someone” in my family (not me) ran over. The latter of these tasks, of course, required that I go out and purchase some tools and supplies – a new post, new box, numbers for the box and a post digger – to get the job done.
My first stop in obtaining the necessary equipment, as always, was my local hardware store. I’m a huge proponent of buying things at local businesses rather than going out to the chain stores. I usually find the more local the store, the more helpful they are.
A similar theme, only from the “provider” point of view, is becoming pervasive among corporations when it comes to making applications available to their employees. Companies are “keeping it local” when it comes to making apps available. As InfoWorld’s app development guru, Paul Krill, points out:
“Companies developing their own custom applications, for example, do not want to make their intellectual property available in a commercial app store, so they set up their own.”
And who could blame them? With “Bring Your Own Device” (i.e., BYOD) one of the driving forces behind companies opening their own, proprietary app stores, organizations need to be certain that their customized apps do not wind up in the hands of those outside the company and they must be sure that the apps being downloaded by employees to access company files can be trusted.
With public app stores of today, that’s not always the case.
Buyer Beware
It’s little wonder companies want to keep their app stores close to the vest and their employees should actually be thankful for that move. If the news over the past 18 months about public app stores has shows us anything it’s that the quality of the available applications doesn’t always meet the highest standards for structural…and that’s being kind.
Take the Android app store for example. There are those who openly question if Google has any application software quality standards that need to be met before being offered in Android Market. Google would cite this laissez faire attitude as being part of its open source nature, but the old adage of programming – “Garbage in, garbage out” – should somehow apply to offering mobile apps in Android Market.
Even Apple, which posts a list of application standards for developers to use if they want to post an application to the App Store, has its issues with applications, but they see fewer of them. Why? Because their example of having a set of standards — and perhaps even a certification process —in place helps ensure structural quality and curbs the availability of malware in app stores.
Start at the Top
When it comes to application software quality, there is plenty of responsibility to be shared at each of level of mobile application distribution, and that’s a big reason why businesses are right in opening their own app stores for employees. Under their control, they can have oversight over the structural quality of the apps being offered and those who handle them – from those who develop the apps to those who administer the app store.
After all, responsibility for the structural quality of application software being offered rightly should begin at the top. The CIO or IT Director needs to establish a set of software quality standards for the apps being made available and then ensure any application that is offered meets those standards. This will help curb new issues and prevent vulnerabilities from older versions creeping into the new versions built on top of them.
Should companies fail to provide the necessary oversight for their app stores, they place their entire operation at risk because not taking responsibility for structural quality through a corporate app store is the same as nobody minding the store.

Did NASDAQ’s App Glitch Cause FB’s IPO Hitch?

Isn’t it ironic?
Facebook, the galactically popular social networking site that for so long has weathered friction regarding weaknesses in its software – particularly around security and privacy issues – may have seen its own IPO effort submarined by a software glitch in the NASDAQ stock exchange.
In reporting on NASDAQ’s response to the technical difficulties it encountered on Facebook’s IPO day, Bloomberg’s Nina Mehta writes:
Computer systems used to establish the opening price were overwhelmed by order cancellations and updates during the “biggest IPO cross in the history of mankind,” Nasdaq Chief Executive Officer Robert Greifeld, 54, said yesterday in a conference call with reporters. Nasdaq’s systems fell into a ‘loop’ that kept the second-largest U.S. stock venue operator from opening the shares on time following the $16 billion deal.
According to Mehta, the reason Greifeld gave for the issues with the IPO was “poor design in the software it uses for driving auctions in initial public offerings.”
One would think that if any exchange out there were to be free of poorly designed software it would be the Tech-heavy NASDAQ exchange. Apparently offering the top Tech companies in the industry, though, does not necessarily mean you run the best software the Tech industry has to offer.
Profile of a Failure
Truth is, software failures like the one experienced by NASDAQ have become quite commonplace lately; so much so that they’re practically met with a shoulder shrug and an “oh well.” We treat news of software failures as though they were inevitable and almost expected. Only when it affects finance – particularly the financial status of a marquis brand name like Facebook – do we step back and even offer so much as a “tsk, tsk, tsk” for the failure.

But why? When exactly did we decide that software failure was an unavoidable part of business, an acceptable excuse for possibly undermining the value of a highly touted IPO?
Facebook reached a high of $45 per share before it dropped back below its initial offering price of $38 per share. Whether the glitches at NASDAQ caused the per performance or whether you agree with Henry Blodget at Business Insider that they were just a convenient excuse for a poor showing, there is still no excuse for application software failure, especially since we know what causes it:

Business Blindspot: Regardless of the industry, most developers are not experts in their particular domain when they begin working for a company. It takes time to learn about the business, but most of the learning, unfortunately, comes only by correcting mistakes after the software has malfunctioned.
Inexperience with Technology: Mission critical business applications are a complex array of multiple computer languages and software platforms. Rather than being built on a single platform or in a single language, they tend to be mash ups of platforms, interfaces, business logic and data management that interact through middleware with enterprise resource systems and legacy applications. Additionally, in the case of some long-standing systems, developers often find themselves programming on top of archaic languages. It is rare to find a developer who knows “everything” when it comes to programming languages and those who don’t may make assumptions that result in software errors that lead to system outages, data corruption and security breaches.
Speed Kills: The pace of business over the past decade has increased exponentially. Things move so fast that software is practically obsolete by the time it’s installed. The break-neck speeds at which developers are asked to ply their craft often means software quality becomes a sacrificial lamb.
Old Code Complexities: A significant majority of software development builds upon existing code. Studies show that developers spend half their time or more trying to figure out what the “old code” did and how it can be modified for use in the current project. The more complex the code, the more time spent trying to unravel it…or not. In the interest of time (see “Speed Kills” above) complexity can also lead to “work arounds” leaving a high potential for mistakes.
Buyer Beware: Mergers and acquisitions are a fact of life in today’s business climate and most large applications from large “acquirers” are built using code from acquired companies. Unfortunately, the acquiring organization can’t control the quality of the software they are receiving and poor structural quality is not immediately visible to them.

A Comment on Facebook’s Status
NASDAQ may need to pay back $13 million to investors who should have received transaction executions but did not because of its software failures. Meanwhile, brokers around the world may lose $100 million repaying investors for mishandled orders. A quick, pre-deployment application of automated analysis and measurement to diagnose the structural quality and health issues within the application software used by NASDAQ or any company would have been a much better investment of time and money.
I guess this is one more reason to lobby for a “DISLIKE” button.

Shortcuts Today Lead to Shortcomings Tomorrow

Catchy slogans are catchy for two good reasons – the put an extremely true point into very simple, succinct language. This is probably why they call these true, simple statements, “catch phrases.”
One of the most effective catch phrases of my youth was for a product called Fram Oil Filters. Sometime in the 1970’s, Fram came out with a set of television and radio commercials where a mechanic would explain how a simple thing like replacing your oil filter on a regular basis could prevent major engine problems. The catch phrase uttered by the mechanic at the end of each commercial was, “You can pay me now, or you can pay me later.”
The connotation behind the final statement was, of course, that by paying now for a relatively inexpensive item – an oil filter – an automobile owner could avoid paying a lot of money for an engine overhaul down the road.
So successful and true was the statement that it would soon extend beyond just the Fram commercials and would become part of the common lexicon. Whenever someone wanted to point out that a little prevention could go a long way toward avoiding a big problem later, that person even today is frequently heard to say some version of “You can pay me now, or you can pay me later.”
One thing this definitely applies to is application software, because the shortcuts taken and money saved during a built can definitely lead to costly problems later.
Analyzing the Path
The growing recognition of the technical debt issue should be evidence enough that companies need to start looking at their build and customization processes for ways to eliminate issues with application software. As Eric Allman recently pointed out over at the ACMQueue Blog, “Roughly speaking, technical debt is acquired when engineers take shortcuts that fall short of best practices.”
Allman goes on to note a number of different types of shortcuts, each of which is certainly avoidable when developers are working on application software. Failing to adhere to best practices, however, inevitably will catch up with the application. As Eric notes in his blog and as we have time and time again noted here, the cost to fix the inevitable problem that arises is what we’ve called in this blog technical debt.
Another point that we’ve often made in this space is echoed by Allman – the fact that not all technical debt is bad. Since the perfect application has yet to be written, and because there will always be upgrades that need to be made, there will always be some maintenance on application software. But that doesn’t mean an application can just be slapped together and pushed out the door without regard for what will happen once it has been deployed.
Further on up the Road
As Allman astutely states, “Technical debt is inevitable. The issue is not eliminating debt, but rather managing it.” Of course, the best way to manage it is to take the time before deployment to eliminate the hiccups and flaws that go into software. The list Allman offers in his blog of examples of shortcuts taken is interesting, although I’m sure even he would admit it’s only a fraction of those developers have undertaken in the name of “expediency.” His list includes:

Sneaking around an abstraction because it is too hard (or impossible) to figure how to “do it right,”
Skipping or scrimping on documentation (both in the code and external documentation),
Using an obscure or incomplete error message because it’s just too hard to create something more informative,
Implementing code using a simple but slow algorithm even though they know that a better algorithm will be needed in production,
Using void* when you really should have created an appropriate union*,
Using build tools that don’t quite work for the system at hand,
Skimping on good security practices,
Not writing unit tests

Just eliminating these shortcuts, though, isn’t enough to manage technical debt efficiently. Even if everything is done by the book, issues can still exist. After all, there’s the “human element” to factor in. And then of course there’s the fact that very few applications are built from scratch anymore, but are instead built upon existing code that may not meet present-day standards.
So to truly manage technical debt, a company should manage its team to make sure it doesn’t take the kinds of shortcuts Allman describes, but it also needs to spend the time during the build process to perform a comprehensive assessment of the application software. The key here is “during the build process” though. Once there’s something worth actually testing to see if it works is too late. With each phase of a software build, issues in the software become more difficult and costly to fix by a factor of 10. By the time a project is completed, a mistake that in the initial phase of the build may have taken 10 minutes to fix, may now take over an hour or more…and as the old cliché says, “time is money.”
By spending a little time and money during the build phases to apply a platform of automated analysis and measurement – which is now becoming more readily available in software as a service (SaaS) form – those little quirks that could bring down an application can be found and then dealt with.
Because after all, when it comes to application software, you can pay a little to fix it now, or pay a lot to fix it later.

Is your Critical Application the next Titanic?

Technical Debt and the Titanic?
Almost everyone has heard about the Titanic and the sinking of the unsinkable.  I guess if you assume your ship is unsinkable, having only 20 lifeboats for a few thousands people seems reasonable.  Maybe it gets overlooked when there are so many important “features” to get right on the maiden voyage.   I’m sure the pressure to ensure the comfort of hundreds of VIP’s must have been immense.  Sometimes it takes a real disaster for change to take place.
Shortly after the sinking of the Titanic, the US Coast Guard formed the IIP “International Ice Patrol” in response to this tragic event.  The IIP has continuously provided ships traveling off the coast of Newfoundland with iceberg information, avoidance best practices and technology to spot trouble before it is too late.  Shorty after the Second World War, the IIP began using radar to locate possible ice masses that could cause danger.  Over time this technology has continuously improved, leveraging Doppler imaging techniques, and is now very effective in distinguishing threating ice masses from other vessels at sea.
In the world of Application Development there are similar pressures to get code out the door quickly.  Recently, there have been numerous articles published on “technical debt” which quantifies the technology defects inherent in software development and often is magnified by fast delivery cycles and business pressure.  What’s interesting is that Application measurement and analysis tools are available today that are capable of identifying source code violations leading to defects but, in many cases they are not used.
Systematically measuring Technical Debt is analogous to Radar – Doppler imaging systems that have been in use by sea vessels, for many years.   Tools like Application Intelligence Platform “AIP” are being utilized by many progressive development organizations and ISV’s to alert Application Development and Maintenance teams of impending structural dangers in a large sea of code.   Many ISV’s now offer Structural Quality Gates as part of their standard ADM methodologies and customer offering to get a deeper vision of where danger lurks in software applications and portfolios.  Increasing visibility and insight into your organizations critical application’s is good business and could help you avoid a real disaster.

Software Analysis and Measurement