On Quality

We’ve been reading quite a bit the last few weeks about iPhone related issues that have had an impact on the security (Citi) and stability (AT&T) of customer data. Beyond the current arms race in the media to see who can write more frequently about Apple, there may actually be something there that spells real news for us in IT.

If you’re in charge of some mission critical systems at an IT-intensive consumer services company, like a bank or a telecom operator, you might be starting to feel a little more nervous than usual. Over the past couple years you’ve finally gotten accustomed to the fallout of the dot com boom that started 10 years ago, since you had to change your architectures, your release cycles, your technology mix, and your team’s (and maybe your own) attitude as your organization got dragged into connecting directly to the end customer. So by now you’re sort of comfortable with the concept that customers can interact directly with back-end systems, including yours. Well, now we have the iPhone, the Droid, the Treo (does anyone still use Palm?), and more real smart phones competition on the way from the likes of Microsoft and Nokia. Having just internalized last Tuesday’s WSJ article, this really does not bode well.

Well, maybe that’s putting too much drama into the equation, but clearly the boom in smart phones is changing the IT landscape. Mobile apps, and the competition around mobile outreach to the consumer, is taking the customer-facing stress to a new level. For IT, the line between the handheld device and the internal system is becoming more blurred. As we can see with some of the commentary coming out of the situation at Citi, the issue is not a straightforward iPhone problem. It’s a combination of factors that include the iPhone OS, the software from mFoundry, and Citi’s internal applications. The recent problems that AT&T had registering new iPhone customers has some similar characteristics. I don’t have any more information about these two companies than what I’ve read in the press, but what’s certain is that we are bound to see more issues like this in the near future.

To me the key question here is what is IT management and customer service line management doing to ensure they stay on top of these problems? Do they take a proactive stance to track their risks to security or stability exposure? Or do we collectively “outsource” or “delegate” the responsibility deep into the IT organization or to our vendors. Collective responsibility is always a winning formula.

As mobile apps come online to be the direct customer interface, the overall IT system we’re customizing to enable our business to compete becomes far-flung. That has an impact on construction and architecture, and the overall structure of these sprawling IT systems. The additional exposure – both in terms of PR and direct customer experience – make these structural issues more important for management to measure and stay on top of.

You could hide security and quality issues when IT was all internally focused. These days badly designed software is going to become more and more of a publicly visible liability. The smart IT managers are getting in front of these issues.

Industry data demonstrate that code reviews are highly effective.

Out-Sized Effectiveness of Code Reviews

The problem, however, is that code reviews are time consuming, expensive, and difficult to get right.

To be effective, code reviews require the following:

1) Software engineering expertise

2) An objective basis for evaluating code

3) Comprehensive code coverage

4) A repeatable method that produces reliable results

5) Practical guidance on how to prioritize and fix problems found, and

6) A way to quantify improvement (and hence the effectiveness of code review activities)

As you can see, these are very difficult to set up in any organization. It quickly becomes too expensive, cumbersome, and unworkable.

So how do you get both low cost AND effective code reviews that overcome all the usual stumbling blocks (1 -6) that otherwise prevent companies from effective code reviews?

Check out the webinar on automating code reviews on the DCG website!

I recently sat down with Fritz Nelson of InformationWeek and walked him through a demo of CAST.

I tried to explain as clearly as I could the value of CAST to CIOs and their IT organizations.

To see the video click here.

Run Your Apps Through C-A-S-T!

(Sung to the tune of YMCA by the Village People)

IT, there’s no need to feel down.

I said, IT, pick yourself off the ground.

I said, IT, ’cause your website is down
And your CIO has left town.

IT, there’s a place you can go.
I said, IT, when your uptime is low.
They will heal you, since I’m sure they will find
All the bugs that cause your downtime.

Let’s run your apps through the C-A-S-T.
Let’s run your apps through the C-A-S-T.

They know everything about Java and SAP,
They can tell when your code is crap …

Let’s run your apps through the C-A-S-T.
Let’s run your apps through the C-A-S-T.

You can get your code clean, you can check your SI,
It will help you with CMMI …

IT, are you listening to me?
I said, IT, how bad can your code be?
I said, IT, you write terrible C.
But you got to know this one thing!

No man finds all bugs by himself.
I said, IT, put your pride on the shelf,
And just go there, give your systems to CAST.
They will find your flaws so damn fast.

Let’s run your apps through the C-A-S-T.
Let’s run your apps through the C-A-S-T.

They know everything about Java and SAP,
They can tell when your code is crap …

Let’s run your apps through the C-A-S-T.
Let’s run your apps through the C-A-S-T.

You can get your code clean, you can check your SI,
It will help you with CMMI …

IT, when the big bugs get missed.
I said, IT, then your QA gets dissed.
I said, IT, cause the business is pissed.
They put you atop their s**t list …

That’s when IT is just way out of luck,

And our VPs are just passing the buck,

And our coders they have all run amok,

And our apps they all really suck …

Let’s run your apps through the C-A-S-T.
Let’s run your apps through the C-A-S-T.

You can get your code clean, you can check your SI,
It will help you with CMMI …

C-A-S-T … we’ll find your bugs with the C-A-S-T.

IT, IT, there’s no need to feel down.
IT, IT, pick your code off the ground.

C-A-S-T … we’ll check your apps with the C-A-S-T.

IT, IT, are you listening to me?
IT, IT, send CAST your Java and C.

C-A-S-T … we’ll measure them with the C-A-S-T.

IT, IT, all your bugs will be found.
IT, IT, all your apps will be sound.

Kickoff 2010! Every year, the entire company gathers in one physical location to strengthen relationships, forge new ones, and internalize CAST’s values and goals. Kickoff 2010 was super! I was delighted to meet some good friends and make many new ones.  The intelligence and creativity of CAST folks is impressive — no wonder we are THE Application Intelligence company.

We spent the day at La Maison de la Chimie in a spectacular auditorium. It was a long day, packed with creative presentations, two short CAST adverts, and a CAST theme song! C-A-S-T for IT!

Maison de la Chimie Auditorium

We had dinner at la Maison des Polytechniciens.

La Maison des Polytechniciens

You can see more pictures on the CAST’s Facebook page. Send me your pictures!

Back in August of 2008,  the FAA reported significant software configuration problems when a software glitch delayed dozens and dozens of national flights. The outage directly affected traffic and ground personnel; cargo had to be manually loaded, and flight plan information manually entered into the system by air traffic controllers. According to an FAA spokesperson, the source of the malfunction was a “packet switch” that “failed due to a database mismatch”.

This was the second glitch of it’s kind, the third was to follow…

November 19th, 2009, Bloomberg reported the FAA systems were down for 4 hours due to a “software configuration problem” within the Federal Telecommunications Infrastructure.

These sorts of issues may commonly be perceived as network issues, but in reality it’s because the software is too complex and badly engineered. The issue is typically around data access, and a “supply-demand” mismatch in how components of the application use the database. Sudden spikes of activity cause hardware to go over capacity because the application forces the network or the CPU to thrash.

These problems often occur because software quality depends on context — you can read more about what CAST’s Chief Scientist Bill Curtis and Olivier Bonsignour say about how context matters in this post.

Software quality issues are extremely serious in the Aviation industry, especially considering the immediate widespread domino effect it has on all sorts of personnel, and more importantly, consumers. Objectively assessing the quality of each moving part of the software system, and its contribution to the system, will improve how the application systems load the hardware, thus aiding the  system’s smooth operation.

It’s that time of year when we look back…and forward to what software development will be like. Here’s a short piece from Bill Curtis, CAST SVP and Chief Scientist from the Wall Street Journal (November 16, 2009).

The Latest Buzz On…

If you’ve been in any airport in the last few years, you’ve seen ads from HSBC – the global bank that prides itself on local knowledge.

The point is, context matters. The same outfit means opposite things depending on where you are or who you’re with. The same thing applies to software quality. Quality is not just a local thing — only when you have global knowledge can you act effectively on the local level.

Two of my colleagues, Olivier Bonsignour and Bill Curtis recently wrote an article explaining this at length. Here is a summary of their excellent article — well worth checking out in full.

“Quality is not an intrinsic property of code: the exact same piece of code can be excellent in quality or highly dangerous depending on the context in which it operates. Analyzing the quality of modern applications in the context of the numerous interconnections with other code, databases, middleware, and APIs is monstrously complex. It can only be accomplished with automated software that analyzes the inner structure of all components and evaluates their interactions in the context of the entire business application.”

They go on to show the quality problems that arise when context is ignored. Again, I’ll summarize. You can find the detailed examples here.

Typical application quality problems are listed below to clarify the distinction between application and code quality. Performance testing alone is not sufficient to detect these application quality problems.

A. Bypassing the Architecture. Components in one tier of a multi-tier application are typically designed to access components in another tier only through an intermediate “traffic management” component. Bypassing this traffic management component will usually result in a cascade of problems.

B. Failure to Control Processing Volumes. Applications can behave erratically when they fail to control the amount of data or processing they allow.  This problem is often caused by a failure to incorporate controls in each of several different architectural tiers.

C. Application Resource Imbalances. When database resources in a connection pool are mismatched with the number of request threads from an application, resource contention will block the threads until a resource becomes available, tying up CPU resources with the waiting threads and slowing application response times to a crawl.

D. Security Weaknesses. Applications are vulnerable to security attacks when they lack appropriate sanitization checks on user inputs in all relevant tiers of the application.

E. Lack of Defensive Mechanisms. Since the developers implementing one tier cannot anticipate every situation, they must implement defensive code that sustains the application’s performance in the face of stresses or failures affecting other tiers.  Tiers that lack these defensive structures are fragile because they fail to protect themselves from problems in their interaction with other tiers.

Each of these application quality problems will result in unpredictable application performance, business disruption, data corruption, and make it difficult to alter the application in response to pressing business needs. Reliably detecting these problems requires an analysis of each application component in the context of the entire application as a whole – an evaluation of application rather than code quality.

Recently, an Australian team studied the performance of the Amazon, Google, and Microsoft Clouds. The results reminded me of Bob on Entourage.

The results are not surprising. The on-demand cloud services from these companies “suffer from regular performance and availability issues.”

Now, not to make too much of this — we already know that blazing performance on the cloud is neither a promise these vendors make nor an economic reality. After all, if you want cheap AND scalable, something’s got to give.

But you can be prepared.

If you could precisely measure the performance and availability of an application on the cloud, would that be something you might be interested in?

If you could do this before you migrated to the cloud, would this be something you might be interested in?

If you’re a vendor of Cloud services, would you be interested in tracking not just usage, but quality of service?

Well, you can. In what follows I’ll show you exactly when and what to measure for optimal migration.

I’d be glad to tell you more – just email me. Or go here.

Now Bob might be a parody of himself, but he really gets to the core of what matters. In software, it’s the only thing that matters in the end is the product, the stuff, aka the code. It’s so difficult to measure that people get frustrated.

But it’s something you should be interested in.