No offense, but I’m not addicted to representative measures. In some areas, I am more than happy to have them. Like when talking about the balance of my checking and savings accounts. In that case, I’d like representative measures, to the nearest cent.
But I don’t need representative measures 100 percent of the time. On the contrary, in some areas, I strongly need non-representative measures to provide me with some efficient guidance.
Did the press club have a meeting? Because this is the second time in two weeks that we’ve been in the press. Last week, I spoke with Business Insider in response to Marc Cuban’s comments about the NYSE’s role in high frequency trading. And this week, I sat down with Karl Flinders at ComputerWeekly.com to discuss some other issues surrounding application outsourcing and outsourced QA. Karl does a nice job summarizing some key points in a crisp way. The story ran today, and we wanted to share it with everyone to help raise awareness about how systematic analysis of your applications can improve our ability to work with outsourcing partners, and our risk profile.
I’ve been asked time and again how CAST is different from performance engineering. And here’s my answer: The CAST discipline of software analysis and measurement versus performance engineering couldn’t be more different. And I’ll explain why and how in a moment. But along with that, it should be noted that they also are like peanut butter and chocolate — they can go very well together.
Here’s the high level explanation, which I’ll drill into further for those of you who like details. Fundamentally, when you’re dealing with CAST, you’re improving code quality during the engineering phase and throughout the development of the product. CAST technology is used as the code is built, and even after you’ve done performance engineering.
Performance engineering, on the other hand, is just one phase late in the game, when the product is essentially considered complete and you’re doing a final round of outcome-driven optimization. Consider performance engineering as fine tuning the finished system to reach specific response time objectives, while CAST is part and parcel of the software engineering process.
Using dynamic analysis and load testing, performance engineering learns how a system is behaving at runtime and if there are things the development team could do to make the software perform faster. CAST is a design-time, proactive measure to avoid common performance problems in the first place.
Performance engineering is like taking a brand-new Mercedes onto the racetrack to see how it performs. The development is over, now you’re getting into fine tuning what you have. With CAST, we’re actually checking for structural problems while the car is still being built.
But I can see where the confusion comes from. The objectives of performance engineering are aligned fairly closely to those of CAST. When it comes to performance, CAST is intended for: eliminating late system deployment due to performance issues; eliminating avoidable system rework due to performance issues; and reducing increased software maintenance costs due to performance problems in production.
So, getting back to our car analogy, working with CAST is more like noticing that one tire on the car is smaller than the rest, or that two axles are not perfectly aligned, or that the engine is diesel while the spark plugs are meant for gasoline engines. These are issues that might not cause problems at low speeds, or right away, but after a few laps they’ll cause the tires to become bald or the engine to stop running — not an ideal situation for a racetrack.
This distinction becomes readily apparent when we take these similar processes and set them on the same task: improving the overall performance of a system. Performance engineers might trace the round-trip time of transactions, figure out which parts of the process take the longest, and then take measures (adjust the network, change the indexes in the database, adjust table spaces, optimize where they have the most processing power, etc.).
With CAST, you start with standard performance rules applied throughout the application’s development. But, this is where performance engineers can have an even greater impact. Along the way, if they notice a certain pattern that’s causing performance to suffer, the performance engineers can turn it into a custom rule. Then CAST can check to make sure all developers avoid such patterns at all times.
I know that some of you are probably reading this and thinking that we’re just splitting hairs. And while it’s true that performance engineering and CAST’s processes might share similar outcomes, the real business value comes in its implementation.
CAST’s platform is best used during the design phase of a product, so that code problems can be checked while there is still a chance to fix them. Performance engineering is done after the product is finished, like a quality control test, to see how well you made out during development.
Combining the two — with CAST streamlining the application during the design phase and performance engineering doing the fine tuning after its release and providing feedback to developers and architects — is a strategic advantage that can be used to create dynamic, high performance, quality applications. This is an essential part of avoiding waste in Lean Application Management. So you can go ahead and call it splitting hairs, but I call it a competitive edge.
Risk detection is the most valid justification to the Software Analysis and Measurement activity: identify any threat that can negatively and severely impact the behavior of applications in operations as well as the application maintenance and development activity.
“Most valid justification” sounds great, but it’s also quite difficult to manage. Few organizations keep track of software issues that originate from the software source code and architecture so that it is difficult to define objective target requirements that could support a “zero defects” approach. Without clear requirements, it is the best way to invest one’s time and resources in the wrong place: removing too few or too much non-compliant situation in the software source code and architecture, or in the wrong part of the application.
One answer is to benchmark analysis and measurement results so as to build a predictive model. This application is likely to be OK in operations for this kind of business because all these similar applications show the same results.
On the one hand, by nature, benchmarking imposes to compare apples with apples and oranges with oranges. In other words, measurement needs to be applicable to benchmarked applications — stability over time — so as to get a fair and valid benchmarking outcome.
On the other hand, risk detection for any given project:
benefits from the use of state-of-the-art “weapons”, i.e., the use of any means to identify serious threat, that should be kept up-to-date every day (as for software virus list)
should not care about fair comparison. It’s never a good excuse to say that the trading applications failed but that it showed better results than average
should heed contextual information about the application to better identify threats (an acquaintance of mine — a security guru — once said to me there are two types of software metrics: generic metrics and useful ones), i.e., the use of information that cannot be automatically found in the source code and architecture but that would turn a non-compliant situation into a major threat. For instance: In which part of the application is it located? Which amount of data is stored in the accessed database tables — in production, not only in the development and testing environment? What is the functional purpose of this transaction? What is the officially vetted input validation component?
Is this ground for a divorce on account of irreconcilable differences?
Are we bound to keep the activities apart with a state-of-the-art risk detection system and a common-denominator benchmarking capability?
That would be a huge mistake as management and project teams would use different indicators and draw different conclusions. Worst case scenario: Project teams identify a major threat they need resource to fix but management indicators tell the opposite so that management deny the request).
Although not so simple, there are steps that can be taken to bridge the gap.
It would be to make sure:
that “contextual information” collection is part of the analysis and measurement process
that a lack of such information would show (using the officially-vetted input validation component example, not knowing which component issues are a problem that would impact the results; not an excuse for poor results which much too often the case
that the quality of the information is also assessed by human auditing
Are your risk detection and benchmarking butting heads ? Let us know in a comment. And keep your eyes on the blog for my next post about the benefits of a well-designed assesment model.
Happy Independence Day everybody! I only hope those of you reading this on your Android device have not turned it sideways or performed some other seemingly innocuous action that has made this application fail.
I say this because I recently read yet another blog about “workarounds” to compensate for application failures inherent in Android devices. These pieces have become almost ubiquitous over the past 18 months to the point where one would think Google would just go back and perform the structural quality analysis it needs to do to address the issues.
Their failure to do so reminds me on this day before Independence Day of the opening lines of Thomas Paine’s “Common Sense”:
These are the times that try men’s souls: The summer soldier and the sunshine patriot will, in this crisis, shrink from the service of his country; but he that stands by it now, deserves the love and thanks of man and woman.
As Google continues to “shrink” from its responsibility to provide application software that is of sound structural quality, they are certainly “trying men’s [and women’s] souls.”
I Have Not Yet Begun to Fight! I continue to be amazed that Google appears more interested in what to call their next Android OS. As “enamored” (can you feel the sarcasm dripping from that word?) as I was last year with “Ice Cream,” I am even more captivated by their latest one – Jelly Bean. I am betting he name really fits the product – looks solid on the outside, but if Android’s history is any indication it will most certainly be a piece of gelatinous mush on the inside.
Maybe Google continues to fall into the trap of believing its own press clippings – the positive ones, at least – because they seem more concerned with marketing than they are with software quality. Google’s mobile operating systems continue to feature one flaw after another with these flaws not being “discovered” until after the system has been rolled out and installed by the consumer. And these flaws are not just minor ones that inconvenience the user like the ones mentioned in the work arounds blog to which I referred above. They include battery-draining and security flaws that cost time and money for those using the devices..
Nevertheless, they continue to build one iteration after another atop mobile platforms they know to have flaws – or at least by now they should know – and they continue to fail to fix them.
We Find these Truths to be Self-Evident
It’s truly a shame Google won’t use the same methodology as Thomas Jefferson and the Continental Congress did in forging one of the world’s greatest documents – the Declaration of Independence. From the time Jefferson began working on the Declaration of Independence on June 11, 1776, he wrote and rewrote, edited and re-edited versions of the document for almost three weeks until he came to what he felt was a product of optimal quality. And yet even after the document with all those versions and all those edits was presented to the Continental Congress on June 28, 1776, for a vote, those men debated for another five days over the contents of the document and made another 33 changes to it!
Obviously there was no Marketing Department pushing Congress to get out the final product. The Declaration of Independence – a document that truly did have urgency behind it as men were fighting and dying for the values it espoused – was edited and changed many dozen times before it was delivered. If that’s the case, why can’t the marketing people at Google allow their developers to perform a bit of automated analysis and measurement on Android software before they declare its “independence” from internal production? Were they to do this (harkening Paine again), they would “deserve the love and thanks of man and woman.”
These truly are the times that try our souls.
I have been an East-Coaster all my life. I’ve lived, worked and even attended college in states that all lie East of the Mississippi. However, throughout my 18 years working in the technology business, my clients have been spread out around the U.S. and abroad. I’ve found myself doing phone calls before the sun rises and well after it has set. That’s just the way it is in this business.
While it is admittedly easier to write about companies that are located in another state, I the remote worker hardly begins and ends with us writers. More often than not I’m working with companies that have developers, architects, managers, directors and even executives spread out over multiple locations. One Canadian-based division of a company I represented showed a real sense of panache and even went so far as to build a robot with an embedded web cam to allow one of its developers to move to another province while still maintaining a “physical” presence in the office.
But the truth of the matter is that communication between people not within a single office is still problematic. You just cannot possibly be as free and open via email or even over the phone when it comes to scheduling meetings and sharing ideas.
As Johanna Rothman points out in her Managing Product Development blog:
“Let’s assume you have what I see in my consulting practice: an architecture group in one location, or an architect in one location and architects around the world; developers and “their” testers in multiple time zones; product owners separated from their developers and testers. The program is called agile because the program is working in iterations. And, because it’s a program, the software pre-existed the existence of the agile transition in the organization, so you have legacy technical debt up the wazoo (the technical term). What do you do?”
It’s Personal…and It Isn’t Being the good consultant and managerial-type she is, Rothman offers a number of ways to mitigate issues between dislocated teams of developers. Her ideas are solid and very much in tune with promoting a good work environment for developers. These ideas encompass many good practices. She suggests establishing teams so that developers can’t be borrowed between offices and ensuring that each team has a specific task or feature on which they always work. She says to establish specific goals for each team and espouses the need to take extra time to map out what needs to be tested in each iteration so that teams in different time zones are not inconvenienced by conference calls outside their zone’s normal work hours.
All of these have great potential to make teams work together better…I’m not sure they solve the problem of the structural quality of the software being developed, though. The problem there is similar to something I learned in my eighth grade Chemistry class. There I learned that every vessel you use to measure something has a certain amount of inaccuracy to it. Therefore, when you use multiple vessels to measure something, you increase the level of inaccuracy for that measurement.
Similarly, when you split up a project, each team yields a certain level of control to its counterparts in other offices and geographic differences – whether teams are split between areas of the U.S. or other countries – play into what goes into the software and how code is written. Those are issues that cannot be resolved easily when you bring the various portions of a project together. However, if an organization implements an across-the-board requirement that all teams perform some form of structural analysis throughout the process, it can potential technical debt as well as provide a measure of visibility into the quality of the application software being developed.
Personally, combining Rothman’s ideas with structural analysis sounds like a good idea…technically speaking.
I’m not one who believes in fortune tellers or those who claim to be able to predict the future. Heck, I don’t even read my horoscope and cringe whenever someone attempts to force it upon me. Only when my wife has attempted to read me my horoscope have I offered even as much as a polite “hmm.” Nevertheless there are many out there who swear by those who claim to be able to predict the future, especially in the financial industry.
And while there were those who predicted a rocky road for Facebook’s IPO, it is doubtful that anybody could have foreseen NASDAQ’s technical melt down that surrounded the Facebook IPO. While the stock price predictions for Facebook may be coming true, surely the technical issues that NASDAQ experienced on Facebook’s IPO day could not have been predicted…or could they?
Not in the Cards
As Scott Sellers points out over on Computerworld, it seemed like NASDAQ understood the kind of volume it would be facing and had taken the necessary precautions. He notes, “Exchange officials claimed that they had tested for all circumstances and for thousands of hours. I believe them.”
I believe them, too, but like we’ve said here many times before, and it’s a point to which Sellers alludes in his post, testing isn’t enough. As Sellers puts it, there needs to be “a resilient underlying infrastructure.” Functionality does not always mean structural quality, yet functionality is all that is needed to ensure that applications pass muster when tested. The functionality issues that might be found in an application are merely the tip of the proverbial iceberg that can potentially sink an application after it sails.
This is what, in all likelihood, happened to the NASDAQ on Facebook IPO day and will probably happen again. Why? Because application failures have happened before on numerous occasions and yet NASDAQ did not take heed from those who had gone (down) before them. Last year alone the London Stock Exchange, Euronext, Borsa Italiana (bought by the LSE in 2007) and the Australian Stock Exchange all suffered outages due to technical flaws.
Obviously there’s a lot more to keeping an exchange running than the functional testing can detect and on this point Sellers adeptly points to the CRASH study on application software health released in December. He notes that:
Exchanges are complex, high-performance systems that can be difficult to build, upgrade and debug. According to CAST Software, “an average mission critical application has just under 400,000 lines of code, 5,000 components, 1000 database tables and just under 1000 stored procedures.”
He later adds that, “Having a robust – and well-reviewed architecture nearly always results in a clear competitive advantage.”
Applying the Crystal Ball
Truth is, software failures like the ones experienced by NASDAQ and the other exchanges have become all too commonplace in all industries. Unless it affects a company’s finances directly – as the NASDAQ failure may have done by holding up trading of the Facebook IPO – we treat news of software failures as though they were inevitable and almost expected. In NASDAQ’s case, however, there are now calls for investigations and answers about what happened. In my book, that’s a good thing. After all, when exactly did we decide that software failure was an unavoidable part of business and an acceptable excuse to leave us hanging and waiting?
NASDAQ, the London Stock Exchange, Euronext…in fact, all exchanges and financial companies need to do a better job of assessing the structural quality of software before it is deployed rather than merely depending on functional or load testing after it is deployment ready. There’s no crystal ball needed here, just automated analysis and measurement, which is now readily available in the marketplace on a SaaS basis. Not doing structural analysis throughout the build is like waiting for an application to fall on its face and fall it will…faster than the share price of Facebook stock.