There were so many great questions from attendees after the “Aligning Vendor SLAs with Long-Term Value” webinar that I moderated last week that we’ve compiled them here for you. Whether you participated in the webinar or not, I’m sure you’ll find the questions — and answers — fascinating. Plus, don’t forget to check out the results from the real-time poll we conducted during the webinar!
If you participated in the webinar but just didn’t get around to asking a question, feel free to email me at steven.hall@isg–one.com
Q. Are you seeing the use of Agile in maintenance as well as development?
A. We are seeing Agile grow in global development projects but not in the maintenance area. Minor maintenance enhancements still tend to be bundled as small releases or included as part of a development release.
Q. Can you give an example how do you use Schedule and Cost Index to calculate Earned Value?
A. CPI (Cost Performance Index) and SPI (Schedule Performance Index) are both outputs from the Earned Value process. Both of the calculations will provide excellent insight into the overall state of the project. Earned Value Project Management (Fleming and Koppelman) and Managing Global Development Risk (Hussey and Hall) both address how to use Earned Value to manage large scale, global projects.
Q. What is your view of standards such as ISO20000 for service management or 9001 etc.?
A. We are seeing much broader adoption of ITIL V3 Service Management and Service Integration and Service Management are both becoming significant market trends in the outsourcing space.
Q. Do you see any trends in the industry toward near-shoring?
A. Near-shoring and domestic sourcing have grown over the past several years. ISG has published multiple articles on this subject, as well as the whitepaper “Gone Country-Rural Outsourcing Gains Ground.”
Q. Does it make sense to have another supplier test defects? Or can we put the right checks in place to have the same supplier test their own work?
A. We’ve seen growing trend towards Testing Centers of Excellence (CoE) that are improving the overall testing process through enhanced automation and focus on testing disciplines. Scale is important, though. If the testing organization is less than 150 FTEs, then it is often best to combine with the service provider doing the development.
Q. Don’t most SIs need to build in a “Risk Premium” into their bids/proposals…to cover them from unknown structural/technical risk? Is there a way to reduce costs by reducing that premium if a customer had a good baseline on the technical quality to share?
A. We believe there can be significant savings in the Run the Business (RtB) or maintenance costs if structural quality is defined and measured in earlier phases. The importance of software quality required is usually defined by the potential consequences of a failure due to poor quality; it therefore depends on specific business objectives and tradeoffs between long-term maintainability and immediate robustness, efficiency and security. Hence, quality objectives and thresholds should, as much as possible, be determined by business lines and project management teams depending on the specific context and business, technical, and functional objectives. Ideally, for every system, “expected targets” are defined as quality scores to achieve, or surpass, and “minimum thresholds” as a limit below which ADM must not go. All the quality indicators must be base lined, by analyzing the applications before the Expected and Threshold Targets are set. Then, based on the baseline Quality Indicators and requirements of the business, Expected & Threshold Targets are agreed upon by end‐customers and their service provider. There is strong market evidence that process discipline reduces re-work, this same rationale should certainly apply to structural quality as well.
Q. How do you see Agile methods affecting outsourcing contracts and relationships? Better, worse, or no difference?
A. Overall better. I have personally led several very large development projects using agile methodologies in a global delivery model and these were highly successful. My thinking has changed in the area over the years, though, as I’ve seen the investments made by so many service providers to better align their teams to Agile models. The fact that you can conduct global Scrums and use shared workspaces and tools to manage requirements and build cycles provides significant advantages in a truly global model. The trick is to have development and design teams at all global locations and have teams at each location that are empowered to make decisions. We had client and supplier personnel at each other’s facilities and teams were integrated throughout the entire lifecycle. This is very different from the traditional onsite/offshore model, but it is extremely effective.
ISG is adding agile methodology language to our contractual exhibits, including Statements of Work, Service Levels, and pricing documents.
Q. How would you weight the importance of each primitive metric?
A. It is difficult to weight them without understanding the context. Each primitive often drives different management decisions from overall quality, application footprints, productivity improvements, etc. The key is to understand what behaviors you want to incent and use the SLAs associated with the primitive metrics to align behaviors with measurable results.
Q. If you recommend test centers of excellence and separate vendors for test, how do you deal with the tension between this and an Agile methodology, which works best with team ethic and co-location of developers and testers in the same Scrum?
A. Agile puts a different spin on Testing CoEs. I haven’t seen an Agile project use a separate testing organization. However, most organizations are still using blended delivery models, with a small sample of projects using Agile methodologies, so there still seems to be value in developing Testing CoEs.
Q. Is testing required for an application if we align CAST?
A. Yes, functional testing is still required even if CAST AIP is fully implemented. Functional testing tools ensure that an application meets the defined requirements for each. The International Organization for Standardization (ISO) calls this “testing for external quality.” External quality measures the behavior of the system and is only possible during the testing and production phases of the lifecycle. It is performed by executing the software in a system environment. Neither manual nor automated (tool-based) testing generally look at the internal quality of an application to identify if the application will withstand changes over time or if the architecture is sound, but rather they focus on ensuring that it does what it is expected to do today. Even though quality assurance tool vendors guarantee that they support testing across the application development lifecycle, it is still necessary to wait until the integration testing occurs to see what effects the overall system may incur.
Q. Is there any industry level efficiency/savings when ADM is managed end-to-end by a single vendor? Compared to multi-vendor, is it better?
A. This is the classic, “it depends” question. In general, we do not see significant differences in efficiencies that lead to costs savings from service providers in a single vs. multi-supplier model. In fact, we often see better pricing, relationship management, and productivity in a multi-supplier model because all suppliers know they have to be competitive every day on accounts that are truly multi-supplier. However, we do see higher retained costs and governance complexities as organizations manage more complex deals with multiple suppliers.
Q. Like FPs or UCPs used in development projects, what measures are typically used for measuring productivity in Apps Support or Production Support areas?
A. Function points can be used in some mature environments to determine the estimated maintenance efforts. We have also developed some interesting complexity matrices that take in account functions points, average # of instances, defect backlog, number of interfaces, number of users, response/resolution times, and availability times. These models look very promising. There is an interesting article on function points and their impact on development and maintenance called, “A Cease Fire in the Function Point Holy War.”
Q. What is a reasonable number of SLAs to manage in maintenance?
A. It’s important to balance the number of maintenance SLAs with the key behaviors you want to incent. In general, you should incent Responsive to Incidents, Timely Resolution of Incidents/Problems, Preventive Maintenance (or reducing recurring problems), and backlog or queue management of lower level issues.
Q. Why and how do you measure EVA in an outsourced environment, as the staff don’t work for the client organization (they work for the provider)?
A. The core EVA metrics, CPI (Cost Performance Index) and SPI (Schedule Performance Index) should be captured and measured by the Service Provider managing the project. If the client is providing the PM support, then Service Providers should be obligated to provide EVA metrics on a weekly basis. Earned Value Project Management (Fleming and Koppelman) and Managing Global Development Risk (Hussey and Hall) both address how to use Earned Value to manage large scale, global projects.
In case you missed it, the full webinar recording can be found here.