ITIL KPIs and the IT Balanced Scorecard

ITIL Key Performance Indicators and the IT Balanced Scorecard

IT-Management 2

Supervisors: Prof. Dr. Andreas Breiter, Dipl.-Inf. Jörg Hofmann 28 August 2009

Tobias Hildebrandt Master Digitale Medien, Fachbereich 3 Universität Bremen Matrikelnummer: 2337614 [email protected]

Abstract This paper deals with the IT Service Management Framework ITIL and the business performance measurement method Balanced Scorecard and its IT equivalent, the IT Balanced Scorecard. The Balanced Scorecard aggregates low level measures, which are called KPIs. Usually there are relations between these measures, which are called cause-and-effect relationships. This paper tries to find out what features current software implementations of the Balanced Scorecard and of the IT Balanced scorecard offer, especially in regards to cause-and-effect relationships.

Keywords: Keywords: ITIL, Performance measurement, Key Performance Indicator (KPI), Balanced Scorecard (BSC), IT Balanced Scorecard (IT BSC), IT Management, IT Service Management, IT Governance

Table of Contents 1

Introduction ............................................ ................................................................... ............................................... ............................................... ............................. ...... 1

2

Balanced Scorecard..................... Scorecard ............................................ .............................................. .............................................. ......................................... .................. 3

3

4

5

2.1

Motivation ........................................... .................................................................. .............................................. .............................................. .......................... ... 3

2.2

Concepts .............................................. ..................................................................... .............................................. .............................................. .......................... ... 4

IT Balanced Scorecard ........................................... .................................................................. .............................................. ..................................... .............. 8 3.1

Differences to the Business Balanced Scorecard .............................................. ........................................................ .......... 9

3.2

Considerations for implementation ............................................ ................................................................... ............................... ........ 10

Linking ITIL Key Performance Indicators and the IT Balanced Scorecard .................... 11 4.1

ITIL Key Performance Indicators .............................................. ..................................................................... ............................... ........ 11

4.2

Combining ITIL Key Performance Indicators with the IT Balanced Scorecard....... 11

4.3

Translating the strategy .............................................. ..................................................................... .............................................. ......................... 13

Cause-and-effect Cause-and-effect Relations ........................................... .................................................................. .............................................. ............................ ..... 16 5.1

State of the art in research .............................................. ..................................................................... ........................................... .................... 17

5.2

Software implementation ........................................... .................................................................. .............................................. ......................... 18

5.3

How a software IT BSC could look like ........................................... .................................................................. ......................... 27

6

Conclusion............................... Conclusion...................................................... .............................................. .............................................. ........................................... .................... 29

7

Sources ............................................ ................................................................... .............................................. .............................................. ................................... ............ 31

List of abbreviations BSC CIO CobiT IT IT BSC ITIL KPI OGC

Balanced Scorecard Chief Information Officer Control Objectives and Related Technology Information Technology IT Balanced Scorecard IT Infrastructure Library Key Performance Indicator Office of Government Commerce

Introduction

1 Introduction The ITIL (Information Technology Infrastructure Library) framework is an approach to IT Service management and according to [Th@] also the most widely accepted one. It was developed by Great Britains’ Office of Government Commerce (OGC) and is a documentation of best practices for IT Service Management. It consists mainly of a series of advising books. The ITIL framework has been revised since its initial development in the 1980s, the current version as this paper is being written is Version 3 which was released in 2007. The OGC claims, that using ITIL might provide benefits like reduced costs, improved IT services and improved customer satisfaction. It provides best practices for all of the lifecycles of an IT service, which according to [Th@] are Service Strategy, Service Design, Service Transition, Service Operation and Continual Service Improvement. This paper will mostly deal with the Continual Service Improvement phase. Since its first introduction the ITIL framework has become more and more accepted, being the most used framework for IT service management and a de facto standard in this area nowadays. This success might be supported by the shift from IT Management to IT Service Management. While IT Management was more concentrated on infrastructure and the operational level of IT, IT Service Management focuses on the services that are provided to its customers. In the course of this shift, new methods were needed to cope with these changes, and the ITIL framework was able to provide these methods. At the same time, the importance of the alignment of business and IT increased, which made the role of the IT Manager more important, leading to the nowadays common role of the CIO (Chief Information Officer), who is often a member of the board of executives. The next shift of roles and mindsets in IT that is frequently discussed is the IT Governance, which is even more business oriented than IT Service Management. While IT Service Management can still be considered to be in the IT domain, the IT Governance responsibilities are to be found almost completely in the business domain. According to [Va07] the ITIL framework is clearly a management framework, and therefore it does not cover IT Governance – in contrary to other frameworks like for example CobiT, which are explicitly frameworks for IT Governance.

1

Introduction

What ITIL is missing however is a business-centered way of controlling the IT and evaluating its support in reaching the business goals. These aspects can be covered by the IT Balanced Scorecard, which can be used to link the business goals and IT Service Management. According to [Sa04@], the IT Balanced Scorecard is a method that can be used to support IT Governance. What are the current features of (IT) This paper tries to answer the question ““What Balanced

Scorecard

software

regarding

cause-and-effect

relationships

between measures in general and ITIL Key Performance Indicators in special and how can they be improved? improved?”” In order to answer this question, firstly the concepts and methods of the Balanced Scorecard (BSC) are explained in chapter 2, which is followed by remarks on the IT Balanced Scorecard (IT BSC) in chapter 3. Afterwards methods to combine ITIL and the IT BSC are discussed in chapter 4, which is then follow by extensive considerations on cause-and-effect relationships in theory and in software products (in chapter 5). This paper ends with the conclusion.

2

Balanced Scorecard

2 Balanced Scorecard The BSC is a method for strategic business performance measuring that was suggested by Robert S. Kaplan and David P. Norton in 1992. Since then it received a lot of attention by managers as well as by scientists of different sectors. These sectors include big companies like banks, insurance companies or software manufacturers, but also smaller or governmental enterprises, like for example hospitals. This chapter first tries to explain the motivation that led to the development of the BSC, and then shows the underlying concepts.

2.1 Motivation Historically, business performance measurement was based mostly on financial techniques and measures like budget planning, Return on Investment (Ratio of money earned or lost on an investment), Cash flow (movement of cash into or out of a business) or earnings per share (Earned money divided by the number of shares) [Ba01@]. The concentration on these aspects may lead to very short term thinking – usually managers plan ahead at most until the next fiscal year, where a new budget plan is being created. Long term goals and developments may be neglected by this paradigm. This kind of thinking may also lead to actions and decisions which can help in the short term, but impede long term improvements. One example might be expenditures on employee trainings. When just focussing on “traditional” financial performance measuring, a manager will, in many cases, come to the conclusion to cut expenditures in this area – in doing so he will better be able to keep his budget plan for the current year. And when a Return on Investment analysis on expenditures on training is being calculated, there might not be any measurable or predictable “return” at all, let alone in the current fiscal year. This is because it is very difficult to calculate the detailed benefits of an improved employee training. But in cutting this expenditure there might be a lot of additional costs in the future years coming up, for example a lower productivity or a decrease in customer satisfaction due to a worse service. This is where the BSC comes into play. It tries to help to see a business from a more “integrated” perspective, and also aims to put performance measurement on a more long term strategic level. 3

Balanced Scorecard

Its’ basic concept is to select a hand full of goals from the following areas, which are called “perspectives”: user orientation, business contribution, operational excellence and future orientation (see figure 1). Each of these perspectives has a mission. One example for such a mission might be “be the supplier of choice for our customers” for the customer perspective. Besides the overall mission, each perspective is made up of several subgoals that support the mission. For each of these goals it has to be defined a way to measure the state of their completion. In this example “reach a high customer satisfaction” might be a good goal to reach the mission of the customer perspective. In this example “Score of evaluation by customer” might be a way to measure this goal. [Ka92]

Figure 1: The four perspectives o f a Balanced Scorecard

It is very important that these values on the lowest hierarchy level of the BSC are actually measurable.

2.2 Concepts One of the reasons why the concept of the BSC is so successful may be that it claims to help seeing a business from an integrated meta-perspective. It especially is intended to help to see interconnections between areas of business (the perspectives), which is illustrated by figure 2.

4

Balanced Scorecard

Figure 2: The interconnections of a Balanced Scorecard

One area that is often discussed about the BSC is the finding of cause-and-effect relationships between the perspectives in general, and between the goals of them in special. These relations shall now be explained by one example. Figure 3 shows on the left side a so-called “strategy map” (an overview of the perspectives of an organization with its goals) of a BSC, which consists of four perspectives - each of which have two subgoals. Now one might come to the conclusion that there might be relationships between the goals, for example between the customer support and the customer satisfaction. A manager of a company could for example suspect that by improving his customer support, a higher customer satisfaction might be the result. This possible relation was added to the strategy map in figure 3. The right side of figure 3 offers a more detailed view with metrics measuring the subgoals defined in the strategy map. The superficial cause-and-effect relation defined between the goals “Customer Support” and “Customer Satisfaction” may here be displayed more detailed by building relations between the metrics themselves. For example assuming that the imaginative company of this example is a data center, especially the metric “% SRs Resolved on Initial Contact” might have a big effect on customer satisfaction. Maybe this relation can even be specified more detailed: the relation may exist especially with the measurement “Customer Satisfaction - Service Support”. But the finding of cause-and-effect relationships usually does not end here – generally this is a process with multiple steps. The relations between goals that have been defined in figure 3 can now be used as a base for finding more relations. 5

Balanced Scorecard

Figure 3: Strategy Map and IT BSC part 1

As pointed out in figure 3, there might also be a cause-and-effect relationship between customer satisfaction and the revenue. Given that improving the customer support leads to a higher customer satisfaction, it can further be concluded that this may lead to increased revenue. Figure 4 links this relationship to the metric “IT Revenue – actual”.

6

Balanced Scorecard

Figure 4: Strategy Map and IT BSC part 2

The examples that were pointed out here can be completed by many more relations that might exist between these goals. Once found, these possible relations should then be evaluated – possibly in group workshops with discussions, using historical business data if available.

7

IT Balanced Scorecard

3 IT Balanced Scorecard After the concept of the BSC was introduced, it was soon picked up by different sectors and areas. Thus, a BSC especially for IT Management / IT Governance was created. The first suggestions to use the BSC in the IT context came up in 1992 by Gold [Go92]. However, Willcocks found out in 1994 [Wi94], that Golds’ approach was leading too much to see IT from an IT departments’ perspective. After that, the concept of the IT BSC has been refined in 1997 [Va97] and 1998 [Va98] again. Since then it has further been improved, for example by Wim Van Grembergen in 2000 [Va00], who pointed out how to link the IT BSC to the business BSC. The main motivation to develop the IT BSC was to demonstrate the value of IT to the business, address IT Governance issues and make IT more efficient and cut costs [Cr07]. According to its popularity, [Cr07] created a hype cycle for the IT BSC. A hype cycle is: “ […] […] a graphic representation of the maturity, adoption and business application of specific technologies […] technologies […]” ” 1. This cycle (see figure 5) was published in 2007, according to which the IT BSC already reached the “Plateau of productivity”. This is the last phase of a Hype Cycle and means, that a product or method is no longer a hype theme which is heavily discussed, but somehow accepted and also widely used.

1

[Un@]

8


Figure 5: Gardner Hype Cycle of the IT BSC [Cr07]

One of the points of motivation to use the IT BSC is that it delivers faster feedback of the status of goals than traditional methods, what van der Zee expresses as follows: “The time lag between business planning processes and IT planning processes is often just too long to be acceptable.” 2 . This chapter first points out the differences to the business BSC, and then states considerations that should be taken into account when implementing an IT BSC.

3.1 Differences to the Business Balanced Scorecard One difference to the business BSC is that often the perspective “financial” is replaced by “business contribution” [Va00]. While the naming of the perspective can differ, the content can be considered the same (IT expenses and business value of IT projects). The term “customer” from the customer perspective is often replaced by the term “user” [Va00], which may sometimes fit better. better. The “learning and growth” perspective is sometimes replaced by “future orientation”, which still contains strategies like offering a good employee training, but also includes for example the research of new technologies and applications in order to provide a better position for the future.

2

[Va99], Page 141

9


3.2 Considerations for implementation There are some things to consider when implementing an IT BSC. For example instead of just one IT BSC there can also be a cascade of hierarchical grouped IT BSCs (which applies to the business BSC also) [Va99]. There could be for example BSCs for each ITIL-Service or for different phases of service lifecycles, like a scorecard for Service Design. These scorecards on the lower levels may then be aggregated to a single high level IT BSC. However, there are some critical success factors that need to be considered when implementing an IT BSC. One of them is the communication between the IT department and business managers, as J.T.M. Van der Zee states: “Potential success is highly dependent on the communication skills of all who are involved. In many organizations, business managers and IT planners have shown that they are unable to express themselves in a common language, […] so that successful links between business objectives, the IT strategy, and the IT architecture were insufficiently build.” 3 . This ability to communicate is especially crucial in the development phase of an IT BSC. In a case study ([Va99]), the IT manager of a company was not involved in the planning process of an IT BSC introduced in his company. In that example, the CIO of said company was not a member of the board and not involved in any decision making - neither in the initial development, nor in the process of finding cause-andeffect relationships. This resulted in unrealistic time estimations of steps that needed to be completed in order to meet goals from the BSC. For example some business executives shared the opinion “ […] […] that the replacement of a core information system could be done almost overnight.” 4 , what resulted in a delay of more than one year. The conclusion of the study is that communication barriers between disciplines have to be recognized and addressed before developing a BSC, as “ […] […] the Balanced Business Scorecard does not strictly enforce an integration of business and IT management processes by itself.” 4 . When business- and IT managers are working together the probability that an IT BSC is successful is higher.

3 4

[Va99], Page 142 [Va99], Page 153

10

Linking ITIL Key Performance Indicators and the IT Balanced Scorecard

4 Linking ITIL Key Performance Indicators Indicators and the IT Balanced Scorecard This chapter tries to point out how to link ITIL Key Performance Indicators (KPIs) to the concept of the IT BSC. After a short introduction about ITIL KPIs, the question is tackled how to combine these with the IT BSC. This chapter ends with remarks on how to translate the organizations strategy top-down up to low level measurements.

4.1 ITIL Key Performance Indicators The lowest level of performance measuring is defined by metrics, which are in the context of BSCs most of the times called KPIs. Although Kaplan and Norton [Ka92] did not speak of KPIs when they first introduced the BSC (instead, they used the more generic term “measure”), the term was soon associated with the BSC. Nowadays, in most cases the measures or metrics in a BSC are called KPIs. ITIL also uses the term KPI to describe its metrics. ITIL defines KPIs as atomic measures that provide low level measurements of how well the ITIL services are running. For every KPI there should be a target value, to which the current value is compared to – without a target, there would simply be no way of telling if a goal is reached or not. There is a number of KPIs suggested in the ITIL framework for every state of the service lifecycle. It is recommended by the ITIL framework that these measurements are used to invoke a continual service improvement – if the actual measures differ from the planned values, actions have to be taken. It has also been recommended frequently, that for every KPI a responsible person is defined (the so called “metric owner”). This person is the responsible for gathering the necessary data to keep up-to-date values of a metric and sometimes also for reaching the goal that is measured by “his” KPI.

4.2 Combining ITIL Key Performance Indicators with the the IT Balanced Scorecard As the goals and strategies of the perspectives of the IT BSC have to be supported by metrics, using ITIL KPIs for that may come to mind. And in fact many of the metrics typically found in IT BSCs are recommend by the ITIL framework.

11


But often ITIL KPIs itself can not cover all aspects of an IT BSC; this may especially be true for the financial perspectives. This is why ITIL KPIs are often coexisting beside other KPIs or also purely financial measures (like displayed in figure 6).

Figure 6: Combining ITIL KPIS and the IT BSC

According to [Ke05] it is a deadly sin to organize an IT BSC in a way that it offers a very IT-centric view of its performance - according to the authors an IT BSCs’ missions and goals should instead be business-centered. One might therefore come to the conclusion that an IT BSC might run the risk of being too IT-centric by focussing too heavily on ITIL KPIs (as they are often very operational and IToriented). On the other hand [Ke05] says that another sin might be using measures that lack standard metric definitions. This problem might occur less when using KPIs that are recommended in the ITIL framework, as there is already existing knowledge about them. And it should not to be forgotten, that even the OGC recommends in its book on Continual Service Improvement [Ca07] to use KPIs generated from ITIL services to measure higher level IT goals and later subsequently aggregate these in IT BSCs. 12


A conclusion might be to use ITIL KPIs, but also other measures and KPIs where adequate. If several KPIs are combined to measure one goal, the risk of having a too short-time-oriented, too IT-focussed measurement can be reduced. When merging several of these KPIs, the fusion of these might help to see the IT from a more distanced, business-like perspective. However, it has to be kept in mind to have a filtering process at work, which helps to select just KPIs that are relevant for the respective business and IT goals.

4.3 Translating the strategy The translation of the business goals into IT goals may be a crucial step when implementing an IT BSC. Figure 7 shows an example of such a transition process. The financial perspective of the Business BSC has in this example, among others, a business goal called “provide a good return on investment of IT-enabled business investments”, which can be mapped to an IT goal “Improve IT’s I T’s cost-efficiency and its contribution to business profitability”. This makes sense, as when the IT cost is going down the investment in IT will also decrease, which may make it easier to get a positive Return on Investment. However, this figure covers only the “investment” part of the business goal to increase the Return on Investment. However, the term Return on investment indicates, that there is besides an investment also some kind of Return (or income). However in this case the IT goal only covers the expenditures of IT, not its incomes or benefits – therefore, the business goal is only partly translated into an IT goal. The moneywise benefit of IT however may be difficult to measure, unless the affected company is a data center, which has concrete revenues for its services. But even if not, with ITIL successfully in place it should be possible to charge the business for the IT services it consumed. Returning to the example in figure 7, the IT goal “Improve IT’s cost-efficiency and its contribution to business profitability” is being covered by two processes, one of them is ”Identify and allocate costs”. This process makes sense, as if the cost-efficiency has to be improved, the costs have to be known. This process is measured by a bunch of KPIs (in figure 7 they are called “IT Performance Metrics”), which in this case are not KPIs recommend by the ITIL framework, as the financial perspective is not covered very deeply by ITIL.

13


Figure 7: Mapping of business goals to IT metrics (screenshot of the software Metricus)

Other IT goals and perspectives could probably be very well measured by ITIL KPIs – for example the goals of the Internal perspective, as it focuses on internal business processes. These internal business processes should be easily translatable into IT processes, which may be managed using the ITIL framework anyway. In this case it would make sense to use also KPIs that are recommended by the ITIL framework. Another example of how to combine different types of KPIs could be the IT BSC for a software developing company. The customer perspective for example could have as a business goal “support the software development team in developing new products”. This business goal could be translated into the more specific IT goal “Support the development team by delivering the necessary tools at high availability”. In order to measure the fulfilment of this goal, the processes could be for example “find out which tools the development team needs”, “provide and maintain the adequate tools” and “provide the availability of development tools”. Concerning the process “provide and maintain the adequate tools”, one could think of the following KPIs to measure the fulfilment of this t his process:

•

Number of requests for change (regarding the tools)

•

Average ratio of number of computers where provided tools are installed compared to average number of computers existing in development department 14

Linking ITIL Key Performance Indicators and the IT Balanced Scorecard •

Percentage of tools that are installed in the latest available product version

•

Number of customer complaints (regarding the tools)

•

Number of incidents (regarding the tools)

•

Percentage of tools that are generally the most popular tools of their category

This list is just for the purpose of giving a general idea, one could certainly think of more KPIs. What is noteworthy in this example is that metrics defined by the ITIL framework (like number of requests, number of customer complaints or number of incidents) coexist beside KPIs that have been developed for this specific process.

15

Cause-and-effect Cause-and-effect Relations

5 Cause-and-effect Relations The principle of cause-and-effect relationships in BSCs was explained in chapter 2.2. There has some theoretical research being done in this field already, for example by Torben Hügens [Hü08], whose results will be picked up later. In comparison to this theoretical existing knowledge, there are a lot of software manufacturers that claim to offer software products which support the creation and maintenance of BSCs or even IT BSCs. This chapter will evaluate the product “Metricus” regarding its functionality of the IT BSC and ITIL KPIs in general and then regarding the finding and evaluation of cause-and-effect relations in special. This is followed by a broader overview over the market of products, stating if the most popular products realize the finding and evaluation of cause-and-effect relationships in their software, and how. This chapter ends with suggestions on how the finding and evaluating of cause-and-effect relations could be implemented in a software product. The selection criteria for choosing the tool “Metricus” for a more detailed evaluation were primarily, that the software supports creating and maintaining a BSC. Besides that another criterion was, as this paper specializes on ITIL KPIs and the IT BSC, that ITIL KPIs and ITIL Service Dashboards are supported. Three software products were evaluated for this: Hyscore BSC, Opalis Business Process Centric IT Dashboard and Metricus. Out of these, “Opalis Business Process Centric IT Dashboard” offered the support of ITIL KPIs and ITIL Service dashboards, but no BSC. Hyscore BSC did this, but without supporting ITIL KPIs and ITIL Service dashboards. Only Metricus managed to support all three criteria (see figure 8).

16


Tool

Balanced Scorecard

ITIL KPIs, ITIL Service Dashboards

Hyscore BSC

Yes

No

Opalis Business Process Centric IT Dashboard

No

Yes

Metricus

Yes

Yes

Figure 8: Selection criteria for the software evaluation

5.1 State of the art in research One of the most comprehensive books in the area of cause-and-effect relationships in BSCs is “Balanced Scorecard und Ursache-Wirkungs-beziehungen” from Torben Hügens [Hü08]. He criticizes that the way of finding relations between measures suggested by Kaplan and Norton - the strategy map - does actually not state causalities, as it is not clear which metric is influencing the other. Further he criticizes that this system is based on subjective expectations from managers, as the possible relations have to be found manually, without a clear model to evaluate these suggestions. He further presents a model suggested by the consulting agency Horváth & Partners, in which the possible relations are determined in the form of a workshop. This seems better to Hügens, as this enables an extensive discussion about the causalities. It is recommended however, to evaluate these possible relations afterwards. Regarding the traditional ways of finding the relations, Hügens sees big problems in the fact that they point out relations, but they do not quantify these. This means they do not show how strong the dependency of a relation is. He concludes this by saying that the status quo in research, as well as in current software products, provides only methods to find qualitative, not quantitative relations. He further states that statistical methods are available, which can help to find correlations between the measures. These are according to the author already in use in order to find cause-and-effect relationships, despite the fact that these methods can just find correlations, not causalities.

17


He suggests to use a method called sequence analysis instead, which can help to find out the time which passes between a cause and the effect it has on another variable. He concludes that there is no simple way to solve this challenge to find these relations due to the complexity of the topic. He sees need for further research in this area, among other things in how to simulate the effects. This might be one area where software would be very helpful. He states that one way to solve this problem might be qualitative reasoning, which is a technique that is usually used in physics, and can help by means as simulation to find causalities even without much quantitative data. Like Hügens, Akkermans et al. [Ak02] also suggest to build a model of the expected relationships, and then test this model using simulation methods which are fed by historical data available in the company. For doing so, they claim that most relations between KPIs are not unidirectional, but that instead most metrics will build cyclic loops of some kind. They tried to prove this by conducting a case study with a company, for which they build a quantified simulation model. The expected (loop) causalities where established in a workshop, the model was filled with company data. The outcome of the study, among other things, established loopcausalities, as well as the successful calculation of time lags between changes in related KPIs. In summary, it can be said that the study proves that simulation can in fact be very helpful to test out relationship models. Unfortunately, according to Hügens, these simulation methods are hardly ever implemented in software products (or at least were at the time his book was written) – if this is still the case will be verified in the course of this chapter. At least, according to Hügens, statistical methods like correlation analysis are already in place and used in developing BSCs. If these methods are also embedded in common BSC software will also be a topic of this chapter.

5.2 Software implementation Metricus Metricus is a web-based Software-As-A-Service application for IT-Management by the Dutch company ITpreneurs. It is a modular system that offers an IT BSC, service Scorecards and dashboards for various ITIL processes such as for example Service Desk and Change Management. These modules come with a set of over 600 KPIs within a “best practice IT KPI library”.

18


Also available are modules to support IT Governance such as a module for project management, as well as a module for green IT and one for Cost-control. This evaluation is giving a general overview of the functionality first, afterwards the features regarding relations are being discussed. The first steps when using Metricus is the data entry. Here Metricus offers some user interfaces to enable the data entry and processing from different sources, such as databases or CSV-files. When building the scorecards, one can use a set of predefined KPIs or define his own ones. When the system is configured and running, one can choose mainly between several dashboards, ITIL-service scorecards and the IT BSC. The dashboards are left aside for this evaluation, as they are not topics of this paper. Figure 9 shows one of the ITIL-service scorecards, the change Management Scorecard. It offers an easy to understand and comprehensive overview of the perspectives, goals and measures.

Figure 9: Change Management Scorecard in Metricus

Figure 10 shows the user interface for IT BSCs, which has a similar structure as the ITIL-service scorecards.

19


Figure 10: IT BSC in Metricus

Besides that view of the IT BSC, the online demo that was used for this evaluation contained no features that seemed to support the finding or simulation of relationships. However, the menu entry “Service Desk Benchmarking” contained a window that was named “Cause and Effect”, which was unfortunately blank and did not offer any options. After asking ITPreneurs about that and about the overall possibilities of Metricus to support the finding or evaluation of these relationships, the company responded that Metricus offers the display of MS-Visio diagrams that show relationships between measures. However, these diagrams have to be created by the customer, and also the relationships itself have to be found, defined and tested by the customer. Therefore it can be concluded that Metricus does not support the process of finding relations.

20


Of course there is more BSC software that might support relations – according to [Fi06@] in 2006 there were 23 vendors whose products were certified by the Balanced Scorecard Collaborative, and around 80 more that were not. The most popular ones are now described according to the features that are stated by their publishers. IBM Cognos 8 IBM Cognos 8 is a business intelligence tool that also contains a BSC. According to IBM it offers cause-and-effect diagrams as well as impact analysis diagrams, which should help to visualize how strong the impact of a change of one KPI on a related KPI is. IBM states the following in its product brochures: “Automatically generated HTML displays of the relationship between metrics visually guide analysis to the root of performance of performance problems.” 5 . IBM claims that its tools can answer the following questions: “What are the factors driving the performance? What other processes or metrics are affected?” 6 . It has, however, not further been researched for this paper, if the cause-and-effect diagrams have to be created manually or if up to which extent Cognos can help to find relations and causalities. In the whitepapers and videos IBM offers about Cognos (at

http://www-01.ibm.com/software/data/cognos/library.html)

there

is

also

no

mentioning of any tools than can simulate changes, so it can be assumed that this is not supported.

CP-BSC The tool CP-BSC by the company “Corporate planning” offers a software supported BSC. It contains views for cause-and-effect relations and their impacts. However, it can be assumed that as in Metricus these relations have to be modelled manually, and that no simulation tool is supported: „Es ist möglich, sowohl positive als auch negative Wirkungen mit verschiedenen

Ausprägungen

(Stärken)

zu

definieren

und

zu

dokumentieren.“ 7 .

5

[Co02@] [Mo09@] 7 [Cp@] 6

21


Artemis Balanced Scorecard The company Artemis offers a product called Artemis Balanced Scorecard. It offers, among other, a view that shows cause-and-effect relations. As shown in figure 11, these relations can also been shown quantitatively with a weighed index.

Figure 11: cause-and-effect relations in Artemis [Ar@]

However, again the product description might lead to the conclusion that the relations have to be established manually: „A simple yet powerful feature links Strategic Objectives, Measures (KPIs) and Initiatives, allowing easy navigation and management of dynamic cause and effect relationships.“ 8 . One might come to this conclusion because the words “navigation” and “management” are used, instead of words like suggestion, finding, exploring or simulation.

8

[Ar@]

22


SBS - STRAT&GO Business Scorecard The product SBS-STRAT&GO Business Scorecard by the company PROCOS AG offers, beside the common features for BSC software, possibilities to simulate the relations between KPIs: “ […] […] Simulation of scenarios using the defined hierarchical KPI relationships […]” 9 . […]” Figure 12 shows that an extensive simulation of the impact of KPIs on other KPIs is possible – regarding [St@] however, the relations itself have to be found and defined manually, PROCOS offers workshops to support the customer in this task.

Figure 12: Cross-Impact-Analysis in STRAT&GO Balanced Scorecard [St@]

SAP SEM Business Planning and Simulation SAP offers the tool SAP Strategic Enterprise Management (SAP SEM), which includes a software enabled BSC. Regarding [Cr@] the tool can help in finding relationships between KPIs:

9

[Sb@]

23


“It supports the development and maintenance process of a Balanced Scorecard through the following functionality […] Definition of an influence diagram (cause-and-effect linkage) to visualize dependencies among strategic objectives on a Balanced Scorecard. “ 10 . However, it is not mentioned that this “definition of an influence diagram” is supported by automatization features or that SAP SEM suggests possible relations. Therefore, it can be assumed that the relations have to be defined manually. This assumption is supported by the following statement: “ […] […] Hypothesis definition through development of influence diagrams to visualize dependencies among strategic objectives and KPIs in a Balanced Scorecard. Influence diagrams are stored in the SEM database for shared access (controlled by defined authorizations) [...] ” 11. [...]” However, Regarding [Sa04@2], the relationships between metrics can be used for “what-if” simulations. And once defined, the relations can also be quantified: “ […] […] Quantification of influence diagram cause-and-effect linkages by creating a dynamic simulation model with SEM Business Planning and Simulation functionality." 12 . Figure 13 shows the simulation functionality of SAP SEM.

10

[Cr@] [No99@] 12 [No99@] 11

24


Figure 13: SAP SEM simulation functionality [Sa@]

In addition, according to [No99@] the created simulation models are even assessed automatically on a regular basis, where actual metric values are compared to their targets. If simulation models are no longer representing the reality, they are automatically adapted. SAS Balanced scorecard: The company SAS offers a tool called Balanced Scorecard. SAS claims to offer extensive automatization and simulation support for their tool: “As noted before, users can create data-driven, dynamic links within strategy diagrams to virtually any element within a SAS Strategic Performance Management project (measures, objectives, initiatives etc…This means that one can “logically” dig deeper into a specific piece of information using appropriate software for developing insight and foresight.” 13 .

13

[Cr07@]

25


In contrast to all other tools evaluated so far, SAS claims not only to be able to simulate user defined relationships, but also to suggest and find these relations at first hand: „Correlation analysis can reveal previously unknown relationships between metrics, making it clear which metrics are important and where to set thresholds. This analysis shows a tight correlation between customer churn and total revenue. Once cause-and-effect relationships are determined and validated, their relationship to one another and the strengths of the relationships can be displayed. Users can better predict potential outcomes based on achieving certain results.“ 14 . The ability of finding relation is further described as follows: “That is, the movement of one variable may have been caused by the movement of another. Using advanced modelling techniques, these causal relationships can then be isolated and highlighted. These capabilities can help uncover cause-and-effect relationships. Regression analysis is a common form of predictive modelling that can reveal previously unknown relationships between KPIs in an easy and intuitive way. In the process, it reveals broader and deeper insights into the way an organization really operates. To further hone your strategy, a variety of analytical methods such as neural nets, genetic algorithms, experimental design and optimization can be applied.” 15 . Simulation also seems to be supported: “Example 2: Forecasting and “what-if analysis” help you model scenarios to determine the best course of action. Change the values for one or more variables to see the effect on the forecasted margin.“ 16 . And also the impact that a change in one KPI has on another KPI seems to be simulatable in form of a sensitivity analysis: “Applying rigorous analytics to business problems enables managers to more quickly anticipate future challenges and opportunities with confidence assess the impact of changing KPI values and respond more quickly with fact-based decisions.” 17 .

14

[Co@] [Pr09@] 16 [Co@] 17 [Pr09@] 15

26


According to [In08@], every SAS program can be loaded into their data mining workbench. One might expect that by means of data mining SAS can offer extensive possibilities to find correlations between measures (for example by the association method) or conduct simulations.

Conclusion After evaluating Metricus practically by the means of an online demo, and the features of six other tools according to the descriptions of their publishers, it can be concluded that software implementations of the BSC leave in many cases a lot to be desired. All of the evaluated tools deliver an implementation of the BSC itself via some kind of dashboard or cockpit view with different degrees of comfort. But only the tool from SAS really seems to help in finding and suggesting possible relations between metrics. The situation regarding simulation features looks a bit brighter: here at least the products from SBS, SAP and SAS seem to offer extensive tools to simulate quantitative impact changes between KPIs. In the short evaluation that was executed here, the tool from SAS seems to offer the best features. As the same company also offers modules for ITIL KPIs and an IT BSC, it seems therefore much more powerful than the practically evaluated Metricus, which lacks any features regarding relation finding or simulation.

5.3 How a software IT BSC could look like It has to be noted up-front that it seems very unrealistic that a BSC system could find out all important cause-and-effect relations with their linkage in terms of the time lag between changes and in terms of their impact on each other without any human intelligence. As the business reality is probably not a deterministic model that can completely be calculated and predicted, it is and will probably stay a task of managers to at least evaluate automatically suggested relations and also to suggest ones that might not be found by correlation analysis. However a well working methodology could be that a system suggests through correlation analysis or other means an initial set of cause-and-effect relations. This list can then be expanded by managers for example in the form of a workshop. For example could a software solution, based on historical business data, create a list of all correlating measures that exceed a predefined correlation threshold. This list could also contain the impact that the metrics have on each other and the time lag that lies between cause and effect. 27


Managers and employees can then use their knowledge to verify or falsify this list, set the causalities of the suggested correlations and add own suspected relations. These new relations can then again be subject of a correlation analysis. Once in place, this system of causalities can then always be used for simulations, for example for sensitivity analyses. And even if such a system fails to provide correct relations, at least the process can help to sensibilize managers, as concluded by [Ho98]. The software from SAS seems to enable all these features - however, one could imagine even more. One thing that could be useful is a feature to suggest possible relations not only using historical data of one company, but maybe using data from several different companies of the same sector or industry. Online based softwareas-a-service solutions like Metricus could collect the data from all its users in an anonymized way to facilitate analysis and simulation by providing a bigger data pool. This might especially be useful for companies that do not have much historical data for KPIs yet, for example because they just introduced ITIL to their company. Such a comparison within companies that operate in the same sector might be especially easy if a company is using a lot of KPIs that are recommend by the ITIL I TIL framework or that are very common. However, the suggested relations have to be evaluated thoroughly, as even in companies of the same sector they might sometimes not be directly transferable. What can be suggested for further research is to find out how KPIs of companies of the same sector resemble each other.

28

Conclusion

6 Conclusion This paper began by pointing out the motivation for the change from IT Management to IT Service Management and finally to IT Governance. It described the need for IT Service Management frameworks like ITIL and performance measurement concepts like the IT BSC. The motivation for and concepts of the BSC were explained, followed by considerations on the IT BSC. It was further evaluated how to link ITIL KPIs to the IT BSC and how to translate the business strategy into goals and measures of an IT BSC. Then the state of the art in research regarding the finding and evaluation of cause-and-effect relations in BSCs was illustrated, which made clear that the research theories are somehow ahead of its implementations in software solutions. The evaluation of the software IT BSC Metricus showed, that even as it offers good support of ITIL and the evaluation of defined goals in an IT BSC, it lacked features regarding cause-and-effect relation concepts, as most of the other evaluated software solutions did too. Three of the seven evaluated products did offer mentionable simulation support, however only one of the seven products (SAS Balanced Scorecard) did show signs of features that help in finding possible relations. This evaluation was followed by considerations on how software enabled (IT) BSCs could be improved, which can be summarized as the implementation of simulation and correlation analysis techniques (an approach that already has been suggested by researchers). The research question of this t his paper “What are the current features of (IT) Balanced Scorecard software regarding causeand-effect relationships between measures in general and ITIL Key Performance Indicators in special and how can they be improved?” has therefore been answered in the course of this software evaluation, as it pointed out the available features by examining a few representative software products. The part of the research question about possible improvements of software products has been answered subsequently. What remains to be seen is if and how fast the industry will adapt the new mindset of IT Governance and replace or adjust its existing concepts of IT Service Management. In this context, it will be interesting to see if ITIL can still be adequate in an environment where IT Governance is in place – maybe the combination with more business-centered concepts of performance measurement like the IT BSC can help ensuring ITIL its popularity. 29

Conclusion

It is hard to predict future features of BSC software but it can be assumed, as there is already indication for that in some products, that software solutions will be more and more connected to business intelligence and data analysis tools. This linkage will provide the desired functionalities by methods like data mining and simulation. There is probably not much need for further theoretical basic research on finding and evaluating cause-and-effect relations, as there has been done a lot of work already. What can and should be done however is to research what mix of methods will provide the best output for practically implemented software solutions. Then it can only be hoped that this knowledge will be subsequently implemented by software manufacturers.

30

Sources

7 Sources [Ak02]

Henk Akkermans and Kim van Oorschot: Developing a Balanced Scorecard with System Dynamics, Proceedings of the 20th International Conference of the System Dynamics Society, July 28 - August 1, at Palermo, Italy, 2002

[Ar@]

Artemis Balanced Scorecard, http://sg.aisc.com/product/3, last checked on 8 August 2009

[Ba01@]

D. R. Backinsell: Performance Measurement and Evaluation, http://www.gfkmysteryshopping.eu/imperia/md/content/gfk_nop/mysterysho pping/cs_performancemeasurement.pdf, pping/cs_performancemeas urement.pdf, 2001, last last checked checked on on 8 August 2009

[Ca07]

Gary Case: Continual service improvement, The Stationary Office, Norwich, 2007

[Co@]

Gary Cokins et. Al: Discover how analytics makes the difference between managing and improving performance, http://www.sas.com/solutions/pm/ pmsupp-analytics.pdf, last checked on 8 August 2009

[Co02@]

Cognos Metrics Manager, https://www.bscol.com/pdf/Cognos https://www.bscol.com/pdf/Cognos_Facts.pdf, _Facts.pdf, 2002, last checked on 8 August 2009

[Cp@]

CP-BSC Balanced Scorecard, http://www.corporateplanning.com/d/pag http://www.corporateplanning.com/d/pages/ es/ soft/cpbsc.php, last checked on 8 August 2009

[Cr@]

Creating Strategic Focus: The Balanced Scorecard, http://help.sap.com/ saphelp_sem40bcs/helpdata/de/5a/546637a0 saphelp_sem40bcs/helpdata/de/5a/546637a04c2367e10000009 4c2367e10000009b38f8cf/ b38f8cf/ content.htm, last checked on 8 August 2009

[Cr07]

Alec Cram: The IT Balanced Scorecard Revisited, Information Systems Control Journal, Volume 5, Pages 33-36, Rolling Meadows, 2007

[Cr07@]

Creating focus, alignment and agility with SAS® Strategic Performance Management, http://www.sas.com/resources/whitepaper/wp_3549.pdf, 2007, last checked on 8 August 2009

[Fi06@]

Joseph M. Firestone: The Balanced Scorecard: Developments and Challenges, KMCI Online Press, 2006, http://www.adaptivemetricscenter.com/media/ BSCdevelopmentsandchallenges.pdf, BSCdevelopmentsand challenges.pdf, last checked on 8 August 2009

[Go92]

C. Gold: Total quality management in information services – IS measures: a balancing act, Research Note, Ernst & Young Center for Information Technology and Strategy, Boston, 1992

[Ho98]

Peter Horvath and Lutz Kaufmann: Balanced Scorecard - ein Werkzeug zur 31

Sources

Umsetzung von Strategien, Harvard Business Manager, Heft 5, Pages 3948, Hamburg, 1998 [Hü08]

Torben Hügens: Balanced Scorecard und Ursache-Wirkungs-bez Ursache-Wirkungs-beziehungen, iehungen, Auflage 1, Gabler Verlag, Wiesbaden, W iesbaden, 2008

[In08@]

Integrating Your Favorite SAS Program into a SAS Data Mining Workbench, http://www.sas.com/resources/whitepaper/wp_6874.pdf, 2008, last checked on 8 August 2009

[Ka92]

Robert S. Kaplan and David P. Norton: The Balanced Scorecard-Measures that drive performance, Harvard Business Review, January-February, Pages 71-79, Boston, 1992

[Ke05]

Jessica Keyes: Implementing the IT balanced scorecard: aligning IT with corporate strategy, CRC Press, Boca Raton, 2005

[Mo09@]

Monitor.Manage.Perform. Scorecarding with IBM Cognos 8 Business Intelligence, http://www.ibm.com/software/data/cognos/pdfs/whitepapers/ wp_c8bi_scorecarding_monitor_manage_perform.pdf, 2009, last checked on 8 August 2009

[No99@]

David Norton Norton et al.: SAP Strategic Enterprise http://www.hochschul-management.de/BSC-SAPSEM.pdf, checked on 8 August 2009

[Pr09@]

Predictive Performance Management, http://www.sas.com/apps/ whitepaper/index.jsp?cid=3496, whitepaper/index.jsp?cid= 3496, 2009, last checked on 8 August 2009

[Sa04@]

Mathias Sallé: IT Service Management and IT Governance: Review, Comparative Analysis and their Impact on Utility Computing, http://www.cioindex.com/nm/articlefiles/67864ITServiceManagementITGovernance.pdf, ITServiceManagementITGov ernance.pdf, 2004

[Sa04@2]

SAP Strategic Enterprise Management (SAP SEM), http://www.sap.com/ germany/media/50071524.pdf, germany/media/50071524 .pdf, 2004, last checked on 8 August 2009

[Sa@]

SAP SEM simulation functionality, http://help.sap.com/saphelp_se http://help.sap.com/saphelp_sem40bcs/ m40bcs/ helpdata/de/f8/4569377f2b8032e10000009b38f8cf/Image113.gif, last checked on 8 August 2009

[Sb@]

SBS-Strat&GO Business Scorecard, http://www.procos.com/en/ pro_mod_sbs.asp, last checked on 8 August 2009

[St@]

Strategieprozess innovativ unterstützt, http://www.procos.com/it/img/pdf/Strategieprozess_innovativ.pdf, checked on 8 August 2009

[Th@]

Management, 1999, last

last

The Official ITIL® Website, http://www.itil-officialsite.com/home/home.asp, last checked on 8 August 2009

32

Sources

[Un@]

Understanding Hype Hype Cycles, http://www.gartner.com/pages/ story.php.id.8795.s.8.jsp, last checked on 8 August 2009

[Va00]

W. Van Grembergen: The Balanced Scorecard and IT Governance, Information Systems Control Journal, Vol. 2, Pages 40-43, Rolling Meadows, 2000

[Va97]

W. Van Grembergen and R. Van Bruggen: Measuring and improving corporate information technology through the balanced scorecard technique, Proceedings of the Fourth European Conference on the Evaluation of Information Technology, October 1997, Pages 163-171, Delft, 1997

[Va98]

W. Van Grembergen and D. Timmerman: Monitoring the IT process through the balanced scorecard, Proceedings of the 9th Information Resources Management (IRMA) International Conference, May 1998, Pages 105-116, Boston, 1998

[Va99]

J.T.M. van der Zee et al.: Alignment is not enough: integrating business and information technology management with the balanced business scorecard, Journal of Management Information Systems archive, Volume 16, Issue 2, Pages 137–156, M. E. Sharpe Inc., Armonk, 1999

[Va07]

Jan Van Bon: IT service management: an introduction, Van Haren Publishing, Zaltbommel, 2007

[Wi94]

L. Willcocks: Information Management: The Evaluation of Information Systems Investments, Chapman & Hall, London, 1994

33

ITIL KPIs and the IT Balanced Scorecard

Recommend Documents