thesis. - Department Of | Electrical And Electronics
Transkript
thesis. - Department Of | Electrical And Electronics
AN ASSESSMENT MODEL FOR WEB-BASED INFORMATION SYSTEM EFFECTIVENESS A THESIS SUBMITTED TO THE GRADUATE SCHOOL OF INFORMATICS OF THE MIDDLE EAST TECHNICAL UNIVERSITY BY GÜL TOKDEMĐR IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY IN THE DEPARTMENT OF INFORMATION SYSTEMS JANUARY 2009 Approval of the Graduate School of Informatics _________________________ Prof. Dr. Nazife BAYKAL Director I certify that this thesis satisfies all the requirements as a thesis for the degree of Doctor of Philosophy. __________________________ Prof. Dr. Yasemin YARDIMCI Head of Department This is to certify that we have read this thesis and that in our opinion it is fully adequate, in scope and quality, as a thesis for the degree of Doctor of Philosophy. _________________________ Prof. Dr. Semih BĐLGEN Supervisor Examining Committee Members Assoc. Prof. Dr. Onur DEMĐRÖRS Prof. Dr. Semih BĐLGEN (METU, IS) _______________________ (METU, EEE) _______________________ Assist. Prof. Dr. Nergiz ÇAĞILTAY (ATILIM, SE) ________________________ Dr. Altan KOÇYĐĞĐT (METU, IS) ________________________ Dr. Sevgi ÖZKAN (METU, IS) ________________________ I hereby declare that all information in this document has been obtained and presented in accordance with academic rules and ethical conduct. I also declare that, as required by these rules and conduct, I have fully cited and referenced all material and results that are not original to this work. Name and Surname: Gül Tokdemir Signature iii :______________ ABSTRACT AN ASSESSMENT MODEL FOR WEB-BASED INFORMATION SYSTEM EFFECTIVENESS Tokdemir, Gül Ph.D., Department of Information Systems Supervisor: Prof. Dr. Semih Bilgen January 2009, 130 pages Information System (IS) effectiveness assessment is an important issue for the organizations as IS have become critical for their survival. With the incorporation of Internet technologies into the business environment, it is now more difficult to measure IS effectiveness, because Internet provides a borderless, non-stop, flexible communication medium. Assessing the effectiveness of web-based information systems (WIS) is vital for survival and competitive advantage which is a complicated iv subject since there are several interacting factors to consider. In the literature there are several methods proposed for IS assessment. However, those studies have been far from providing a broad, comprehensive evaluation framework for any type of web-based IS independent of its domain. In this study, a generic WIS effectiveness assessment framework is proposed. The framework is applied in case studies consisting of four organizations in e-commerce and e-banking domains. Keywords: Web-based IS, IS effectiveness assessment v ÖZ WEB-TABANLI BĐLGĐ SĐSTEMLERĐ ĐÇĐN ETKĐLĐLĐK DEĞERLENDĐRME MODELĐ Tokdemir, Gül Doktora, Bilişim Sistemleri Tez Yöneticisi: Prof. Dr. Semih Bilgen Ocak 2009, 130 sayfa Bilgi sistemlerinin (BS) etkililiğinin değerlendirilmesi, organizasyonların yaşam döngüsünde kritik önem arzetmektedir. Çünkü günümüzde. Internet teknolojilerinin iş ortamlarına girişiyle birlikte, bilgi sistemlerinin etkiliğinin ölçülmesi daha da zor hale gelmiştir, çünkü Internet sınırları olmayan, sürekli çalışabilen, esnek bir iletişim ortamı sağlamaktadır. Web Tabanlı Bilgi Sistemlerinin (WBS) etkliliğinin ölçülmesi organizasyonların devamlılığını sağlamaları ve rekabetçi ortamda varolabilmeleri olabilmeleri için gerekli olan, ancak birbiriyle etkileşimde olan faktörleri içermesi dolayısıylada karmaşık bir konudur. Literatürde bilgi sistemlerinin değerlendirilmesi konusunda çeşitli çalışmalar mevcuttur. Ancak bu çalışmalar, genel, kapsamlı ve etki vi alanından bağımsız bir WBS etkililik değerlendirmesinden çok uzaktır. Bu çalışmada, genel bir WBS etkililik değerlendirme modeli sunulmaktadır. Bu model örnek olay incelemesi olarak e-ticaret ve e-banka alanlarındaki dört şirkete uygulanmıştır. Anahtar Kelimeler: Web-tabanlı bilgi sistemleri, Bilgi sistemleri etkililik değerlendirmesi vii To My Parents; Fecriye and Faruk viii ACKNOWLEDGMENTS I would like to thank; My advisor Prof. Dr. Semih Bilgen; for his guidance, his precious support and his creative comments. His contributions were invaluable, extraordinary and his way of directing a student was unique. I have learned a lot from him about research and advising. I am privileged to be his student. My parents; for their endless support, my sister Gonca; for her extraordinary help in every step of this work and my brother Kemal; for his patience and motivation. They enlighten my life continuously. My Colleagues at Atilim University, Computer Engineering Department; for their kind support throughout this period. Dr. Ali Arifoğlu and my friends Hülya & Ferit Gökbel, Haldun Aydoğdu, Ali Çınar; for their astonishing help in arranging the interviews and in data collection process. ix TABLE OF CONTENTS ABSTRACT ............................................................................................................... IV ÖZ .............................................................................................................................. VI DEDICATION ........................................................................................................ VIII ACKNOWLEDGMENTS ......................................................................................... IX TABLE OF CONTENTS ............................................................................................ X LIST OF TABLES .................................................................................................. XIII LIST OF FIGURES ................................................................................................. XV CHAPTER 1. INTRODUCTION ................................................................................................... 1 2. LITERATURE SURVEY ........................................................................................ 6 2.1. IS EFFECTIVENESS ....................................................................................... 7 2.2 TRADITIONAL IS EFFECTIVENESS STUDIES .......................................... 9 2.3 WEB-BASED IS ASSESSMENT STUDIES .................................................. 14 2.4 GOAL QUESTION METRIC –GQM ............................................................. 22 2.5 SUMMARY AND DISCUSSION ................................................................... 24 3. SEWISS: WIS SUCCESS ASSESSMENT FRAMEWORK ................................ 32 3.1. IDENTIFICATION OF DIMENSIONS IN SEWISS FRAMEWORK ......... 34 3.1.1 WIS Relationships..................................................................................... 34 3.1.2 Strategy ..................................................................................................... 36 3.2 SUCCESS ASSESSMENT IN SEWISS FRAMEWORK .............................. 39 x 3.2.1 SEWISS Framework Weights and Measures ............................................ 41 3.3 SEWISS SUCCESS CALCULATION STEPS ............................................... 42 4. RESEARCH METHODOLOGY ........................................................................... 50 4.1. CASE STUDY RESEARCH IN INFORMATION SYSTEMS FIELD ......... 53 4.2 DATA COLLECTION METHOD .................................................................. 54 4.3. CASE STUDY PLAN- ASSESSMENT FRAMEWORK .............................. 56 4.4. PRELIMINARY CASES ................................................................................ 58 4.5 CASE STUDIES .............................................................................................. 59 5. APPLICATION OF SEWISS FRAMEWORK TO THE CASES......................... 61 5.1 PRELIMINARY CASE STUDIES .................................................................. 61 5.1.1 Preliminary Case 1- Organization A ......................................................... 62 5.1.2 Preliminary Case 2- Organization B ......................................................... 65 5.1.3 Preliminary Case 3- Organization C ......................................................... 66 5.1.4 Discussions of Preliminary Case Studies .................................................. 68 5.2 CASES FROM E-BANKING SECTOR ......................................................... 68 5.2.1 Organization D .......................................................................................... 68 5.2.2 Organization E .......................................................................................... 72 5.2.3 Outcomes of the Cases in E-banking Domain .......................................... 75 5.3 CASES FROM E-COMMERCE SECTOR ..................................................... 77 5.3.1 Organization F........................................................................................... 77 5.3.2 Organization G .......................................................................................... 80 5.3.3. Outcomes of the Cases in E-commerce Domain...................................... 83 5.4 DISCUSSION ON CASE STUDY RESULTS................................................85 6. CONCLUSION ...................................................................................................... 86 REFERENCES........................................................................................................... 92 APPENDICES A: FULL LIST OF MEASURES ............................................................................. 104 B: INTERVIEW QUESTIONS................................................................................ 109 C: SEWISS CALCULATION ................................................................................. 113 D: SHORT LIST OF MEASURES .......................................................................... 116 xi E: QUESTIONNAIRES ........................................................................................... 119 VITAE ...................................................................................................................... 129 xii LIST OF TABLES 1. 2. 3. 4. Literature Summary .......................................................................................25 Organizations Summary .................................................................................60 WIS Relationship Goals for Organization A .................................................63 WIS Relationship Weights .............................................................................64 5. 6. 7. 8. SEWISS value calculation .............................................................................64 WIS Relationship Goals for Organization B..................................................65 WIS Relationship Weights .............................................................................66 WIS Relationship Goals for Organization C..................................................67 9. WIS Relationship Weights .............................................................................67 10. WIS Relationship Goals for Organization D .................................................69 11. WIS Relationship Weights .............................................................................70 12. 13. 14. 15. 16. 17. SEWISS value calculation .............................................................................71 SEWISS values for Organization D ...............................................................71 WIS Relationship Goals for Organization E ..................................................72 WIS Relationship Weights .............................................................................73 SEWISS value calculation .............................................................................74 SEWISS values for Organization E ...............................................................74 18. Organization D and E Properties ...................................................................77 19. WIS Relationship Goals for Organization F ..................................................78 20. WIS Relationship Weights .............................................................................78 21. 22. 23. 24. SEWISS value calculation .............................................................................79 SEWISS values for Organization F ...............................................................80 WIS Relationship Goals for Organization G .................................................80 WIS Relationship Weights .............................................................................81 xiii 25. SEWISS value calculation .............................................................................82 26. SEWISS values for Organization G ...............................................................82 27. Organization F and G Properties ....................................................................85 xiv LIST OF FIGURES 1. GQM Technique.................................................................................................23 2. WIS Success Aspects .........................................................................................34 3. WIS Relationships..............................................................................................35 4. WIS Success Assessment Framework Dimensions ...........................................39 5. Assessment steps at the top level .......................................................................42 6. Steps of Measuring WIS relationships’ Success ................................................43 7. Steps of Applying GQM ....................................................................................44 8. Steps of preparing and applying the questionnaire ............................................45 9. Steps of Gathering CSF related data ..................................................................46 10. Case Study Steps ..............................................................................................55 11. WIS Success Assessment Stages .....................................................................57 12. SEWISS comparison of Organization D and E................................................75 13. Comparison of DailyReach values for Organization D&E ..............................76 14. Comparison of DailyPageview values of Organization D&E..........................76 15. Comparison of SEWISS values for Organization F and G ..............................83 16. Comparison of Daily Reach values for Organization F & G ...........................84 17. Comparison of DailyPageview values of Organization F&G ..........................84 xv CHAPTER 1 INTRODUCTION Information Systems (IS) have become crucial for organizations to survive in today’s technology-focused environment. Increasing amounts of resources are invested in IS infrastructures in organizations to give better services and to produce better value products. This boost brings the question of how much those systems add value to the business or to the organization compared to their investment. A recent survey (McKinsey, 2008) has reported that the huge sums invested in information technologies (IT) have not really served organizations’ business strategies as effectively as expected. Therefore, there is a need to justify IS expenditure by examining its contribution to achieving organizational goals. Because of the advances in Internet and IT, many IS systems turned into web-based information systems (WIS), enabling access through multiple channels in a dynamic and competitive environment. Considering this new environment together with the increased need for system assessment, this study aims to create a framework to assess effectiveness of WISs of organizations, which will also enable them to create a baseline for future investment decisions. The objectives of this study were: ► To explore the previous studies in IS effectiveness assessment area and to identify their inadequacies in the context of WIS; 1 ► To present a novel framework for WIS effectiveness assessment; ► To enhance the proposed framework and to appraise its validity via multiple case studies. In the literature, effectiveness is concerned with the influence or results caused by a system on the environment, thus has an external focus (Myers , Kappelman and Prybutok, 1997). Similarly, success is considered as the achievement of an intended or expected effect. The IS literature has many studies on IS effectiveness assessment concept. These studies can be grouped into two main categories: studies performed for traditional ISs, and those that focus specifically on WISs. In the former category, IS researchers have considered different aspects of the information systems being assessed like product, process, service, stakeholders and user satisfaction, as examined thoroughly in this study. Traditional ISs were closed systems, having a single access channel, where only staff running the IS had access to the system (Taniar and Rahayu, 2004). With the emergence of Internet and information technologies, many IS systems turned into WISs, enabling everyone to access it through multiple channels which transferred them into open systems. Naturally, this new dynamic environment has generated the necessity to look into the IS assessment topic through a new understanding. For WIS assessment, initially, the former traditional IS conceptual models were applied to the web-based domain by adopting these models specifically to e-business and e-government environments (DeLone and McLean, 2004; Hu, Xiao, Pang and Xie, 2005; Peters, Janssen and Engers, 2004). However, it is observed by many researchers that Internet generates a wide communication medium with many users from different cultures, with diverse expectations which makes business environment more complex (Bremser and Chung, 2005; Jarvenpaa and Tiller, 1999; Jones, Wilikens, Morris and Masera, 2000; Pires and Aisbett, 2003). Hence, parallel to that, the assessment of WISs should account this complexity, considering the system and 2 its interactions with its environment as a whole through a new perspective. Other studies on WIS evaluation focus on domain specific WIS assessment like assessment of e-government or e-business WISs, to consider the effect of Internet. These studies mainly concentrate on one of the following dimensions: consumer satisfaction, organizational strategy, risk, service quality and web-site assessment. In general, these studies, which are discussed broadly in this research, are far from providing a broad, comprehensive framework for the evaluation of WISs independent of the application domain. Besides, they ignore the organizational targets and WIS’s effect on those targets. Thus, a specific insight is needed for effectiveness assessment in web-based domain. The studies in IS area are generally conceptual studies, providing a research infrastructure and discussing which measures are more important than the others. However, these measures are usually not applied to assess WISs and to decide which system is more preferable in comparison to the others in a quantitative fashion. The proposed framework aims to fill this gap by developing a WIS effectiveness assessment framework which allows comparisons between different WISs. WISs have several interactions with different stakeholders having different expectations from those systems. Through Internet, users can access a broad range of information quickly, compare product prices, shop by fast transactions, exchange views about products and services easily, which creates a complex environment with security being an important issue. In such a competitive and complex environment, WIS’s success becomes vital for organization’s survival, and thus a WIS should contribute to the realization of the organizational targets. Therefore, WIS success assessment should take into account its contribution to the achievement of organizational goals. As a result, there is a need for a novel approach to evaluate WISs, considering Internet as a system characteristic rather than as an add-on property, regardless of 3 which domain they run in like e-commerce or e-government or e-health. This study proposes a comprehensive WIS effectiveness assessment framework based on both WIS-related and organization-related aspects, since WIS effectiveness or success is considered as “the level of achievement of system’s interactions with its environment as expected from it”. In this regard, WIS relationships and organizational strategy will be included as framework dimensions. A WIS interacts with users, with other IS’s (web-based or non web-based) in the domain and with the entities constituting the organization. These interactions, namely WIS Relationships, form one of the proposed framework dimensions. On the other hand, investment requirements for IT are shaped by business goals and so the evaluation process should measure accomplishment of reaching these goals (Serafeimidis and Smithson, 2003). A WIS strong in its relationships with the environment cannot be assessed as successful if it does not serve the strategies of the organization. Consequently, organizational strategy has been included as the second dimension of the proposed framework, which is specified in critical success factors (CSF). Achievements of the objectives of WIS relationships will be explored through GQM method, then they will be filtered through the CSFs, which in turn will yield a SEWISS success value for the WIS under assessment. This way the operationalisation of the organizational strategy in assessment will be addressed. The proposed SEWISS framework has been enhanced and its validity has been appraised through multiple case studies. Interviews with IS Specialist and organizational management and web-based questionnaires have been performed to gather necessary data for the assessment. Initially, preliminary case studies have been performed in different WISs in diverse domains to check and refine the framework dimensions and the assessment process. Then the enhanced framework has been applied in different organizations from e-commerce and e-banking domains. The WIS effectiveness results obtained through the framework have been analyzed through some organizational or domain specific factors to demonstrate concurrent 4 validity of the outcomes. The SEWISS framework is proposed as a generic framework for any type of WIS, for any organizational domain, whether it is e-business, e-government, etc. Nevertheless, it also allows organization specific assessment based on organizational CSFs and WIS relationship measures for different domains. This dissertation is organized as follows: in Chapter 2, related IS assessment literature is reviewed and evaluated from the viewpoint of today’s information systems in general and WIS, in particular. This is followed by the presentation of the proposed model and WIS success calculation framework in Chapter 3. Chapter 4 discusses the case study methodology implemented for first, enhancing the model, and then assessing its validity. The performed case studies are presented and discussed in Chapter 5 and finally Chapter 6 concludes the work by presenting an overall evaluation of the outcomes of the study and suggesting future work. 5 CHAPTER 2 LITERATURE SURVEY This chapter summarizes the current approaches and practices on IS effectiveness assessment. The subject is explored by considering the studies with various viewpoints like different stakeholders' views, consumer satisfaction, and processes. Additionally, the assessment studies for current web-based information systems like e-business, and e-government are investigated. Researchers have considered different aspects of the information systems being assessed as product, process, service dimension, stakeholders, and user satisfaction. The most comprehensive study on IS success assessment is DeLone and McLean’s IS success model (DeLone and McLean, 1992), which classifies the IS success measures under six dimensions building a conceptual model. With the emergence of Internet technologies, researchers tried to modify DeLone and McLean model to include the effect of Internet by applying the same model to ebusiness and e-government environments with some modifications. Other studies proposed different measures specific to e-business and e-government applications each of which mainly focuses on web-site evaluation, consumer satisfaction, firm strategy, and risk or service quality dimensions. In the following sections, the 6 concept of effectiveness, success and efficiency are discussed followed by the literature on these concepts. 2.1. IS Effectiveness Total worldwide information technology investment exceeded one trillion US dollars per year in 2001, which grows 10% each year (Seddon, Greaser and Willcocks, 2002). This huge amount of expenditure brings the question if those systems provide enough advantage for firms’ competitiveness. To be competitive in today’s shrinking markets, managements try to lower down the costs and increase the value for money (Kanungo, Duda and Srinivas, 1999). In this regard, IS expenditure should be justified by examining its benefits to the organization. Therefore, evaluation of the information system (IS) has been a popular research area over the past years in terms of effectiveness, efficiency and success. 2.1.1. Effectiveness, Efficiency and Success Several measures have been proposed to assess IS expenditure. In this regard, many studies focused on concepts of efficiency, effectiveness or success as a measure. These measures are used interchangeably creating an ambiguity in the field. Efficiency is considered as the ratio of the output to the input of any system. It is concerned with getting maximum benefit with less cost, so it focuses on doing the thing with minimum cost. On the other hand, effectiveness is considered as the power to be effective; or the quality of being able to bring about an effect. It is concerned with the influence on the environment, results caused by a system, thus has an external focus (Myers et al., 1997). Hamilton and Chervany used effectiveness as the accomplishment of organizational goals (Hamilton and Chervany, 1981). They claimed that effectiveness could be measured in two different ways; goal-centered view and system-resource view. The goal-centered view was concerned by assessing the organization with respect to its task objectives by finding the difference between performance and objectives. In system-resource view, effectiveness was concerned with resource viability. For the assessment of system 7 effectiveness, these considerations should converge. They also stated that, IS provided information to improve organization in reaching to its goals, which could be evaluated from two perspectives; efficiency of IS resources utilization, and effectiveness of the IS to users in achieving organizational objectives. They claimed that to assess the IS goals, performance measures developed for both of the perspectives could be used. Fitzgerald discusses completing some tasks efficiently, with minimum possible cost, does not guarantee that those tasks satisfy their intending objectives. Thus, using efficiency as an assessment measure may not give correct results since systems that may not provide cost savings can provide significant opportunities to organizations in the future (Fitzgerald, 1998). IS can be called effective if it supports organization to reach to its objectives as a whole (Malik, 2001). Early studies used efficiency as an IS evaluation measure, however it is replaced by effectiveness since IS systems are goal-oriented systems, thus need to measure its influence on the environment. An effective measure should be relevant to organizational performance with a future dimension and should provide internal and external comparisons and commonality between companies (Smith and McKeen, 1996). Success is considered as an incident that accomplishes its intended objective, and it is related with the achievement of an intended or expected effect. It has an external focus like effectiveness. IS success measure is elusive to researchers, and it is one of the controversial issues (Molla, 2001). In the context of information systems, success has been defined by DeLone and McLean and studied by many researchers. Measurement of IS success or effectiveness used interchangeably in these studies, which is mentioned as not an easy task, since these systems are not isolated from the environment and have several interactions with different stakeholders having different expectations from those systems. Since these two terms have a very close 8 meaning, in this study, success and effectiveness are used interchangeably. IS effectiveness assessment studies found in the literature can be grouped into two categories; studies performed for traditional ISs, and studies carried out for WIS assessment. The following sections elaborate those studies. 2.2 Traditional IS Effectiveness Studies For the traditional IS studies, IS researchers have considered different aspects of the information systems being assessed as product, process, service dimension, stakeholders, and user satisfaction. The following sections will explore those studies in detail. DeLone and McLean’s IS Effectiveness Model A major contribution to this area was by DeLone and McLean’s study which proposed a comprehensive framework for IS success measurement (DeLone and McLean, 1992). DeLone and McLean specified six different dimensions of IS success; System Quality, Information Quality, Use, User Satisfaction, Individual Impact and Organizational Impact (D&M model). They examined studies performed on measurement, and concluded that those studies fell into one of the dimensions they proposed. Many studies used and supported the validity of D&M framework. Myers et al. added service quality and workgroup effect to D&M framework (Myers et al., 1997). Jennex, Olfman, Panthawani and Park applied D&M model to Organizational Memory Information systems by proposing some modifications specific to the context (Jennex, Olfman, Panthawi and Park, 1998). Malik proposed that IS effectiveness should be performed in an integrated approach considering product, process and environment views (Malik, 2001). Pitt, Watson and Kavan argued that IS effectiveness measures focus on product, and service dimension is ignored (Pitt, Watson and Kavan, 1995). They suggested modifying D&M model by adding a new 9 dimension, service quality. They proposed to use the Servqual instrument, which has originally been used in the marketing area to measure the service quality. Seddon, Staples, Patnayakuni and Bowtell argued that D&M’s six success dimensions lack individual stakeholders’ interpretations of success, which is the dependent variable (Seddon, Staples, Patnayakuni and Bowtell, 1999). They also argue that since different systems may require different measurements for effectiveness, the system being evaluated must be considered. DeLone and McLean, in their 10-year update study (DeLone and McLean, 2003) evaluated the arguments against their six dimensions of IS success, and added a new item service quality to their framework. Iivari tested D&M model empirically, proposed using perceived system quality and perceived information quality instead of system use to measure user satisfaction with the system (Iivari, 2005). Chang and King suggested IS functional success evaluation in three dimensions; namely system performance, information effectiveness and service performance (Chang and King, 2005). They proposed functional score card composed of 18 factors in three dimensions, evaluated by IS managers and validated their results statistically. Similarly, Gable, Darshana and Chan reviewed D&M dimensions and modified it for Enterprise Systems omitting use construct, keeping its causal/process model of success, they added some new measures like customization, increased capacity, e-government and business process change and considered satisfaction as an overall measure of success (Gable, Darshana and Chan, 2003). They applied factor analysis to reduce measures number and validated it with a survey. Considering D&M model, Sabherwal, Jeyaraj and Chowa provided an IS theoretical model based on three constructs; context-related, user-related and IS success related (Sabherwal, Jeyaraj and Chowa, 2006). They applied meta-analysis and statistical analysis to find the correlations between the constructs by analyzing various studies in this area. Peter, DeLone and McLean reviewed the literature and performed meta-analysis of the studies that partly or fully applied or discussed D&M model and elaborated which causal interactions of D&M model were supported by these studies (Peter, DeLone and McLean, 2008). They analyzed 90 studies citing 10 D&M model, which is an evidence that D&M model has been a benchmark in IS success domain in understanding IS success and its complexity with a generic view. Özkan’s IS Effectiveness Model In her PhD thesis (Özkan, 2006) and her research paper (Özkan, Hackney and Bilgen, 2007), Özkan proposed a conceptual model and an assessment framework for IS effectiveness. In the studies, IS is conceptually modeled as being composed of three interacting constructs: people, resources and services & benefits. Requirements for each construct and processes related to each construct were defined using Cobit/ITIL/CMM frameworks. These processes were evaluated according to maturity level, processes and three components of the model. The process-based objectives were evaluated in three case studies and interviews with stakeholders. In this model, processes were treated as equally important; regardless of the context of the system they were applied. Even though, IS was considered as an open system having interaction with its environment, the effect of Internet was not considered explicitly in the assessment. Additionally, although some evaluation metrics were proposed, only qualitative methods were applied to assess the effectiveness. Stakeholders' Viewpoints in Effectiveness Assessment There are several studies in the literature that consider different stakeholders’ views in assessing IS success. Among them, management constitutes an important group since management has point of views related to business goals and strategies for a company. Ifinedo and Nahar examined top and middle level managers and compared their perceptions of IS success applying stakeholder theory (Ifinedo and Nahar, 2006). Critical success factors (CSFs) are considered as success criteria needed to be satisfied for system success (Poon and Wagner, 2001), but they have not been determined to be general in different organizations, business environment or culture. However, CSF’s are considered to be the key factors for a business to reach its goals (Lu, Huang and Heng, 2005). 11 Opponents of D&M model argue that it is not possible to have single measure for IS effectiveness, a huge range of measures incorporating views of different stakeholders and the systems characteristics are needed (Seddon, Staples, Patnayakuni and Bowtell, 1998). In 2002 study, Seddon et al. interviewed 80 senior IT managers to find out how they evaluated their IT investments and they suggested two dimensional IS effectiveness matrix to include different stakeholders views and nature of the system (Seddon et al., 2002). This matrix provides a new insight into the IS success measurement, considering IT application and stakeholder type, which is questionable since for a given stakeholder and IT application type, the proposed measurement may not be applicable to the organizational context. Therefore, the organizational domain where the assessment is applied to should also be considered. Other Effectiveness Assessment Models Kanungo et al. considered six major planning objectives for an organization claiming effective planning results for IS (Kanungo et al., 1999). These objectives are: predicting future trends, improving decision making, avoiding problem areas, increasing user satisfaction, improving systems integration, improving resource allocation. They also interviewed managers and gathered more factors like improving speed of operations and control over system and system standardization, facilitating information retrieval, minimizing errors in functional areas. They used interpretive structural modeling (ISM) to gain insight into the causal relationships between these factors. Applying MICMAC1 analysis they concluded that improving systems integration turns out to be the most important factor for effectiveness. Facilitating information retrieval, increased user satisfaction, improving quality of product/service and minimizing mistakes in functional areas are the other significant effectiveness factors. This model considers different stakeholder’s views like IS manager, CEO and users, thus incorporating both organizational goals and user satisfaction factors, therefore it can be considered as a comprehensive approach. 1 MICMAC: cross-impact matrix, used for analysis of the indirect and hidden relationships among elements of the structure obtained using interpretive structural modeling. 12 However, it seems that it is more concerned with the internal stakeholders’ perceptions. The views of the related and supporting parties like suppliers and customers are not included in the model. This neglects the effect of the interaction between firm’s IS and other parties IS systems on IS success. Performance is considered as any recognized achievement or the act of doing something successfully. Applying to IS area, performance is defined as “how well a system interacts with its environment to gain value and avoid loss”, (Whitworth, Cheikna and Whitworth, 2006). According to Whitworth et al., in nature every system has a boundary, internal structure, effectors (changes the environment), and a receptor (senses the environment) so as IS; each of these aspects has some functions referring to system dimensions, which can be expressed in different contexts with different terms meaning the same things (Whitworth et al., 2006). For example, considering IS boundary aspect, extendibility, security can be goals; for IS internal structure aspect, flexibility and reliability can be considered; for IS effectors aspect, functionality and usability are the factors; and for IS receptor aspect, connectivity and privacy can be the factors. Performance of any system can be considered to be based on these functions and their interactions. Therefore, by improving one dimension will not improve the overall system performance. They consider four levels for IS, mechanical, information, cognitive and social. The model provides a conceptual view for system performance without creating a framework that proposes measurement approach. In this perspective it is not clear whether proposed dimensions can be measured by qualitative or quantitative techniques. Cronholm and Goldkuhl discuss different strategies for IS evaluation (Cronholm and Goldkuhl, 2003). They identify three evaluation conducts; namely goal-based, goalfree and criteria- based evaluation according to the drives of the evaluation. They combine those strategies with the system type and propose six different ways for the evaluation. 13 2.3 Web-based IS Assessment Studies With the emergence of Internet and information technologies, many IS systems turned into WISs, enabling everyone to access it through multiple channels which transferred them into open systems. Naturally, this new dynamic environment has generated the necessity to look into the IS assessment topic through a new understanding. For the web-based IS assessment, the initial trend was to apply traditional IS conceptual models like D&M model to WISs. In that respect, researchers tried to modify D&M model to include the effect of Internet by adopting the same model specifically to e-business and e-government environments. The other studies proposed models for specific “e-“ domains. The following sections discuss WIS evaluation studies in detail. E-Business Success Assessment With the advances of information technologies, the way of conducting business has also changed. Effective use of Internet technologies can provide competitive advantage, market penetration, innovation, technology transfer and even management competency (Torkzadeh and Dhillon, 2002). Internet has a broad aspect which can be used to reach large populations very easily. It basically changes the way the business is done. It can provide new business sectors; make firms to advertise their products and services fast, offer a flexible business environment and presents customization for end-users. These unique characteristics of WISs may require new metrics or at least careful evaluation of the existing ones (Straub, Hoffman, Weber and Steinfield, 2002a). In this regard, there are several studies that aim to evaluate e-commerce system success and effectiveness. To be successful, organizations should have customer oriented businesses, and therefore success assessment should include the customer dimension. Keeney proposed focusing on the customers’ net value of the benefits and cost of both a product and the processes of finding, ordering, and receiving it (Keeney, 1999). Interviewing almost one hundred customers, data is collected about all the 14 pros and cons of using Internet commerce that they experienced or envisioned. Using this data Keeney identifies two sets of objectives, namely means objectives, which helps businesses to provide what is important to customers and fundamental objectives that provide ends that a decision-maker may value in a given context. He argues that these objectives can be used designing an Internet commerce system for a business, creating and redesigning products, and increasing value to customers which may create the question: but to achieve these targets, should or could a company conduct this research for all of its products and services? Keeney enables the customization of objectives in terms of the type of the products or the services offered for Internet commerce (Keeney, 1999). However, considering the variety of products a company has, it would be inefficient or non-practical to run a research on customer values for each product/service a company sells. So, before conducting such a comprehensive research it may be necessary to identify the most value adding products for the company and optimize the number and the type of the products that will be considered. Torkzadeh and Dhillon used Keeney’s approach and proposed a model to measure means objectives and fundamental objectives to evaluate factors for Internet commerce (Torkzadeh and Dhillon, 2002). Then they re-examined and validated these two instruments empirically using more samples statistically (Torkzadeh, Chang and Dhillon, 2003). In the late 90’s, Internet allowed many quick start-up companies to be established and introduced to the market. They did not have definitive business models, and they focused on future earnings potential with methods like web site traffic, and earnings before interests (Carr, 2001). However these companies were attractive since Internet provided easy access to consumers and low barriers to entry to the market as stated in the interview with a business-strategies expert at Harvard Business School, Peter Coughlan in the article of Kharif (Kharif, 2000). These companies attracted many investors causing their stock values to climb up with high demand and limited sources (Carr, 2001). Unfortunately, this rise did not last long, and many of start-up companies started to close down their businesses. While dot-com companies were 15 failing, some traditional companies who had brand recognition, purchasing power, cash flow, customer bases, and intimate knowledge started to market their products on the Internet (Kharif, 2000). According to the interview of Stone (Stone, 2000) with web strategy consultant P. Cohan, pure dot-com companies would be removed from the market and traditional companies would start benefiting from Internet. The dot-com bust shows that, using Internet as a marketing arena for businesses and reaching to customers easily is not enough to be successful; companies still need business strategies, well-established business models to maintain customer loyalty and to survive in the fast changing business environment. Lai and Wong surveyed e-commerce and non-e-commerce companies and compared their strategies for survival after 2001 dot-com bust (Lai and Wong, 2005). They considered four groups of strategies by interviews with executives; savings-related, structure-related, policy-related and marketing-related strategies and analyzed each companies focus on those strategies to survive. They mentioned that savings-related strategies were the most important for B2B and B2C companies whereas marketing related strategies were effective for non-e-commerce companies only. Generally, a company decides on its strategies by identifying and focusing on its critical success factors (CSF). However, in determining company strategies, relying only on past data may not create successful and healthy results, because the environment in which the company runs its business changes continuously. For example, a successful strategy that worked well before the "2001 dot-com boost" may not work well in today’s environment, so should not be considered as a benchmark for currently operating companies. In addition to this, strategies must be driven by analyzing different factors which should include past performances in the industry as well as views of the company’s stakeholders. Reviewing the IS success literature, Pather, Erwin and Remenyi proposed the concept of e-customer who interacts with the business website for some transaction, and created a model to evaluate e-commerce effectiveness based on combination of 16 user satisfaction and service quality theories (Pather, Erwin and Remenyi, 2003). It is observed that this model does not include the integration of internal organizational factors such as production processes, stakeholders and also does not take into account the CSF’s. The fundamental D&M IS success model has also been an inspiration for ecommerce studies. Molla re-specified D&M model for e-commerce systems, by considering customer e-commerce satisfaction as a dependent variable (Molla, 2001). After their famous success model, DeLone and McLean discussed to adopt their model to e-commerce systems using their six dimensions, proposing new measures for each dimension (DeLone and McLean, 2004). They argued that e-commerce measures should include net benefits measures but not surrogate measures like web site hits, which could be measured in individual, group, organizational and industry levels. As web technology is used extensively for e-commerce, research on success factors for the web sites get a lot of attention too. A successful web site should provide trustworthy, dependable and reliable transactions, and should attract customers by advertising, new product and services, and provide good after sales services (Liu and Arnett, 2000). Schonberg, Cofino, Hoch, Podlaseck and Spraragen discuss the best metrics to evaluate effectiveness of websites; two metrics are addressed: clickthrough and look-to-buy metrics (Schonberg, Cofino, Hoch, Podlaseck and Spraragen, 2000). D’ambra and Rice study some factors for web performance measurement (D'ambra and Rice, 2001). These measures focus on the web sites, the front office, ignoring the organizational work domain. This approach assesses the IS as isolated from the systems running in the organization. Back of the web site, the organizational elements that provide the business running like users, processes and their interactions cannot be ignored. These interactions can affect the success of the web site or the front office, for example low user satisfaction in the organization may result in ineffective processes and therefore failure of the business. Focusing only on 17 the front office and disregarding the organizational domain may result in misevaluation of the system as a whole. Consumer satisfaction is an important dimension of e-commerce; consumers must be satisfied with e-commerce systems to get more goods and services on-line. It is an essential measure to determine success and failure of e-commerce (Schaupp, 2005). E-satisfaction has three dimensions; technology, shopping and product. The relationship between what the customer does& perceives and why the customer does it should be explored (Minocha et al., 2006). Not just customer interaction with the website and usability of the user interface design, but the overall experience (CX) and satisfaction a customer has when buying and using the product/service should be considered. According to Minocha et al. CX includes prior experiences, personal values, attitudes to technology and e-commerce, and preferences that shape the customer’s expectations (Minocha et al., 2006). Just considering usability and user interface design is limiting, excluding social, organizational, and individual factors effecting CX. Cheung and Lee proposed a theoretical research framework for customer satisfaction with Internet shopping (Cheung and Lee, 2005). They consider key dimensions of information quality, system quality, and service quality. Customer satisfaction is analyzed in two frameworks; end-user computing and Servqual. Lai proposed a framework for service quality and user satisfaction with e-business considering employee’s perceived service quality revising Servqual instrument for ebusiness context (Lai 2006). In order to have competitive advantage over Internet based businesses, the company strategies should be aligned with e-commerce strategies. Internet based e-commerce strategy should be considered as the integral part of the firm strategy (Auger, 2003). Auger searches for the relationship between Internet based e-commerce strategies; transaction and support strategy and firm’s differentiation strategies; price, image, support, design and quality. By employing questionnaires to several firms, Auger found that strong relation exists between these strategies, and using firm’s Internet 18 based e-commerce strategy to create business strategy may establish a competitive advantage. He also notes that Internet-based e-commerce transaction strategy is utilized more by smaller firms to survive in the market. It can be concluded that, the level of integration between business processes and company’s IS which also includes e-commerce infrastructure, will affect the aligning of business strategy and e-commerce strategy. Considering the assessment studies above, it can be concluded that, none of them provides a comprehensive approach to measure e-business success. Each proposed method focuses on specific success factors like strategy, customer satisfaction, service quality, leaving out the other factors which may be unique for the organization. A comprehensive framework should take into account the different stakeholders’ views, company strategy, and relationships of WIS with other systems. E-Government Success Assessment Internet has been used by governmental organizations, beside private companies, to provide services to citizens. Internet provides increased operational efficiency and better service quality to citizens (Gil-Garcia and Pardo, 2005), however governments have been unable to get the benefits of Internet because of lack of focus on organization but in technology, lack of competition force and doing the things from scratch (The Economist 2008). In order to have comparisons between different applications and benchmarking, to identify good practices and bad examples, egovernment effectiveness should be measured. Since it involves collaboration and communication between different stakeholders and integration of cross-agency business processes, a well-based theory is needed (Peters et al., 2004). However, it is a complex process since it also involves the legal and political context. Peters et al. examines available instruments to measure e-government effectiveness and concludes that they are not adequate to measure multi-service organizations, ignoring different stakeholder views, focusing only on front desk, based on a short-term approach (Peters et al., 2004). Similarly, Hu et al. compare measurement frameworks 19 and conclude that these are one-sided, not objective, concerned only with quantifiable results (Hu et al., 2005). Therefore they suggest using D&M model for e-government success. However, this study does not provide success factors for D&M dimensions. Government administrations focus on information quality and do not consider the reorganization potential of communication and transaction processes (Becker, Niehaves, Algermissen, Delfman and Falk, 2004). Therefore, external perspective that deals with citizen’s business perception of e-government activities, evaluating web-portals; and internal perspectives that consider the internal self-assessment of local public administrations using questionnaires must be examined. Becker et al. suggests several success factors like organizational responsibility for e-government, e-government awareness, budgetary funding, and organizational change (Becker et al., 2004). They mention that to get significant benefits, public services should be offered as e-services and instead of providing traditional public services, it is better to reorganize the underlying processes and organizational structure in the back office. However, such reorganization may not be possible because of country’s political regime. Therefore, the e-government success factors should also include the country-based context. Siegfried, Grabow and Drüke propose that long term success is determined by many factors like organizational measures, strategic procedures, qualifications, communication, and partnerships; obtaining resources and provide 10 factors for successful e-government like guiding principles and strategy, organization, project and change management, applications, benefits and costs, legality, competence, motivation and qualification (Siegfried, Grabow and Drüke, 2003). In their 2006 study, Elpez and Fink claim that the success of an IS project depends on the stakeholders perceptions (Elpez and Fink, 2006). Supporting Seddon’s stakeholders view, they interviewed end-users, project managers, project sponsors, IT managers and CIO’s to find the most important IS success factors. They compared 20 the differences between private and public sector and linked the public sector characteristics to IS success by providing a model based on meeting of user requirements, system usability and performance, information quality, use, user acceptance and IS ownership, interaction with IT infrastructure, expenditure control, accountability, long term perspective. Different from D&M model, this model takes into account the organizational characteristics. It may also be possible to identify the failure reasons of e-government systems and provide success measures accordingly (Evangelidis, Akomode, Bendiab and Taylor, 2002). Evangelidis et al. identify risk categories in e-government into classes: technological and implementation risk factors, social and human risk factors, financial risk factors, legal risk factors. Using these risk factors they provide a checklist for success factors. Likewise, Gil-Garcia and Pardo consider e-government challenges and define success strategies for each challenge and examine four practitioner tools used for e-government guidance according to the strategies (GilGarcia and Pardo, 2005). They identify risks for e-government success which could change according to the environmental dynamics like government change with elections or change in regulations (Gil-Garcia and Pardo, 2005). Managing a government is like managing lots of companies all having different characteristics. Unlike private organizations, in government, integrating different institutions’ IS system is much more difficult because of different characteristics of those institutions such as legislation, culture, nature of transactions in each institution, variation in civil servants profiles etc. and so, success of a specific unit is impossible without the success of all other units. Website Assessment The interface quality of WIS interacting with users has also become a popular subject studied by many researchers. Schonberg et al. proposed to use click-through and look-to-buy metrics to evaluate website effectiveness (Schonberg et al., 2000). Park and Baek modified Servqual instrument for websites and applied Exploratory Factor 21 analysis to find out the most important factors for web-site quality evaluation (Park and Baek, 2007). Aladwani and Palvia proposed a 25-item instrument to measure website quality and performed the empirical evaluation of a website (Aladwani and Palvia, 2002). Seethamraju validated this instrument with structural equation modeling and concluded that it was not comprehensive and did not reflect the website quality fully (Seethamraju, 2004). He concluded that website quality was a multi-dimensional subject and using a single instrument was difficult and in addition depending on the website objective, factors contributing to the quality and user satisfaction were different. Additionally, Loiacono, Watson and Goodhue created a tool WebQual with 12 dimensions and 36 factors to measure website quality by applying Technology Acceptance Model (Loiacono, Watson and Goodhue, 2007) which was applied by D’ambra to assess if a website satisfies the information needs (D'ambra, 2001). Liu and Arnett performed factor analysis to identify which measures were more important to measure success of websites (Liu and Arnett, 2000). These studies are valuable in their own ways, however focusing mostly on the static, front-office properties of web-based systems and neglecting the organizational backoffice issues. 2.4 Goal Question Metric –GQM Goal Question Metric technique is used for software measures. It was introduced to identify problems in a software process or product and define improvement goals for them for software process improvement. It builds a connection between software goals; questions to be answered for each goal and metrics as answers to the questions (Mendonça and Basili, 2000) as illustrated in Figure 1. 22 Figure 1. GQM Technique (Dumke, 2007) Basili, Kan and Shapiro state that “GQM represents a systematic approach for tailoring and integrating goals to models of the software processes, products and quality perspectives of interest based on the specific needs of the project, the customer and the organization.”, ( Basili, Kan and Shapiro, 1994). Since the aim is to improve particular processes or product, measurement goals related to those are defined, which are converted into questions and to metrics which will be used to answer those questions. GQM method provides a framework to improve particular part of the project, by gathering data related to project problems based on the satisfaction of its goals. The technique begins by identification of organizational or project goals based on products, processes, or resources with respect to different viewpoints. Goals are identified in terms of object of the study, the purpose, the quality focus, the point of view and the context (Lindvall, Donzelli, Asgari and Basili, 2005). Then, for each goal, questions are formed on how a specific goal shall be assessed or achieved, with respect to a quality issue from a particular viewpoint. As a last step, objective and subjective metrics are identified to answer each question quantitatively. These 23 metrics are used to measure if the goals are achieved or not, resulting in the improvement of the particular process or the product. 2.5 Summary and Discussion Considering the studies mentioned above, it can be concluded that ISs are not easy to assess and there are many aspects to be considered in the assessment process. With the emergence of Internet technologies, it is now more difficult to measure IS effectiveness. Internet provides a borderless, non-stop, flexible communication medium. Companies engage in web-based business, they can now reach many people, market their products easily, reduce transaction costs, increase productivity, reduce procurement costs, lower the number of employees, and can gain competitive advantage with respect to traditionally run businesses. Similarly, with Internet, users can access a broad range of information quickly, compare product prices, shop by fast transactions, exchange views about products and services. So, Internet provides a very advantageous environment to do business that is very appealing for many companies. However, by reducing the barriers to entry to the market, competition is tougher in this new world. Therefore, assessing web-based information systems is more vital for organizations for survival and competitive advantage. The various IS success assessment proposals in the literature usually are not comprehensive focusing on some dimensions of IS construct. They are mostly theoretical studies, providing research infrastructures for the construct, concerned on what measures should be used for the assessment and so, far from providing a success measure with which the ISs can be compared as summarized in Table 1. 24 Table 1. Context Literature Summary Study Dimensions IS success System quality, Information Quality, Use, User Satisfaction, DeLone&McLean(1992) Individual impact, Organizational impact Evaluation - Myer et al. (1997) Added Service quality and Workgroup Effect to D&M model - Pitt et al.(1995) Added Service quality to D&M model, suggested using Servqual tool - Seddon et al. (1999) Created a matrix based on stakeholder’s view and system to be evaluated - Iivari (2005) Perceived system quality &perceived information quality instead of system Use in D&M Model - Gable et al. (2003) Omitted Use in D&M model, added customization, increased capacity, e-government, business process change - Özkan (2006) People, resources, Services and benefits 25 10 processes on 3 constructs are evaluated by stakeholders to find maturity level Table 1. (cont.) Study Performance e-business Dimensions Evaluation Functional score card with 18 factors in 3 dimensions are evaluated by IS managers Interviews with IS manager, CEO to get more factors. Applied MICMAC analysis to decide which factor is more important for measurement Chang &King. (2005) System performance, information effectiveness, service performance Kanungo et al. (1999) Six objectives: predicting future trends, improving decision making, avoiding problem areas, increasing user satisfaction, improving systems integration, improving resource allocation Whitworth et al. (2006) 8 Factors in 4 dimensions : extendibility, security, flexibility, reliability, functionality, usability - Lai(2006) Revise servqual by considering perceived service quality, user satisfaction - IS Success Context Keeney(1999) Customer’s net value of the benefits; cost of product and processes Interviewed 100 customers to identify means and fundamental objectives of ecommerce systems Torkzadeh et al. (2003) Means and fundamental objectives like Keeney Provided measurement model 26 Table 1. (cont.) e-business Context Study Dimensions Loiacono et al. (2007) Website quality Park and Baek (2007) Website quality Evaluation WebQual instrument is developed and applied to ecommerce sites Applied WebQual to online bookstores DeLone&McLean(2004) System quality, Information Quality, Use, User Satisfaction, Individual impact, Organizational impact - Liu et al. (2000) Web-site effectiveness - Schonberget al. (2000) Click-through, look-to-buy ratio - D’ambra and Rice (2001) Web usage, satisfaction, individual performance, impact of information technology Schaupp (2005) Consumer satisfaction - Minocha et al. (2006) E-satisfaction. Overall customer experience - Cheung and Lee (2002) Satisfaction: information quality, system quality, service quality 27 Applied questionnaire in 3 surveys 2frameworks: end-user, servqual Table 1. (cont.) Context Study Dimensions Website quality 25-item instrument is developed Seethamraju (2004) Website quality Applied 25item instrument to e-commerce site Gil-Garcia and Pardo (2005) Examining challenges to reach success strategies - Peters et al. (2004) D&M Model - Becker et al. (2004) Responsibility of egovernment, awareness, budgetary funding, organizational change - e-business Aladwani and Palvia (2002) e-government Evaluation Siegfried et al. (2003) Elpez et al. (2006) 28 Organizational measures, strategic procedures, qualifications, communication, partnership, obtaining resources Stakeholders perceptions: meeting of user requirements, system usability, performance, information quality, use, user acceptance, IS ownership, interaction with IT infrastructure, expenditure control, accountability, long term perspective - - Table 1. (cont.) e-government Context Study Evangelidis et al. (2002) Hu et al. (2005) Dimensions Identified risk factors: technological and implementation risk factors, social and human risk factors, financial risk factors, legal risk factors. D&M Model Evaluation - - Considering the complexity of information systems, measuring their effectiveness in one dimension and defining a single metric for assessment is not realistic. Instead, IS effectiveness should be considered as a multi-dimensional entity. In this regard, Internet should be considered as an entity to be assessed and integrated to the system assessment. Consequently, application of traditional IS assessment methods to WIS domain is not suitable or does not capture all of the details of WIS success. DeLone and McLean’s comprehensive framework considers six constructs which provides a generic research infrastructure for IS assessment (DeLone and McLean, 1992). In order to account for Internet and e-commerce, they proposed (DeLone and McLean, 2004), some new measures for each construct additional to their original model. However, DeLone and McLean do not suggest how and which measures should be chosen according to the type of the organization. The model specifies some measures from the literature which were previously defined by other studies and lists them under the six generic success categories. However, this study do not provide any information how these measures are to be applied to real world cases, and how they are used to decide whether a system is successful or not. Furthermore, this proposal focuses on only e-commerce applications among various types of webbased applications. For a generic WIS assessment, the context in which the organization acts should be taken into account, for example WIS for e-commerce or 29 e-government cases require different measures since organizational goals of each differs. Özkan has proposed a conceptual model for IS effectiveness, and a framework for IS assessment (Özkan, 2006). In her study, IS is considered to be composed of three interacting constructs: people, resources and services and benefits in the conceptual model. The effect of Internet is said to be considered in the assessment, and it is included only in resources dimension as a single process. Therefore, this study does not explicitly and extensively account for the existence of Internet in terms of different organizational dimensions. There are many other studies focusing on specific WIS assessment, for example egovernment and e-business, which are far from providing a broad, comprehensive generic web-based IS evaluation framework. Some of them propose various measures with traditional assessment approaches. With the emergence of Internet, the way the business is operated has changed, so Internet affects all processes performed in an organization directly or indirectly. Therefore, assessment of WIS should not be considered as an add-on dimension to the traditional IS assessment frameworks, but its effect should be considered effecting organization as a whole. An organization is a group of people organized for a particular purpose. It is constructed to realize its goals. Therefore, IS should support organization to fulfill its goals, it should serve to organizational objectives. In this regard, assessment of IS effectiveness should consider how well it boosts organization to reach to its goals as a whole (Malik, 2001). The generic WIS evaluation should not ignore the effect of Internet on organization and its objectives, for example agility of a company in this new environment, impacts of other WIS systems, or strategic competitiveness considerations of the organization. 30 As stated above, organizational objectives are important constructs to consider when evaluating a WIS. Because, what you measure for WIS success should add value to organizational goals. Existence of Internet enlarges the IS assessment context and content. WIS changes the organization’s internal business processes, provides more time savings and efficiency. With Internet, users can switch between different web sites easily; can use some sites more, if it is user-friendlier or more entertaining. Therefore, users constitute an important dimension in evaluating whether a WIS is successful or not. Customers can compare products and prices; can influence each other with comments. Therefore, differences between WISs can, in turn, influence each other and so, the environmental effect on assessment of WIS should also be considered. Based on these effects, in the following chapter, a novel model and a framework for WIS effectiveness assessment will be explored. 31 CHAPTER 3 SEWISS: WIS SUCCESS ASSESSMENT FRAMEWORK In this chapter, a generalized framework that allows the evaluation of WIS success is proposed. The framework called SEWISS, is based on two dimensions specific to organization and to WIS. IS success is a multi-dimensional construct (DeLone and McLean, 1992), hence, to decide whether a WIS is effective or not or to what degree, several dimensions of WIS success should be reflected on the assessment process. In other words, WIS success is a function of several variables. Functions are rules that map different sets to each other. They are used frequently in mathematics, as well as in other sciences and engineering. They represent dependence between different entities and they associate an output entity with the input entities. For example, y=f(x1,x2,x3,…, xn) represents output entity y depending on input entities x1,x2,..,xn through function f. Put another way, y is the resultant entity of entities x1, x2, x3,.., xn ruled by f. Similarly, the WIS success can be represented as: WIS success = S(d1, d2, d3, ..,dn) 32 Equation (1) where d1, d2,.., dn are the dimensions constituting the WIS success and S is the rule that associates the dimensions to WIS success. In equation 1, dimensions d1, d2…, dn are assumed to be related to WIS success through function S. Hence, the dependent variable of this study is the WIS success that is being influenced by independent variables d1, d2,.., dn. It is obvious that numerous dimensions d1, d2,.., dn, constituting the WIS success can be listed (Chang and King, 2005; DeLone and McLean, 1992; Hamilton and Chervany, 1981; Iivari, 2005; Lu et al., 2005; Malik, 2001; Pitt et al., 1995; Poon and Wagner, 2001; Sabherwal et al., 2006; Seddon et al., 1999; Smith and McKeen, 1996). For example, effects like devaluation emerging in the economy or arising of a war in the country can affect the WIS through affecting the organization uncontrollably. Since the goal of this study is to assess WIS success of an organization, the framework focuses on the controllable dimensions that can be improved by organizational efforts. In this regard, at the basic level, a comprehensive assessment framework for WIS success will be the result of the interaction of WIS and Organization in the business environment. In this study, organizational aspect influencing WIS success will be represented by organizational strategy, while for WIS-related aspects, the relationships of WIS with its environmental entities will be considered as illustrated in Figure 2. Consequently, Success Function, S, constituting the main subject of this study has the structure given in Equation (2). WISsuccess = S(OrganizationStrategy, WISRelationships) 33 Equation (2) Figure 2. WIS Success Aspects 3.1. Identification of Dimensions in SEWISS Framework The following sections describe the realization of SEWISS Framework aspects or dimensions in relation to WIS Relationships and Organizational Strategy. 3.1.1 WIS Relationships Effectiveness is considered as the quality of being able to bring about an effect. It is related with the system’s influence on its environment, and the results caused by a system, thus effectiveness has an external focus (Myers et al., 1997). In the parallel way, success is considered as the achievement of an intended or expected effect. In this study, the terms effectiveness and success are used interchangeably meaning “level of achievement of system’s interactions with its environment as expected from it”. IT is a coordination-oriented technology that provides interactions between different parties (Barut, Faisst and Kanet, 2002). Similarly, WIS systems usually promotes the work being done and interact with database and transaction processing ISs that are not web-based (Isakowitz, Bieber and Vitali, 1998). Beise proposes a model for IS effectiveness in relation to IS and organization interaction (Beise, 34 1994). It is concluded that IS effectiveness is associated with the relation between IS function and organizational departments. Therefore, when considering WIS effectiveness, its effect and relations on the environmental entities should be taken into account. Figure 3. WIS Relationships A web-based IS interacts with users, with other IS’s (web-based or non web-based) in the domain and with the entities constituting the organization, as depicted in Figure 3. These interactions form one of the SEWISS framework dimensions: WIS relationships. This dimension consists of three categories of relationships, each possessing various characteristics that must be considered in the assessment. • User-WIS relationship (Rel1): consists of the interaction between system users, both internal and external, and the system itself. It takes into account the perspectives of different stakeholders using the WIS system (related characteristics: user friendliness, ease of use, understandability etc). 35 • Other Systems-WIS relationship (Rel2): consists of the interaction of organization’s WIS with other systems in the external environment. It covers the effect of other systems in the environment such as compatibility between different systems (related characteristics: security, compatibility, timeliness, etc). • Organization-WIS relationship (Rel3): consists of the interaction of WIS with organizational units. Culture, structure, standards, processes, possibly other non-web-based or web-based ISs in the organization, financial indicators, communication factors by which the organization is influenced are some of the factors considered under this relationship (related characteristics: privacy, scalability, standards, etc). It can be concluded that WIS relationships dimension takes into account the stakeholders’ views, environmental factors and also the organizational characteristics under the above-listed relationship categories. The relationship categories may differ in different organizational structures, however, the three categories stated above are proposed as a comprehensive set considering the current systems. 3.1.2 Strategy While WIS relationships dimension identifies the WIS’s interaction with its environment, it is vital to understand to what extent WIS creates value to the organization in line with its goals, in order to have a comprehensive WIS assessment framework. In this regard, organizational strategy is considered as the second dimension of the SEWISS framework. A strategy is a plan of actions that directs an organization in its environment, affects its processes and characteristics, and thus its performance (Hambrick, 1980). Any change in organizational strategy means changes in information system to provide new products or services (Sobczak and Berry, 2006). Thus, information system characteristics should be the result of the organizational strategy (Sobczak and Berry, 36 2006). Auger argues that using Internet based e-commerce strategy establishes a competitive advantage (Auger, 2003). Similarly, Siegfried et al. suggest strategy as one of the ten factors for successful e-government (Siegfried et al., 2003). Organizations seek to distribute better value products and services in order not only to gain competitive advantage in the market place but also to continue to stay in the market that they operate in. The investment in IS fundamentally emerges from these impulses and like in any other investment, the outcomes of it needs to be justified in terms of its strategic, operational and tactical contributions (Irani and Love, 2008). Gunasekaran, Ngai and McGaughey mention that IT/IS involvement has an effect on organizational performance, therefore, there is a relation between organizational strategy and IT/IS selections and actions (Gunasekaran, Ngai and McGaughey, 2006). In that respect, investment requirements for IT are shaped by business goals and so the evaluation process should measure accomplishment of reaching these goals (Serafeimidis and Smithson, 2003). Powell and Dent-Micallef have examined the literature related to strategy and IT relationship and have noted that some researchers have shown the existence of strong two-way interaction between IT and organizational strategy (Powell and DentMicallef, 1997). Accordingly, a WIS strong in its relationships with the environment cannot be assessed as successful, if it does not serve the strategies of the organization. Web-based organizations owe their existence to information systems. Even using the same technological tools, providing same services and products, organizations could employ them in variety of ways to reach their goals. Web-based organizations could have different goals, they can be classified as non-profit web-based organizations and profit-oriented web-based organizations. Non-profit organizations (e- government, e-health and charity organizations) provide services and products in order to enhance the transactions; on the other hand, profit-oriented organizations (e37 business organizations) provide services and products in order to make profit, earn market share, etc. Additionally, other than being profit or non-profit organization, each organization has its own specific strategies identified by top management or shaped by the market conditions. These strategies are specified in organization’s critical success factors (CSF). CSFs are the important areas of action that must be accomplished effectively to achieve the mission, objectives, quality and high performance (Alazmi and Zairi, 2003; Rockart, 1980). CSFs are those factors that provide a reference point to direct and measure if the company or business unit is successful or not (Munro and Wheeler, 1980; Poon and Wagner, 2001). They are considered to be the key factors for a business to reach its goals (Lu et al., 2005). CSF’s are one of the important factors to be considered for IS success (Poon and Wagner, 2001). Through company life cycle, CSF’s may need to be changed or modified according to the environmental dynamics. For example, in early stages, reaching the maximum number of consumers could be the major critical success factor, while at a more mature stage; customer loyalty can emerge as a dominant CSF which could affect the IS success. IT/IS constitutes the most important investment for organizations in today’s sophisticated, “e-directed” business world and like other investments, expected to contribute to the organizational objectives, strategies thus CSFs. Organizations seek to distribute better value products and services in order not only to gain competitive advantage in the market place but also to continue to stay in the market that they operate in. The investment in IS fundamentally emerges from these impulses and like in any other investment, the outcomes of it needs to be justified in terms of its contribution to the organization’s goals and targets. In this study, WIS success is perceived as “how much the achievement of WIS relationship goals contribute the organizational CSFs”, as illustrated in Figure 4. Hence, WIS success is perceived as the satisfaction of WIS relationship goals as its contribution to reach organizational CSFs. 38 Figure 4. WIS Success Assessment Framework Dimensions The WIS assessment framework based on organizational CSFs and WIS relationships as illustrated above will be called Strategy-based Evaluation of WIS Success-SEWISS framework. 3.2 Success Assessment in SEWISS Framework In IS evaluation studies, it is mentioned that in the system assessment, stakeholders constitute an important aspect (Ammenwerth, Graber, Herrmann, Bürkle and König, 2003; Irani and Love, 2008; Seddon et al., 1999), and IS success depends on stakeholders perceptions (Elpez and Fink, 2006). Web can be considered as a source providing information for users’ needs (D'ambra and Wilson, 2004) to support the accomplishment of operations easily. Serafeimidis and Smithson reviewed the literature on subjective IS evaluation and concluded that because of the “organizational and subjective” character of IS evaluation, human factor became critical and so, the interpretive knowledge would improve IS assessment (Serafeimidis and Smithson, 2003). Nevertheless, in IS 39 evaluation studies, since user perspective is considered valueless, user aspect is usually neglected and interpretive IS evaluation is usually not employed (Irani, Love, Elliman, Jones and Themistocleous, 2005). However, with the web-based systems employed by e-business, e-government, e-learning organizations, users and their perceptions become the major necessity for organization’s survival. Therefore, in WISs, interpretive IS evaluation approach may be more suitable where the users are the critical elements for the continuation of the organization. WISs are systems actualized by stakeholders, and external systems, so their success largely depends on their interaction with the environment, services and products are presented to thousands of users, and this interaction is alive, as long as the user is pleased with the interaction value. Different stakeholders may differentiate about their perceptions level for a particular WIS. An experienced stakeholder may have higher expectations from a WIS and may not find particular characteristics successful and may switch to the alternative system that satisfies his/her expectations better. Internet supports direct marketing (Jarvenpaa and Tiller, 1999), which has given power to the customers: they can select variety of products, move store to store virtually with a single mouse click. They would attach to the Internet sellers who provide the goods and services that satisfy their needs and expectations better (Cheung and Lee, 2005) and success of electronic market is achieved by customer’s motivation to use it (Chen, Gillenson and Sherrell, 2008). Therefore, long-term problem for Internet based service providers are to keep customer trust up, which puts the human factor as the major concern (Jarvenpaa and Tiller, 1999). When above mentioned views are considered, it can be concluded that human factor and thus the perceptions are the crucial elements of a successful WIS assessment framework. Consequently, in SEWISS framework, stakeholder perception is one of the main sources of the assessment data. 40 3.2.1 SEWISS Framework Weights and Measures In WIS success assessment framework, the aim is to find a WIS Success of an organization based on WIS relationships and CSFs by taking into account each relationship’s importance with respect to CSFs. Therefore, the following factors are included as the framework elements which are used in success calculation. 3.2.1.1 CSF definition and importance rankings CSFs are the key areas to focus on in order to achieve strategic objectives. Management defines the strategy and thus the CSFs, so organizational strategy, in fact, is a subjective concept and differs from organization to organization Since an organization’s strategic objectives have different priorities, CSFs related to these objectives also have precedence among each other. In this framework, the information related to CSFs and their importance rankings are gathered from the management denoted as ri for each CSFi. 3.2.1.2 WIS relationship weights Each WIS-relationship category considered under three main headings namely UserWIS; other systems-WIS; Organization-WIS (referred as Rel1, Rel2, Rel3 respectively), are not necessarily equally important, since some categories may be more significant for the organization to reach to its goals than others. For example, rather than WIS-Organization Relationship, User-WIS Relationship category has higher significance for achieving a CSF of attracting new customers. To state more clearly, if a category in WIS relationships dimension does not provide any value for a specific CSF, then its contribution to the system success will be null. So as to address the discussed significance variances, each relationship category will be weighted by the management to reflect how important it is with respect to a specific CSF based on 1 to 10 scale (10 as very important and 1 as least important) symbolized by wij as importance of Relj for CSFi. The weighting out of 10 is used to increase the scale sensitivity. The ranking “0” means that specific WIS relationship does not contribute 41 to the specific CSF and the value “10” means it contributes to achieving the CSF fully. 3.2.1.3 Measures for WIS Relationships Numerous measures can be listed under the WIS relationship categories by examining previous studies for IS effectiveness assessment literature (Chang and King, 2005; DeLone and McLean, 1992; Hamilton and Chervany, 1981; Iivari, 2005; Lu et al., 2005; Malik, 2001; Pitt et al., 1995; Poon and Wagner, 2001; Sabherwal et al., 2006; Seddon et al., 2002; Seddon et al., 1999; Smith and McKeen, 1996; Straub et al., 2002a; Straub, Hoffman, Weber and Steinfield, 2002b; Torkzadeh et al., 2003). Thus, to determine which of these measures are more meaningful or vital for measuring the success of WIS relationships of a particular organization, method based on GQM is applied as explained in the following sections. 3.3 SEWISS Success Calculation Steps SEWISS success calculation steps are presented in Figure 5 at the top most level, and the assessment process is explained in detail in the following sections. The step numbers used below refer to the steps in the associated figures. Figure 5. Assessment steps at the top level 42 Step 1. Measure WIS Relationships’ Success The steps of the WIS Relationships’ success measurement is depicted in Figure 6. The aim of this step is to measure the success of each WIS relationship (Rj, where j=1,2,3), which eventually contributes to the WIS success (S). The success of each WIS relationship is based on how much each relationship satisfies its intended goals. The assumption is that, if an individual WIS relationship provides the functionalities and tasks as expected from it; then it will contribute positively to WIS success. Figure 6. Steps of Measuring WIS relationships’ Success There are numerous measures proposed in IS assessment literature (Chang and King, 2005; Malik, 2001; Sabherwal et al., 2006). In order to determine which of these measures are more relevant for measuring the WIS relationships success of a specific organization, a method based on Goal-Question-Metric technique (GQM) (Basili et al., 1994) is applied. In SEWISS, a method based on GQM is applied in interviews with IT/IS Specialists who have the comprehensive knowledge of what the system should provide to its stakeholders as products and services in order to specify what to measure for WIS relationship success. Step 1.1 Apply GQM In order to assess the level of success of each WIS relationship, IT/IS Specialist is asked to specify the goals of each WIS relationship. For example, one of the goals of User-WIS relationship may be “to provide 7/24 service”. In specifying goals in GQM method, the parameters of GQM are depicted as: Object of the study: WIS effectiveness assessment 43 Purpose: to assess WIS effectiveness Quality focus: WIS relationships Perspective: to examine the success of WIS relationships from the point of view of stakeholders Context: all interacting parties around WIS, including other web-based and non-web-based systems, users, organizational units After identifying the goals of each WIS relationship, the next step is to gather questions that can be asked to check if the goals are satisfied or not (Figure 7). As an example, for the above mentioned goal, one of the questions would be “how many times do the system collapse during the day?” In order to answer each question, the measures which should be investigated are specified by IT/IS Specialist. Figure 7. Steps of Applying GQM For measures that can be suggested to interviewees who has difficulty specifying measures in this step, sample measures (Eralp, 2004; Jun and Cai, 2001; Liao and Cheung, 2002; Straub et al., 2002a, 2002b; Torkzadeh et al., 2003) can be used as provided in Appendix A and D . Step 1.2 Prepare and apply questionnaire The resultant measures identified by IT/IS Specialists are used to compile a webbased questionnaire which will be answered by the system stakeholders. The questionnaire is composed of statements each of which corresponds to the chosen measure. Statements in the questionnaire are designed in such a way that higher values show success/satisfaction regarding that specific statement. Each statement is rated by users to check the level of existence of the measure in the WIS based on 5point Likert scale (1: strongly disagree, 5: strongly agree) (Figure 8). 44 Figure 8. Steps of Preparing and applying the questionnaire A WIS relationship satisfying the specific measure fully, as rated by stakeholders, has the value 5, and value 1, if it does not contribute to that measure at all. For the respondents having no idea about the measure, “I do not have an idea” option was provided, which is valued as 0 and excluded in the calculations. These evaluation values (Mjk, where j is the WIS relationship, k is the measure number) are used to find a single value (Rj) for each WIS relationship success by averaging the results for that specific relationship (over the number of measures mj). The answers to the questions are converted to the satisfaction level of the user for a specific question through 5-point Likert scale by using the corresponding value of the Likert category. Thus taking the average of the questionnaire answers will result in a general satisfaction level for WIS relationships. These calculation details are illustrated in Appendix C. Step 2. Obtain CSF related data CSF related data is composed of CSF importance rankings and WIS relationship weights as shown in Figure 9. Step 2.1 Gather CSF Importance Rankings Since each organization has different priorities in terms of its strategic objectives, CSFs related to these objectives may differ in their priority to the organization. Organizational CSFs are identified and ranked by organization’s management according to their importance to the organization. For example, if the management of the organization specifies four CFSs, the most important CSF (CSFi) will have an importance ranking (ri) of 4 according to their ranking. The CSF importance rankings will be input to overall WIS success calculation. 45 Figure 9. Steps of Gathering CSF related data Step 2.2 Create WIS Relationship Weight Matrix Each WIS-relationship category considered under three main headings namely UserWIS; other systems-WIS; Organization-WIS (referred as Rel1, Rel2, Rel3 respectively), are not necessarily equally important. If a category in WIS relationships dimension does not provide any value to achieve the CSFs, then its contribution to the system success will be null. So as to address the discussed significance variances, each relationship category must be weighted by the management to reflect how important it is with respect to a specific CSF based on a 1 to 10 scale (10 as very important and 1 as not important). This is accomplished by creating the WIS relationships weight matrix, presented by wij weights, in the interviews with management. By this way, a relationship contributing to a more important CSF will have more effect on the WIS success. Additionally, success of the WIS relationship (Rj) can contribute to organizational strategies by its importance weight value (wij) which will be reflected to overall success calculation as described in the following section. 46 Step 3. Calculate Overall WIS Success After gathering questionnaire responses, identifying CSF importance rankings and constructing the WIS relationship weight matrix, the final step in the framework is to calculate overall WIS success (Figure 5). The following terminology outlines the discussion above: n : number of CSFs (obtained from management) CSFi : critical success factor i (i=1,…,n) (obtained from management) ri : importance ranking for CSFi (i=1,…,n) (obtained from management) Reli : WIS relationship i (i=1,2,3) (obtained from IT/IS Specialist) Rj : success for Relj (j=1,2,3) (average of the stakeholder responses to questionnaire) wij : importance weight of Relj for CSFi (i=1,..,n; j=1,2,3) (obtained from management) mj : the number of measures for Relj (obtained from IT/IS Specialist) Mjk : value of measure k for Relj (k=1,..,mj ; j=1,2,3) (stakeholder answers to each questionnaire questions) s : number of stakeholders participated in the assessment SEWISS success value will be calculated as S= n 1 3 ∑ ri ∑ R j wij n 3 wmax Rmax ∑ ri i =1 i =1 47 j =1 Equation (3) with 1 s 1 Rj = ∑ s i =1 m j mj ∑M k =1 jk Equation (4) As explained in step 1.2, success of each WIS relationship category (Rj) is the average of the questionnaire answer values over the measure number for that relationship category which are based on 5-point Likert scale, calculated for each respondent. The Rj values obtained for all respondents are averaged over the number of respondents that yield to the resultant Rj’s. Alternatively, simple summation of answer values could have been used to reach a single relationship success value, but to address the possibility of differences in the number of measures under each relationship category, and the diverse questionnaire responses, average function is preferred. There are several methods to find a result based on rankings and weights (Sobczak and Berry, 2006). In the proposed framework, contribution of WIS relationships to success is calculated using weighted sum model as used in several studies (Sobczak and Berry, 2006; Zahedi and Ashrafi, 1991), in which the factors of the sum are multiplied by their weights. Similarly, success of each WIS relationship category calculated as Rj in Equation (4) above is reflected to overall WIS success based on its contribution to CSFi by wij. In this calculation, the influence of the differences in importance values of CSFs is also reflected by ri. Finally, the value found is normalized by number of CSF’s and maximum values of weights as in Equation (3) that yields a SEWISS success value out of 100. In order to analyze the results, the following categories are proposed as effectiveness levels as the outcomes of the SEWISS calculation: 0-20: Very poor: WIS does not satisfy its intended purpose and needs major improvements. 21-40: Poor: WIS is poor in satisfying its intended purpose and it needs improvements. 48 41-60: Satisfactory: WIS satisfies its intended purpose on average and it can be improved more. 61-80: Effective: WIS satisfies its intended purpose. 81-100: Very Effective: WIS satisfies its intended purpose fully. The resultant SEWISS values will give information about the ranking of the effectiveness of the WISs. Therefore, it will allow ordering the systems in regard which one is more effective than the other. The proposed WIS effectiveness assessment dimensions and whether the framework results in a meaningful value for the WIS effectiveness when applied to organizations need to be verified through multiple case studies. In this regard, cases from two different sectors namely banking sector and e-commerce sector shall be employed to verify the framework elements namely, framework dimensions, CSF importance rankings, and WIS relationship weights and the assessment process. In the next chapter, the research methodology will be explored and in Chapter 5, the results of the case studies carried out for this purpose will be discussed. 49 CHAPTER 4 RESEARCH METHODOLOGY With the purpose of assessing WIS effectiveness, an assessment framework have been constructed as described in Chapter 3. To validate the proposed framework and its elements, the following research questions are explored in this study: 1. Does the proposed framework reflect WIS success? a. Do the proposed SEWISS dimensions exist in e-commerce, e-banking and e-learning systems? b. Do WIS relationships exist in e-commerce, e-banking and e-learning domains? c. Do CSF’s differ in e-commerce, e-banking and e-learning organizational contexts? d. Do CSF’s differ in their importance to the organization? 2. Does the effectiveness value found within the proposed framework measure WIS’s success in e-commerce and e-banking organizations? 3. Does the framework distinguish between fully and partly web-based organizations in success assessment? 4. Is the proposed framework applicable to the organizational domains subject to the case studies, namely e-banking and e-business organizations? 50 Out of the scope of the present study is another question that merits future study: Is the proposed framework applicable for a wide variety of organizational domains that use WIS, such as e-government, or e-health? Several studies in the literature strive to find how an IS adds value to the organization. Business value is the benefit in dollars for the organization as a whole resulting from IT solutions, which can be achieved through direct contribution for market position or revenue; through solutions to customer needs and challenges; through financial benefits and through investments to provide industry wide improvement (Sward, 2006). In this regard, correlation between IS and organizational performance is explored in different studies by considering different measures like organizational effectiveness, efficiency, profitability, productivity, return on investment (ROI), return on assets (ROA). These studies have resulted in some conflicting remarks, some finding strong relations whereas others indicating no or weak relations between IS and organizational benefits. Brynjolfsson states that with the increase of IT usage, there is an unexpected drop in productivity (Brynjolfsson, 1993). Dedrick, Gurbaxani and Kenneth classify the previous studies on IT returns and conclude that nearly all the studies performed after the mid-1990s resulted in positive returns of IT investments (Dedrick, Gurbaxani and Kenneth, 2003). Therefore, IT can be considered as an enabler that can result in productivity gains. We believe that this difference could be the result of technological progresses in this field and with the advancements of the technology, user satisfaction and system effectiveness has increased abruptly. The main aim of an IS is generally accepted to be able to contribute to the organization to perform better. Zhang in the PhD thesis, states that firms with better IT capability, can have better competitive advantage in the international markets and hence better performance (Zhang, 2005). Firms using IT systems effectively can differentiate themselves better with respect to their less successful competitors (Alexander and Randolph, 1985). It can reduce the production cost, can help firm to 51 differentiate within the competitors, therefore providing a competitive advantage (Gurbaxani and Whang, 1991; Ives and Learmonth, 1984). Vandenbosch and Huff discuss the relationship of managers’ use of Executive Information Systems and organizational performance in terms of efficiency and effectiveness (Vandenbosch and Huff, 1997). Similarly, a study carried out in Chrysler Corporation (Mukhopadhyay, Kekre and Kalathur, 1995) revealed that using electronic data interchange with the suppliers has provided financial benefits to the company. Mata, Fuerst and Barney discuss that IT adds value to a company in wide variety of cases, but creating sustained competitive advantage requires proper strategy (Mata, Fuerst and Barney, 1995). Consequently, it can be said that more successful the information system in an organization, the more advantageous the organization with respect to its competitors in its business environment. With this token, if the organization performs better than its competitors, its WIS is expected to be a successful one contributing to organization’s competitiveness. In searching for the correlation between IT and firm performance, the measures used for firm performance are mostly economical measures like, labor productivity, labor hours, IT capital stock, non-IT capital stock, GDP, profitability. In this study, to check the validity of the SEWISS success results, the concurrent validity approach has been employed. Concurrent validity searches for correlation between the instrument developed and other factors which could be related to the subject (Muijs, 2004). It allows comparisons of the instrument measurement items and known or accepted standard measures or criteria (Garson, where direct objective measures are unavailable. In this respect, SEWISS success values obtained through the proposed framework are compared by web-site statistics from Alexa company and some organizational factors like organization age, employee number, and revenue. Alexa is a web information company that keeps a large database of web sites 52 information, their statistics and traffic data. Reach measures the number of users of a specific site. It is the percentage of all Internet users to the users of a specific site. Similarly, Page views measure the number of pages viewed by site visitors (Alexa, 2008). The SEWISS success values obtained through the proposed framework were analyzed by checking the parallelism of it with web-site statistics from Alexa company and organizational factors like organization age, employee number, and revenue. In order to answer the research questions of this study, non-experimental research methods were used. In that token, multiple case studies were performed. In the following sections, justification of case study research in our context, the data collection method and case study plan are explained. 4.1. Case Study Research in Information Systems Field According to Benbasat, Goldstein and Mead, the IS field has a shift from technological to managerial and organizational questions, and consequently it interests in how context and innovations interact (Benbasat, Goldstein and Mead, 1987). They mention that case study is suitable when research and theory at their premature stages and the players and the problem domain are important where the investigator needs to understand the practice based problems. Similarly, in this study, the theory is at its early stage and the web-based system environment and stakeholders are basic entities to be considered. According to Yin, a case study is, “a research strategy comprises on allencompassing methods with the logic of design incorporating specific approaches to data collection and to data analyses” (Yin, 1993). Case studies could be designed as single or multiple cases. If the study is comparable to a single experiment then single-case study is applied otherwise, multiple-case study is employed (Yin, 1993). 53 When research aim is description, theory building, or theory testing, multiple case designs are desirable (Benbasat et al., 1987). In addition, multiple-case studies allow cross check between cases and allowing the extension of theory. In this study, the aim is to build up a framework for WIS effectiveness assessment; choosing a single case study would not be enough to reflect the characteristics of WIS of organizations in different environments. Therefore, the proposed method is applied to four different organizations to have a complete consideration. It would be better to have more case studies in various organization types, however, because of time limitation, two cases are employed in each of e-banking and e-business domain for the case study research. 4.2 Data Collection Method In this study, qualitative as well as quantitative data collection methods are applied. Quantitative research methods were initially used in natural sciences which include experiments, formal and numerical methods. Qualitative research methods were used in social science which include interviews, questionnaires, document analysis, texts, observations (Myers, 1997). In computer science, most studies use quantitative methods that are based on technical, economic, effectiveness and performance measures. However, these methods ignore the effect of cultural environment, and the social interactions in that environment. Consequently, some of the studies combine both qualitative and quantitative methods (Kaplan and Duchon, 1988). In this study, proposed framework considers the relationships of WIS with other systems and environmental entities, which includes both qualitative and quantitative aspects. Therefore, both research methods are employed in data collection process. Accordingly, in order to explore if the proposed framework elements are valid or not, interviews were held with IS Specialist and the management. Then, for effectiveness assessment, a questionnaire was applied to the stakeholders of WIS relations to rate the effectiveness of their WISs. The interviews were recorded and afterwards transcribed for data analysis which were provided to the interviewees for their approval. 54 After the proposed framework elements were verified through the interviews, the results were evaluated and the missing elements were added. In order to mature the WIS assessment process and three cases were used as preliminary test cases. The preliminary cases were used as a verification of framework elements (Figure 10). After verifying the framework elements in the preliminary case studies, it was applied to cases in the e-commerce and e-banking domains to confirm its validity in different domains, in which there are well known references available for checking concurrent validity. The interviews and questionnaires were applied to two organizations in e-banking and to two organizations in e-commerce domain. Figure 10. Case Study Steps 55 4.3. Case Study Plan- Assessment Framework The case studies were chosen from two different domains. The e-commerce and ebanking domains were chosen because there are some organizational performance factors that can be used for comparing the results with. Some reference data for sectoral performance evaluations is available in those domains, therefore these domains are chosen for the case studies to compare the results with. In e-commerce domain we have chosen similar companies for reliability. Both of the organizations sell the same type of goods, both have only web-based business. In e-banking domain, both of the organizations rank among the first 10 banks of Turkey. Both develop their software in-house and both organizations are currently in the process of acquiring CMMI- level 3 certification for software development. In the interviews, an IS/IT specialist who was responsible, knowledged in WIS used in the organizations were chosen to gather the data related to the WIS relationships. Similarly, management was interviewed to gather CSF related data, since management knows and decides on the organizational strategies. We planned to address a single interviewee in each case since the aim was to gather the necessary data through in-depth interviews. In some cases where the interviewees could not provide in-depth data, two interviewees were contacted. In order to collect the questionnaire answers, convenience sampling was used to decide on the participants. The questionnaires were posted on a website and the link was sent to the volunteers who were available for the study. In each case study, semi-structured interviews with open-ended questions were conducted with IT/IS Specialist and also with management of the organizations (see Appendix B for interview questions). The case study plan is depicted in Figure 11. The interviews lasted approximately 2 hours in each case study. After gathering the necessary data, questionnaires were prepared and applied within one month time frame to stakeholders. 56 Figure 11. WIS Success Assessment Stages An interview was performed with IT/IS Specialist to gather the framework elements considering the WIS of the organization. Then, model based on GQM was applied determining goals and related measures to assess WIS effectiveness for each WIS relationship which were used to create questionnaire. Parallel to that stage, this data was used to introduce the WIS assessment framework and measures to the 57 management and gather CSFs rankings and weights for WIS relations with respect to contribution of each relationship on each CSF. After verifying the framework elements and assessment process in the preliminary case studies, and analyzing the results, framework was modified and applied to two banks and to two e-commerce companies and WIS effectiveness values were calculated as shown in Figure 11. Next section describes the organizations studied for case studies. 4.4. Preliminary Cases For the preliminary cases both profit and non-profit organizations were explored to consider domain related differences. Organization A The first preliminary case study was performed for an electronics retailer company to test the framework elements and the assessment process. Organization A has both online and store based business and has been in this sector for almost 10 years. It has 160 employees, 8 of which work for IT department and responsible for development and maintenance of the e-commerce IS. Organization B This organization is one of the private universities of Turkey with 3.000 students. The focus of the consideration was the system developed as part of a project to provide Remote Electronics Laboratory (RL) for Electrical Engineering Department students all over the world through a WIS. Organization C This organization is one of the biggest universities of Turkey with 20.000 students and almost 2000 academics. It has several web-based systems like student registration WIS, course management WIS, library management WIS, human resources WIS, financial WIS, academical CV WIS, academical evaluation WIS as 58 well as a distant education WIS. 4.5 Case Studies The SEWISS framework was applied to four organizations in e-banking and ecommerce sectors. Organization D Organization D is one of the major banks of Turkey with 600 branches country wide, having approximately 15.000 personnel by 2008. It has an IT department with 600 employees, 20 of who work for Internet banking channel. Organization D has several awards related to its Internet Banking WIS. Organization E This organization is one of the biggest and oldest financial service providers in Turkey. It has almost 900 branches country wide, having approximately 19.000 employees by 2008. As a leading bank in Internet banking services, it has 180 employees working in IT department, 10 of who work for the development of the Internet channel. Organization F Organization F is a leading e-commerce company selling various consumer products; electronics, clothing, cosmetics, books, DVDs etc. They have been in business for 10 years. Organization F has 156 employees, 10 of who work for IT department and responsible for development and maintenance of the B2C system. Organization G Organization G is an e-commerce company which is newly started up, selling various types of consumer goods. It has total of 50 employees, 15 of who work for the B2C WIS. 59 Table 2 below summarizes the organization properties used for the case studies. In the next chapter, the results of the case studies are explained. Table 2. Case Organization Profit/ Business Nonprofit Domain A P e-business 9 11 B N Academical 12 - C N Academical 52 - D P Banking 62 53 E P Banking 84 91 F P e-business 10 68 G P e-business 2 17 Preliminary Type Actual Organizations Summary Age Number of Respondents Total Respondents 240 Average Organizational Age 33 60 CHAPTER 5 APPLICATION OF SEWISS FRAMEWORK TO THE CASES This chapter discusses the case studies performed to elaborate the proposed framework elements and the assessment process. The cases are chosen from ebanking and e-commerce sectors. Section 1 discusses the preliminary case studies and their findings; Section 2 and Section 3 explain e-banking and e-commerce case studies respectively. 5.1 Preliminary Case Studies This study was conducted to construct an assessment framework for WIS success. Since WIS could be used in wide range of sectors, in order to develop a generic framework, three organizations from different domains namely e-commerce, egovernment, and e-learning were chosen as preliminary cases. These cases were used to validate the model elements exist in different WISs and to test the assessment framework. The reason was to capture the differences between different domains and as a result modify the proposed assessment framework. The cases were chosen with the objectives of checking framework elements also the applicability of GQM, domain differences, and assessment process in real cases. The case study for Organization A was also used to apply the assessment framework and calculate WIS success. 61 5.1.1 Preliminary Case 1- Organization A An interview was arranged with IS Specialist to gather information about the WIS and with a manager to gather company’s CSFs. Organization A has different IS’s like accounting, personnel intranet working together with e-commerce WIS. The relations of e-commerce WIS with the environmental entities include User-WIS relationship utilized by customers, personnel, and managers; Organization-WIS relationship which contains the relationship of WIS with accounting IS and local intranet. Other systems-WIS relationship does not exist in the company since the information gathered from suppliers is inputted to the system manually. However, they were working on to enable this connection which would allow electronic data transfer between companies. WIS for e-commerce system provides different functionalities for managers, users and for sales representatives. Organization A did not use any tool to measure WIS success, but they just check the customer complaints and improve the system accordingly. IS Specialist specified the goals of each WIS relationship and has chosen measures to determine the success of each goal. The manager stated that at the beginning of their business, the price was an important reason to catch new customers and to be popular in sector. However, as time passes, the quality and speed of the service they gave to the customers became more important to keep the customers. It was observed that Organization-WIS relationship may include both web-based and non web-based IS’s exist in the organizational environment, so the proposed framework was modified accordingly. GQM method was applied to gather relationship measures. IS Specialist specified goals for each relationship, as given in Table 3, for each of which questions to be answered had to be provided to check if each goal was accomplished or not. However, IS Specialist preferred directly to specify a goal and then related measures without providing the questions. GQM method requires experts who are experienced in application of this method. The well62 known organizational performance evaluation method called Balanced Score Card (BSC) (Kaplan and Norton, 1992), also suggests specifying goals of the system and then for each goal identifying the measures to be used in the assessment. Parallel to this idea, the GQM method was modified, such that goals were identified for each relationship and then related measures were specified which would make sure the objectives were successfully met. Additionally, in specifying the measures IS Specialist had a difficulty, therefore a list was created from various studies performed in this area, and it was provided to IS Specialist and asked to choose among them as provided in Appendix A. Since the list was too long to choose from, for the following case studies, a shorter list was used as given in Appendix D. Table 3. RELATIONSHIPS User-WIS Other Systems-WIS Organization-WIS WIS Relationship Goals for Organization A GOALS -serving 7/24 -providing fast access -providing correct price and product information -maintaining customer loyalty -enabling flexible and correct reporting Does not exist -providing price consistency -enabling message transfer -providing correct exchange rate The interview with the manager revealed that company had several CSFs as stated according to their priority below, which have different importance for the company which supports the proposed framework. CSF1. Provide high quality and fast customer service CSF2. Supply high product variety CSF3. Create good employee motivation CSF4. Maintain up to date, working e-commerce system The importance values of CSFs and also the CSF rankings with respect to each relationship were gathered from the manager, which resulted in differences among them, which supports the proposed framework suggestion that “the success of each 63 WIS relationship affects the realization of each CSF differently”. The data gathered from the manager is illustrated in Table 4. Table 4. WIS Relationship Weights CSF1 CSF2 CSF3 CSF4 CSF Importance rankings (ri) 4 3 2 1 Rel1 (User-WIS) 5 9 7 10 Rel2 (Other Systems- WIS) Does not exist Rel3 (Organization- WIS) 10 10 6 10 A questionnaire was prepared by using the measures obtained in the interview with IS Specialist, which was then sent to system users by email (Appendix E). By following the calculation steps illustrated in Appendix C, WIS relationship success values Rj’s were obtained. Then, these values were filtered through CSFs using WIS relationship weight matrix. The SEWISS value was calculated as shown in Table 5. Table 5. SEWISS value calculation CSF Importance rankings(ri) CSF1 CSF2 CSF3 CSF4 4 3 2 1 j Relationships Rj 1 Rel1 (User-WIS) 3,8 5 9 7 10 2 Rel2 (Other Systems - WIS) 0 0 0 0 0 3 Rel3 (Organization- WIS) 3,4 10 10 6 10 Success for each CSF (wij*Rj) 53 68 47 72 Success for each CSF with priority(wij*Rj*ri) 212 204 94 72 Total success for each CSF 583 SEWISS value (out of 100) 39 SEWISS effectiveness level Poor In SEWISS value calculation, average of the questionnaire answers were used. The calculation was tested using both median and average of the answers, which did not 64 differ significantly; therefore, to include the answers at diverse ends average is preferred. Additionally, since the data gathered from questionnaires were continuous data, average function was more suitable for the analysis. 5.1.2 Preliminary Case 2- Organization B Remote Laboratory (RL) system considered in this case was used as part of the course content, RL should work with other systems like course management or content management systems, so the integration between them is important for system to be successful. The RL system serves for students, technicians, educators and engineers from all over Europe. The requirements of all these stakeholders are different from each other which makes the system more complicated. An interview was conducted with project technical manager to identify remote lab (RL) system relationships and their goals. During the interviews, the existence of RL relationships was validated. Three relationships, namely User-RL, Other systemsRL and Organization-RL were identified as proposed by the framework. For each relationship category, relationship goals were gathered as shown in Table 6. Table 6. WIS Relationship Goals for Organization B RELATIONSHIPS User-WIS GOALS -finding the required information easily -having different types of information on a subject -no problems in performing experiments Other systems-WIS -good integration Organization-WIS -supporting teaching activities The CSFs for this organization are collected and their link with each WIS relationship has explored as given in Table 7. CSF1. Have all the graduates being employed CSF2. Increase in publications CSF3. Increase in student number 65 CSF4. Low level of instructor leave In organization B, CSF importance values collected were lower than the previous cases as shown in Table 7, which brought the consideration that the WIS, namely remote lab system does not affect this organization as much as the WISs explored in previous cases. Since the RL project affects small portion of the organization as a whole, the rankings are small out of 10. Therefore, the organizational environment that WIS works in was considered in the success assessment through WIS relationship weight matrix. Table 7. WIS Relationship Weights CSF1 CSF2 CSF3 CSF4 CSF Importance rankings (ri) 4 3 2 1 Rel1 (User-WIS) 3 3 3 1 Rel2 (Other Systems- WIS) 1 1 1 1 Rel3 (Organization - WIS) 1 3 1 1 The WIS success in this case was not measured but the aim was to use it to verify the assessment framework elements in a different domain. 5.1.3 Preliminary Case 3- Organization C In Organization C, the focus of the case study was the student registration WIS. The relations of WIS with the environmental entities include User-WIS utilized by students, academics, departmental secretaries and student affairs office employees; Other systems-WIS relationship including the communication of the student registration system with banks, hostel office, OYS; and Organization-WIS relationship containing the relationship of WIS with university rules and regulations, human resources department, health center, library, and traffic management office. An interview was performed with software development manager to validate the proposed framework elements. The goals gathered are displayed in Table 8. 66 Table 8. WIS Relationship Goals for Organization C RELATIONSHIPS User-WIS Other systems-WIS Organization-WIS GOALS -providing fast transactions -enabling access from anywhere -synchronization -data consistency -non-repetitive data -correct data -fast access -providing answers to requests The CSFs for this organization are collected and their link with each WIS relationship has explored as depicted in Table 9. CSF1: Being a leader in community improvement CSF2: Educating future leaders CSF3: Being innovator, creative CSF4: Having an international reputation CSF5: Having successful organizational management CSF6: Having rich resources CSF7: Having strong research orientation CSF8: Creating synergy between disciplines Table 9. WIS Relationship Weights CSF1 CSF2 CSF3 CSF4 CSF5 CSF6 CSF7 CSF8 8 7 6 5 4 3 2 1 Rel1 (User-WIS) 10 9 7 10 10 5 7 9 Rel2(OtherSystems- 10 9 7 10 10 5 7 9 10 9 7 10 10 5 7 9 CSF Importance rankings (ri) WIS) Rel3(Organization-WIS) This case was used to check the existence of framework elements. 67 5.1.4 Discussions of Preliminary Case Studies Three cases revealed some points that were not considered at the initial framework development phase. Choosing WISs from profit and non-profit domains allowed exploring the domain differences and their effect on the proposed framework. The following points were identified in the preliminary case studies. • Organization-WIS relationship was modified to include both web-based and non web-based systems that may exist in the organization. • Other systems-WIS relationship was modified to include both web-based and non web-based systems that may exist in the external environment. • Interviewees had difficulty in specifying first the goals, then the questions and finally the measures as in the GQM method. They preferred to skip the middle step of questions since they were not skillful in GQM method. Therefore, the step of asking questions was eliminated and the method was modified as reaching measures directly from goals by eliminating step 1.1.2 in Figure 7. • In GQM application, a list of suggestions for measures was provided to the interviewees in case they have difficulty specifying the measures. 5.2 Cases from E-banking sector 5.2.1 Organization D The interview was conducted with Assistant Vice President of Internet banking channel and a senior IT specialist. Their main responsibility was to develop and manage Internet banking channel. Organization D monitored Internet banking success based on some key performance indicators like increase in customers, transaction amount, products sold through web, costs related to Internet banking, security and customer satisfaction which were checked through monthly reports. These indicators, however, were not made available to the researcher, due to bank policy. 68 The interviewees identified three WIS relationships of the framework: User-WIS relationship; Organization-WIS relationship containing the interaction of Internet banking WIS with the core banking system, which is a non-web based system, and also the interaction of Internet banking with non web-based Management Information System; Other systems-WIS relationship including the communication of Internet banking system with insurance company and also the interaction of Internet Banking system with external systems like ĐMKB, Turkcell, SSK, Telecom. For each relationship, the goals expected from that specific relationship were specified by the interviewees and presented in Table 10. For each goal specified in Table 10, IS Specialist was asked to choose measures (Appendix D) to meet each goal. Table 10. WIS Relationship Goals for Organization D RELATIONSHIPS User -WIS Other systems-WIS Organization-WIS GOALS -successful transactions -easy search -high number of successful transactions -minimum error -increase in product sales -easy data update -ability to replace physical banking branch -fast transactions -correct payments&timely payments -continuity -increase customer number -creating convenience -increase product sales -consistency in data exchange -real-time and fast connection -low error rate -correct reports -flexible report creation The next step was to identify the company CSFs. Organization D had several critical success factors as listed below according to their priorities. CSFs had different 69 importance levels for the company, which supports using CSF importance levels as input in the SEWISS calculation. They were identified as changeable periodically according to the economical dynamics. CSF1. Gain/increase customer number CSF2. Increase profit based on products and distribution channels CSF3. Increase revenue CSF4. Decrease operational costs CSF5. Increase creativity in services and being the pioneer in new services CSF6. Possession of retention in the sector The importance of CSF’s and also the CSF rankings over 10, with respect to each relationship were gathered which is illustrated in Table 11. The values below should be read as: in achieving CSF1, Organization-WIS relationship has contribution of 9/10. Table 11. WIS Relationship Weights CSF1 CSF2 CSF3 CSF4 CSF5 CSF6 CSF Importance rankings (ri) 6 5 4 3 2 1 Rel1 (User-WIS) 8 8 7 10 10 9 Rel2 (Other Systems- WIS) 8 8,5 8,5 9 7,5 8 Rel3 (Organization - WIS) 9 8,5 9 9 7 9,5 Calculation of SEWISS success value A web-based questionnaire was prepared by using the measures obtained in the interview with IS Specialist, which was then sent to system users by email (Appendix E). 67% of the respondents were older than 30 years. All of them were at least university graduates. 81% of the respondents use Internet continuously during the day, 86% of which use Internet banking more than several times a week, 95% performing active transactions with Organization D’s e-banking system. 70 By following the calculation steps illustrated in Appendix C, WIS relationship success values Rj’s were obtained. Then, these values were filtered through CSFs using WIS relationship weight matrix. Calculation details are shown in Table 12. Table 12. SEWISS value calculation CSF Importance rankings(ri) CSF1 CSF2 CSF3 CSF4 CSF5 CSF6 6 5 4 3 2 1 j Relationships Rj 1 Rel1 (User-WIS) 4,4 8 8 7 10 10 9 2 Rel2 (Other Systems- WIS) 4,5 8 8,5 8,5 9 7,5 8 3 Rel3 (Organization-WIS) 4,4 9 8,5 9 9 7 9,5 Success for each CSF (wij*Rj) 110 110 108 124 108 117 Success for eachCSF with priority(wij*Rj*ri) 662 552 433 371 216 117 Total success for each CSF 2352 SEWISS value (out of 100) 75 SEWISS effectiveness level Effective Considering different respondent profiles, SEWISS values were calculated as in Table 13 for Organization D. Table 13. SEWISS values for Organization D Respondent Profile SEWISS All respondents 75 Using Internet banking continuously 77 Using Internet continuously 75 Using Organization D WIS several times a week 74 Using Organization D WIS continuously 78 71 5.2.2 Organization E An interview was conducted with Assistant Manager of the Internet banking channel who was responsible from analysis, design, implementation, testing and maintenance of the system development. There were several ISs in Organization E, namely Bankamatic IS, IVR-phone banking IS, mobile banking IS, main banking IS, and credits management IS. Being a leader in Internet channel, Organization E provided almost all the banking transactions through Internet banking. There were 140 banking operations offered through Internet banking. Organization E’s WIS success definition was “to be available 7/24 with high performance”. Internet banking success was examined by measuring transaction durations, system-off durations, extensively used transactions and through feedback from advisory agencies which were not made available to the researcher because of security reasons. Three WIS relationships were identified considering Organization E’s Internet banking IS with their priority. Organization-WIS relationship contained the connection with the main banking system and Other systems-WIS relationship included the communication of Internet banking WIS with insurance company, retirement insurance, SSK, THY, Turkcell, Avea etc. For each relationship, goals were specified as shown in Table 14. Table 14. WIS Relationship Goals for Organization E RELATIONSHIPS User-WIS Other Systems-WIS Organization- WIS GOALS -7/24 service -enabling all the transactions -easy and fast transactions -security -secure operation -electronic data stability -consistent, flexible data structure -7/24 operation - available access -operational system 72 In the next step, for each goal specified in Table 14, related measures (Appendix D) were chosen. Organization E had several critical success factors as listed below according to their priorities, which had different importance for the company which supports the proposed framework. CSF1. Increase in profit CSF2. Increase in customer number CSF3. Offer fast, effective, quality solutions CSF4. Being reliable CSF5. Being pioneer in services and products CSF6. Increase personnel motivation The importance of CSF’s and also the CSF rankings over 10, with respect to each relationship were gathered. The data gathered in the interview are illustrated in Table 15. Table 15. WIS Relationship Weights CSF1 CSF2 CSF3 CSF4 CSF5 CSF6 CSF Importance rankings(ri) 6 5 4 3 2 1 Rel1 (User-WIS) 10 10 4 2 4 1 Rel2 (Other systems - WIS) 2 4 6 6 9 1 Rel3 (Organization - WIS) 2 4 8 6 10 1 Calculation of SEWISS success value A web-based questionnaire was prepared by using the measures obtained in the interview, which was then sent to system users by email (Appendix E). Results revealed that, 98% of the respondents were older than 30 years. All of them were at least university graduates. 69% of the respondents use Internet continuously during the day, 86% of which use Internet banking more than several times a week, 93% performing active transactions. 47% of the respondents use Organization E’s internet banking continuously. 73 By following the calculation steps illustrated in Appendix C, WIS relationship success values Rj’s were calculated which were then filtered through CSFs using WIS relationship weight matrix. Calculation details are shown in Table 16. Considering different respondent profiles, SEWISS values were calculated as in Table 17 for Organization E. Table 16. SEWISS value calculation CSF Importance rankings(ri) CSF1 CSF2 CSF3 CSF4 CSF5 CSF6 6 5 4 3 2 1 j Relationships Rj 1 Rel1 (User-WIS) 4,3 10 10 4 2 4 1 2 Rel2 (Other systems-WIS) 4,4 2 4 6 6 9 1 3 Rel3 (Organization-WIS) 4,5 2 4 8 6 10 1 61 79 79 62 101 13 364 392 317 185 202 13 Success for each CSF (wij*Rj) Success for each CSF with priority(wij*Rj*ri) Total success for each CSF 1467 SEWISS value (out of 100) 47 SEWISS effectiveness level Satisfactory Table 17. SEWISS values for Organization E Respondent Profile SEWISS All respondents 47 Using Internet banking continuously Using Internet continuously Using Organization E WIS several times a week Using Organization E WIS continuously 74 49 47 46 49 5.2.3 Outcomes of the Cases in E-banking Domain SEWISS framework has applied to Organization D and Organization E and SEWISS values based on measures chosen by decision makers of both organizations were obtained. When SEWISS values for Organization D and Organization E for different respondent categories are compared, Organization D got higher SEWISS values in every category as shown Figure 12 below. Figure 12. SEWISS comparison of Organization D and E The parallelism between the obtained SEWISS values and organizations’ websites statistics from Alexa.com and other organizational factors is shown in Figures 13 and 14. When these organizations reach and pageview measures from Alexa.com are examined, Organization D is seen to outperform Organization E in website access. When SEWISS outcomes are compared, it is seen that Organization D has a level 75Effective, whereas, that of Organization E is 47-Satisfactory. It is clear that SEWISS results and Alexa.com statistics display an encouraging parallelism. Even though, the latter, which indicates levels of external access, cannot be taken as a direct measure 75 of WIS effectiveness for obvious reasons. Figure 13. Comparison of Daily Reach values for Organization D & E(Alexa, 2008) Figure 14. Comparison of DailyPageview values of Organization D&E(Alexa, 2008) In Table 18, some performance ratios proposed by Ekodialog, for banking sector, which are also used in other studies (Berger, Clarke, Cull, Klapper and Udell, 2005; 76 Berger, Davies and Flannery, 1998; Bonin, Hasan and Watchtel, 2003; Zimmerman, 1996), are analyzed for Organizations D and E. These ratios are obtained from BDDK yearly report (BDDK, 2008) shows that even though Organization D is a younger bank, the bank performance ratios have the same parallelism with SEWISS results. Table 18. Organization D and E Properties FACTORS ORGANIZATION D 62 ORGANIZATION E 84 15354 19949 ROA-Return on Assets (2007) 3.4 2.1 ROE- Return on equity (2007) 110.3 61.7 Net interest Margin (2007) 136.3 108.7 75-Effective 47-Poor Bank age Employee number (2008) SEWISS Additionally, Global Finance Organization awarded Organization D as having the best Internet banking channel in Europe in 2008, which also supports the SEWISS results. 5.3 Cases from E-Commerce sector 5.3.1 Organization F An interview was conducted with Sales Manager who was also one of the three shareholders of the company. Three WIS relationships proposed by the SEWISS framework existed in Organization F’s e-commerce WIS. Their related objectives are given in Table 19. For Organization F, factors like delivery speed, problem solving speed, revenue target, daily visitors to new customer ratio, customer satisfaction, and product return rate were constantly monitored to check the problems of the WIS system. 77 Table 19. WIS Relationship Goals for Organization F RELATIONSHIPS GOALS -easy use -fast operations -understandable -secure data exchange -fast -easy reporting -correct financial reporting User- WIS Other Systems-WIS Organization-WIS The company had five critical success factors as listed below according to their priority, which had different importance values for the company, which supports the proposed framework. CSF1. Correct Delivery CSF2. Increased Customer Satisfaction CSF3. Increased Revenue CSF4. Increased number of sold products CSF5. Have high revenue per bill ratio The importance of CSFs and the CSF rankings with respect to each relationship were gathered, which resulted in differences among them. The data gathered in the interview are illustrated in Table 20. Table 20. WIS Relationship Weights CSF1 CSF2 CSF3 CSF4 CSF5 CSF Importance rankings (ri) 5 4 3 2 1 Rel1 (User-WIS) 4 9 9 10 7 Rel2 (Other Systems - WIS) 10 10 10 8 4 Rel3 (Organization- WIS) 10 10 10 4 4 78 Calculation of SEWISS success value The questionnaire was prepared by using the measures obtained in the interview, which was then rated by system users (Appendix E). 64% of the respondents were older than 30 years, 84% of them were at least university graduates. 73% of the respondents used Internet continuously during the day, 40% of whom used Internet shopping several times a year, 33% of them used online shopping several times a month. 65% of the respondents used online shopping within 1 month. 52% used the e-commerce system to perform active transactions and 36% just to get information. 31% of the respondents used Organization F’s Internet shopping several times a year, 27% accessed it several times per month and 21% were connected continuously. WIS relationship success values Rj’s are calculated as given in Table 21. These values are then filtered through CSFs using WIS relationship weight matrix. Table 21. SEWISS value calculation CSF Importance rankings(ri) CSF1 CSF2 CSF3 CSF4 CSF5 5 4 3 2 1 j Relationships Rj 1 Rel1 (User-WIS) 4,1 4 9 9 10 7 2 Rel2 (Other WIS- WIS) 3,8 10 10 10 8 4 3 Rel3 (Organization - WIS) 3,7 10 10 10 4 4 Success for each CSF (wij*Rj) 91 112 112 86 58 Success for each CSF with priority(wij*Rj*ri) 455 448 336 172 58 Total success for each CSF 1469 SEWISS value (out of 100) 65 SEWISS effectiveness level Effective 79 Considering different respondent profiles, SEWISS values were calculated as in Table 22 for Organization F. Table 22. SEWISS values for Organization F Respondent Profile SEWISS All respondents 65 Using Internet shopping several times a year 65 Using Internet shopping several times a month 66 Using Internet shopping continuously 61 Using Internet continuously 66 Using Organization F WIS several times a year 62 Using Organization F WIS several times a month 69 5.3.2 Organization G An interview was conducted with Software Development Manager who was also responsible for sales activities. Three WIS relationships were specified and their goals are represented in Table 23. Table 23. WIS Relationship Goals for Organization G RELATIONSHIP User-WIS Other systems-WIS Organization-WIS GOALS -fast shopping -easy to search -no technical problems -user-friendly -fast data exchange -correct operations -easy to detect errors -available functions -fast promotion definition -display correct customer information -enabling information update -deciding new products to be sold 80 For Organization G, the e-commerce system’s success was specified as “7/24 availability with fast and easy shopping, controllable and effective with no IT dependence”. Some factors like time spent on the site, checkout duration, and customer feedback were monitored. Organization G had several CSFs, which were stated as “tightly connected” to each other, described below. CSF1. Increase sales CSF2. Increase customer quantity CSF3. Cheap product prices CSF4. Increase Product variety CSF5. Increase payment options The importance values of those CSFs to the specified WIS relationships are shown in Table 24. Table 24. WIS Relationship Weights CSF1 CSF2 CSF3 CSF4 CSF5 CSF Importance rankings (ri) 5 4 3 2 1 Rel1 (User-WIS) 6 8 4 2 8 Rel2 (Other Systems- WIS) 8 6 2 8 1 Rel3 (Organization - WIS) 8 6 3 9 3 Calculation of SEWISS success value The questionnaire was prepared by using the measures obtained in the interview, which was rated by system users (Appendix E). 56% of the respondents were older than 30 years. 87% of them were at least university graduates. 81% of the respondents used Internet continuously during the day, 38% of whom used Internet shopping several times a year, 31% of them used online shopping several times a month. 63% of the respondents used online shopping within 1 month. 63% used the e-commerce system to perform active transactions and 38% just to get information. 81 50% of the respondents used Organization G’s Internet shopping several times a year, 19% accessed it several times every month and 7% connected continuously. WIS relationship success values Rj’s were obtained which were then filtered through CSFs using WIS relationship weight matrix as given in Table 25. Table 25. SEWISS value calculation CSF Importance rankings(ri) CSF1 CSF2 CSF3 CSF4 CSF5 5 4 3 2 1 j Relationships Rj 1 Rel1 (User-WIS) 3,6 6 8 4 2 8 2 Rel2 (Other WIS- WIS) 3,4 8 6 2 8 1 3 Rel3 (Organization - WIS) 3,9 8 6 3 9 3 Success for each CSF (wij*Rj) 80 73 33 70 44 Success for each CSF with priority(wij*Rj*ri) 401 291 99 139 44 Total success for each CSF 974 SEWISS value (out of 100) 43 SEWISS effectiveness level Satisfactory Considering different respondent profiles, SEWISS values were calculated as in Table 26 for Organization G. Table 26. SEWISS values for Organization G Respondent Profile SEWISS All respondents 43 Using Internet shopping several times a year 44 Using Internet shopping several times a month 44 Using Internet shopping continuously 43 Using Internet continuously 44 Using Organization G WIS several times a year 46 Using Organization G WIS several times a month 47 82 5.3.3. Outcomes of the Cases in E-commerce Domain SEWISS framework has applied to Organization F and Organization G and SEWISS results based on measures chosen by decision makers of both organizations were obtained. When SEWISS results for Organization F and Organization G for different respondent categories are compared, Organization F performs better in every category as shown in Figure 15. The concurrent validity of the results was checked by comparing SEWISS results with organizations’ websites statistics and some organizational factors as illustrated in Figures 16 and 17. Figure 15. Comparison of SEWISS values for Organization F and G 83 Figure 16. Comparison of Daily Reach values for Organization F & G (Alexa, 2008) Figure 17. Comparison of DailyPageview values of Organization F&G (Alexa, 2008) When these organizations reach and pageview measures from Alexa.com were examined, parallel to SEWISS results, Organization F does better than Organization G in website access. These results supports SEWISS as a promising framework for 84 WIS success assessment. Additionally, for concurrent validity, some organizational characteristics were compared with the SEWISS results as shown in Table 27 below. Table 27. Organization F and G Properties FACTORS ORGANIZATION F ORGANIZATION G Age of the Organization 10 2 Number of employees 156 50 Revenue $150 million $10 million SEWISS 65-Effective 43-Satisfactory Furthermore, Organization F was placed as 58th in the first 500 IT companies in a research conducted by Interpromedya in 2007 in Turkey. They were also awarded as the best IT Company in B2C category, in 500 IT companies in Turkey, which also supports the SEWISS results. 5.4 Discussion on Case Study Results The framework resulted in a WIS effectiveness level that was anticipated to order the WISs. SEWISS allows ordering of different WISs with respect to their effectiveness and thus makes classification possible at five different levels. The validation of the results is a complex subject. Because, there is no reference measure that can be used to compare the effectiveness results to validate SEWISS values against. Therefore, the SEWISS effectiveness results were justified by comparing the outcomes with some organizational factors and website statistics. The SEWISS results and those factors were parallel by supporting the fact that the organization that got a higher SEWISS success value, measured through the SEWISS framework, was the one that operated better in terms of organizational or domain specific factors. 85 CHAPTER 6 CONCLUSION Internet has changed the characteristics of the information systems and their environment by providing a multi-access, borderless communication medium. This change brings the necessity of considering IS assessment through a novel approach. Consequently, this study aimed to create a WIS effectiveness assessment framework, which would allow better judgments about information system investments for organizations. The objectives of this study were: ► To explore the previous studies in IS effectiveness assessment area and to identify their inadequacies in the context of WIS; ► To present a novel framework for WIS effectiveness assessment; ► To enhance the proposed framework and to appraise its validity via multiple case studies In order to fulfill these objectives, the starting point was to review the previous studies on information system effectiveness assessment. In this regard, literature on both traditional IS assessment and on WIS assessment, namely assessment of ebusiness, e-government and website evaluation were explored and their insufficiencies were identified. Even though the studies on IS effectiveness assessment literature are valuable in their own perspectives, they are far from providing a broad, comprehensive framework for the evaluation of WISs 86 independent of the application domain. Hence, a novel WIS success assessment framework, SEWISS, was developed which encompasses both WIS related and organization related aspects for the assessment. In this regard, the relationships of WIS with its environment were taken into account as the first dimension. WIS success was evaluated in terms of its relationships with its users, other ISs and the organization. Considering solely WIS relationships in success assessment isolated from the organization provides a limited view of the WIS success. Therefore, organizational CSFs were considered under the Strategy dimension of the proposed framework. For effectiveness assessment, initial consideration was that if a WIS relationship interacts with a CSF, it adds value to the accomplishment of the specific CSF and therefore, it contributes to WIS success. Hence, relationships and CSF’s are essential framework elements. Considering these elements, the framework resulted in a WIS effectiveness level that was anticipated to quantitatively rank the systems. SEWISS allows ordering of different WISs with respect to their effectiveness and thus makes ordinal classification possible between different WISs. After the development stage, framework was enhanced and its validity was justified through multiple case studies. In this regard, SEWISS framework was applied in a total of four different organizations in two different sectors, namely in e-banking and e-commerce domains. Before those case studies, three preliminary cases were investigated for WISs in business, university, and remote lab domains, to test the framework elements. The following research questions were answered through the analyses of the case studies: 1. Does the proposed framework reflect WIS success? a. Do the proposed SEWISS dimensions exist in e-commerce, e-banking and e-learning systems? b. Do WIS relationships exist in e-commerce, e-banking and e-learning 87 domains? c. Do CSFs differ in e-commerce, e-banking and e-learning organizational contexts? d. Do CSFs differ in their importance to the organization? 2. Does the effectiveness value found within the proposed framework measure WIS’s success in e-commerce and e-banking organizations? 3. Does the framework distinguish between fully and partly web-based organizations in success assessment? 4. Is the proposed framework applicable to the organizational domains subject to the case studies, namely e-banking and e-business organizations? First of all, in WISs of all the case studies, including the preliminary cases, it was observed that the framework dimensions; WIS relationships and CSFs, existed. Secondly, CSFs gathered from different organizations of the same domain have some similarities, but also some differences, which mean even in the same domain, organizations may have different views of doing business. In different domains, CSFs are more diverse supporting the proposed framework. The CSFs differed in their importance levels to the organization; some were more important and crucial than the others, which supports the proposed framework. The verification of the results is a convoluted subject. Because, using a reference measure to validate SEWISS values against, means accepting a valid WIS success measure already exists. In that case, there would be no need to propose a method for success calculation; that reference measure could have been used for success assessment instead. Nevertheless, there is not such a measure that can be used to compare SEWISS values against, which creates a validity problem. Therefore, in this study, concurrent validity of the obtained results was justified by comparing the outcomes with some organizational factors and website statistics. The comparison of SEWISS values and those factors gave promising results; by supporting the fact that organization which got a higher SEWISS success value, measured through the 88 SEWISS framework, was the one that operated better in terms of organizational or domain specific factors. SEWISS framework aimed to provide an empirical formulation to measure effectiveness of WISs, which may not be unique for the assessment. Other formulations can be proposed by other studies. SEWISS framework is proposed as a generic framework for any type of WIS, for any organizational domain, whether it is e-business, e-government, e-learning, or e-health system. But, it also allows organization specific assessment based on organizational CSFs and WIS relationship measures for different domains. For a specific domain, the WIS relationship measures may differ and can be chosen according to the characteristics of the domain. For example, an evaluation criterion like delivery time used for the assessment of an e-commerce WIS would meaningless for the assessment of an e-banking WIS. Therefore, flexibility is provided in the assessment process according to the domain of the organization. Even though SEWISS proposes a structured assessment framework, it allows customization of the WIS assessment according to the characteristics of the organization. Evaluation of a WIS cannot be independent of its users. In this regard, SEWISS framework proposes assessment based on stakeholder perceptions. This study was an interpretive study, it took into account not only the user perceptions but also their influence to achievement of organizational goals. Consequently, in assessment, expert knowledge (IS Specialist), management concerns (CSF related data) and other stakeholder perceptions (questionnaires) were considered. Limitations The questionnaires were applied to the users of the WIS in the organizations in which the case studies were conducted; however some of the stakeholder groups could not have been reached, since the process of filling out the questionnaires was 89 based on volunteer participation. The answers collected from stakeholders were not equally distributed in terms of stakeholder types. Additionally, the data collection process through interviews were difficult to conduct because of low motivation and time constraints. These limitations can be mitigated once the framework is applied by organizational management incentives rather than external research purposes. In gathering the measures, GQM could not be applied fully, the middle step of getting questions was eliminated because the interviewees were not experienced in GQM method and they preferred to specify goals and jump directly to the measure specification step. A sample list of measures for success assessment was provided to the interviewees to ease GQM application and their contributions were also gathered. The provided list was just an example used in this study, and its content could be expanded in other studies. The SEWISS framework was applied to limited number of organizations because of time constraints, which is another limitation of this study. Future Study The proposed framework could be applied to different domains like e-government and e-health. In a specific sector, the framework can be applied to a higher number of organizations to develop domain-specific list of measures. This would create fixed sector specific questionnaires to be applied to users. The framework can also be applied to a specific organization at different times of the organization’s life span to monitor the improvement of their WISs in time and to test which measures are chosen for the different ages of the organization for WIS assessment. A web-based tool based on the framework proposed in this study has been developed, it is in the process of being applied and evaluated in different domains. As a future research, the SEWISS assessment framework could also be applied together with other assessment models to provide a comparison between them. The 90 frequency of applying the SEWISS framework is another important point to consider in the dynamic business and Internet environment. Because, in time, as the organization adjusts itself to reach a more successful state, it may also need to change the measures it uses for the assessment. Additionally, the cost of performing WIS assessments for organizations deserves some effort as a future study. Since the numbers of respondents were not high, statistical reliability was not the concern of this study. The aim was to develop a descriptive assessment framework and to show its applicability in different organizations. Hence, statistical analysis of the results is left as a future study. Technology Acceptance Model (TAM) searches for the measurement of the determinants of computer usage (McCloskey, 2003). As a future work, the integration of TAM to SEWISS framework in specification of the measures for assessment can be attempted. With web 2.0, now Internet provides two way interaction between users, therefore WISs have become more interactive environments than social environments. This also deserves attention and its effect in WIS assessment could be explored further. 91 REFERENCES Aladwani, A. & Palvia, P. (2002). Developing and Validating an instrument for measuring user perceived web quality. Information and Management(39), 467-476. Alazmi, M. & Zairi, M. (2003). Knowledge Management critical success factors. Total Quality Management, 14(2), 199-204. Alexa (Producer). (2008) Alexa information http://www.alexa.com/. Alexander, W. J. & Randolph, W. A. (1985). The Fit between Technology and Structure as a Predictor of Performance in Nursing Subunits. The Academy of Management Journal, 28(4), 844-859. Ammenwerth, E., Graber, S., Herrmann, G., Bürkle, T. & König, J. (2003). Evaluation of health information systems- Problems and Challenges. International Journal of Medical Informatics(71), 125-135. Auger, P. (2003). The Relationship between Differentiation Strategies and Internetbased Electronic Commerce Strategies. Unpublished Working Paper. Melburne University. Barut, M., Faisst, W. & Kanet, J. J. (2002). Measuring Supply chain Coupling: An Information System Perspective. European Journal of Purchasing and Supply Management(8), 161-171. 92 Basili, V. R., Kan, S. H. & Shapiro, L. N. (1994). Software Quality: An overview from the perspective of total quality management. IBM Systems Journal, March. BDDK (2008). Seçilmiş Rasyolar-2007. Becker, J., Niehaves, B., Algermissen, L., Delfman, P. & Falk, T. (2004). Egovernment Success Factors. Paper presented at the E-Government. Beise, C. M. (1994). A Model of the IS/Organizational Interface and Users’ Perceptions of IS Effectiveness. Computer Personnel, July. Benbasat, I., Goldstein, D. K. & Mead, M. (1987). The Case Research Strategy in Studies of Information Systems. MIS Quarterly, 369-386. Berger, A. N., Clarke, G. R. G., Cull, R., Klapper, L. & Udell, G. F. (2005). Corporate Governance and Bank Performance : A joint analysis of the static, selection, and dynamic effects of domestic, foreign and state ownership. Journal of Banking and Finance, 29, 2179-2221. Berger, A. N., Davies, S. M. & Flannery, M. J. (1998). Comparing Market and Supervisory Assessments of Bank Performance: Who Knows What When? FEDS Paper No. 98-32. Retrieved from http://ssrn.com/abstract=121651 or DOI: 10.2139/ssrn.121651. doi:10.2139/ssrn.121651 Bonin, J. P., Hasan, I. & Watchtel, P. (2003). Privatization Matters: Bank Performance in Transition Countries. Journal of Banking and Finance, 29(89), 2155-2178. Bremser, W. & Chung, Q. B. (2005). A framework for performance measurement in the e-business environment. Electronic Commerce Research and Applications(4), 395-412. Brynjolfsson, E. (1993). The Productivity Paradox of Information Technology. Communications of the ACM, 36(12). 93 Carr, S. (2001). Book Review: The Internet Bubble. Business Book Review, 18. Chang, C. J. & King, R. W. (2005). Measuring the Performance of Information Systems: A Functional Scorecard. Journal of Management Information Systems, 22(1), 85-115. Chen, L., Gillenson, M. & Sherrell, D. L. (2008). Consumer Acceptance of Virtual Stores: A Theoretical Consumer Acceptance and Critical Success Factors for Virtual Stores. International Journal of Mobile Communications. Cheung, C. M. K. & Lee, M. K. O. (2005). Consumer Satisfaction with Internet shopping: A research framework and Propositions for Future Research. Paper presented at the Proceedings of the 7th international conference on Electronic commerce. Cronholm, S. & Goldkuhl, G. (2003). Six generic Types of Information Systems Evaluation. Paper presented at the 10th European Conference on Information Technology Evaluation (ECITE-2003). D'ambra, J. (2001). Measuring the effcetiveness of the www as an information source. Paper presented at the IEEE International Conference on Systems, Man, and Cybernetics. D'ambra, J. & Rice, R. E. (2001). Emerging Factors in User Evaluation of the WWW. Information and Management, 38, 373-384. D'ambra, J. & Wilson, C. S. (2004). Explaining perceived performance of the World Wide Web. Internet Research, 14(4), 294-310. Dedrick, J., Gurbaxani, V. & Kenneth, L. K. (2003). Information technology and Economic Performance: A critical Review of the Emprical Evidence. ACM Computing Surveys, 35(1), 1-28. DeLone, W. H. & McLean, E. R. (1992). Information systems success: The Quest for the Dependent Variable. Information Systems Research,, 3(1), 60-95. 94 DeLone, W. H. & McLean, E. R. (2003). The DeLone and McLean Model of Information Systems Success: A Ten-Year Update. Journal of Information Systems, 19(4), 9-30. DeLone, W. H. & McLean, E. R. (2004). Measuring e-Commerce Success: Applying the DeLone and McLean Model of Information Systems Success Model. International Journal of Electronic Commerce, 9(1), 31-47. Dumke (2007). GQM Method, eng/us/java/GQM/link1.shtml from http://ivs.cs.uni-magdeburg.de/sw- Ekodialog. Bankacılık: Yönetim ve Performans Retrieved December 10, 2008, from http://www.ekodialog.com/finansal_eko/fin_eko_konulari6.html Elpez, I. & Fink, D. (2006). Information Systems Success in the Public Sector: Stakeholders’ Perspectives and Emerging Alignment Model. Issues in Informing Science and Information Technology, 3. Eralp, Ö. (2004). Design and Implementation of a Software Development Process Measurement System. Middle East Technical University. Evangelidis, A., Akomode, J., Bendiab, A. & Taylor, M. (2002). Risk Assessment& success factors for e-government in a UK establishment. Paper presented at the EGOV 2002. Fitzgerald, G. (1998). Evaluating Information Systems Projects: A Multidimensional Approach. Journal of Information Technology, 13, 15-27. Gable, G. G., Darshana, S. & Chan, T. (2003). Enterprise System Success: a Measurement Model. Paper presented at the 24th International Conference on Information Systems. Garson, G. D. Statnotes: Topics in Multivariate Analysis Retrieved December 1, 2008, from http://www2.chass.ncsu.edu/garson/pa765/statnote.htm 95 Gil-Garcia, R. & Pardo, T. (2005). E-government success factors: Mapping practical tools to theoretical foundations. Government Information Quarterly, 22, 187216. Gunasekaran, A., Ngai, E. W. T. & McGaughey, R. E. (2006). Information Technology and Systems Justification: A review for research and Applications. European Journal of Operational Research, 173, 957-983. Gurbaxani, V. & Whang, S. (1991). The Impact of Information Systems on Organizations and Markets. Communications of the ACM, 34(1), 59-73. Hambrick, D. C. (1980). Operationalizing the concept of Business-Level Strategy in Research. Academy of Management Review, 5(4), 567-575. Hamilton, S. & Chervany, N. (1981). Evaluating Information Systems EffectivenessPart 1: Comparing Evaluation Approaches. MIS Quarterly. Hu, Y., Xiao, J., Pang, J. & Xie, K. (2005). A research on the appraisal Framework of e-government Project Success. Paper presented at the 7th international conference on Electronic commerce. Ifinedo, P. & Nahar, N. (2006). Prioritization of Prioritization of Enterprise Resource Planning (ERP) Systems Success Measures: Viewpoints of Two Stakeholder Groups. Paper presented at the ACM symposium on Applied computing. Iivari, J. (2005). An Empirical Test of the DeLone-McLean Model of Information System Success. The DATA BASE for Advances in Information Systems, 36(2). Irani, Z. & Love, P. (2008). Information systems evaluation: A crisis of understanding Evaluation of Information Systems, Public and Private sector (pp. ix-xxxvi): Elsevier Ltd. 96 Irani, Z., Love, P. E. D., Elliman, T., Jones, S. & Themistocleous, M. (2005). Evaluating e-government: Learning from the experiences of two UK local authorities. Information Systems Journal, 15, 61-82. Isakowitz, T., Bieber, M. & Vitali, F. (1998). Web Information Systems. Communication of the ACM, 41(7). Ives, B. & Learmonth, G. P. (1984). An Information System as a Competitive Weapon. Communications of ACM, 27(12). Jarvenpaa, S. L. & Tiller, E. H. (1999). Integrating Market, Technology, and Policy Opportunities In E-Business Strategy. Journal of Strategic Information Systems, 8, 235-249. Jennex, M., Olfman, L., Panthawi, P. & Park, Y. (1998). An Organizational Memory Information Systems Success Model: An Extension of DeLone and McLean’s IS Success Model. Paper presented at the 31st Annual Hawaii International Conference on System Sciences. Jones, S., Wilikens, M., Morris, P. & Masera, M. (2000). Trust Requirements in Ebusiness: A conceptual framework for understanding the needs and concerns of different stakeholders. Communications of ACM, 43(12). Jun, M. & Cai, S. (2001). The Key Determinants of Internet Banking Service Quality: A Content Analysis. International Journal of Bank Marketing, 19(7), 276-291. Kanungo, S., Duda, S. & Srinivas, Y. (1999). A Structured Model for Evaluating Information Systems Effectiveness. Systems Research and Behavioral Science, 16, 495-518. Kaplan, B. & Duchon, D. (1988). Combining Qualitative and Quantitative Methods in Information Systems Research: Case study. Management Information Systems Quarterly, 4, 571-581. 97 Kaplan, R.S. & Norton, D.P. (1992). The Balanced Scorecard-Measures that Drive Performance. Harward Business Review, Jan – Feb, 71-80. Keeney, R. (1999). The Value of Internet Commerce to the Customer. Management Science, 45(4). Kharif, O. (2000). Broken Mantras of the Cyberworld. Business Week Online. Lai , J. Y. (2006). Assessment of Employees’ Perceptions of Service Quality and satisfaction with e-business. Paper presented at the SIGMIS-CPR’06. Lai, V. S. & Wong, B. K. (2005). Business Types, E-Strategies, and Performance. Communication of the ACM, 48(5). Liao, Z. & Cheung, M. T. (2002). Internet-based Banking And Consumer Attitudes: An Emprical Study. Information and Management, 39, 283-295. Lindvall, M., Donzelli, P., Asgari, S. & Basili, V. (2005). Towards Reusable Measurement Patters. Paper presented at the 11th IEEE International Software Metrics Symposium. Liu, C. & Arnett, K. P. (2000). Exploring the factors associated with Web site success in the context of electronic commerce. Information and Management, 38, 23-33. Loiacono, E. T., Watson, R. T. & Goodhue, D. L. (2007). WebQual: An Instrument for Consumer Evaluation of Web Sites. International Journal of Electronic Commerce, 11(3), 51-87. Lu, X., Huang, L. & Heng, M. (2005). Critical Success Factors of interorganizational Information Systems- A case study of Cisco and Xiao Tong in China. Information and Management, 43, 395-408. 98 Malik, K. (2001). Information Systems Effectiveness: An Integrated Approach. Paper presented at the Change Management and the New Industrial Revolution, IEMC '01. Mata, F. J., Fuerst, W. L. & Barney, J. B. (1995). Information Technology and Sustained Competitive Advantage: A Resource-Based Analysis. MIS Quarterly, 19(4), 487-505. McCloskey D. (2003). Evaluating Electronic Commerce Acceptance with the Technology Acceptance Model. The Journal of Computer Information Systems, 44(2), 49-57. McKinsey (2008). IT’s unmet potential Retrieved December 10, 2008 Mendonça, M. G. & Basili, V. R. (2000). Validation of an Approach for Improving Existing Measurement Frameworks. IEEE Transactions on Software Engineering, 26(6). Minocha, S., Petre, M., Tzanidou, E., Dijk, G. V., Millard, N., Roberts, D. et al. (2006). Evaluating E-Commerce Environments: Approaches to Crossdisciplinary Investigation. Paper presented at the Conference on Human Factors in Computing Systems CHI 2006. Molla, A. (2001). E-Commerce Systems Success: An Attempt to Extend and Respecify the DeLone and McLean Model for Success. Journal of Electronic Commerce Research, 2(4). Muijs, D. (2004). Doing quantitative research in education with SPSS: Sage Publications. Mukhopadhyay, T., Kekre, S. & Kalathur, S. (1995). Business Value of Information Technology: A study of Electronic Data Interchange. MIS Quarterly, 137156. Munro, M. & Wheeler, B. (1980). Planning, Critical Success Factors, and Management’s Information Requirements. MIS Quarterly, 4(4), 27-38. 99 Myers , B. L., Kappelman, C. K. & Prybutok, V. R. (1997). A Comprehensive Model for Assessing the Quality and Productivity of the Information Systems Function: Toward a Theory for Information Systems Assessment. Information Resources Management Journal, 20(1), 1-15. Myers, M. D. (1997). Qualitative Research in Information Systems, . MIS Quarterly 21(2). Özkan, S. (2006). PB-ISAM: A Process-based Framework For Information Systems Effectiveness Assessment in Organizational Contexts. Middle East Technical University, Ankara. Özkan, S., Hackney, R. & Bilgen, S. (2007). Process based Information Systems Evaluation: towards the attributes of “PRISE”,. Journal of Enterprise Information Management, 20(6), 700-725. Park, H. & Baek, S. (2007). Measuring Service Quality of Online Bookstore with WebQual. Paper presented at the Human-Computer Interaction HCII. Pather, S., Erwin, G. & Remenyi, D. (2003). Measuring E-commerce Effectiveness: A Conceptual Model. Paper presented at the SAICSIT South African Institute. Peter, S., DeLone, W. H. & McLean, E. R. (2008). Measuring Information Systems Success: models, dimensions, measures, and interrelationships. European Journal of Information Systems, 17, 236-263. Peters, R., Janssen, M. & Engers, T. V. (2004). Measuring e-government impact: existing practices and shortcomings. Paper presented at the 6th International Conference on Electronic Commerce ICEC’04. Pires, G. & Aisbett, J. (2003). A Relationship Between Technology Adoption and Strategy in Business- To- Business Markets: The Case of E-Commerce. Industrial Marketing Management, 32, 291-300. 100 Pitt, L. F., Watson, R. T. & Kavan, C. B. (1995). Service quality: A measure of information systems effectiveness. MIS Quarterly, 19(2), 173-187. Poon, P. P. & Wagner, C. (2001). Critical Success Factors revisited: Success and Failure Cases of Information Systems for Senior Executives. Decision Support Systems, 30, 393-418. Powell, T. C. & Dent-Micallef, A. (1997). Information Technology as Competitive Advantage: The role of human, business and technology resources. Strategic Management Journal, 18(5), 375-405. Rockart, J. F. (1980) Tilte. In &: Vol. 1297-82: Center for Information Systems Research, Alfred P. Sloan School of Management. Sabherwal, R., Jeyaraj, A. & Chowa, C. (2006). Information System Success: Individual and Oranisational determinants. Management Science, 52(12), 1849-1864. Schaupp, C. L. (2005). A Conjoint Analysis of Online Consumer Satisfaction. Journal of Electronic Commerce Research, 6(2). Schonberg, E., Cofino, T., Hoch, R., Podlaseck, M. & Spraragen, S. (2000). Measuring success. Communications of ACM, 43(8). Seddon, P. B., Greaser, V. & Willcocks, L. P. (2002). Measuring Organizational IS Effectiveness: An Overview and Update of Senior Management Perspectives. The DATA BASE for Advances in Information Systems, 33(2). Seddon, P. B., Staples, S., Patnayakuni, R. & Bowtell, M. (1998). The IS Effectiveness Matrix: The Importance of Stakeholder and System in Measuring IS Success. Paper presented at the International Conference on Information Systems. Seddon, P. B., Staples, S., Patnayakuni, R. & Bowtell, M. (1999). Dimensions of Information Systems Success. Communications of the Association for Information Systems, 2(20). 101 Seethamraju, R. C. (2004). Measurement of user-perceived web quality. Paper presented at the 12th European Conference on Information Systems (ECIS). Serafeimidis, V. & Smithson, S. (2003). Information Systems Evaluation as an Organizational Institution- experience from a Case Study. Information Systems Journal, 13, 251-274. Siegfried, T., Grabow, B. & Drüke, H. (2003). Ten factors for success for Local Community E-Goverment. Paper presented at the EGOV 2003. Smith, H. A. & McKeen, J. D. (1996). Measuring IS: How Does Your Organization Rate? Database Advances, 27(1). Sobczak, A. & Berry, D. M. (2006). Distributed priority ranking of strategic preliminary requirements for management information systems in economic organizations. Information and Software technology, 49, 960-984. Stone, A. (2000). Crawling from the Dot-com Wreckage. Business Week Online. Straub, D. W., Hoffman, D. L., Weber, B. W. & Steinfield, C. (2002a). Measuring eCommerce in Net-enabled Organizations: An Introduction to the Special Issue. Information Systems Research, 13(2), 115-124. Straub, D. W., Hoffman, D. L., Weber, B. W. & Steinfield, C. (2002b). Toward NetMetrics for Net-Enhanced Organizations. Information Systems Research, 13(3), 227-238. Sward, D. (2006). Measuring the Business Value of Information Technology. Intel Press. Taniar, D. & Rahayu, J. W. (2004). Web Information Systems: IGI Publishing. The Economist (2008). Government offline: Why business succeeds on the web and government mostly fails. The Economist, from http://www.economist.com/opinion/displaystory.cfm?story_id=10689634 102 Torkzadeh, G., Chang, J. & Dhillon, G. (2003). Re-examining the Measurement Models of success for Internet Commerce. Information and Management, 41, 577-584. Torkzadeh, G. & Dhillon, G. (2002). Measuring Factors that Influence the Success of Internet Commerce. Information Systems Research, 13(2), 187-204. Vandenbosch, B. & Huff, S. L. (1997). Searching And Scanning: How Executives Obtain Information From Executive Information Systems. MIS Quarterly, 21, 81–108. Whitworth, B., Cheikna, S. & Whitworth, E. (2006). Assessing Emergent Business IT using the Web of System Performance. Paper presented at the The Fifth Wuhan International Conference on E-Business. Yin, R. K. (1993). Applications of the Case Study Research (Vol. 34): SAGE Publications. Zahedi, F. & Ashrafi, N. (1991). Software Reliability allocation Based on Structure, Utility, Price, and Cost. IEEE Transactions on Software Engineering, 17(4). Zhang, M. (2005). Information Technology Capability, Organizational Culture, and Export Performance. Washington State University, Washington. Zimmerman, G. C. (1996). Factors Influencing Community Bank Performance in Californaia. FRBS Economic Review, 1. 103 APPENDICES APPENDIX A: FULL LIST OF MEASURES Letters in codes refer to measures referred in the following studies: A: Kumar (1990) ; G: Palmer (2002); H: Agarwal and Venkatesh measures stated in Straub et al. (2002a) ; J: Torkzadeh and Dhillon (2003); K: Kim et. al measures stated in Straub et al. (2002b); L: Zhu and Kraemer measures in Straub et al. (2002b); M: McKinney et. al measures in Straub et al. (2002b); N: Devaraj et. al measures in Straub et al. (2002b); O: Eralp (2004); P: Joseph et al. (1999) ;Q: Liao and Cheung (2002) ;R: Jun and Cai (2001) Code N1/J21 N2 N3/M40 N4 N5 N6 N8 N9 N10 N11/J22 N12 N13 Measure kolay alışveriş imkanı Code N14 anlaşılabilir alışveriş etkileşimi sitede dolaşma kolaylığı N15 M1 alışverişte daha fazla kontrol imkanı effektif alışveriş imkanı alışveriş kararlarını kolaylaştırma yeterli fiyat, vergi bilgisi farklı alışveriş alternatifleri değerlendirme hızlı alışveriş alışverişiçin harcanan çaba ödeme opsiyonları M2 M3 M4 M5 M6 M7 M8 M9 müşteriyi hatırlama 104 Measure çeşitli ödeme opsiyonları online alışveriş güveni uygulanabilen bilgi verilen bilginin anlamlı olması verilen bilginin yapılan işlerle ilgili olması verilen bilginin açık olması verilen bilginin anlaşılır olması verilen bilginin okunurluğunun kolay olması sitenin güvenilirliği sitenin doğruluğu sitenin başkaları tarafından benimesenmesi Code M10 M11 M12 M13 M16 M18 M19 M20 M21 M22 M23 M24 M25 M26 M27 M28 M29 M30 M31 M32 M33 M34 M35 M36 M37 M38 M39 M41 L2 L3 L6 Measure sitede sunulan bilginin, servislerin yeterli olması sitede sunulan bilginin, servislerin tam olması sitede sunulan bilginin, servislerin gerekli konuları içermesi ürün ve servislerin çeşitliliği Code L7 L8 L9 L10 L11 bilgilendirici verilen bilginin kalitesi(kullanılışlı, anlaşılabilir, güvenilir) verilen bilginin tatmin edici olması tüm web sitesinin tatmin ediciliği kullanıcıyla etkileşim içinde olması/ cevap vermesi hızlı yüklenebilmesi basit planının olması L12 L13 L14 L15 L16 L17 L18 kolay kullanılabilir olması L19 iyi organize edilmiş olması L20 açık bir tasarımının olması K1 görsel olarak etkileyici olması eğlendirici olması K2 K3 ilginç olması K4 yeterli link olması link açıklamalarının açık olması ileri/geri gitmenin kolay olması az sayıda klik gerektirmesi ürün listesi oluşturabilme ürün listesi değiştirebilme kişiye özel ürün yaratabilme değişik ürün özellikleri seçebilme kolay ulaşım kullanılabilirlik sitenin tatmin edici olması arama yapabilme yeteneği ürün yorumları hesap yönetimi 105 Measure sipariş takibi imkanı müşterinin tekrar gelmesi güvenlik konfigurasyon müşteri kayıt online tavsiye içerik kişileştirme gerçek zamanlı destek online ürün tedariği elektronik veri değişimi tedarikçi sanal topluluğu diğer tedarikçilerle entegre bilgi sistemleri lojistik stok verisi paylaşımı her ortamda hızlı yüklenebilme ilk sayfanın yüklenme hızı dışardan giriş yapmaya çalışanlara karşı koruma sistem güvenliği için kesin kurallar ürün ve servisler için doğru bilgi K5/N7/L1 ögeler hakkında bilgi K6 teslimat şekli ve tarihi için opsiyonlar K7 sipariş işlemi kullanışlı K8 şekil ve görünüş stili K9 yerleşimin ekrana uygunluğu K10 müşteriyle farklı irtibat yolları K11 duyuruların olması K12 ürün seçimi J1 kaliteli ürün çeşitliliği J2 ürün çeşitliliği J3 ürün ulaşılabilirliği J4 kolay karşılaştırma J5 geniş ürün yelpazesi J6 Code J8 J9 J10 J11 J12 J13 J14 J15 J16 J17 J18 J19 J20 J21 J22 J23 J24 J25 J26 J27 J28 J29 J30 J31 J32 J33 H2 H3 H4/L4 H5 H6 H7 Measure izinsiz kullanım Code H8 kişisel bilgilerin kötüye kullanılması kredi kartı bilgilerin kötüye kullanılması kişisel bilginin başkalarıyla paylaşımı H9 H10 H11 tedarikçi meşruluğu H12 satıcı meşruluğu H13 tedarikçi güvenilirliği H14 e-ticaret güvenliği H15 yanlış ürün H16 nakliyat hataları H17 e-ticaret kirliliği önleme H18 e-ticaret çevreye etki G1 e-ticaret çevreye zarar verme G2 kolay alışveriş G3 alişveriş çabası G4 sırada zaman G5 seçme zamanı G6 kişilerle tartışma G7 ödeme zamamnı G8 zaman kısıtı G9 geri verme garantisi G10 geri verme kolaylığı G11 satış sonrası hizmet G12 vergi G13 ürün fiyatı G14 ürün değeri O1 farklı araçların kullanımı O2 içeriğin derinliği O3 yeni bilgi O4 sitenin açık amaçları olması O5 websitesi yapısı O6 ilerleme hakkında geribildirim O7 106 Measure satışı teşfik online grup kişileştirilme en yeni trend ve bilgiyi sağlama sitenin iddialı-gösterişli olması bir hikaye oluşturması güvenilir kişileri tutabilme etkileşilen bilgi ilerlemesi kontrolü kullanılırlık tasarım edinilen izlenim kullanıcı tatmini müşterinin tekrargelme olasılığı kullanım sıklığı erişim hızı gösterme hızı sitenin organizssyonu operasyonların sırası site düzeni yerleşimi miktar çeştlilik kişileştirme etkleşimi sık sorulan sorular geribildirim rapor edilen problem sayısı çözülen problem sayısı ortalama problem çözme süresi sistem hata sayısı çalışma saati miktarı personel sayısı tecrübe yılı Code O8 O9 O10 O11 O12 O13 O14 O15 O16 O17 O18 O19 O20 O21 O22 P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 P11 P12 P13 P14 P15 P16 P17 Measure edinilen personel sayısı Code Q1 kaybedilen personel sayısı Q2 database tablo sayısı Q3 database record sayısı Q4 database kelime sayısı Q5 componnet/unit sayısı Q6 eklenen componnet/unit sayısı değiştirilen componnet/unit sayısı silinen componnet/unit sayısı arayüz sayısı Q7 Q8 Q9 Q10 silinen arayüz sayısı Q11 eklenen arayüz sayısı Q12 değiştirilen arayüz sayısı Q13 kod satır sayısı Q14 kaynak kodu dosyası sayısı Q15 işlem doğruluğu işlemlerin yapılması garantisi Q16 Q17 doğru işlem bilgisi Q18 kolay kullanabilirlik R1 müşteri geri bildirimi 7 gün/24 saat işlem yapabilme 24 saat içinde şikayet ceveplama beklenmeyen sorulara cevap verme R2 R3 R4 R5 profesyonel görünüm hesap oluşturma ve işlem yapma çabukluğu işlem sırasında bekleme menü yeterliliği R6 R7 R8 R9 sistemin kullanma eğitimi R10 yeni kullanıcılara yol gösterme engelliler için özel işlemler R11 R12 kişiye özel bir ortam R13 müşteriyi ismiyle karşılama R14 107 Measure heran erişim herhangi yerden erişim çeşitli işlemlere erişim zevkli stressiz kolay izlenebilir talimat basit operasyonel işlemler basit donanım ve yazılım ihtiyaçları geniş yardım menüleri verimli website tasarımı işlemleri teyit öncesinde kontrol edebilme işlemleri istenen hızda yapma yetkili erişim müşteri bilgisi gizliliği yüksek meblalı işlemlerde kısıtlamalar sıkı güvenlik taahhüdü geleneksel bankacılıktan fazla hız hızlı cevap verme ürün çeşitliliği ürün özellikleri doğru servis yardım vaadedilen servisin sunulması doğru kayıt email ulaşımı telefon ulaşımı hızlı problem çözme açık cevaplama uygun servis müşteriye önemli bilgilerin ulaştırılması işlemlerin kullanılır durumda olması problem çözmeye izin vermesi Code R15 R16 R17 R18 R19 R20 R21 R22 R23 R24 R25 R26 R27 R28 R29 R30 R31 R32 Measure banka servislerine inanç Code R33 iyi itibar müşteri servislerinde sürekli iyileştirme banka ürünlerinde sürekli iyileştirme online sistemde sürekli iyileştirme ürün ve servisler hakkında bilgi yenilenen bilgi müşterinin ihtiyacı olan diğer bilgiler doğru işlemler website çekiciliği R34 A1 A2 A3 A4 A5 A6 A7 A8 A9 A10 A11 A12 A13 arayüzdeki hatalar içerikteki hatalar gizlilik işlem yapma güzenliği kolay kullanım kolay giriş hızlı cevap verme erişim 108 Measure ihtiyaç olunan işelme çeşitleri menüler arasıdna kolay dolaşma doğru bilgi zamanlı ve geçerli bilgi iç kontrol yeterli bilgi uygun bilgi sistem güvenliği ve felaket önleme donanım performansı sistem kullanımı kolay kullanabilirlik sistemin kullanıcılar ve işlerine olan etkisi operasyon masrafı program kalitesi sistemin uyumluluğu ve organizasyon üzerindeki etkisi APPENDIX B: INTERVIEW QUESTIONS IS/IT specialist Đsim: Tel: e-mail: Bu çalışma web-based IS’ lerin effectivenss-etkililik-tesirlilik/başarılarını ölçmek için tasarlanmıştır. WIS’in ne kadar etkili olduğunu veya başarılı olduğunu araştırır. Konu Hakkında 1. Đş tanımınız nedir? 2. Sorumluluklarınız ve bunların önem sırası nedir? Đş ve Organizsayon Hakkında 3. Şirkette kaç kişi çalışıyor? IS/IT bölümünde kaç kişi çalışıyor? 4. Şirketin büyüklüğü? (Ciro etc..) 5. Kime rapor veriyorsunuz? 6. Kimler size rapor veriyor? 7. Kimlere servis sağlıyorsunuz? 8. Ne tür isteklere cevap veriyorsunuz? (e.g. system development, maintenance etc..) 9. Hangi farklı IS’ler var şirketinizde çalışan? 10. WIS kendinizmi ürettiniz, hazır mı aldınız? 11. Şirketinizde kullanılan farklı WIS’leri tanıtır mısınız? Model Doğrulama 12. WIS kullanıcıları kimler? 109 13. WIS hangi fonksiyonları sağlıyor? 14. WIS başarısı/etkililik denince ne anlıyorsunuz? 15. WIS başarısı/ etkililik nasıl ölçüyorsunuz? Bir tool kullanıyormusunuz? 16. Modeldeki WIS ilişkileri mevcut mu şirkette?. BAşka ilişkiler var mı? 17. Modeldeki WIS ilişkileri ne içeriyor bu şirket için? 18. Modeldeki herbir WIS ilişkisi için hangi fonksiyonlar sağlanıyor WIS tarafından? Değerlendirme Ölçütlerinin Toplanması 19. WIS ilişkilerinin önem ağırlıkları nedir? 20. WIS ilişkileri için şirketin ulaşmak istediği amaçlar var mı? 21. Her bir amacın ulaşılıp ulaşılmadığını anlamak için hangi sorular sorulmalı? 22. Bu soruları cevaplamak için neler ölçülmeli? Yönetim Đsim: Tel: e-mail: Bu çalışma web-based IS’ lerin effectivenss-etkililik, tesirlik/başarılarını ölçmek için tasarlanmıştır. WIS’in ne kadar effective olduğu, yapması gerekenleri ne kadar yaptığını araştırır. Organizasyon Hakkında 1. Vizyon bir şirketin gelecekte kendini nerede gördüğüdür. Şirket vizyonunuz nedir? 2. Şirketinizin gelecekte ulaşmak istediği hedefi, misyonu nedir? 3. Kritik başarı faktörleri başarıldığında iş sürecinin başarısından emin olunmasını sağlayan kriterlerdir. Şirketin başarılı olması için ulaşmanız gereken kritik başarı faktörleri nelerdir? (Ör: Etkin ve Verimli Đç Üretim, 110 Doğru ve Zamanında Sevkıyat, Müşteri sayısı, çalışanların motivasyonu, çalışsanların gelişimi, kişiselleştirilmiş servis sağlamak, inovasyon, güvenlik vs.) Model Doğrulama 4. Bu başarı faktörleri arasında önem sırası var mı? Hangisi daha öncelikli? 5. Bu başarı faktörlerini önem sırasına gore sıralayınız. 6. Bu başarı faktörleri hangi sıklıkla değişiyor? WIS bir bilgi sistemidir. Web üzerinden çalışır ve şirket içinde ve dışında kullanıcıları olabilir, kullanıcılara web üzerinden servis/ürün sunar. 7. Siz hangi WIS fonksiyonlarını kullanıyorsunuz? Ne amaçla? 8. Sizce başarılı WIS kritik başarı faktörlere ulaşmanıza yardımcı mıdır? Başarısız WIS, başarı faktörlerine ulaşmanızı engelleyebilir mi? Başarısız WIS şirketin başarısını etkiler mi? 9. WIS şirketiniz için ne kadar önemli? Eğer ortadan kaldırsak etkisi ne olur ? 10. WIS , diğer şirketlere göre şirketinize rekabet avantajı sağlıyor mu? Nasıl? Hesaplama ağırlıklarının Toplanması 11. Herbir ilişki için: herbir ilişkinin başarısı/tesirliliği –etkililiği herbir CSF’i nasıl etkiler? Buna gore sıralayınız. CSF1 CSF2 ..... CSF1 CSF importance(ri) R1 Kullanıcı-WIS R2 WIS-WIS R3 WIS-organizasyon 111 CSF2 CSF3 CSF4 CSF5 wij: (10 üzerinden; 10-maksimum, 0- hiç önemli değil) Bu ilişkinin başarısı (Rj) bu CSF’in (CSFi) gerçekleşmesini ne kadar etkiler? Bu ilişkinin başarısı (Rj) bu CSF’in (CSFi) gerçekleşmesini için ne kadar önemli? 112 APPENDIX C: SEWISS CALCULATION n : number of CSFs (obtained from management) CSFi : critical success factor i (i=1,…,n) (obtained from management) ri : importance ranking for CSFi (i=1,…,n) (obtained from management) Reli : WIS relationship i (i=1,2,3) (obtained from IT/IS Specialist) Rj : success for Relj (j=1,2,3) (average of the stakeholder responses to questionnaire) wij : importance weight of Relj for CSFi (i=1,..,n; j=1,2,3) (obtained from management) mj : the number of measures for Relj (obtained from IT/IS Specialist) Mjk : value of measure k for Relj (k=1,..,mj ; j=1,2,3) (stakeholder answers to each questionnaire questions) s : number of stakeholders participated in the assessment SEWISS success calculation: S= n 1 n 3 wmax Rmax ∑ ri 3 ∑r ∑ i =1 i j =1 R j wij i =1 1 s 1 Rj = ∑ s i =1 m j mj ∑M k =1 jk 113 114 115 APPENDIX D: SHORT LIST OF MEASURES Organizations Code Measures A D E F G N3/M40 sitede dolaşma kolaylığı N8 yeterli fiyat, vergi bilgisi N9 farklı ürünleri karşılaştırma imkanı N10 hızlı alışveriş N13 müşteriyi hatırlama N15 online alışveriş güveni X X M5 verilen bilginin anlaşılır olması X X M7 sistemin güvenilirliği X M10 sunulan bilgilerin ve servislerin yeterli olması M13 ürün ve servislerin çeşitliliği X X X M21 kullanıcıyla etkileşim içinde olması/cevap vermesi X M22 hızlı yüklenebilmesi M24 kolay kullanılabilir olması M27 görsel olarak etkileyici olması M28 eğlendirici olması M30 yeterli link olması M31 link açıklamalarının açık olması M32 ileri/geri gitmenin kolay olması M33 az sayıda klik gerektirmesi M34 ürün listesi oluşturabilme X M35 ürün listesi değiştirebilme X X L2 arama yapabilme yeteneği X X X L3 ürün yorumları X X L7 sipariş takibi imkanı X L8 müşterinin tekrar gelmesi X L9 güvenlik L13 içerik kişiselleştirme X X X X X X X X X X X X X X X X X X X X X X X X X X X 116 Organizations Code Measures A D E F L14 gerçek zamanlı destek L15 online ürün tedariği L16 elektronik veri değişimi X X X L18 diğer tedarikçilerle entegre bilgi sistemleri X L20 stok verisi paylaşımı K1 her ortamda hızlı yüklenebilme X X K2 ilk sayfanın yüklenme hızı X K3 dışardan giriş yapmaya çalışanlara karşı koruma R5 vaadedilen servisin sunulması R9 hızlı problem çözme R17 müşteri servislerinde sürekli iyileştirme X R19 online sistemde sürekli iyileştirme X K5/N7/L1 ürün ve servisler için doğru bilgi K7 teslimat şekli ve tarihi için opsiyonlar X K12 duyuruların olması X J6 geniş ürün yelpazesi J12 tedarikçi meşruluğu J13 satıcı meşruluğu J29 geri verme kolaylığı J30 satış sonrası hizmet J32 ürün fiyatı H11 en yeni trend ve bilgiyi sağlama G3 kullanım sıklığı G14 geribildirim O1 rapor edilen problem sayısı O2 çözülen problem sayısı O3 ortalama problem çözme süresi O4 sistem hata sayısı O5 aynı müşterinin geri gelme oranı X O7 tecrübe yılı X G X X X X Y X X X X X X X X X X X X X X X 117 X Organizations Code Measures A D E F P1 işlem doğruluğu X X P3 doğru işlem bilgisi X P6 7 gün/24 saat işlem yapabilme P9 profesyonel görünüm P10 hesap oluşturma ve işlem yapma çabukluğu P12 menü yeterliliği P13 sistemin kullanma eğitimi P14 yeni kullanıcılara yol gösterme P15 engelliler için özel işlemler P16 işlem hızı P17 müşteriyi ismiyle karşılama Q4 zevkli olması Q7 basit operasyonel işlemler Q8 basit donanım ve yazılım ihtiyaçları Q9 geniş yardım menüleri Q11 işlemleri teyit öncesinde kontrol edebilme Q13 yetkili erişim X Q14 müşteri bilgisi gizliliği X Q15 yüksek meblalı işlemlerde kısıtlamalar Q16 sıkı güvenlik taahhüdü X X X A7 donanım performansı X G X X X X X X X X X X X 118 APPENDIX E: QUESTIONNAIRES Organization A 119 120 Organization D 121 122 Organization E 123 124 Organization F 125 126 Organization G 127 128 VITAE Gül Tokdemir was born in Ankara, on April 17, 1973. She received her B.S. degree in Aeronautical Engineering from the Middle East Technical University in 1995 ranking as the first student. She has completed her Master’s degree in Georgia Institute of Technology, Aerospace Engineering Department, where she was the member of the winning team in American Helicopter Society Design Competition in 1997. She worked for Aselsan MGEO Division until 2001. She has been teaching at Atilim University, in Computer Engineering Department since 2001. Publications: Tanik M., Dogru A., Tokdemir G. (2003) Interface Engineering: A ComponentOriented Approach, IDPT Conference, Austin, USA, December 2003. Sezer N., Tokdemir G., Eryılmaz M., Emiroğlu B. G. (2003) Computer & Information Systems Laboratory Manual, Atilim University Publications. Çağıltay N.E., Tokdemir G. (2004) The Role of Learning styles in engineering education, National Engineering Congress, May, 20-21 2004, Izmir, Turkey. Cagiltay N.E., Selbes C. F., Tokdemir G., Turhan C. (2006) C Dersi: Programlamaya Giriş,ISBN-978-9944-62-337-7,SeçkinYayıncılık. 129 Kılıç Ö., Çağıltay N.E., Tokdemir G. (2006) Yazılım Mühendisliği diyagramlarının kullanımındaki bilişsel ve davranışsal özellikler, II. Ulusal Mühendislik Kongresi, Zonguldak, Bildiri ve Poster Kitabı, Sayfa 349-355, Türkiye, 2006. Tokdemir G., Cagiltay N.E. (2007) Using Learning Style Theory in Remote Laboratory Applications, The 18th Annual IEEE International Symposium on Personal, Indoor and Mobile Radio Communications (PIMRC'07), Athens, Greece. Selbes C.F., Cagiltay N.E., Tokdemir, G., Turhan C. (2008) C Dersi: Çözümlü Problem Kitabı , Seçkin Yayıncılık. Tokdemir G., Bilgen S (2008) Remote Lab Effectiveness Assessment Model, in Proceedings of the IEEE OPTIM’08 Conference, Brasov, Romania. Tokdemir G., Bilgen S. (2008) Internet Banking Success Assessment based on Strategy. E-society 2009 Conference, accepted. Tokdemir G., Bilgen S. (2008) Strategy-based Evaluation of Web-based Information Systems, WEBIST Conference 2009, accepted. Tokdemir G., Bilgen S. (2008) Web-Based Information System Assessment Framework: SEWISS, submitted to European Journal of Information Systems-EJIS, in progress. Tokdemir G., Bilgen S. (2009) Assessing Remote Lab Success through SEWISS Framework: A Case Study, in preparation. 130