Uncorrected Proof To appear as: Ley, T.; Albert, D.: Quality Criteria for Competency Assignments: Examples from a Project Management Case Study. Proceedings of Informatik 2004, Gesellschaft für Informatik, Ulm, 2004 Quality Criteria for Competency Assignments: Examples from a Project Management Case Study Tobias Ley1, Dietrich Albert2 1 Know-Center Graz tley@know-center.at 2 Cognitive Science Section Graz University dietrich.albert@uni-graz.at Abstract: We present several quality criteria for skills management initiatives based on the competence-performance approach [Kor97]. The criteria can be measured empirically, they help to determine the suitability of competency assignments in an organisation. The approach is illustrated by a case study. 1 The need for Quality Criteria in Skills Management Whenever databases are being used in skills management initiatives, they usually contain assignments of competencies to people or competencies to jobs. These assignments are established through employee profiles which provide information on the competencies possessed by a certain employee, or job profiles which detail out the competency requirements of a certain job. In this paper, we define competencies as personal characteristics (knowledge, skills, abilities) of employees which are relatively stable across different situations (see also [LA03a]). Competencies can be described in terms of distinguishable elements of underlying capacities or potentials which allow job incumbents to act competently in certain situations [Ber00]. Employees dynamically combine these elements according to the requirements of the situation in a self-organising process [EH99]. From a psychological perspective, competencies constitute theoretical constructs which can not be directly assessed or observed. Instead, they have to be inferred from some form of observable performance. For example, whether a project manager has the capacity to communicate effectively with customers, whether she is proficient at planning a project, or whether she is an expert in eLearning Systems, can not be directly observed. We would have to observe her performance in several situations (or ask someone who has done so), in order to arrive at an assessment of these competencies for that particular person. In order to assess the suitability of these constructs, quality criteria are needed that measure the degree of correspondence of the theoretical constructs to the “real world”. For example, sometimes skills are being rated by assigning a person’s proficiency in a certain skill (or skill requirements in a certain job) a value on some scale (e.g. beginner, proficient, expert etc.). The meaning of these ratings remains unclear: the assignment is usually not related to performance in the job and the rating is dependent on the frame of reference of the one providing the rating. Therefore, the assignments are of limited value. Psychological research has established quality criteria for the use and assessment of constructs that can be of use in the domain of skills management: objectivity, reliability and validity. The degree to which these criteria have been attained, is dependent on a multitude of conditions, for example whether the labels used for describing competencies are shared and understood equally well across the organisation, or whether the procedure employed for assigning competencies produces reliable results. When these criteria are operationalised in a specific case, they can be made measurable and thereby provide information on the appropriateness of the assignments and the skills management methodology as a whole. From a theoretical perspective, several requirements for establishing these quality criteria seem to be important. (a) The skills management initiative should be based on a theoretical model that relates employee competencies to performance outcomes. A framework for such a model is presented in section 2. (b) The operationalisation of the quality criteria should take into account the purpose of the skills management initiative in the organisation. Additionally, practical considerations demand that (c) assessing the quality criteria makes use of existing data and does not require additional effort for the organisation. These last two issues will be addressed in section 3. 2 A Competence-performance Approach for Skills Management Korossy [Kor97] has introduced a competence-performance approach. His framework and related approaches (e.g. [DF99]) have subsequently been used for skills management (e.g. [LA03b], [AZ03]) and for other applications in the learning domain (see [AL99]). The approach provides the basis for the quality criteria that will be introduced in the next section. A case study briefly illustrates the use of the framework (for a more thorough discussion see [LA03a]). It should be noted that the case study makes use of a simplified version of Korossy’s theory. The case study was conducted at a project based R&D institute with about 40 employees. The institute conducts pre-competitive industry research projects in the fields of knowledge management and eLearning. The purpose of the skills management initiative was to support the qualification process of project managers (i.e. “which competencies does an effective project manager need in the typical situations she encounters?”) and to show specific fields of expertise of individual project managers. The following three steps were taken. Defining Performances and Competencies. The competence-performance approach requires that performance outcomes be specified which can be observed and evaluated to some standards of performance. In the case study, documents produced by the project managers were used as performance outcomes. These give good indication of past work requirements and also reflect fields of expertise employees have developed. Competencies were then derived in interviews with 7 employees in a repertory grid type interview (see [Jan90]) to extract competencies used for creating the documents. According to the skills management purpose, the resulting competencies were grouped into two categories: general job competencies and domain specific competencies. Providing Document-Competency Assignments. The employees were then asked to indicate which competency they had used for creating a certain document. The result was a matrix which – in the terminology of [Kor97] – provides the interpretation function which maps each document to the set of competencies required for producing it. Deriving Competence-performance Structures. From the interpretation function, prerequisite relations on the competence level can be derived. Figure 1 shows an example for such a prerequisite structure which outlines prerequisites for general job competencies. The figure gives a condensed version of the structure by showing categories of the single competencies named by the respondents. Prerequisite relations are given by the lines between the boxes, i.e. competency 22 is a prerequisite for competency 10, and both are prerequisites for competency 11. Only direct relations have been outlined, others can be derived by transitivity. From the prerequisite relations, a “competence space” can be derived, which outlines development paths for project managers in the particular company [LA03a]. Additionally, in a competence space one can identify the situations in which a particular employee is expected to perform well (via the representation function, [Kor97]). 12 Targeted information production 14 Logical argumentation, reasoning 11 Information synthesis and structuring 15 Communication with others 18 Project management 16 Analysing requirements of target groups 10 Information search, research skills 22 Judgment, Critical Thinking Figure 1: Prerequisite relations on the competence level for project managers 3 Empirical Quality Criteria for Skills Management: Some Examples The document-competency assignments as well as the prerequisite relations between the competencies that have been outlined in the previous section are hypothetical in nature. The purpose of establishing quality criteria is to find out whether the assignments and relations are appropriate for the organisation where they have been established. The following section gives examples for measures that can support a person in charge of a skills management initiative in determining whether the results found correspond to the “true” affairs in the organisation. They also help to determine which part of the results should be subject to revision because the quality does not reach a certain standard. 3.1 Reliability The reliability of the document-competency assignment determines how consistent the assignment is. In our case, we had employees repeat the assignment after six months and derived a “test-retest” reliability coefficient for each of the 7 employees (see Table 1). The coefficient λr [GK54] measures how well the second assignments can be predicted from the first one. It is clear from the table that some of the employees made more consistent assignments than others. The reasons for these inconsistencies can then be explored further. Subject 1 2 3 4 5 6 7 Total N 90 144 112 102 112 160 55 775 λr 0.136 0.500 0.558 0.231 0.125 0.641 0.659 0.494 z 1.03 5.47 5.93 1.78 1.15 9.72 5.18 13.95 sig n.s. *** *** * n.s. *** *** *** Table 2: Reliability of the assignments (λr), z-value and level of significance 3.2 Validity We take a view of validity that corresponds to classical test theory [Ros96]. Specifically, we have used a cross validation technique to determine how well the documentcompetency assignments correspond to a “true score”. This was achieved by examining how well the assignments correspond with a-priori document categories. Each of the documents used in the case study originated from a certain business area (“Knowledge Management” or “eLearning”) and each was also categorised according to a certain document type (i.e. “formal project documentation” or “project result”). A validity coefficient can be obtained by determining whether a certain document is classified to the correct a-priori category using its feature vector of competency assignments. Category “Business Area” Domain specific competencies General job competencies Category “Document Type” Domain specific competencies General job competencies κ N Po Pe z sig 39 39 0.872 0.692 0.503 0.503 0.742 0.381 4.691 2.387 *** ** 47 47 0.660 0.660 0.600 0.600 0.149 0.149 1.416 1.416 + + Table 3: Cohen’s κ, Z-Value and level of significance for a “leave-one-out” cross validation A “leave one out” cross validation method was used [Bor93]. Distance to a cluster was measured by Euclidian distance of the feature vector of a document to the centroid of the a-priori clusters made up of the other N-1 documents. Cohen’s κ (Cohen (1960), cited in [Bor93]) measures the quality of the classification. Table 2 gives the results. It is clear that the “business area” classification could be well replicated, especially by using domain specific competencies. This was not the case for the “document type” classification which resulted in little more than chance agreement. Apparently, similar competencies are being used for the different document types. 4 Conclusion and Outlook We have presented some methods for assessing the quality of competence-performance assignments in a skills management initiative based on the competence-performance approach. Examples for reliability and validity measures were presented. These measures can be the basis for evaluating the quality of the skills management methodology and for making decisions about necessary adjustments. References [AZ03] Alan, Y.; Zelewski, S.: Ontologiebasierte Wissensräume. In (Reimer, U.; Abecker, A.; Staab, S.; Stumme, G., Eds.): WM 2003: Professionelles Wissensmanagement Erfahrungen und Visionen, Gesellschaft für Informatik, Bonn, 2003; p. 5-8. [AL99] Albert, D.; Lukas, J., Ed.: Knowledge Spaces: Theories, Empirical Research, and Applications. Lawrence Erlbaum, Mahwah, 1999. [Ber00] Bergmann, B.: Arbeitsimmanente Kompetenzentwicklung. In (Bergmann, B.; Fritsch, A.; Göpfert, P.; Richter, F.; Wardanjan, B.; Wilczek, S., Eds.): Kompetenzentwicklung und Berufsarbeit, Waxmann, Münster, 2000; p. 11-39. [Bor93] Bortz, J.: Statistik für Sozialwissenschaftler. Springer, Heidelberg, 1993. [DF99] Doignon, J.; Falmagne, J.: Knowledge Spaces. Springer, Heidelberg, 1999. [EH99] Erpenbeck, J.; Heyse, V.: Die Kompetenzbiographie: Strategien der Kompetenzentwicklung durch selbstorganisiertes Lernen und multimediale Kommunikation. Waxmann, Münster, 1999. [GK54] Goodman, L. A.; Kruskal, W. H.: Measures of Association for Cross Classifications; In: Journal of the American Statistical Association, 49, 1954; p. 732-764. [Jan90] Jankowicz, A. D.: Applications of Personal Construct Psychology in Business Practice. In (Neimeyer, G. J.; Neimeyer, R. A., Eds.): Advances in Personal Construct Psychology, JAI Press, Greenwich, 1990; p. 257-287. [Kor97] Korossy, K.: Extending the theory of knowledge spaces: A competence-performance approach; In: Zeitschrift für Psychologie, 205, 1997; p. 53-82. [LA03a] Ley, T.; Albert, D.: Identifying Employee Competencies in Dynamic Work Domains: Methodological Considerations and a Case Study; In: Journal of Universal Computer Science, 9, 12, 2003; p. 1500-1518. [LA03b] Ley, T.; Albert, D.: Kompetenzmanagement als formalisierbare Abbildung von Wissen und Handeln für das Personalwesen; In: Wirtschaftspsychologie, 5, 3, 2003; p. 86-93. [Ros96] Rost, J.: Lehrbuch Testtheorie, Testkonstruktion. Hans Huber, Bern, 1996. View publication stats