Ms. Trudi Cooper and Dr. Terence Love
The paper explores how portfolio-based assessment, particularly its online forms, can address some of the challenges presented by new trends in tertiary education in fast changing subject areas such as Information Systems.
Keywords: Information Systems, Assessment, Portfolios, Online
The problem: How can university assessment methods be developed to meet new challenges and address contemporary assessment problems in the field of Information Systems?
Six trends in tertiary education indicate that university assessment methods need to be reviewed:
Outcomes-based assessment: Increased focus on outcomes-based assessment changes the emphasis towards ‘demonstrating the application of a body of knowledge’ rather than ‘knowing about’ a body of knowledge. Different forms of assessment are needed for educational processes that focus on the skills and the practical application of knowledge in fast changing situations. Traditional assessment is less appropriate for this purpose because it was originally developed for assessment in:
The trend towards outcomes-based assessment is found throughout education: in the university systems (Graduate Attributes) (DETYA, 1999, p 16, DETYA, 2001, p 32) , in the school education system (Curriculum Council, 1998) , and in the professions (see, for example, Australian Computer Society, 2001a, Australian Nursing Council Inc., 2000) .
Graduate attributes: By definition, graduate attributes are skills that students have at graduation. Assessment modalities for graduate attributes cannot easily be integrated with traditional unit assessment processes where courses are modularised and where modules can be taken in a variety of different sequences. There is tension between existing university assessment structures and assessment structures needed for the assessment of graduate attributes because generic skills assessment crosses unit boundaries and existing assessment processes are focused around topic-based units.
High rates of change: In fast changing professional disciplines, such as IS, it is unsatisfactory to assess ability through the demonstration of detailed knowledge of content likely to be obsolete soon. Professional institutions such as the Australian Computer Society (2001a) regard it is more important for students and practitioners to:
Assessment of these skills is not usually regarded as a strength of traditional assessment modalities, which assess students’ grasp of pre-defined topic information rather than their abilities to develop their knowledge and skills.
Employability: In fields of study in which knowledge becomes quickly obsolete, one-time mastery of a body of knowledge does not imply ongoing employability or continued professional competence (Australian Computer Society, 2001a) . There is tension between education appropriate to a fast-changing knowledge situation, and the expectation that students completing a course should be immediately employable without further training. Pedagogically, it is complex but feasible to create educational processes that fulfil all these aims; helping students acquire a deep understanding of fundamental concepts, skills at life-long learning and continued professional development, and skills of immediate use in employment. Combining these educational processes presents assessment problems. These assessment problems are more easily resolved by portfolio-based assessment because the role of a portfolio as a ‘container’ allows the inclusion of different forms of evidence.
Plagiarism: There is increasing concern about levels of student plagiarism, especially in fields such as IS in which students have high levels of computer and Internet skills (Kearns, 2000) . Portfolio-based assessment can assist in this situation by facilitating the triangulation of assessment. Traditional forms of assessment that do not co-locate individual assessments from different units do not offer a ready means for examiners to correlate standards across an individual student’s multiple assessment items.
Improving Assessment Quality: Assessment processes are becoming increasingly subject to quality assessment and quality assurance processes (DETYA, 1999, Kemp, 1999) . Improving the quality of assessment depends on a direct and transparent relationship between the aims of education and practicalities of assessment. In addition, improving assessment quality requires sound moderation between and across units, courses and institutions.
Minimising problems associated with equity issues: The shift to tertiary mass education has resulted in culturally, educationally, and socially more diverse student populations (DETYA, 1999) . This significant reduction in homogeneity increases the potential for equity issues in assessment. Minimising equity problems is an important aspect of choosing assessment methods. Portfolio-based assessment can resolve many of these equity issues (Cooper, 1999) .
Disintegration of education processes and assessment: Trends towards commercialisation, modularisation and globalisation in education have led to courses and units being delivered by partner institutions, and through flexible delivery modes including external study or Internet-based study, and with marking processes often outsourced to junior staff or postgraduate students (Technology and Industry Advisory Council, 2000, Bradley, 2000) . Each of these factors reinforces the problems of traditional assessment processes and reduces the possibility for appropriate, effective, efficient, equitable, high quality assessment suited to professional education in fast-changing knowledge areas.
This paper argues that the above factors point to the need to review assessment processes. It uses Cooper’s (1999, 1997) six-step model of portfolio-based education to show how portfolio-based assessment offers benefits over traditional modes of assessment in addressing the assessment challenges raised by these factors. The paper explores the role of professional accreditation in IS assessment and then concludes by discussing how online portfolio-based assessment can address the above assessment challenges.
Portfolio-based assessment is now well established as a valuable assessment tool (see, for example, Barrett, 2000c, Biggs and Tang, 1997, Cooper, 1997, Education Department of Western Australia, 2000a, Education Department of Western Australia, 2000b) . Portfolio-based assessment is beneficial pedagogically because the format can encompass evidence from a wide variety of sources (Education Department of Western Australia, 2000b) , it can help educators overcome many assessment difficulties, especially in relation to equity and moderation (Cooper, 1999, Cooper and Love, 2000) , it provides a ‘richer picture’ of the student (Barrett, 2000c) , portfolio-building actively involves students in the learning process (Bowie et al., 2000) , and is valuable in supporting lifelong learning (ANTA/AVCC, 2000) . For IS and IT related disciplines, online portfolios are especially appropriate (Bowie et al., 2000, Barrett, 2000c) .
Cooper’s (Cooper, 1999, Cooper, 1997) definition of a portfolio is:
Portfolio: Collection of evidence that demonstrates skills, achievements, learning or competencies.
Cooper’s ‘six step’ portfolio building process is well established as a basis for planning and teaching outcome-based professional assessment in tertiary courses (see, for example, Cooper, 1997, Cooper, 1999, Cooper and Emden, 2000, Cooper et al., 1999, Cooper and Love, 2000) . The six-step process ensures that portfolio-based assessment is applied in a purposive way that relates transparently and directly to predefined pedagogic processes and outcomes.
The portfolio-building process consists of:
Step 1: Identify the areas of skills that the student is intended to develop.
Step 2: From these skill areas, develop specific learning outcomes for the student to achieve.
Step 3: Identify appropriate learning strategies for the student to achieve their learning outcomes.
Step 4: Identify performance indicators that establish the student has achieved their learning outcomes, and indicate the evidence the student needs to collect.
Step 5: Collect evidence that demonstrates the student has met the performance indicators.
Step 6: Organise this evidence in a portfolio so assessors can easily understand how the evidence relates to each performance indicator.
Visually, the sequence is:
(Reference: Cooper (1999, 1997) , copied with permission)
The table below shows how the six step process might be applied in an IS context.
|
Six Step Process |
Information Systems Course Design Issues |
|
Skill area |
Core skill areas determined by the pedagogic aims of the course, ACS requirements, and graduate attributes |
|
Learning outcomes |
Learning outcomes are either prescribed by the course outline, or individually negotiated with students after their level of expertise in each skill area |
|
Learning Strategy |
Learning strategies are suggested by lecturers, but ultimately the responsibility of each student |
|
Performance indicators |
Performance indicators are defined by course design, or discussed with students so that they understand how to make decisions about standards of performance relevant to key IS skills |
|
Evidence |
Students collect evidence of their skills and knowledge from a variety of sources including: marked assignments, references and testimonials, certified courses, external projects, employment |
|
Organisation of portfolio |
The portfolio is assembled and presented according to a predefined structure. This structure can be defined more or less completely. For example, it may consist of tables of contents, commentary relating evidence to performance indicators, and items of evidence. A more tightly defined portfolio schema might involve a rigid online template structure within which students enter evidence and explanations of the relationships of evidence to performance indicators. Background server-side processing then shapes how each portfolio is viewed by examiners online or in print. |
Portfolios are either summative or formative, summative portfolios demonstrate learning outcomes while formative portfolios demonstrate learning processes. This paper focuses on summative portfolios, because professional accreditation such as that of the Australian Computer Society (ACS) is primarily concerned with final achievements rather than the students’ learning processes.
Summative portfolio-based assessment has three main forms (Barrett, 2000c, Cooper, 1999) :
In the competency-based portfolio, course outlines specify the learning outcomes for a student. In the negotiated learning portfolio, students negotiate intended learning outcomes. Competency and negotiated learning based portfolios can be combined into a hybrid approach for which the lecturer stipulates some compulsory learning outcomes and allows students to define additional learning outcomes relevant to their personal interests and future career directions. The biographic portfolio or profile is an expanded Curriculum Vitae that is not generally sufficient for assessment purposes because it is not usually related to performance indicators and specified learning outcomes.
Most students do not enter university with the skills necessary to plan their professional development or satisfactorily organise a professional portfolio. Experience in other disciplines has shown it is necessary to teach students these skills. The six-step portfolio building process above was developed for this purpose. Within IS curricula, this process can be taught in whichever unit focuses on developing students’ lifelong learning skills to facilitate continued professional development.
In tertiary IS education in Australia, the Australian Computer Society (ACS) is a key body for accreditation of courses and individuals. The ACS empasises continuous professional development, and their accreditation of courses requires IS professionals to recognise their training needs and participate in devising suitable means of meeting those needs (Australian Computer Society, 2001a) . They state that,
"Because knowledge about information technology is still expanding rapidly, it is important that IT professionals continue to learn throughout their careers. It should be both an employer and an employee expectation to undertake continuing education courses in various forms, either at academic institutions, through publicly available seminars and conferences, or the ACS certification scheme. IT professionals also should undertake studies in management or in the fields of application of IT that are relevant to their daily work, such as accountancy, business, or any other appropriate discipline."
As part of their university education, IS students should learn how to recognise their own training needs and how to satisfy them. The above statement from the ACS guidelines, acknowledges it is impossible for any undergraduate course to adequately prepare students for all possible IS/IT careers.
The ACS syllabus (Australian Computer Society, 2001b) identifies the following skills as fundamental in information fields. The ACS require that students must:
This demonstrates an explicit focus on generic fundamental skills that avoids assessing students’ skills at using proprietary computer languages or applications because they are subject to rapid change.
The ACS approach aligns well with portfolio-based assessment because it provides a medium for students to collate evidence of fundamental skills over a wide variety of contexts. Students can organise their portfolios around mandatory areas such as those defined in the documentation of their courses and the syllabi of the ACS. They can choose additional skill areas, such as those outlined in the optional ACS examination syllabus, and graduate skills relevant to IS/IT in the graduate attributes list. If students undertake specific professional training during their degree, they may wish to include in their portfolio any relevant proprietary certification. Students may also choose to supplement their portfolio according to their personal interests or future career goals. This helps students to build skills to manage their own professional development. Online portfolio based assessment is very appropriate in developing the skills required for ongoing professional development such as that required by the ACS, because the medium provides an authentic assessment opportunity where students can demonstrate their skills (Barrett, 2000b) .
Online variants of portfolio-assessment offer several additional practical advantages to those of hardcopy portfolios (Barrett, 2000a, Cooper and Love, 2001) because they:
Online processes for creating and assessing portfolios depend on technology, and students’ technology skills. Potentially, these dependencies may have a significant and adverse affect on assessment quality by reducing equity. Access to technology to build online portfolios is not ubiquitous. In some cases, it is tied to the socio-economic status of individuals and institutions. To reduce equity problems the portfolio format should be specified so as to avoid disadvantaging students because of lack of access to hardware or software. This minimises students being unfairly disadvantaged through limited access to more sophisticated software.
Barrett (2000a) separates electronic portfolios into different categories on the basis of the levels of skills, hardware and software needed. ‘Electronic’ portfolios, for Barrett, include ‘any electronic representation’ such as video and audiotapes, and are not necessarily computerised. Barrett’s categories are outdated by technology changes, but her approach is useful for categorising modes of online portfolio assessment. The categories provide the basis for choosing technology platforms and presentation specifications to minimise equity issues.
Most equity issues for online portfolio assessment that are caused by technological issues can be resolved by limiting the technologies that can be used to a common framework. This can be done, for example, by the use of online templates designed to work with a set of standardised hardware and software that is available to all students, and in which all students are trained. This limits the scope for more affluent students to gain assessment advantage through access to more sophisticated software.
Computed-aided plagiarism is an assessment problem for student work submitted in both paper-based and online formats where online access and computers are available (Kearns, 2000) . Lecturers cannot easily address the plagiarism problems that the Internet presents using traditional approaches. Recent experience, anecdotal and reported, indicates plagiarism is a serious assessment concern (for example, Terrell and May, 2001) . Suspicions about plagiarism undermine confidence in assessment procedures, and negatively affect the reputation of the examining institution. This provides pressure for cases of plagiarism to pass apparently undetected or be resolved without publicity, increasing the level of suspicion. New forms of addressing plagiarism are needed.
Assessment by online portfolios facilitates the detection of plagiarism. New technology means electronically submitted student work, especially structured submissions such as online portfolios, can be automatically scanned for plagiarism. Recently developed software from Glatt, iParadigm, Integriguard and others (IntegriGuard.com, 2001, iParadigms.com, 2001, Plagiarism.com, 2001) offers a means to test for plagiarism in electronically submitted documents. Glatt’s program, using cloze theory and the uniqueness of each individual’s writing ‘fingerprint’, claims to detect plagiarism where it would not have been detected by manual systems without producing ‘false positives’. This is an important consideration given the seriousness of plagiarism. This process can be applied to hardcopy but is easier where documents are electronic. This offers a significant improvement in strategies for reducing plagiarism, and maintaining credibility in assessment processes.
In IT courses, the opportunities for fraud may represent a particular temptation for some students because they have access to equipment and expertise that facilitates fraud. It is easier to alter undetected an electronic copy of a certificate or electronic reference than to credibly alter an original hardcopy.
Tampering with electronic evidence is a serious issue. It falls under university misrepresentation regulations in the same way that applicants who lie about qualifications are subject to a summary dismissal if their lie is discovered. Fraud in evidence invalidates a portfolio in assessment terms, and should lead to severe penalties including possible exclusion from the university.
The triangulation facilitated by portfolio assessment offers some protection from assessment fraud where gross differences in skill level are evident. Currently, there are emerging standards in online certification of documentation. The main weakness that remains is in the processes of converting paper-based documentation into electronic form. In terms of quality assurance processes, one way addressing this issue is for students to sign a declaration that they have the original evidence on which electronic portfolio submission is based, and that they have not in any material way altered or edited certification, references or evidence provided by other people and included in the portfolio.
Server-based security is well developed and there are several fraud-hardened approaches to assessment certification possible. For example, external and associate examiners can enter marks or competency certification using web-based password protected forms that use secure server-side scripts to place an authenticated certificate in a student’s online portfolio container.
Online portfolios offer an efficient means of demonstrating and assessing graduate attributes. Portfolio based assessment uses a combination of performance criteria and evidence. Graduate attributes provide an additional set of performance criteria and can be included in portfolio assessment processes in a similar way to other performance criteria. Up to final assessment, any evidence that a student collects to satisfy unit-based performance indicators may be also presented as evidence that they possess graduate attributes. This removes the need for a separate graduate assessment mechanism and minimises the need for students to prove that they posses particular graduate attributes.
This paper presents a strong argument for the inclusion of online portfolio-based assessment in IS education (alongside other assessment strategies) to resolve identified challenges due to changes in tertiary education environments. Online portfolio-based assessment offers efficient and effective ways to include graduate attributes, quality assurance processes, and professional certification. It offers the basis for addressing equity issues and controlling plagiarism and fraud, it provides students with an authentic opportunity to demonstrate their ability to use software and hardware on a real task, perhaps most importantly, it help students to develop skills that enable them to plan and documents their own continued professional development. This skill is essential to IS students as future professionals.
ANTA/AVCC (2000) Pathways to Partnerships. Australian Vice-Chancellors Committee. Available: www.avcc.edu.au/policies_activities/teaching_learning/credit_transfer/index.htm [2001, July]
Australian Computer Society (2001a) Accredited Tertiary Courses, [html file]. Australian Computer Society. Available: www.acs.org.au/national/accreditation/intro.htm [2001, June]
Australian Computer Society (2001b) Fundamentals of Information Technology, [html file]. Australian Computer Society. Available: http://203.58.197.209/acs/events_admin/static/training/exam2.htm [2001, June]
Australian Nursing Council Inc. (2000) National Competency standards for the Registered Nurse, Australian Nursing Council Inc, Adelaide.
Barrett, H. C. (2000a) Collaborative Planning for Electronic Portfolios: Asking Strategic Questions, [html file]. Barrett, Helen C. Available: http://transition.alaska.edu/www/portfolios/planning.html [2001, May]
Barrett, H. C. (2000b) The Electronic Portfolio Development Process, [html file]. Barrett, Helen C. Available: http://transition.alaska.edu/www/portfolios/aahe2000.html [2001, May]
Barrett, H. C. (2000c) Electronic Portfolios, School Reform and Standards, [html file]. Barrett, Helen C. Available: http://transition.alaska.edu/www/portfolios/PBS2.html [2001, May]
Biggs, J. and Tang, C. (1997) 'Assessment by portfolio: Constructing learning and designing teaching', Research and Development in Higher Education, 79-87.
Bowie, C., Taylor, P., Zimitat, C. and Young, B. (2000) 'Electronic Course portfolio in a new on-line Graduate certificate in Flexible Learning', HERDSA News, (April), pp.
Bradley, D. (2000) Distance education: An Open Question?, [html document]. University of South Australia. Available: www.com.unisa.edu.au/cccc/papers/keynote_address.htm [2001, July]
Cooper, T. (1997) Portfolio assessment: a guide for students, Praxis Education, Perth.
Cooper, T. (1999) Portfolio assessment: A guide for lecturers teachers and course designers, Praxis Education, Perth.
Cooper, T. and Emden, C. (2000) Portfolio Assessment: A Guide for Nurses and Midwives, Praxis Education, Perth, WA.
Cooper, T., Hutchins, T. and Sims, M. (1999) 'Developing a Portfolio which demonstrates Competencies', In Learning materials: Certificate in Children’s Services; 0-6 years (bilingual support), (Eds, Sims, M. and Hutchins, T.) Ethnic Childcare Resource Inc. Western Australia, Perth, pp. 3-29.
Cooper, T. and Love, T. (2000) 'Portfolios in university-based design education', In Re-inventing Design Education in the University, (Eds, Swann, C. and Young, E.) School of Design, Curtin University, Perth, pp. 159-166.
Cooper, T. and Love, T. (2001) Online Portfolios: Issues of Assessment and Pedagogy (Working Paper ). Edith Cowan University, Perth
Curriculum Council (1998) Curriculum Framework for Kindergarten to Year 12 Education in Western Australia, Curriculum Council, Perth, WA.
DETYA (1999) The Quality of Australia Higher Education: An Overview, [html document]. DETYA. Available: www.detya.gov.au/archive/highered/pubs/quality/overview.htm [2000, March]
DETYA (2001) Higher Education Report for the 2001-2003 Triennium ( 6655HEPA01A). Department of Education, Training and Youth Affairs, Canberra
Education Department of Western Australia (2000a, March 2000) Assessment - Portfolios, [html file]. Education Department of Western Australia. Available: www.eddept.wa.edu.au/centoff/outcomes/focus/fc429.htm [2001, May]
Education Department of Western Australia (2000b, March 2000) Review Classroom Approaches to Student Assessment, [html file]. Education Department of Western Australia. Available: www.eddept.wa.edu.au/centoff/outcomes/focus/fc42.htm [2001, May]
IntegriGuard.com (2001) IntegriGuard, [html file]. IntegriGuard.com. Available: www.integriguard.com/ [2001, May]
iParadigms.com (2001) Turnitin, [html file]. iParadigms.com. Available: http://iparadigms.com/turnitin.html [2001, May]
Kearns, L. (2000) In Theage.com.au, Vol. 2001 Theage.com.au.
Kemp, D. (1999) Knowledge and Innovation: A policy statement on research and research training, Legislative Services, AusInfo, Canberra.
Plagiarism.com (2001) Glatt Plagiarism Screening Program, [html file]. Plagiarism.com. Available: www.plagiarism.com/screen.id.htm [2001, May]
Technology and Industry Advisory Council (2000) Export of Western Australian Education & Training: Constraints & Opportunities, [pdf file]. Technology and Industry Advisory Council. Available: www.wa.gov.au/tiac/exportedu/index.html [2001, June]
Terrell, D. and May (2001) The Deane Terrell Inquiry Report of the External Reviewer appointed to examine issues arising from a case of alleged plagiarism. A report commissioned by: Professor Lance Twomey Vice-Chancellor Curtin University of Technology Perth, Western Australia (Report ). Curtin University of Technology, Perth, Western Australia