Trudi Cooper & Dr. Terence Love
Edith Cowan University, Perth, Western Australia
The online environment offers many advantages that complement the benefits that portfolios offer in a wide variety of educational settings. This paper focuses on the differences between online portfolios and traditional "hard copy" portfolios in relation to a six step portfolio-building process and model of course design developed by one of the authors. The authors contend that obtaining the enhanced outcomes possible using on-line portfolio-based assessment depends on a review of courses’ pedagogical and assessment foundations, and on teachers’ assessment decisions being consistent with underlying pedagogic principles.
The use of portfolio-based assessment is now well established as a valuable assessment tool (see, for example, Barrett, 2000b; Biggs & Tang, 1997; Cooper, 1997; Education Department of Western Australia, 2000b, 2000c). Portfolio-based assessment is beneficial pedagogically because the format can encompass evidence from a wide variety of sources, (Education Department of Western Australia, 2000c), it can help educators overcome many assessment difficulties, especially in relation to equity and moderation (Cooper, 1999; Cooper & Love, 2000), it provides a ‘richer picture’ of the student (Barrett, 2000b), and portfolio-building actively involves students in the learning process (Bowie, Taylor, Zimitat, & Young, 2000). Within the Curriculum Framework process, portfolio-based assessment appropriately applied, can form a valuable assessment and reporting tool for schools (Education Department of Western Australia, 2000c).
Recently, online portfolios have been suggested as a useful extension of portfolio-based assessment (Barrett, 2000b; Bowie et al., 2000). This paper explores the benefits and limitations of online portfolios in the school context in relation to the Curriculum Framework of Western Australia.
Hamm & Adams suggest (Education Department of Western Australia, 2000b),
A portfolio is more than a ‘folder of student work; it is a deliberate specific collection of accomplishments.
This can be expressed as a positive definition (Cooper, 1997, 1999),
Portfolio: Collection of evidence that demonstrates skills, achievements, learning or competencies.
Portfolios are useful in educational contexts for both summative and formative purposes (Cooper, 1997, 1999; Education Department of Western Australia, 2000c).
Cooper (1999) suggests that summative portfolio-based assessment has three main distinct forms; the competency based portfolio, the negotiated learning portfolio, and the biographic profile, or record of achievement. These categories align with similar categories suggested by Barrett (2000b). In a school context, the main forms of portfolio are:
In this paper, the focus of discussion is the first type of portfolio, the outcomes-based portfolio as used in conjunction with the Curriculum Framework of Western Australia (Curriculum Council, 1998). It is recognised that online portfolios have other possible uses in schools, including formative uses and uses in whole school assessment, but these are not discussed here.
Several ‘key’ issues are important in assessment, and these apply equally to traditional and online portfolios. The Curriculum Framework (Curriculum Council, 1998, pp 37-38) identifies that all assessment should be:
Valid: The assessment should be based directly upon judgments about whether each student has achieved a defined outcome, rather than indirectly on assessment of their knowledge of facts, concepts or skills.
Educative: The assessment process should contribute positively to the student’s learning.
Explicit: The basis of each assessment judgment should be transparent.
Fair: The assessment should not discriminate on grounds that are irrelevant to achievement of the defined outcome.
Comprehensive: The assessment should be based upon multiple sources of evidence.
In addition to these criteria, Sutton (Education Department of Western Australia, 2000a, p2) suggest that the ‘art’ of assessment also involves finding the ‘best fit’ between purpose, validity, reliability and manageability. This concept of ‘best fit’ recognises that in practice, if not in theory, there is always some trade off between the competing requirements of perfect assessment. In line with this, Cooper (Cooper, 1999; Cooper & Love, 2000) has suggested that assessment should be:
Fitting: The assessment process should be effective and appropriate to its intended purpose. (This corresponds with ‘validity’ and ‘comprehensiveness’ in the Curriculum Framework)
Fair: The assessment process should minimize unfair bias or advantage in assessment. (This corresponds with ‘fairness’ in the Curriculum Framework.)
Capable of moderation: The assessment process should make it possible to be consistent in assessing students’ work. (This corresponds with ‘explicitness’ in the Curriculum Framework.)
Efficient: The assessment process should be efficient in terms of time and other costs. (This corresponds with ‘manageability’ in Sutton’s schema)
Monitored: The assessment process should be designed in such a way as to include measures that permit its own evaluation and ensure that the assessment process functions as intended relative to the desired pedagogic outcomes.
Against this background, portfolios provide a useful assessment tool because correctly implemented they satisfy the requirements of all the assessment criteria discussed above. Portfolio-based assessment linked to the outcomes of the Curriculum Framework can contribute to the comprehensiveness of assessment, be educative, provide valid assessment, support fair assessment, be used to make assessment processes explicit, and can be monitored.
Cooper’s ‘six step’ portfolio building process was developed to provide the basis of outcomes-based placement assessment in tertiary courses (see, for example, Cooper, 1997; Cooper & Emden, 2000; Cooper, Hutchins, & Sims, 1999). It helps ensure that portfolio-based assessment is applied in a purposive way that relates transparently to the predefined pedagogic outcomes. The ‘six step’ model of portfolio-building process is used here as a method of linking traditional and online portfolio assessment to the Curriculum Framework.
The six steps in Cooper’s portfolio-building process are:
Step 1: Identify the areas of skills that the student is intended to develop.
Step 2: From these skill areas, develop specific learning outcomes for the student to achieve.
Step 3: Identify appropriate learning strategies for the student to achieve their learning outcomes.
Step 4: Identify performance indicators that establish the student has achieved their learning outcomes and indicate the evidence the student needs to collect.
Step 5: Collect evidence that demonstrates the student has met the performance indicators.
Step 6: Organise this evidence in a portfolio so assessors can easily understand how the evidence relates to each performance indicator.
Visually, the sequence is:
(Reference: Cooper, 1997, 1999)
All of the above steps are found in most formalized methods of course design. Within the Curriculum Framework, the above model applies in the following way:
Step 1: Skills: These are described in the overarching statements and foundation strand statements within the Curriculum Framework.
Step 2: Learning outcomes: The strand outcome statements in the Curriculum Framework describe the learning outcomes.
Step 3: Strategies: Decisions about preferred strategies for designing the ways in which the learning outcomes will be achieved by individual students and school-wide are the responsibilities of teachers and the School Council, and should be addressed on a school by school basis within individual school plans.
Step 4: Performance indicators: The performance indicators are contained in the evidence statements for each strand.
Step 5: Evidence: The evidence (listed under pointers) is what teachers assess in order to determine the level of attainment of each student.
Step 6: Organisation of evidence: For portfolio-based assessment, the strand outcome statements provide the basis for organising evidence within a portfolio. An appropriate cross-referencing system or duplication process may be necessary in each portfolio because a single piece of evidence might be applicable to more than one learning outcome. A commentary may also be needed to explain the relevance of evidence to each strand outcome statement where it is not self-evident.
There are two distinct ways in which online portfolios have implications for the assessment process. The first is their contribution to the effectiveness of the assessment process. The second is their contribution to its efficiency, particularly of the management of the assessment process. In some instances, online portfolios may simultaneously contribute to both efficiency and effectiveness of assessment. For example, material that students prepare in electronic formats for portfolios may also be submitted electronically. The combination of the two digitally based processes has potential gains for the effectiveness of assessment processes where students are authentically assessed on their ability to produce their work in an electronic environment, and has potential for improving the efficiency of assessment because the submission process can be managed electronically.
In the remainder of this paper, we consider the advantages and disadvantages of online portfolios in terms of their impact on the effectiveness and efficiency of assessment processes and the management of assessment. In relation to the effectiveness of assessment we focus on the organisation of student work relative to the Curriculum Framework, plagiarism, equity issues, student ownership of their work, and teachers’ assessment skills. In relation to the efficiency of assessment the focus is on the management of assessment records, facilitation of assessment moderation processes, and automated marking. The selection of topics is not intended to be exhaustive. They are areas where online portfolios offer obvious benefits, or present possibilities that are controversial or potentially problematic. The discussions are speculative to the extent that the ideas discussed at this stage remain relatively untested in the school context. They outline significant issues that may be relevant to school-based trials of online portfolios.
The effectiveness of portfolio-based assessment may be enhanced by the use of templates and electronic guides that help students structure their evidence This is especially valuable for students who have not yet reached the level of independence where they can structure their own learning and present it in a manner that facilitates its assessment. Many presentation issues can be resolved by the standardisation that comes with the use of online templates and style sheets.
The effectiveness of assessment can be enhanced by:
· The use of a standard set of templates to helpfully structure students’ online portfolios.
Plagiarism has been identified as an important assessment issue by Kearns (Kearns, 2000), who believed plagiarism is much easier for students to embark on in an online computerised environment, but considers many rumours of computerised plagiarism may be exaggerated or without foundation. The experience of universities, however, anecdotal and reported (for example, Terrell & May 2001)) indicates plagiarism continues to be a serious and embarrassing, assessment concern. Suspicions about undetected plagiarism are embarrassing for educational institutions because they undermine confidence in assessment procedures and ultimately this negatively affects the reputation of the examining body. For this reason, it is likely that many cases of plagiarism either pass undetected or are resolved without publicity. Computed-aided plagiarism is an assessment problem for student work submitted in both paper-based and online formats. Familiarity with all formulations of information on the Internet easily accessible for copying by students is now beyond the capacity of discipline experts. Therefore, teachers can no longer be expected to control plagiarism by being aware of all possible sources for plagiarised material.
The implementation of assessment by online portfolios facilitates the detection of plagiarism. Recently developed software from Glatt, iParadigm, Integriguard and others (IntegriGuard.com, 2001; iParadigms.com, 2001; Plagiarism.com, 2001) provides a means to test for plagiarism in electronically submitted documents, mainly through the use of Internet-based search engines to cross compare material. The Glatt program is based upon cloze theory and the uniqueness of each individual’s writing ‘fingerprint’. (While this process can be applied to paper based documents, it is easier to apply if documents are already in electronic form.) Glatt claim to be able to detect plagiarism where it would not have been detected by manual systems, and that their method does not produce ‘false positives’: an important consideration where a test is used as the basis for alleging plagiarism. The implication of this new technology is that any electronically submitted student work, and especially that submitted in a structured format such as an online portfolio, could be electronically scanned for evidence of plagiarism. In future this may be the best strategy for reducing the incidence of plagiarism, and may ultimately be the only way in which the credibility of assessment processes can be maintained.
The effectiveness of assessment may be improved by:
· Using digital technologies available through the online medium to identify plagiarism. There are several possibilities emerging but, at this stage, the most straightforward is likely to be the automatic review of the contents of online portfolios using propriety plagiarism testing software.
Online processes for the creation and assessment of portfolios depend on technology and on students’ technology skills. These dependencies may have a significant and potentially adverse affect on assessment quality by reducing equity. Access to technology to build online portfolios is not ubiquitous: in some cases it is tied to the socio-economic status of individuals or the institution within which learning is undertaken. It is not at present reasonable to assume that all, or even most, students have the additional technical skills necessary to build highly sophisticated online portfolios and this presents equity issues. Most of these issues can be resolved if the technical sophistication required for building online portfolios is matched to the equipment and skills supported by existing base-line levels of resourcing and teaching in schools. This minimises students being unfairly advantaged because of limited access to more sophisticated software, especially if the assessment process is strictly tied to specified learning outcomes. This can be achieved by students being provided with online templates designed to work with base-line software.
There is anecdotal evidence that many students become personally involved with, and attached to, traditional portfolios. This is sometimes seen as desirable from a pedagogically perspective because it indicates that students claim ownership of their work. With traditional portfolios, some students create decorated covers and take trouble over the visual presentation of the portfolio contents. For paper-based portfolios, the technologies and skills used by students are relatively ubiquitous, have a low technology requirement (coloured pens), and the costs of training the students is not an issue because the necessary skills have usually been acquired as a normal part of early childhood education. For online portfolios, it is also possible for students to enhance the presentation of their work and to personalise their contributions. The simplest computerised methods use clip art and drawing packages that students now entering school learn how to use in early childhood years. More complex forms of enhancement make full use of multimedia technologies. At present, there remains a great diversity in student access to this technology, and in students’ opportunities for gaining advanced computer skills. Avoiding equity issues from this source requires assessment processes to be constructed carefully.
The effectiveness of assessment may be improved by:
· Matching the technical sophistication needed to build online portfolios to the base-line level of computer resourcing and skills for schools.
· Defining online templates for the portfolios that are designed to be used with base-line software and skill levels that are regarded as standard.
New assessment methods, such as using portfolio-based assessment in conjunction with the Curriculum Framework, have differential rates of acceptance within schools. Teachers can enhance their own assessment skills if they have opportunities to share their experiences, judgements and questions with their peers. Sharing with peers enhances the effectiveness of portfolio-based assessment as teachers share their experiences and their ways of resolving problems. Sharing samples of portfolios can also enhance the effectiveness of assessment processes by increasing the skills of individual teachers through peer learning. Online portfolios can easily be shared between teachers in different schools, and this is likely to be especially significant for teachers in small or remote schools.
The effectiveness of assessment may be improved because:
Using online portfolios for assessment offers the possibility of a significant contribution to the efficiency of assessment processes through the automatic management of routine tasks that many teachers find both irksome and time consuming. Online intelligent agents are available capable of automating many submission management issues, for example, using computerised systems of recording that automatically record where and when submissions for assessment were received (Wenn & Darbyshire, 2000). This approach also enables the possibility for identifying whether parts of a portfolio are missing or not adequately addressed. In addition, the online environment offers several features as standard that can result in efficiencies compared to physical processes for undertaking the same tasks.
Online portfolio-based assessment systems may improve the efficiency of assessment because:
The efficient moderation of physical samples of students’ work across a State presents practical difficulties. Co-ordinating statewide moderation of paper-based portfolios involves physical transportation of cumbersome packages containing the sole copies of students’ work. Online portfolios, however, can be efficiently moderated without physical transport of hard copy media. Moderation of online portfolios can be achieved by electronic circulation of portfolios allowing moderators to simultaneous access identical copies of the original student work: a process that depends only on Internet access and compatible software This relieves the necessity for moderators to be physically co-located, and offers savings on travel and transportation costs, and travel time. It minimises the possibility of student work being lost or mislaid, especially where secure digital backups are in place, and reduces the impediments to full participation by teachers in remote areas.
Online portfolio-based assessment systems may improve the efficiency of assessment because:
New software developments offer the possibility of automating assessment. Automatic assessment of portfolios is at present at the ‘cutting edge’ of pedagogic discussions about online education system. A broad debate about the place of automated marking for ‘open-ended’ student work (as opposed to closed question tasks) has yet to happen. Whilst some portfolio-based assessment tasks are likely to remain subjective, and may always require a skilled human assessor who can exercise their professional judgement, some aspects of assessment of online portfolios could be fairly easily automated using current software technology. The use of intelligent agents undertaking these sorts of functions is already implemented in part in current versions of Microsoft Word, Grammatika and Corel WordPerfect (trademarks of relevant organisations).
Online portfolio-based assessment systems may facilitate improvements to assessment efficiency through automating functions such as:
· Identifying the number and type of grammatical and spelling errors.
· Assessing the complexity of writing using standard software that provides feedback about the minimum required reading age of the intended readership.
· Identifying missing items, where parts of the portfolio have not been submitted.
· Providing a report on the relative weighting by word count given to different elements within a portfolio.
Online portfolios may be implemented in various ways using a range of technologies from the commonplace home computer solutions to advanced solutions involving expensive and complex computer software, hardware and high levels of programming skill. Barrett (2000a) categorised five levels of technological sophistication for building online portfolios. Increasing sophistication requires more expensive hardware, more extensive software and requires students to possess higher levels of skill in using computer packages and peripherals. According to Barrett, the most basic form of online portfolio is one that uses only the word-processing software and LAN server storage of files: an approach requiring the student to have only basic word-processing and file saving / retrieval skills.
From an assessment perspective, using increased levels of technology reduces the validity of assessment for students who are less skilled in computer applications in situations where online portfolios assume greater competence in computer skills than students commonly possess. Only where the purpose of the portfolio is to demonstrate computer skills can a more sophisticated online portfolio be justified pedagogically.
Barrett’s (2000a) appears to favour the use of medium levels of technology for online portfolios, but there are some inconsistencies between the three tables in her paper. She supports her claim by reference to the superior technical features possible when students present portfolios as PDF files and use hypertext links. From an equity perspective in a school context, these advantages need to be carefully weighed against the disadvantages of increased technological sophistication and skill. The technical advantages she describes undoubtedly can lead to more impressive documentation than is possible with simpler technologies, and where portfolios are placed on websites there is advantage of having them in an uneditable format. Online portfolios that school students produce for their assessment are not, however, produced primarily for display on schools’ websites.
To summarise, we contend that the apparent technical superiority of technically sophisticated solutions is outweighed in the school context by equity considerations and considerations about the validity of assessment. Taking the above factors together indicates that only the lowest level of technical sophistication is appropriate for on-line portfolios in the school context. Although there may be technically superior ways of producing online portfolios, the lowest level of online portfolio, identified by Barrett as making use only of word-processing and file saving and retrieving skills, poses few equity or validity problems. Selecting a low level of technology has the added benefit that most teachers have at least these levels of computing skills, and the implementation of technically simple online portfolios would, therefore, not require additional commitments to professional development.
There are some assessment tasks for which online portfolios are unsuitable. Online portfolios preclude the inclusion of physical objects that students have constructed e.g. games, non-computerised work of art, and craft items. Anecdotally, teaching staff have raised issues relating to the benefits of the physicality and tactility of hard copy traditional portfolios. Physical portfolios offer one of the few practical avenues for students to use their skills in situations that can produce authentic tangible products. Another objection that bears consideration is that online portfolios do not allow the assessor to take a handful of portfolios with them to mark in the garden, at the beach or at a café, (although a good laptop computer may resolve this gripe except for the beach).
We conclude that online portfolios are useful as assessment tools in conjunction with the Curriculum Framework. Online portfolios offer benefits in the ‘process of student assessment’ and the ‘management of the process of student assessment’. Used in conjunction with appropriate software solutions, online portfolio-based assessment may relieve teachers of some of the more tedious aspects of assessment ‘housekeeping’ and permit parts of the assessment process to be automated. This latter issue probably needs further debate to clarify the limits of the role of automation associated with pedagogic factors. For senior school students, the use of online assessment processes may be an important strategy for deterring plagiarism because it facilitates the use of anti-plagiarism software. Any systematic introduction of online portfolio-based assessment needs, however, to take into account the existing levels of technology, computer skills, and technical support in schools. We argue that, for equity reasons, online portfolio-based assessment should not exceed the current minimum standards of computer technology in schools, notwithstanding the fact that better technical solutions may be possible if more sophisticated hardware and software are used.
The use of portfolio-based assessment, online or paper-based, in conjunction with outcomes statements, requires that teachers have a thorough understanding of a systematic portfolio-building process such as the ‘six step’ method outlined in this paper. This implies a need for professional development courses for teachers connecting the practical and pedagogical issues of using portfolio-building processes to strand outcome statements in the Curriculum Framework
Barrett, H. C. (2000a). Create Your Own Electronic Portfolio, [html file]. Barrett, Helen C. Available: http://transition.alaska.edu/www/portfolios/iste2k.html [2001, May].
Barrett, H. C. (2000b). Electronic Portfolios, School Reform and Standards, [html file]. Barrett, Helen C. Available: http://transition.alaska.edu/www/portfolios/PBS2.html [2001, May].
Biggs, J., & Tang, C. (1997). Assessment by portfolio: Constructing learning and designing teaching. Research and Development in Higher Education, 79-87.
Bowie, C., Taylor, P., Zimitat, C., & Young, B. (2000). Electronic Course portfolio in a new on-line Graduate certificate in Flexible Learning. HERDSA News(April).
Cooper, T. (1997). Portfolio assessment: a guide for students. Perth: Praxis Education.
Cooper, T. (1999). Portfolio assessment: A guide for lecturers teachers and course designers. Perth: Praxis Education.
Cooper, T., & Emden, C. (2000). Portfolio Assessment: A Guide for Nurses and Midwives. Perth, WA: Praxis Education.
Cooper, T., Hutchins, T., & Sims, M. (1999). Developing a Portfolio which demonstrates Competencies. In M. Sims & T. Hutchins (Eds.), Learning materials: Certificate in Children’s Services; 0-6 years (bilingual support) (pp. 3-29). Perth: Ethnic Childcare Resource Inc. Western Australia.
Cooper, T., & Love, T. (2000). Portfolios in university-based design education. In C. Swann & E. Young (Eds.), Re-inventing Design Education in the University (pp. 159-166). Perth: School of Design, Curtin University.
Curriculum Council. (1998). Curriculum Framework for Kindergarten to Year 12 Education in Western Australia. Perth, WA: Curriculum Council.
Education Department of Western Australia. (2000a, March 2000). Assessment - Introduction, [html file]. Education Department of Western Australia. Available: www.eddept.wa.edu.au/centoff/outcomes/focus/fc41.htm [2001, May].
Education Department of Western Australia. (2000b, March 2000). Assessment - Portfolios, [html file]. Education Department of Western Australia. Available: www.eddept.wa.edu.au/centoff/outcomes/focus/fc429.htm [2001, May].
Education Department of Western Australia. (2000c, March 2000). Review Classroom Approaches to Student Assessment, [html file]. Education Department of Western Australia. Available: www.eddept.wa.edu.au/centoff/outcomes/focus/fc42.htm [2001, May].
IntegriGuard.com. (2001). IntegriGuard, [html file]. IntegriGuard.com. Available: www.integriguard.com/ [2001, May].
iParadigms.com. (2001). Turnitin, [html file]. iParadigms.com. Available: http://iparadigms.com/turnitin.html [2001, May].
Kearns, L. (2000, Thursday 11 May 2000). School assessment: does it pass the test? Theage.com.au.
Plagiarism.com. (2001). Glatt Plagiarism Screening Program, [html file]. Plagiarism.com. Available: www.plagiarism.com/screen.id.htm [2001, May].
Terrell, D., & May. (2001). The Deane Terrell Inquiry Report of the External Reviewer appointed to examine issues arising from a case of
alleged plagiarism. A report commissioned by: Professor Lance Twomey Vice-Chancellor Curtin University of Technology Perth, Western Australia (Report ). Perth, Western Australia: Curtin University of Technology.
Wenn, A., & Darbyshire, P. (2000). Evolution of an Interface for Web-based Teaching. In S. Stoney (Ed.), Challenges of the New e-Conomy. Perth: We-B Research Centre.