Feeds:
Comments

Posts Tagged ‘advice for a young academic’

WHAT IS “PATCHWORK ASSESSMENT” ALL ABOUT?

Summary (the short version)

  • Patchwork Assessment (or Patchwork Text Assessment) is an alternative to exams or traditional coursework, and may be particularly appropriate for introduction as part of a module being redesigned for flexible digital delivery.
  • It involves students creating or curating numerous “patches” of content or activity through the module, and using them as the basis of a final summative assessment that adds up to more than the sum of its component patches. It is not merely a portfolio, as the components themselves are not summatively assessed.
  • Advantages include opportunities for improvements in inclusivity,  student engagement, and the deployment of strategies for generative learning. 
  • Disadvantages include unfamiliarity to both students and staff, and potentially significant workload in developing new assessments and inducting students into the process.

Introduction:

I have been asked to put together some notes about Patchwork Assessment for the benefit of colleagues who might be encountering it for the first time while developing alternative assessments in place of traditional unseen exams. As a “beginner” in this myself, I have organised the notes in Q&A format, and I invite colleagues to correct me if they think I have misunderstood anything, missed any big questions or given any wrong answers. My approach here has been to adopt and adapt rather than to copy slavishly from a template, and I imagine that colleagues reading this will do the same: take these notes as a starting point for their own ideas.

Here are my responses (PGK) to imaginary Questions from Curious Colleagues (QCC).

Q & A about Patchwork Assessment

QCC: Before we start, are these just your own ideas or is there a framework of pedagogic research and experience behind this?

PGK: There is an extensive framework of published research and experience around Patchwork Assessment, but my answers here are personal opinions based partly on that framework and partly on my own experience. For a starting point into the literature about Patchwork Assessment, please see the reference list at the end of these Q&As. Two  key sources to begin with would be Winter (2003) and Jones-Devitt et al. (2016).

QCC: So, is Patchwork Assessment just a fancy term for “Portfolio”?

PGK: No. Some of the literature is quite adamant that thinking about Patchwork Assessment as simply a portfolio assessment is unhelpful, because we are not assessing a collection of material but assessing something developed from that collection, the collection being an extended student activity that precedes the assessment item. However, I confess that when I first started to think about Patchworks, the idea felt at-least somewhat related to the idea of a portfolio. Now, I find it helpful to differentiate between the patchwork collection (the patches or items that you might think of as a non-assessed portfolio) and the final patchwork assessment (the assessed item that emerges from, or uses, the patchwork collection). We are not assessing the collection, but assessing something that the student creates after they complete the collection. Some sources do refer to the final assessment as a stitching together of the patches, but I think of it as needing to be more than just that: more than just the sum of its parts.

QCC: If it’s not a portfolio, what how does a Patchwork Assessment work, then?

PGK: There are two stages to the activity.

(1) Throughout a module, students create and assemble (short) items of work (patches) that are not themselves summatively assessed, (but may be formatively assessed to provide constructive feedback, or may be shared and discussed in peer groups). These patches can and should be quite varied, ideally with students having a choice of which ones to engage with. They might include personal notes on a set reading, a short reinforcement exercise, an online group discussion, a reflective commentary on a lecture, a topic summary, or a student choosing their own local case study or application of a theme from a taught session and producing a development, commentary or illustration of that in their own preferred format. Blog or vlog entries might work well, or some type of course-long workbook. A lot of these activities could be things that we like to image students might be doing anyway as part of their independent work, but we are giving them a bit of a steer and a push. Tutors might “prompt” activities alongside each lecture or online session. These items are not themselves summatively assessed and it is up to the module leader how closely they want to “enforce” engagement. Having students share and discuss their activities with their peers in a shared module space can be helpful.

(2) At the end of the module, when students (should) have assembled (at least a partial) collection of different items from those that have been recommended through the module, the summative assessment requires them to do something with their collection. This might be an applied or practice-based task where the collection is used as the basis of something such as a planning application or a grant application or a professional report or a job application, or at its simplest it  could be that they are asked to produce a written reflection on their collection, with items from the collection being presented as illustration or evidence within that summative report but not themselves being assessed.

If it is set up properly, the quality of the final summative piece will depend partly on students having effectively engaged with the collection of formative patchwork items along the way, without the collection itself actually being assessed.

QCC: Does the tutor have to mark all these little pieces of work!??

PGK: No. The patchwork items are not themselves assessed. They are materials that the students are compiling for themselves, to use when they do the final component. It is up to module leaders whether the students have chances for interim submissions, peer-group discussions, etc in order to get formative feedback as they work up their patchwork, but certainly there is no formal assessment of this stage of the activity. Most staff will need to manage this process carefully to keep their workloads appropriate, and there is no requirement that staff will be heavily engaged in this stage of the students’ work once the framework of instructions to students is set up.

QCC: What if a student simply doesn’t do the in-course formative items? Can they just write the report at the end and somehow get away with it?

PGK: If they haven’t collected their items, they can (and must) still do the summative assessment, but if it has been set correctly their work will be severely limited by the lack of evidence or illustration from their patchwork items. The final piece has to be designed to build on (rely on) the collected items, so the quality of the overall assessment will depend on the quality of the collection even though the collection was not in itself assessed.

QCC: What if a student does excellent work on the (formative) patchwork items but does a poor job on the summative write-up: do they get credit for the component items?

PGK: The assessment is based entirely on the summative item. This will include reference to the patchwork components as examples, illustrations or evidence, but the mark is assigned for the summative piece, not the patchwork parts. Theoretically, a student could do good work creating their component items but then get a poor result if they do not use them effectively in the actual assessment.

QCC: Do the students present the patchwork items within the final report?

PGK: This will depend on the exact nature of the final report that you have set. It could be set up so that the patchwork items are “behind the scenes”, or so that they have to be incorporated in some way within the report, or as an appendix. If students are be asked to submit their collection there is a danger that the exercise reverts to the level of a portfolio. Alternatively they could just insert items (or parts of items) from their portfolio as bits of evidence or examples for things they are doing in the summative assessment. The tutor does not need to read and mark all the collected items… the final report should be free-standing (albeit with those references back to patchwork items or activities).

QCC: So what is the final report…  is  it “about” the patchwork items?

PGK: This is up to the tutor to decide and will depend on the nature of the module. A simple approach would be to have students write a final report that basically describes their patchwork collection, but I don’t think this really makes much of a step up from basic portfolio-plus-reflection. A more sophisticated approach could be, say, to have students complete a new task or solve a problem using their patchwork items. They could build from their patchwork to develop a new textbook outline, or create a public-facing or industry-facing document about the module, the topic, or their experience… Or based on a patchwork of local case studies a student might compile a national-scale proposal.  It is really up to the imagination of the module designer to find activities or missions that will enable students to create something new and freestanding that somehow draws on their experience of assembling the patchwork and somehow uses their patchwork items towards some greater outcome. In some contexts there could be a specific “question” for students to write in response to: “Making specific reference to items from your patchwork and/or to the patchwork as a whole, demonstrate how x, y, z, and propose how A and B in the next ten years”. This is for individual tutors to decide in their own module contexts, and provides a huge amount of flexibility.

QCC: Do you announce the final assessment up front, or at the end?

PGK: Again, either way could work depending on how you organise the module. If you want students to knowingly build a patchwork for a particular purpose, and if you want them to make decisions for themselves about what to collect en route,  then giving them the final assessment “question” at the very start of the module would make sense. On the other had if you set the exact components of the patchwork yourself (for example as a basic year-long workbook of exercises) then it might make sense to reveal the exact assignment at the end, a bit more like an unseen exam. At the moment I think that announcing everything up front gives the students a greater degree of responsibility and autonomy.

QCC: How does the student know what to do and when to do it… is there a roadmap?

PGK: Some modules might build the patchwork creation into a weekly “task list”, basically telling students what to do and when to do it. (eg Week 1: complete KLE exercise 1, read reference items 1 & 2,  and write a 200-word reflection on your existing knowledge…”). Other modules could leave it much more to students to decide what to do when, but in that case very clear suggestions, examples, etc would be needed. Certainly, for students who have not done this before, I think that really very clear guidance will be necessary, even if the guidance is based on sets of choices. For example, there might be pathways leading towards different optional final assessments, so students collect items relevant to their particular target. In my own context, I can imagine having suggestions such as “At this point in the course for your patchwork collection either identify the most intensively researched examples of a soft-bed and a rigid-bed surging glacier (if you are working towards patchwork-text assessment option A) or identify a research programme that has used numerical modelling to simulate surge behaviour (if you are working towards patchwork-text assessment B)”.

QCC: What’s the advantage of this over a more traditional assessment?

PGK: There are several advantages: for example – inclusivity and motivation to engage. Students may be better motivated to engage with their module throughout, as they can collect, create and compile their patchwork components based on activities week by week as the module progresses. Also, it is usually up to the student to identify the pivotal moments in learning that they want to include as patches in their collection. It will be hard for students to leave everything till the end and rely on last-minute revision as they might do for an exam, as opportunities such as group activities, lecture-connected exercises, etc roll by throughout the course. Another advantage (especially if the module offers lots of choice or flexibility in what a student can include in their patchwork) is inclusivity, as students will have a variety of different activities underpinning the assessment, and should have the opportunity to select what items to work on or what media, methods or platforms to use. Patchworks promote assessment for learning rather than just assessment of learning, and   encourage the use of  student-centred teaching methods. There is more about this in the literature (see reference list).

QCC: And what are the disadvantages?

PGK: Having just embarked on introducing this type of assessment into my own modules I would say the initial disadvantage is the set-up cost in the tutor’s time. It would be very quick and easy to replace a traditional exam with, say, a take-home exam or yet another essay, whereas inserting a Patchwork Assessment into an existing module feels a bit like taking the sugar out of a hot cup of tea, or threading the veins into a raspberry ripple ice cream. Patchwork Assessment doesn’t feel like a bolt-on; it is something that needs to be integrated into the fabric of a module in a fairly complex way. Students will require a significant amount of direction at the start of the module to understand the importance on continuous engagement and the value of developing and assembling worthwhile patches that will eventually stitch together as a component of the final assessment. This will involve a more substantial fraction of the module’s total hours than would be needed to explain a simple essay-plus-exam assessment structure. 

The HEA practice guide identifies one particular issue that is with thinking about at the outset: resistance! Both students and colleagues may find this novel and initially disconcerting, and may need to unlearn some of their preconceptions about how learning and assessment are aligned.

QCC: Your introduction said Patchworks could help with “generative learning”. What’s that?

PGK: Generative learning is a term often used alongside “active learning” to differentiate between “passive” learning activities such as listening or reading that involve memorization for inserting knowledge into the brain, and activities such as  organising, applying, summarising or re-teaching that involve learners using the material that they are trying to learn, which leads to a deeper level of learning and retention (eg Fiorella and Mayer, 2015). In our context, that might be achieved by setting patchwork activities that include these types of activity and will lead students away from over-reliance on time sitting in front of a lecturer or reading pages of text over and over again. Inviting students to summarise and re-teach something from their reading would be an example of a generative learning activity. Enser (2020) provides a short discussion of the value of summaries (summarising a lecture, summarising set readings) as a way of engaging students with generative learning.  Patchwork assessments can give tutors good opportunities to embed these sorts of activities in a distributed manner throughout the module, also enabling the implementation of strategies such as spacing, interleaving and dual coding in such a way that students do those things as part of their own generative learning rather than simply “receiving” them from the tutor standing and delivering.

QCC: Can you provide a concrete example, with student resources, of how to do this in Geography?

PGK: Watch this space!

References:

Enser, M. (2020). What’s the key to remote learning? You already use it. TES Times Educational Supplement 21st June, 2020. https://www.tes.com/news/whats-key-remote-learning-you-already-use-it

Fiorella, L., & Mayer, R. (2015). Introduction to Learning as a Generative Activity. In Learning as a Generative Activity: Eight Learning Strategies that Promote Understanding (pp. 1-19). Cambridge: Cambridge University Press. doi:10.1017/CBO9781107707085.003 https://www.researchgate.net/publication/264233729

Jones-Devitt, S.,  Lawton, M. and Mayne, W. (2016). HEA Patchwork Assessment Practice Guide. 18pp. Higher Education Academy, York. https://www.advance-he.ac.uk/knowledge-hub/patchwork-assessment-practice-guide

Winter, R. (2003).  Contextualising the patchwork text: addressing problems of coursework assessment in higher education. Innovations in Education and Teaching International Vol. 40, No.2, 112-122.  http://www.cetl.org.uk/UserFiles/File/reflective-writing-project/PatchworkText-winter.pdf

Read Full Post »

I have spent a lot of my career for the last 30 years or giving advice to young academics, and hearing their advice for me. Early on in that period I was young myself and was fortunate to have fantastic mentors and advisors throughout my early career. Thank you to all of you both for the advice you have given me and for allowing me to think through and reflect on my own ideas about so many of the issues that young academics – and not-so-young academics – face in their careers. In the last few years I’ve put random bits of advice onto social media here and there, and onto my own web page, and looking back at those I realise that my “advice to young academics” posts have have generated the most interest and stimulated the most conversation. Therefore I have decided to try and be a bit more organised about posting up that kind of material. From now on I will try to include tips for young academics more often into my Twitter feed @petergknight http://www.twitter.com/petergknight and I will try to develop that theme here too, with an “advice for a young academic” tag, or theme, or category here on my wordpress blog. A tag? A category? A book? Who knows… give me a break – I’m not that well organised: I’m an academic. My first bit of advice for a young academic is always to be aware that you are going to get a lot of advice from a lot of people, and most of it will be useless to you. Taking advice wisely is an important skill to cultivate as many senior academics have not cultivated the skill of giving it wisely.

Read Full Post »