Part 3: The Tool

The Tool

If you cast your mind way, way back to the beginning of this workshop, you’ll recall that we introduced the idea of the Chemistry Threshold Learning Outcomes (CTLOs), but we didn’t go into detail. You can find them here!

After you've had a look, let’s take note of a couple of things:

• Firstly, there are five threshold learning outcomes, each with between two and five parts. A given assessment task doesn’t need to cover all five! In fact, as we’ll see in a moment, it might be difficult for a task to even cover more than one.
• Secondly, the CTLOs are relatively agnostic to complexity. This means we can deal with CTLOs at any level (from the developing through to the graduate), which is going to be important when it comes to our assessments.

The basic premise of the tool is this:

This means that an assessment task that claims to assess a part of a CTLO must not only credit the part, but also require it to pass. In evaluating an assessment task, we thus move through two questions:

1. Task design: At what level does the task engage with a CTLO?
2. Assessment criteria: Which elements are required to pass, and at what level of complexity?

1. Task design

We divide engagement into three levels: Addressed (A), Demonstrated (D), and Credited (C). The increase correlates with the weight attributed to the CTLO, from mention through to allocation of marks.

For a task to address an outcome, it merely needs to make reference to the outcome. At this level, students are exposed to any part of the TLO in any way, either implicitly or explicitly, to any level of achievement through completion of the task as instructed.

At a deeper level of engagement, a task might require the demonstration of an outcome. In this case, students are directly instructed to demonstrate or evidence their attainment of any parts of the TLO, to any level of achievement, through completion of the task as instructed. Note that ‘demonstrated’ does NOT apply if:

• The task is structured as a learning experience in the TLO, not an evaluation of capacity.
• Evidence of student understanding of the TLO is not required to be submitted, despite relevance to the task.
• Students’ capabilities with respect to the TLO are unable to be discerned, regardless of exposure to the TLO.

Finally, and ideally, a task that credits an outcome assigns some affirmation of attainment to it. These can be, for example, ‘marks’, ‘grades’, compulsory ‘hurdles’ that are dependent on students providing clear evidence of achievement with respect to the TLO, within their submitted work. ‘Credited’ does NOT apply if:

• Demonstration of the TLO is absent from the assessment criteria and is not presented as a compulsory hurdle.
• Assessment criteria are too vague or non-specific to be clearly said to include the TLO.
• Assessment criteria have poor (or no) connection to student capability with respect to the TLO.

As an example, let’s consider a mock assessment task on black body radiation that is designed to target only CTLO 1,

There is a myriad of ways we might design this assessment—the CTLO is intentionally broad. Here are just a few that we might think of, classified according to their level of engagement with the outcome:

2. Assessment criteria

Even if a task credits a CTLO, it doesn’t accomplish the fundamental goals of the Threshold Learning Outcomes project—in order for the claim of CTLOs to mean anything, assessments must credit the CTLO to the extent that a student would fail if they did not adequately display their mastery of the requirement.

The obvious way to do that would be to assign enough weight to the CTLO in the marking criteria to achieve the pass requirement. But this might not be the best way; after all, you can really only assess one CTLO per assessment if you need to allot half the marks to one.

The main alternative is to structure assessments such that sufficient demonstration of a CTLO (or a selection of them) is a hurdle to passing the assessment or course. There may only be a few marks associated with the CTLO in the final grade, but a student would fail if they did not successfully demonstrate that CTLO at the requisite level.

The tool itself

A downloadable version of the tool can be found here. An online version is coming soon. To use the tool, simply download the Microsoft Excel file and answer the questions from the drop-down menus on the third sheet with respect to the assessment you are evaluating.

There are two keys to understanding the output of the evaluation. Firstly, this Venn diagram summarises the categories we’ve just described. The ideal assessment task sits at the centre:

The full or partial colouring comes into play in the final output of the tool: when we assess a task or series of tasks for a particular CTLO, the output is a 2x2 grid that summarises the efficacy of the task:

The way the grid is filled (along each axis, and with the full or partial colouring of the Venn diagram) tells us about how well the assessment treats a particular CTLO:

A couple of examples

Now that we've seen the tool and understand its components, we're going to take a deeper dive into some assessments to see how it's applied. We’ve already seen some examples of assessment tasks and their learning outcomes in this workshop, but we’re going to go through two to highlight some common mistakes (example 1) and how they can be avoided (example 2).

Case Study 1: Preparation and Investigation of Surfaces of Partially Wetting, Hydrophobic, and Superhydrophobic Surfaces.

For the sake of continuity and reinforcement, let’s return to our original assessment example: the lab task that required the preparation and analysis of different glass coatings.

Recall that students work in pairs to

1. Prepare a set of superhydrophobic surfaces by coating small glass tiles with a series of silicon-based organic polymers.
2. Record the contact angle of water droplets on each surface.
3. Analyse the differences in hydrophobicity of the different surfaces.
4. Evaluate the efficacy of the different coating methods.

And are assessed with a short (2 page) summary task. No grade is given for laboratory skills, except insofar as they are represented through the quality of results obtained.

The distribution of marks for the assessment is:

And, using our tool, the assessment is evaluated as:

In particular, note that CTLO 3.3 (“Applying recognised methods and appropriate practical techniques and tools, and being able to adapt these techniques when necessary”) is only demonstrated—even though the task is an assessment of a laboratory, and this CTLO is implicit in the assessment structure, it’s not explicitly credited at all.

CTLO 3.5 is a similar case: even though the lab work is conducted in pairs, the assessment does not even require demonstration of any teamwork.

Though this task has some wins, in attempting to cover many outcomes, it fails to cover almost any effectively. The next case study provides an excellent alternative to structuring the laboratory and project assessment.

Case Study 2: Team-Based Analytical Lab Report

In this third-year ‘capstone’ project, which spans an entire semester-long subject, students work in teams to “plan, execute and report on an investigation of a complex authentic industrial ‘waste’ material (solid or liquid or combination) including background research into the nature of their sample and possible contaminants, and state regulations of disposal options.” The focus is on building teamwork skills, so that ‘teamwork’ constitutes 40% of the assessment (as an explicit component (20%), an oral project pitch (10%), and a presentation of project outcomes (10%)). The rest of a student’s grade comes from their laboratory practice and recordkeeping (20%) and a 30-40 page technical written report (40%).

More explicitly, the assessment rubric looks like this:

What is interesting—and highly effective—is the inclusion of so-called hurdle requirements. Their inclusion makes fully crediting multiple CTLOs within a single unit possible, and easily so: students cannot pass unless they demonstrate associated CTLOs.

Because of the hurdle requirements, we’ve given this assessment scheme the following evaluation:

Note that three CTLOs are completely covered at a graduate level; some parts of four others are covered at a graduate level. This is remarkable! The hurdle requirements and simple focus of the assessment scheme made this possible—because the assessment tasks build on each other and cover an entire unit, they are able to do far more.

This assessment is also a great example of an effective application of the backwards design process. You can find out more about it here.