EdTech thought leaders and innovators from across Australia and India meet in this panel discussion to consider the changes in assessment and online learning during the COVID era and beyond.

In this two-part panel discussion, education innovators and thought leaders from across Australia and India meet to consider the state of student assessment and engagement in Higher Education, in Australia, in India, and in the global context.

Moderated by Cherie Diaz, Managing Director of OpenLearning in Australia, and Margo Griffith, Head of Business Development of Edalex, this conversation’s panelists include:

• Dr. Dino Willox, Director Student Employability at the University of Queensland
• Raju Ganapathy, CEO of Conscience
• Dr. Adam Bridgeman, Pro Vice Chancellor of Educational Innovation at the University of Sydney
• Nimmish Chaudhary, Vice President of International Business at WheeBox

These education and EdTech experts explore the world of education in India, Australia, and globally from the context of a current- and post-pandemic world, considering which changes will remain, which will fade, and which will develop as the education ecosystem innovates into the future.

Highlights of their discussion and a full transcript follow below:

Engagement and interaction

In the physical classroom, engagement and interaction are straightforward. Engagement translates easily into the online space, where a lecture isn’t very different when delivered on a screen. Interaction, however, requires some ingenuity. Dr. Dino Willox and Raju Ganapathy offer some advice:

• Quality learning is not a passive experience. Students benefit most when they are participating in some fashion.
• Engage students in speaking as much as possible online.
• Include polls and other input tools so students can easily give their feedback.
• With the right structure, EdTech opens the possibility of making classes actually more interactive online than in person. But educators need to partner with instructional designers to make this happen.

“There’s always a difference between engagement and interaction. You can engage people by getting them logged on, like we’ve got 54 people watching us today. But are they really interacting?”

Dr. Dino Willox, University of Queensland

Online and hybrid classrooms

When properly equipped, a hybrid-style class — combining both in-person and online interaction — may offer students a more quality learning experience than just one or the other. However:

• Content, structure, and technology should work together in the online space. You cannot simply move a lesson from face-to-face to online and expect the quality to remain.
• It is important to incorporate students’ feedback to improve the online learning experience.
• In moving to an online or hybrid style class, it is okay to fail. Students and faculty should work together to learn from failures and improve the remote learning environment.
• The online architecture and content can work symbiotically to create a greater experience than a traditional learning environment can produce.

Dr. Willox and Raju Ganapathy stress that moving into online and hybrid learning spaces is new and unfamiliar to both students and educators. It will take some time to adapt to this new environment and take full advantage of it.

“Good quality learning is not, and should not be, a sit-back-and-watch passive experience like looking at video headshots preceded by or followed by multiple choice questions. That’s not a great learning experience.”

Raju Ganapathy, Conscience

Considerations for online content

In the classroom students find themselves on more-or-less even footing. However, experiences in the online realm can be hampered by a poor internet connection, audio or video quality disruptions, and even time zones. Dr. Willox cautions us to take several factors into consideration when planning online content:

• Refer students to campus resources such as laptop loans and internet dongles.
• Establish the “minimal viable product” that can be accessible to as many people as possible.
• Be aware of which students are studying from other countries or time zones, and do your best to accommodate them.
• Consider making lectures and other resources available outside of the scheduled class time.

Moving into the new learning realm is a collaborative process. No one is entering this space with all the knowledge and skills to make it perfect. Educators can bring the content, course designers and EdTech institutions help with the structure, and students provide the essential feedback necessary to know what to adjust. It is important to become comfortable with failure and change.

“Recognizing that everyone brings to that environment something unique, and something valuable, and it’s the combination and the collision of all of those things that creates something really new.”

Dr. Dino Willox, University of Queensland

In addition to collaboration, comfort in failure, and accessibility, the panelists discuss several other considerations for online teaching:

• Students now have the choice not only of what to learn, but who to learn from. Academic institutions must work to keep up with the academic trends of new generations.
• More than ever, content should be developed to be student-facing rather than teacher-facing.
• The best learning structure may actually be a hybrid of both physical and remote learning.

“In education there’s this idea that, as a university, or as an academic, you can’t be seen to fail. We need to get comfortable with that and say this is actually part of innovation, this is part of creativity.”

Dr. Dino Willox, University of Queensland

The challenges facing assessment

In the second session of this panel discussion, Dr. Adam Bridgeman and Nimmish Chaudhary speak with Margo Griffith about the challenges that higher ed institutions face with examination in remote learning.

• With examination occurring more and more online, and with the proliferation of the internet, assessing pure knowledge of a subject is becoming insufficient.
• Educators should look to assess more than just students’ knowledge and consider assessing their skills and abilities as well.
• The largest problem facing current assessment practices is scale.
• Testing hundreds of thousands to millions of students, while assessing skill over knowledge, and ensuring integrity is a very complex issue.

“Assessment [of knowledge] is probably pretty not fit for the modern world because Google and other search engines are just there and available, and so why would we restrict our students from them?”

Dr. Adam Bridgeman, University of Sydney

The workforce is also encouraging skills over knowledge through their hiring practices. Employers want to know what their new hires can contribute to the organisation, not just what they have been able to memorise in school. In light of that:

• Assessments may change entirely to be more portfolio- or micro-credential-based so that employers can see a student’s abilities rather than just their knowledge.
• Educators also need to distinguish between what methods are appropriate for formative assessments and which are preferable for summative assessments.
• To maintain integrity in online and remote assessment, educators must authenticate the person who is taking the exam, ensure the exam is not leaked, and ensure that the exam is being authentically taken on the other end.

“It has to be assessment for learning to make sure that we’re assessing the things that we want that are valuable to students, and valuable to their employers in the future, and valuable to us for giving them rich feedback; not simply, “Yes, this is correct or incorrect.”

Dr. Adam Bridgeman, University of Sydney

Full Transcript

Cherie:

Hi everyone, thanks for making the time to join us today. I’m going to introduce two important people that will be part of today’s conversation, and first of all I’d like to introduce Dr. Dino Willox, who is the Director of Student Employability at the University of Queensland. Dino’s work spans professional, academic, and extracurricular spaces, taking a multi-dimensional strategic approach to embedding employability across the University of Queensland. Dino is the Chair of the Employability Group, a member of student experience, and global mobility groups for under 21, a global network of research intensive universities. Dino, I think it’s fair to say you keep yourself fairly busy.

Thanks for joining us. Rather than me put in my words what you do from a day-to-day perspective, do you want to maybe introduce yourself and some of the cool stuff that you’re working on?

Dino:

I work across the University of Queensland with academics, and professional staff, and students to create a wealth of different enrichment opportunities for students to get involved in, both within the classroom, but also in the co-curricular and extracurricular space. The focus at UQ is really on experiential learning, and that can be in any aspect of your life. Really, what we’re trying to do is make sure that we offer as many different opportunities to as many different students as possible, to engage in as many different places and spaces in the virtual as well as the physical so that they can actually reflect on those experiences, and learn from them, and really translate that into value. Value for themselves, for any organization they work with, or value in their own institutions, or their own organizations if they want to actually become founders. So a whole range of different ways of doing that, with a whole range of different people, in a whole range of different places and spaces. It’s fun.

Cherie:

Awesome. Thank you. And lots of points to come back to. I’d also like to introduce Raju Ganapathy, who is the CEO of Conscience, an education technology company. They partner with the world’s leading education and professional publishers, K-12 schools, colleges, universities, associations, test preparation companies, and training companies that teach and assess as a distinctly measurable student outcome. Raju, thank you for joining us, and I’m going to ask you the same question, or ask you to also give an overview of what you do in a day-to-day perspective as well, please.

Raju:

Thank you, Cherie. I think it’s my pleasure to join this meeting. Just to add to what you just mentioned of my first level description, I think I’m front and center wherever I have the opportunity to really participate on structural and instructional design methodologies that we undertake in creating these compelling learning experiences. So if you’re talking about student experiences in a digital world, and more so now in a hybrid world where you have touch points or high touch classroom environments to a less touch or a no-touch experience, which is completely self-paced and on demand. How are we really focusing on better learning experiences? How do we create sufficient learning books? How do we get students coming back for more because there is so much out there? I think I was listening to the last five minutes of the earlier session that Mr. Aditya Gupta was talking about, and I allude to a lot of those points. There are so many EdTech companies in India and globally, and everybody has a solution. But which one is better than the other, or which one is more suitable and more appropriate? I focus quickly on the design principles that underpin the structural sides of these learning dissemination processes, as well as obviously the pedagogical, which is either andragogy or pedagogy in the K-12 and higher ed space. That’s me at a high level. Again, I’m the CEO of Conscience, which is an Indian EdTech company. But we’re servicing largely the global markets, the US, the UK, Australia, and of course all of these companies that we work with have found themselves an audience in India as well. So by that virtue, we’re pretty much working across continents in delivering services.

Cherie:

Awesome. Thank you. This is a conversation about student engagement, and one of the interesting points I always find is to ask what’s your definition of student engagement, or what does that look like for you? Dino, I might put you on the spot and ask for the things that you look for.

Dino:

I think one of the things that I always think about is there’s always a difference between engagement and interaction. You can engage people by getting them logged on, like we’ve got 54 people watching us today. But are they really interacting? So one of the questions for me is, “How do we make people interact, especially in the virtual space?” It’s very easy to get people in the physical space to interact and get involved, but engagement and interaction are two different things, and really what you need is data from the systems that you’re using to be able to say, well are people just logged on? Are they engaging? Are they talking to us? Have we got polls going? How do we make this an interactive space? So it’s not just a passive reception of information. It’s really engaging people almost in a kinesthetic way, getting them to actually interact with the technology, and I think Raju can actually speak to how some of the design of EdTech can make that happen, and cause that to happen, and I think there’s some really interesting ways that new EdTech is causing that to happen in ways that are actually better than in the physical space.

So being able to have people working on a virtual whiteboard where they’re actually engaging in ways where they probably wouldn’t in person, but they can through the technology. So I don’t know if you want to speak a little bit to some of those really innovative technology spaces that are better than the physical space.

Cherie:

That’s a great one. Raju, over to you. How do you define engagement?

Raju:

Absolutely, Cherie. I think I would only add to everything that Dino has just talked about on this opening note, which is that when we’re looking at the design process, which is where I squarely participate, I would break this down into two parts: one is the structural design, and the other is the content design, because you need for structure to meet with content, especially when you’re creating the digital experience. Now, more often than not, or in almost every scenario, you need a platform which more likely is a learning management system, or it could be something else, but for the most part it’s basically a learning management system which houses the content delivery experience.

Whatever it is you take from an instructional design perspective, whatever methodologies you apply scientifically, we heard about micro-credentialing, we talk about basically the learner journey that you can see as a course designer, as a course developer, as faculty. You need to make sure that the two aspects of the platform, which is what I call the structural design, mix with the instructional design that you come up with in your learning journey. And of course, through that process you have so many touch points.

I also oftentimes say that good quality learning is not, and should not be, a sit-back-and-watch passive experience like looking at video headshots preceded by or followed by multiple choice questions. That’s not a great learning experience. Yes, we may hit check boxes here and there, but is that really going to increase the efficacy? Is that going to retain your students and their knowledge? More often than not, we’ve seen disappointing statistics in those areas. So we focus on looking at structural design. We’re looking at how the platforms are conceived, and I think, Sherry, the real starting point is, and this is a problem, this is a 30,000-feet above the ground, I say that as a problem statement, mindfully, but really if you look at any good tech product that’s there in the ed space, you really need to look under the hood and see that first thing first, which is, “Have we got the instructional designers or the people that come from the teaching and pedagogical background to really get on the same table, to get on the same page with people who’ve gone ahead and built the platform?”

So it’s getting the software developers and the learning designers on the same page to conceive your product from step one to step last. The ones who’ve done it well are the ones who are really reaping the rewards, and we’re just getting started.

Dino:

I think, just to pick up on that as well, one of the things that Eddie talked about in the previous presentation was actually about game-based learning, and one of the things there is it’s really thinking about how to use the structure of that technology to engage people, and have them interact with each other, and actually then link that to the content, and link that to the learning that’s happening. So the structure of the EdTech causes the learning to happen by virtue of that framework, and the architecture behind it. So you have to have the architecture and the content working symbiotically to actually create something that’s new, that’s actually more than what you would get if you were in a traditional learning environment. That’s what I’m more excited about, really, in this space, because you are getting new ways of connecting those things together, and I think the other thing that’s really important is that you have to valorise the ability to create that as much as create the content. So, to create the structure, and the framing, and the technology that goes with that is equally as important as the content if you really want this to become a truly exceptional educational process.

Cherie:

I’d have to completely agree with you on that one, and if I draw upon a particular experience for open learning, one of the key things that we stand behind is that our learning platform was designed and is founded in a learning science, which is Social Constructivism. The entire platform is designed with that in mind, and our learning design approach is carried throughout.

But one of the points of conversations with the almost 180 educational partners that we partner with is often how do they get started or how do they optimise what the platform enables them to do. That’s not a one-size-fits-all approach. So it’s not just who’s the target audience, and the delivery mode, and that piece. But it’s also the consideration when you’ve worked in a particular learning management system or platform, quite often there’s a risk when you change platforms or you test out new platforms that you take the exact same content and use the platform in the same way you’ve used the previous one. So how do you vary that? And if I talk about a particular use case for one of our partners, which is High Resolves, and they deliver ethics and social justice programs primarily in K-12 schools, not just in Australia, but globally as well, and they already had astounding NPS ratings for their face-to-face delivery. But one of the pieces is they had highly engaging face-to-face workshops. So how do you redesign that in an online environment?

And what’s been quite cool is that actually the concern of losing out that interaction and that interactivity, some of their programs are not only more scalable and repeatable now, but in many instances they’re getting NPS scores of 90 to 100 for those programs.

So, with that in mind, I might hand over to Dino. With your programs that you run. How did you make that shift to an online environment over the last 18 months? What were your consideration points?

Dino:

The first consideration point, honestly, was to get it done quickly. So we transitioned all of our learning to online within a week. That speed of activity really doesn’t necessarily give you the opportunity to think through how could you make this or that thing that I was talking about, that symbiotic relationship between the technology and the content, to make it really exciting. But what we have done at the University of Queensland is take a student’s partner’s approach. So everything that we’ve done, we’ve actually worked with our students to say, “Okay, what works with you, and for you, and how can we actually develop this, and co-create something going forward?”

So the initial switch was just quick, and it was a lot of that just lift-and-drop, and lift-and-drop into an LMS that was there so that we could actually maintain the learning happening. But really what we then did is that, okay, we know that’s not ideal and we know that we need to do better. So let’s actually do it with students so that we’re working with students, with EdTech companies, and with third-party providers as well, to say, “Well, what products are there in the market that we can use?” And there’s a whole wealth of different products that are working, which can be great from the student perspective and from the learner’s perspective, because it’s very agile. Universities are not necessarily very agile places, so you’ve always got the issues in the background of connecting the data, and connecting the systems, and making sure that we’re getting from all of those different systems data that actually helps us continue to develop, and grow, and co-create those learning environments with students.

But I think that really has been a fundamental priority for the university, to not assume that we know, neither assume that the student necessarily knows, neither assume that the tech company knows, but actually work cooperatively together to create something more because, again, and I think it’s one of the things that has been very challenging for academics, I would say is all of a sudden they’re not the experts. They’re one expert within an environment or within an ecosystem where we have other experts. So we have EdTech people and EdTech companies who can say we’re actually the experts in this design space, and we’ve got students going, “Well, we might be experts in how to use some of this technology because we’ve grown up with it, and we’re digital natives, and maybe some of our academics haven’t got the same experience, the same life skills.” Recognizing that everyone brings to that environment something unique, and something valuable, and it’s the combination and the collision of all of those things that creates something really new.

So being open to that possibility and that co-creation, I think, was really essential in pushing things forward, and that’s just my experience. I’m sure with 52,000 students and six faculties that there’ll be other experiences out there, but that’s certainly been my experience.

Cherie:

Awesome. I’m going to throw out both of you now a question that’s come up in the chat. How do you take into consideration, in that design process, student access to devices and connectivity in terms of being able to provide reliability in that learning experience?

Dino:

That’s really interesting because a lot of our students do work integrated learning in remote-rural and remote areas where the connectivity to the internet is not necessarily great. So making sure that what you’re providing is something that can be accessed online, either through that at the moment I’m at home and I’m working off a dongle. Seems to be working okay. But making sure that people have access to that. But also, at the University of Queensland we have laptop loan options. Students can get access to laptops if they don’t have their own things like that.

But you really do have to think about what do people need and what’s the minimum viable product that we can produce that is accessible to as many different people as possible? We’re not just looking at people on campus. We’re looking at people at home, we’re looking at people who are remote, we were looking at people offshore who are also still connecting, and we’ve got a lot of students even now who are still studying in-country offshore. So we’ve got to make sure that we’ve considered not only the tech, but the bandwidth potential and also time zones. There’s a whole bunch of things we need to think about and do you provide things live as well as to watch afterwards. It’s trying to balance that is always difficult.

Raju:

I couldn’t agree more with what Dino just mentioned. If I’m just drawing the angle of the design aspect of access to devices, I think again when we go through a design process of how do you deliver this training and how do you redesign this training for the new tech stack that’s available? I think we also look at it from the perspective of which content lends itself best to what type of environment when students actually have to include with it. So if you go through a learner journey, as an example, where you start with, let’s say, some amount of teaching content, then you have a lot of material to read with pictures, images, visuals, etc. Then you start introducing media, probably as a way of reinforcement. Do you reinforce the learning that you’ve just given in the last 15 minutes? Some of those pieces can actually be given away for a mobile first environment, as against a desktop environment, and it’s not a sit back and watch, a read experience, even of a printed textbook or a didactic learning experience.

We’ve been trying this for much before the pandemic, for the last 15 years or so, but you can very well see students actually taking some type of formative assessments, even on their mobile phones, when they’re in the underground train in New York, or London, or any other place, and you can apply the same. Even with offline access to content, you could do that in other developing countries where the network and connectivity is not that high as well. So, which content lends itself where best in the entire learner journey is one that also needs to be paid attention to when you start at the outset, just to add to the network and connectivity part, that data just talked about.

Cherie:

I think it reinforces Dino’s point earlier about working collectively through that design process rather than feeling that you have to get it to a certain point before handing it over or being ready to go live. It can be an iterative process, working with a particular partner and students, etc. And then, how do you use that data for each iteration to improve or feed through for further intakes or cohorts of learners that are going through the process, as well?

Dino

I think that, just to pick up on that challenge, that idea, of it being an iterative process, learning itself is an iterative process. So I think we’ve almost, through this pandemic, we’ve become more honest about the fact that we’re trying things, we’re trying different things, and maybe they don’t quite work. But, okay, let’s work together and find a better way of doing that. I think within that, there is a recognition that what we create within an educational environment is a place or a space where people should be trying things out, and being safe to fail, and have early failures, and going, “Okay. We’ll learn from that, and we can do better.”

Even within EdTech and within the entrepreneurial space, it’s really par for the course that you try something and it fails, and you learn from it, you move on, and almost in education there’s this idea that you can’t, as a university, or as an academic, you can’t be seen to fail. We need to get comfortable with that and say this is actually part of innovation, this is part of creativity, this is just actually what happens, and we have to recognise that sometimes we don’t quite get it right, and so we’ll learn from that, and we’ll make it better next time. So that iterative process really needs to be front and center, I think.

Cherie:

That’s one of the challenges, even myself having come from a teaching background is that sense that I have to have all of the answers and I have to be able to do it all myself. There’s the concern or risk in being in an online environment that often comes up is that sense of isolation or that you lose that human connection, whereas actually there’s the positives that come out of it in that human connection of being able to say this is an iterative process, and then the other side is how you build that community of practice and community of learning so that it isn’t you as an academic, or a teacher, or facilitator having to be there, the sage on the stage the entire way through.

I guess that leads me to the question that’s come up, and it’s one that I get asked often, as well, on a daily basis. “How do you support or guide academics or teaching staff through that change process?” Raju, I’m going to hand over to you on this one because you have a wealth of experience, not just during the pandemic, but well before that. So what’s your guidance or advice?

Raju:

I’m going to take a lead from the sage on the stage example that you just cited here, Cherie. I think we’re just dwelling deeper into points we’ve already opened up in the last five minutes, talking through things. The first thing I think is, again, if you look at the higher education system globally, Australia, India, US, anywhere, we’re talking about hundreds of years of institutional practice to get it to where we are today. Five years, we’re looking at brick and mortar. We’re looking at physical campuses being built. We’re looking at getting students inside the classroom. We’re looking at putting the facilitator or the faculty in the center of the class and then going about the process of conducting teaching, learning, and dissemination.

What has happened over 300 years in that format is now being quickly questioned, challenged. I say that respectfully and mindfully here, but that’s what we’re dealing with. With the new Millennials and the new students, it’s not about just what I want to learn. It’s also about who can I learn from, which is also a choice now in the hands of the learners. So obviously when you have these rather evolutionary changes happening under the feet, we really need to look at a redesigned process, which alludes to the fact that when content was originally created in the traditional format, which is the high-touch classroom environment that we all know, the content was really created with the perspective of actually making it more teacher-facing and faculty-facing than directly student-facing. It’s a facilitation process where faculty takes that PowerPoint slide from the overhead projector and uses his or her soft skills to teach and control the students, and have that interaction in a one-on-one, completely 100% high-touch environment.

With a lockdown and a pandemic, unfortunately, the world needed a pandemic to talk about the change of scenario. But that has all changed. Right now we are the position in the triangle, as I like to call it. You have the student, you have the learner, and you have the content. The position of these three pivotal points has rapidly changed in just about a few years, and moreso in the last 18 months. To answer your question, I think I would say it’s about an overhaul of the content design process, such that you make that content not teacher-facing but directly student-facing. It’s easier said than done. That’s a huge transformational processes, and for the ones who understood it, have done it well, have released. I have really started to see the rewards already in so many years, and we’ve seen some of that very closely.

That’s where I oftentimes say that the “what” of the teaching, if you ask me, is the prerogative of the subject matter expert because he or she knows what it is they teach. They’re talking about, in any subject area, but the how-to of that teaching is where you can get the handy help of seasoned instruction, good quality instructional designers, because they do not understand the content, the “what”, the subject matter. But they definitely can help with how you can take this to the last mod in terms of the design process, in terms of the distillation process.

So that is the first phase. It is about really taking that content and making it from a didactic, teacher-facing experience, to making it more directly student-facing in that you can bring in all those things. Just a few minutes ago Dino talked about creating scenario-based experiences where it’s perfectly fine to fail. That is what you need. You want students to go through a process where there’s not always one way of getting enlightened at the end of it. You have to make your mistakes. You have to fail before you come back on the right path.

So using media and using interactive videos… Again, in a video experience like this, it should be an environment where, if it’s a roleplay, if it’s a dialogue, if it’s a scenario-based learning experience, you could create that using animations, graphics, illustrations, or you can even put up a head shot of a key opinion leader for a five minute video. But you do not have to run that video for five minutes, and students are then going off to their WhatsApp messaging. You can actually intervene the five minute video with cue points where you’re actually allowing learners to participate. I’ll just give that as one example, but that’s a classic interactive video experience in a good quality course.

So it’s about bringing in those touch points. On the other extreme, you can create simulations like a Physics or Biology lab simulation. Let’s take a gram stain. It’s a 37-step process of how you do a stain and how you find the results, and students have to do that, and that’s an extreme form of creating simulations with rich media interactions. But even a basic one could be taking a five-minute video headshot of a faculty or key opinion leader, but not letting it play on its own. But getting those interactions, allowing for objections, and probably even looking at remediation pathways. So you’re actually taking them into different parts and getting them out. I’m going in a different direction and another student is going off in a different direction from a key intersection gateway point.

Cherie:

Great thoughts. Do you have any guidance or texts as to how you approach it?

Dino:

I agree with everything that Raju said, and I think that the thing is that it’s the “what” and the “how”, and I think in bringing those two things together the critical thing for me is to have students involved in those conversations as well because that really gives you the “why”. Why are you doing this? So there’s an element there about why do the students want to know this and how can we make it better for them? But is it working? And get that feedback. So getting real-time feedback, and that comes back to the data again. That very first question about the engagement and interaction, getting that from the students and making sure that that’s feeding all the time the co-creation and the continued development of that learning environment, because that’s how you’re going to make it work.

Cherie:

Definitely. I think the piece that ties that all together is how does this all align to the learning outcomes that we started out with the intention of delivering and being able to demonstrate against as well. I think reflection and building opportunities, not just for the learners, but for the learning design staff and the academic staff, is a key part of that process. I could keep talking for hours, but we’re running out of time very quickly. So if I can ask Dino, do you have two or three key takeaways from a student engagement perspective to leave everyone with today?

Dino:

I think that the virtual and the physical spaces and places for learning provide different opportunities, but they actually need to be connected to have maximum effects. You can’t just be virtual or just be physical. You’re going to have to somehow bring those together, and thinking about collaboration and creating synergies between all of the people involved in that, the human connection, is fundamentally important, and that human connection can be facilitated in a virtual environment. But ultimately we need to make sure that that connection is two-way, that we’re listening as well as speaking, and that we’re actually designing things that engage people and get them to interact. It’s not just a didactic one-way thing.

I did get asked before we started this, what would your one takeaway be, and I want to leave you with a quote. I studied philosophy originally. “The only constant in life is change.” That’s from Heraclitus, born in 540 BCE. So we’re still in that place now. Everything is changing still, and we need to embrace that change, and enjoy it.

Cherie:

Amazing. Thank you so much for your contribution and your time today. Raju, what would be your takeaways?

Raju:

Sure, Cherie. I think I would allude to the fact that everything that we do here in terms of design and experience also has to be one that’s actionable and data driven, which I think was a point that you led into in the first part. So that is a very important piece: getting actionable insights from anything that you do. Especially in an iterative process, you have version one and then you end up with a version five over a period of time. We need to be able to get the right type of actionable insights and we need to design that effectively for all stakeholders in the system. So if it’s a learner, I mean as a student, and these are some other things that coincidentally we’re working on very closely at this moment, even in the form of a product. But as a student, what do I need to see? I need to see how I am doing, what my progress is in a course if I have a 12-week course or a 20-week course. How far along have I gone? What activities have I taken? What is still in progress? What is completed? What are the mandatory activities? I need to be able to get all of this data. If you talk about student engagement, they need to be again hooked to their experience, whether it’s physical or whether it’s digital.

So looking at some of these parameters, like my progress in the course, my pace, I would say the pace is the equivalent of your heart rate when you step up on a treadmill. If I have 300 students in my peer group, how am I doing events up against the others? So the pace indicator is a vital indicator, and of course, finally, the performance, which again is looking at formative, summative, and really how have I done in this course? How do I look at my assessments? How am I scored, and how do I go to my remediation? All of this vested with a good interaction, with a student success manager or a director of student success if you have that through a learner journey will call for a nice, compelling experience. I can see that people are looking at their watch.

Cherie:

I think David’s popped back up, so that’s the final wrap-up.

As soon as I start hearing researchers and academics quoting the philosopher who created a saying that we use every single day from 545 BCE then I realise it’s time for me to come back to bring the conversation back to EduGrowth normal. So, thank you. Thank you, Dino, and thank you Cherie for an amazing conversation about engaging students in the digital world.

David:

That gives us an opportunity now to welcome Margo Griffith to the stage, who is from Edalex. Margo leads growth at Edalex, which is an Australian EdTech platform that is doing amazing things around micro-credentialing, and supporting learners, and showing the evidence of that work. But Margo’s been in the higher education space for a very long time and we won’t put numbers on things. Margo, we’ll just say that you are well known to our audience and you have got a long career here. So I’ll hand over the stage to you.

Margo:

Thank you very much, David. Very excited to be here and I appreciate you not dating me with the number of years I’ve been in the education market here in Australia. Yes, super excited to be here to talk about assessment and opportunities in assessment, but also innovative assessment models. I’m going to be joined by two wonderful people, the first of which is professor Adam Bridgeman. Welcome, Adam. So Adam is or has, according to my piece of paper here, decades of experience as well as an educator, and he’s also been the Pro Vice Chancellor for Educational Innovation in the Education Portfolio since 2018 at the University of Sydney, with a focus on using electronic resources in chemical education. Adam has specialised in the effectiveness of electronic resources for enhancing education, including the use of simulations, calculators, and games on the web and on mobile phones. Adam holds a PhD in Inorganic Chemistry from Trinity Hall in Cambridge. Welcome, Adam.

Adam:

Thanks, Margo.

Margo:

And I will introduce our other colleague here, Nimmish Chaudhary. Nimmish is the current Vice President of International Business at Wheebox, an IT company that offers technical assistance to aid seamless pedagogy. Wheebox provides proctored, AI-driven, safe, secure, and scalable assessment and also LMS solutions. Nimmish received an MIM of Management at Charles Darwin University and also has a Master of Arts from Chaudhary Charan Singh University. So, welcome Nimmish. Thank you, both, for joining today.

We’ve only got 30 minutes today, so I guess we’ll jump right in. If I can throw to you, Adam, please. We are talking about innovation and innovative models in assessment. I’m hoping you can share with us the pandemic and also post-pandemic plans for assessment at the University of Sydney for your 70,000 students.

Adam:

Thanks, Margo. Yes, 70000 students, about 4,000 or 5,000 staff who are research focused. We’ve been thinking about changing assessment for years. Suddenly last April everything went online, so obviously we started having to really rethink our assessment very quickly, and I’m getting through last year, and we’re now planning for what is going to be next year and what’s going to be the new normal, which is definitely going to be much more online than we ever would have planned.

Making the best of online and face-to-face, so that’s the challenge. I’ve spent the last 14 months trying to get online assessment to actually work at scale, and now we’re thinking about what that’s going to look like in the future.

Margo:

There’s some huge challenges there, Nimmish. I wonder if you’d be able to share with us around the Asian environment within India. We know COVID from afar, but what did that mean, in particular, what does it mean for assessment in India?

Nish:

Margo, there were abundant challenges which were there in terms of how to assess people, especially with about 1.35 billion people. And, in universities as Adam said, there are about seven thousand students in our universities. We are close to about two-and-a-half hundred thousand students which actually appear for assessments. Managing that crowd and managing that scalability was missing, and that is one of the biggest challenges the entire economy faced. The entire education fraternity phase, and I think since we’ve been operational for the last 11 years in India, we had some large ideas as how to plug in and fit in the model where we can offer an AI-driven remote proctor, customizable assessment solutions,  I think we seeped in into that change which was forced upon us. As somebody said, that change is inevitable. That’s the only thing which is actually constant and permanent.

Without 11 years of experience we realised that this is the time probably where we’ll have to innovate more, because by the time we were sitting in 2018-19 we thought that we had invented and reinvented the wheel. But by the time we came to mid-2020, we realised there are a whole bunch of new requirements which suddenly mushroomed-up, and we didn’t know how to address them, and probably today sitting in 2021, we as Wheebox believe that we take another three to four years to innovate more to address what are the requirements in terms of the industry’s expectations.

Margo:

There are many challenges and I’m understanding that completely around the move to online assessment. Adam, would you care to outline some of those challenges I’m sure you’re facing as we speak?

Adam:

Those challenges I think are partly in the way that we have assessed, we’ve tended to assess knowledge and testing knowledge is just unsuitable for somebody that’s got access from home to the web. But that’s telling us something. That’s telling us that assessment was probably pretty not fit for the modern world anyway because the Google search engines are just there and available, and so why would we restrict our students from them. Doing this on a scale is a massive problem. Ensuring integrity is a really big problem. So when we’re rethinking assessment for the modern world, it has to be assessment for learning to make sure that we’re assessing the things that we want that are valuable to students, and valuable to their employers in the future, and valuable to us for giving them rich feedback, not not simply, “Yes, this is correct or incorrect.”

It’s those skills that the students will need that we need to assess, and doing that online is obviously a huge challenge, and some things you probably don’t want to do that way because you don’t want your surgeons necessarily to be assessed in a VR model for forever. You want them to actually have done that on a patient at some point, and so there’s lots of challenges. Students crave feedback. We know across the world that that’s something that we do really badly, and the University of Sydney is certainly no exception to that. So how can we improve feedback?

I did like at the end of the last talk where Dino was talking about the importance of failure. I think that assessments where students can get feedback, and act on that feedback, and that feedback helps them to improve and have high expectations for the students that they will fail. Life’s about failure and learning from failures, and feedback is about doing that. So those are some of the challenges that we face.

Margo:

That’s so important isn’t it, that the personalization of that feedback too, that’s specific to the individual’s requirements. Nimmish, from a technology perspective, how do you feel that technology enables or supports some of the challenges that Adam had highlighted?

Nimmish:

Margo, see, every challenge we believe is a source for opportunity. I mean the moment we have something which is actually bottling us up, we realise that there is tremendous potential where we can explore the entire opportunity we have and turn it into an advantage, a technological advantage. You could serve mankind with about trillions and trillions of brain cells that we have. Eventually technology will be only an assistance to mankind. It’s a very strong misbelief that technology might overcome something. But then we will be the masters for the entire technology.

Absolutely. I agree with that. The challenges are in terms of security. So, how do we make sure that the entire platform is safe? It is secure. There is no data pilferage. And I reckon with different continents and with different countries having their own standards in terms of maintaining security, if I look at Australia there is a 1988 privacy act, and when you come to Europe there is a GDPR protocol. Then there are ISO certifications. So there are a whole bunch of security issues which need to be addressed, which need to be incorporated, that give us the opportunity to offer a wholesome, holistic solution where we can come down to our clients and tell them, “Guys, this is what we are able to offer you in terms of security, in terms of your data, in terms of data privacy.”

I’ll just give you one small quote. A couple of months back we had one client who said, “All is fine, but I still do not want to share my students’ details.” Well, we said, “Well, that’s quite a challenge we have.” But after a few deliberations we just gave him a solution. “Okay, what you do is you give me a ten digit code allocated to every student. I do not need the entire details or nuances of that particular person and we’ll be able to execute a test.” And that’s exactly what we did.

So the second challenge, which I believe is the scalability issue. If you’re talking about a couple of hundred I guess that’s manageable. A thousand, that’s a little difficult. But when you come to about hundreds and hundreds and thousands of students being conducted at the same time, and having that scalability.

In February or March, 2021, we came across a challenge where we were supposed to address assessment to one million students in a span of 8 days. So that was like conducting 180,000 examinations every day, seamlessly, flawlessly. That actually gave us shivers. But by the time we sailed through that process, we realised we can do it because we’ve done it. It’s just a matter of scalability. So that’s exactly what we did.

And the third challenge was about customization, in terms of how do you support what exactly is the requirement, and today I think, as I said, by 2018-19, we realised we’ve achieved everything. But today we just started to walk. There’s a long way that we need to go and achieve in terms of the solutions that we offer to our clients across the globe.

Margo:

Yes, I don’t think we really have a real understanding of the sheer scope of the numbers that you’re dealing with in India. And with the high stakes exams that you would be conducting as well, that security is just simply everything.

Can we talk a little bit about how things are shifting, because there is a new normal out there. There is a recognition that authentic assessment, personalization in the assessment and the feedback. Adam, how do you feel that assessment strategies are changing? And there’s a two part to this too. What will it take from a professional development perspective to enable that to happen as well, because we know that there has to be a shift and lift with the academics themselves?

Adam:

You’re absolutely right. The fact that you can’t just assess knowledge, or it’s very hard to assess knowledge, and why would you in the online environment, means that we do have to shift our assessments. Clearly employers are telling us that they want students to have certain skills and those skills aren’t all about writing an academic essay or learning the elements of the periodic table. They’re much richer skills. They’re much more about their thinking and their understanding rather than just their ability to memorise. So that is a real shift. We also need students, of course, who are able to work in some multimodal assessments, which of course is so much better suited to the online environment than it is to an examination hall. It’s something that the online environment really provides in much richer ways than we had before.

So taking hold of that and bringing those two things together to make assessments that are authentic. But to come back to your point at the end there, absolutely. It’s a real struggle because one of my roles is to improve the teaching of the academics are absolutely success stories of assessment systems where memorization and cramming were the main things. So getting them to mind shift to think that they should assess in the different ways and then to give them just the skills to do that is a real challenge. It’s something that I spend many hours every week on and, although we’re getting somewhere, we’re getting somewhere quite slowly because of the sheer volume of change.

To go back to that word again, the sheer volume of change in work that’s required at the moment. It’s something that’s pretty difficult to fit into everybody’s working day. But it is something that has to be done if we’re going to make this next jump away from emergency online assessment to something.

Margo:

I’m just going to pick up on one of the things that you said earlier around I guess assessing non-domain specific skills. Where I see it and the work that we do at Edalex, it’s really around discovering and surfacing the workplace skills from your perspective. I think during the pandemic, and correct me if I’m wrong, there was and it was achieved the move to online and an assessment of those domain specific skills. But how are you going to tackle these employability skills, these soft skills, these transferable skills? How are you going to tackle assessing those, and is there a way to tackle it at scale? I love your thoughts. Nimmish, I’ll get yours next.

Adam:

I think it is possible. I think the scale is a problem, as I said. Also, the background, the skills of the academics, is a problem. Just thinking, just looking at one of the questions in the chat is, “Academics aren’t great at giving feedback. Don’t want to spend too much time giving feedback. And they consider the feedback that they give to be good enough. Whereas a lot of students want something much richer.” And so it is possible I think. We just need to place ourselves in the students’ shoes, think about the journey that they’re going on, the challenges that they face, and the employers, the skills that the employers will want, and then shift our assessments to do that.

It might be portfolios, it might be micro-credentials, it might be small skills that they pick up along the way to personalise that journey for them, but also making sure those assessments are authentic to where they’re going, to the profession that they’re being trained for.

Margo:

Nimmish, what’s your thoughts on that? And that assessing domain specific skills and also assessing those employability skills, what’s your thoughts on that particularly with reference to the Indian environment?

Nimmish:

Margo, just adding on to what Adam said, and probably pinpointing the two main challenges that every faculty member faces, because I have witnessed my mother as a teacher. See, every faculty member has about 300 or 400 students to cater to and, presuming that it takes about 10 days or 15 days to complete one chapter, the biggest challenge is how do you judiciously actually assess every student in terms of their learning, and in terms of their outcome of the deliverables?

The only chalk and talk way was that you make them undergo a paper and pen examination mode. You assess every copy. That means if there are 300 students you are looking at about 3,000 man hours being spent accessing those 300, which is about roughly, I think, 45-50 hours of human labor going into it. So that was one challenge.

The second challenge is how do you actually come to the point where you judiciously realise that this particular student or this group of students did not understand topic A, or topic B, or topic C of a particular chapter because I know what you’ve learned. But then, how do you compute the fact that, okay, out of these seven topics that are there, in a particular chapter these three topics, two topics were not understood or learned by this segment. So these are the two main challenges now, as far as the soft skills are concerned.I think we have one product which is based on John Holland’s psychometric assessment, which is the Barrow Test.

There will have to be innovation. There will have to be new emergences of the techniques that you would involve. It could be technologically and it could be a psychometric base. It could be personality based, so there will have to be ways which will have to be evolved which will have to be mature enough to make sure that we are able to deliver. And we are able to offer a wholesome solution where these things can be taken into account.

Just last Friday we were actually celebrating and our CEO launched a couple of initiatives. One of the first new product features that will launch is called a watermark. Now see, the challenge is how do you ensure that the examination paper, which especially in case of high-stakes examinations, how do you ensure that that is not leaked or that is not passed on to a particular process? So what we’ve innovated now is we’ve called something a watermark. The moment I pass on this question paper to somebody else, I can actually locate and find the path where it was leaked. So that was another challenge which came to us, how do you ensure that the examination paper which I’m giving online does not get leaked and passed on to the other students. So there are challenges, absolutely, and as I said challenges will help you with opportunities. And that’s what will have to be taken into account.

Margo:

That’s really interesting, and thank you, Nimmish. Adam, we’ve touched on before this notion around authentic assessment and we’ve also been talking a lot about scale in this conversation here, obviously things like multiple choice questions are economies of scale. That’s the way things have been done and the way that things will probably be part of the assessment arsenal into the future. What are your thoughts around authentic assessment? Can multiple choice questions be authentic, or what does it mean to you around what would you define as authentic assessment?

Adam:

I think authentic assessment is about problem solving that means something in that discipline area. Do I believe in those questions? I find myself not doing as much anymore as I used to. It’s really difficult. I think it’s actually more time consuming to write good, exact MCQ questions. Although you save time on the marking, you have to refresh them because you have to assume that they’re out there I think, and so writing a good exam, a good MCQ is certainly possible. It’s very challenging, and I think in most cases it’s actually easier perhaps to write a scenario that requires maybe a few multiple choice questions and a short answer just to justify the approach that the student’s doing so.

MCQs are wonderful for formative purposes. They’re wonderful for giving rich feedback to the student and giving rich feedback to the teacher. But for summative assessments I find them harder to justify. I think the workload that they used to save is not the key thing. It used to be authentic assessment. We don’t tend to, in our lives, have MCQs. We tend to have things which are much more nuanced. We don’t have the right answer in front of us to choose from. Sometimes we have to work it out without the answer. It’s not that he wants to be a millionaire. It’s much more about thinking about the answer without being given those clues. Formative, yes. Summative, I’m not so sure.

Margo:

It’s really interesting. So how would you do that online? How would you do performance-based assessments online?

Adam:

I think the scale is the difficult thing. I think that I don’t necessarily believe that the traditional exam is the way that we should sit. I think we got to have these assessments which are staged, which have multiple points for feedback, multiple points for the students to build on their understanding, and I think that also helps to build integrity into the assessments because it’s much easier to see that you’re dealing with the same student consistently, but doing it at scale is definitely a problem. Maybe some combination of online assessment with oral exams, for example, is important. It comes down to scale is always a difficult problem.

Nimmish:

What Adam said is also right. But then there are another two components to it. The first component is how do you authenticate the person who’s coming to appear for the examination? That’s one. Another challenge, so I might be a bona fide student, but how do I ensure that it’s me who’s actually coming on the platform to appeal? And third would be how do you authenticate the entire flow and process during which the examination has been taken into process? So these are another two challenges. These are another two processes of authentication that need to be derived over here.

Margo:

We need to wrap up, so I would really love it if you could give the three key takeaways. I think in terms of either assessments in a digital environment or where you would like to see assessments in a digital environment. Go, Nimmish, if you can quickly tell us.

Nimmish:

Margo, again, as I said, the first part is the security and the scalability. That’s something which is uncompromisable, that is something which is of paramount importance. With the paradigm shift that we’ve seen, I think there’s still a lot of long way and a lot of innovation that needs to be incorporated in terms of technology and in terms of what solutions we bring on the platform, that there still are I think a few parameters which need to be addressed in terms of security, in terms of a cheat proof examination being conducted, in terms of the associations that you form within the peer groups which definitely is missing from the conventional classroom.

And the third challenge, and the third takeaway, is how do we ensure that we have a very strong customizable solution? Every person, every client, would have a different need, and so our challenges are to what extent can we help customise the solution that way that we require, and probably help you with those solutions? That’s our key motors and that’s the key parameters that we have.

Margo:

Thanks, Nimmish. Adam, your three key takeaways around assessment.

Adam:

Thanks, Margo. I think first of all, don’t try to replicate what works in the face-to-face environment. For formative and summative purposes use the best things that the online environment can give you. Secondly, that means I use it as an opportunity for richer, for more media, multimodal, assessment and more authentic assessment too, and thirdly I think we haven’t really touched on students’ concerns around privacy, but try to explain you have to students the benefits and the flexibility getting them through their degrees in a pandemic. But also the flexibility and the richness of the assessment, and I think then everybody’s responsibility, the students’ and the staff’s responsibility, and integrity being as beneficial for the students as it is for the staff. And that’s why it’s important.

Margo:

Oh, and look at that! Here’s David.

David:

Look at that. Right on time as well. So, Adam, Nimish, Margo, thank you so very much for giving us some insights into assessment in the digital world, and there’s no doubt that this hybrid and online education environment is not going away anytime soon. So it’s exciting to hear that we’ve got some deep people, some people thinking deeply about what the future looks like. So thank you very much.