In the recent issue of the Ex4Edu.Report we started the discussion about how technologies such as ChatGPT and the growing array of a.i. tools will change higher education. In this issue of the Report, the topic of adapting assessment strategies and processes in light of the ChatGPT and the growing array of a.i. tools will be the key focus.

Share Student360.Report

More than tests and grades

A few years ago, I was making a presentation to an economic development agency in Berlin about locating a venture in the region for data analytics to support higher education institutions. The meeting was fairly typical until the end when the proposal was dismissed as “just tests” and not worthy of further consideration or funding.

The view of some highly trained development officials, unfortunately, reflects a prevalent but mistaken view. A good friend, Ken Gergen of Swarthmore College, and I have had several similar discussions around his book written with Scherto Gill, “Beyond the Tyranny of Testing: Relational Evaluation in Education.”

What Ken and Scherto clearly identify is that educational assessment has been captured by a grades and testing mentality, or as Ken has put it, “the testing enterprise.” Breaking out of the testing and grades box requires moving beyond the assessment instruments (such as tests and grades) to thinking about what we can and hope to accomplish in education.

When I read articles predicting systems collapse in academics due to ChatGPT or any other ai technology, the authors are usually stuck in the mud of viewing assessment instruments as the whole rather than taking a broader view. It is almost like a farmer thinking that buying the latest tractor (i.e. technology) will increase crop yield. Yes, it might help, but it is not enough.

Dimensions of effective assessment

The first line of confusion in assessment is neglecting the level of focus. There are significant differences in evaluating the success of a course, an academic program, or an academic department or institution. Effective assessment plays an important but different role at each level.

One tool that I have used to assist clients in understanding the difference is shown in the diagram below.

The diagram bundles several concepts into a single presentation. First, determining the effectiveness of a program differs from a course. An academic program should be “assessed” based on the overall success of its parts, which are courses. Courses, by contrast, are more granular, which directs the assessment focus to what happens as a course is taught.

Second, there is a distinction between summative and formative elements. Summative elements are what the name suggests—the summation of efforts. Formative elements are checkpoints along the way to achieving the final result.

Last, a distinction between internal and external is drawn at the program level and direct and indirect at the course level. The internal vs. external dimension is used at the program level because it is relevant to how stakeholders would view the success of a program in preparing students for functioning in the world. At the more granular level of the course, a direct vs. indirect distinction is drawn. Such a definition helps decide how to evaluate the outcomes of various learning activities. Tests appear at this level as a direct assessment but as one of many potential measures.

The internal vs. external and direct vs. indirect distinctions cannot capture the complexity of success at the institutional level.

There is a higher or “institutional” level, which is not shown in this diagram. When the discussion transitions to the success of an educational institution, department, or organization, the way of evaluating success also changes. Instead of assessment, the organizational level is more suitably evaluated based on what is termed institutional effectiveness. The effectiveness of an institution often reflects many non-academic factors, such as the facilities, funding, the nature of the population served, etc. Although these items certainly impact student success, the mixture of academic and non-academic areas requires a different approach. A future Student360.Report post will tackle the institutional level issues.

Moving beyond impoverished assessment

The current consternation with AI tools and their potential to disrupt higher education appears overblown. The current suite of tools consists primarily of large language models, which rely on available content to assemble what is essentially a web query on steroids. As explained by Peter Ziehen, these tools are programming and programmer intensive. Replacing the entirety of human learning capability is still years or even decades away at the earliest.

The infancy of ai models does not mean that impovished assessment is without risk.

Tools such as ChatGPT can pose a risk when an institution engages in what I term “impoverished assessment.” The hallmarks of such poor assessment are:

• overreliance on a single type of assessment instrument, e.g. papers or simplistic multiple-choice tests,

• neglecting the value of observational tools and methods common in indirect and external types of assessment, and

• failure to consider the full dimensions of learning, such as experiential learning.

Moving beyond impoverished assessment requires thinking differently about the roles of the learner and teacher in an educational setting. It also requires recognition of the relational nature of the learning experience.

Embracing the multiple dimensions of the student

The shift to a balanced view of assessment starts with looking at capability as the central focal point. In short, what will be the impact of a learner or student after completing a program? Further, how will the collective impact of learners or students play out in society?

A full view of a student’s role in education recognizes that the student is a recipient of instruction (customer), the key actor (means of production), and the beneficiay (the outcome).

Framing capabilities in terms of the multiple roles of the student should be the starting design point for an assessment strategy. Key questions that should be addressed include:

• How and how well are students learning and experiencing the learning journey?

• Are students adequately equipped to learn successfully?

• Can the institution demonstrate that students have mastered and have the abilities and attitudes to use the knowledge and skills in their lives?

Leave a comment

Assessment by design - The imperative

The “jobs to be done” to develop effective and helpful assessment processes and instruments begin with a thoughtful design effort. Discussions of how assessment by design can be structured will be contained in future issues of the Student360.Report.

In the past three years, higher education globally has survived epic disruption and change. The sector has moved from pre-Covid business-as-usual, through a pandemic remote-learning pivot, and now to the post-pandemic realities challenging nearly every aspect of the existing ways of working. A sector that takes pride in its slow pace of change is discovering that gradualist change is not enough. Facing existential disruption from changing generational demographics, unstable employment markets, and EdTech 3.0+ will require a different orientation.

Going forward into 2023, the focus of our two newsletters—Ex4Edu.Report and Student360.Report—will be refined yet remain with the purpose to support navigation of the changing world of higher education. 

The Ex4Edu.Report will focus on significant trends and events at a macro level that impact the sector’s viability and ability to deliver value. The guiding idea is navigating change rather than attempting to manage unmanageable change. Understanding how to navigate change is one of the “5 Disciplines of Highly-effective Organizations” observed through years of observing and working with role-model, resilient organizations. The Ex4Edu.Report will focus on identifying and interpreting key trends from the viewpoint of how to navigate change effectively in higher education. 

The Student360.Report will concentrate more on the nuts and bolts of delivering value from a student-lifecycle optic. The emphasis will be on good practices in designing, delivering, supporting, and evaluating learning in higher education. The term “good practice” is intentional and avoids the pitfalls of notions such as best practices—which are, in reality, mythical—and confusion of one-off successes—something that works in limited surroundings—with real solutions fit for multiple settings and cultures. 

Both newsletters will remain free of charge for now. Our development of a cooperative venture in 2023 could lead us toward new, specific levels of subscriber content. The more specific desires and needs of those joining us in the co-op may suggest paid tiers as we co-create our economic model. Information will be provided as the developments unfold.

Share Student360.Report

We also plan to develop expert chat sessions—using the new feature of Substack—once we develop a sufficient subscriber base. For now, please enjoy the content of one or both newsletters. And, of course, your helpful comments on the topics posted are genuinely appreciated.

Leave a comment

Subscribe now

Bridging student touchpoints to insights requires an understanding of identifying data that matters to student success.

Avoiding two common mistakes with data points

A key part of using the Student360 model is using data to guide our work in helping students grow and succeed.  Many educators buy into this idea and tout themselves as “data-driven”. At one university where I served, the mantra was that “we measure everything that moves”. Both approaches typically lack a focus on data that matters for student success.

What is typically described by data-driven often amounts to using standardized testing to guide curriculum and instruction decisions. This approach is simply too narrow. Student success, looked at as persisting and completing education, is influenced by more than what is measured in standardized test scores.

Those claiming to measure everything that moves commit one of the deadly sins of performance management. Measuring anything and everything leads to unneeded costs, bloated information systems, and piles of unused data. More critically, oversampling of data dilutes focus from what is key to helping students succeed. The movement towards key result areas (KRAs) in evaluating the performance of companies provides a good contrast.

How to avoid these mistakes? For one, think of measuring across the student lifecycle rather than just during the admissions or learning phases. Also, for each phase of the student lifecycle, measures should be selected that have meaning in terms of reflecting student success.

A Data Architecture for the Student Lifecycle

An understanding of student success needs to start with a clear focus on key result areas or KRAs. The word “key” in key results areas tells it all. The process of identifying and selecting data points from student touchpoints should focus on strategies that will help students stay in programs and successfully complete their studies. It really is as simple as that.

We need to counter the promises of IT that come with each generation of new technology that we can scoop up massive amounts of data and magically derive helpful insights. Today it may be AI or machine learning but two decades ago it was relational databases and a decade ago it was data warehousing. A reasonable process actually does not take massive pre-investments in technology but good use of smart people.

Start by asking who has the most contact with students. In a typical student lifecycle in higher education, the three groups with the most contact with the dynamics of student success are, in order of level of engagement, faculty, student affairs, admissions, and alumni relations. These groups often have not only insights from an institution’s current and past students but also the broader range of academia. 

Workshops can be used to help tease out the key result areas (KRAs) in each of the student lifecycle phases. A technique that has worked very well in my practice has been to facilitate towards a limit of two to three KRAs per lifecycle stage. Such an approach ensures that the results areas and indicators are truly key for success rather than just representing activity. It also forces a focus on the student rather than the activities of the institution.

There are many techniques to leverage institutional knowledge to develop KRAs and ultimately ways to measure and represent success. One technique that I have used is based on the Ishikawa or fishbone quality tool and uses the approach to focus on meaningful data. Briefly how this approach works is to alter the focus from problem-solving to solution generation.

Ideally, the approach should reflect all student lifecycle phases either with a separate analysis for each or grouping two together.  My favorite framework is the 6M model as shown below with education-specific designations and the lifecycle phases.

Rather than a problem being diagnosed, the focus is on what defines a successful student in each lifecycle phase. The branches of the diagram can then be used to brainstorm success approaches in an appreciative manner. Once a picture of success is painted for each of the lifecycle phases the analysis turns to developing result areas and supporting data with measures. 

From KRAs to Data to Success Insights

The use of KRAs to develop data and success measures next leads to insights for success. Future issues of the Student360.Report will focus on ways to glean and use insights for student success.

About the Student 360.Report

The Student360.Report is a commentary of key topics in higher education from a student-centered perspective. The Report is a product of EduPartner.Solutions OÜ, a data analytics as a service (DAaaS) company focused on assisting higher education with understanding and driving to higher levels of student success.

Share Student360.Report

Leave a comment

2021:4

Results of a survey by Third Way, a U.S.-based Think Tank, has some concerning news for the future of college enrollments. The survey examined a number of post-pandemic attitudes of from over 2,000 interviews of students and caregivers.  

Although universities and colleges received high marks in the survey for many areas of pandemic response, a concluding set of questions should leave a chill with any admissions or recruitment office. When asked to respond to the question "Higher education is not worth the cost to students anymore", 2/3rds of all respondents across the demographics surveyed agreed.

This may not mean enrollments falling off a cliff but when over 60% of customers are unhappy with a product or service, it spells trouble. So, what do such results mean? 

Educational myopia - Producing what is not wanted

Reaching back over 60 years, there is a good answer in a 1960 Harvard Business School article Marketing Myopia by Ted Levitt. The idea is that companies get tied up with the busyness of marketing and fail to focus on real customer needs. Instead of producing what customers want and need, companies often produce expensive market flops.

Higher education is replete with examples of educational myopia from faculty research for the sake of research, layers of administration, fundraising for buildings that will be obsolete when built, etc. Supposedly those of us in higher education are smarter and better equipped to know what students and other stakeholders need from our institutions. This survey says otherwise.

Seeing more clearly - Success across the student lifecycle

How does higher education get past its short-sightedness? My suggestion is a student-centric view focused on success for the student across the entire student lifecycle or what we call the Student360 success model.

Student success needs to be first and foremost. That does not just mean recruiting students or high grades but making the entire student lifecycle the focus. Using a combination of touchpoints and data, it is possible to develop the insights needed to deliver value.

Future installments of the Student360.Report will focus on how student lifecycle touchpoints translate into data and insights.

About the Student 360.Report

The Student360.Report is a bi-weekly commentary of key topics in higher education from a student-centered perspective. The Report is a product of EduPartner.Solutions OÜ, a data analytics as a service company focused on assisting higher education with understanding and driving to higher levels of student success.

2021:3

It’s more than teaching and learning

In this edition of the Student360.Report the focus is on the importance of touchpoints as a tool of focus for the prediction and assessment of student success in higher education.

Our model focuses on touchpoints across the student lifecycle beginning with a prospective student’s first interest in a higher education experience. They continue through the success that comes with the completion of a degree or credential. Touchpoints can also be seen as steps in the journey of an educational experience. 

Educators are prone to think of knowledge as the sole stock in trade. Certainly, knowledge is a critical component of what we do in teaching and learning. Notions that student success is defined solely by knowledge gained in a classroom or other learning situation are incomplete and lead to lack of success for students and the institution. Such a narrow focus creates book smart, street dumb for those who do manage to persist to completion.

In our Student360.Design model, educational experiences should be evaluated based on a holistic approach at each student touchpoint. A full analysis often includes a rich analysis of the thoughts and feelings of the student or prospective student, what actions result, and how the touchpoint can be improved to promote success.

Touchpoints - Two Examples

Let’s look at how this works through two examples, one in the learning stage and one in the attract phase. 

A few years ago I was in a conference with a renowned professor from the Harvard Business School. As often happens in prof to prof conversations, we exchanged frustrations student progress. His students are often the crème de la crème of business students. Yet, we had a common problem that our students often do not read assigned materials and come unprepared to class. 

Engagement with course materials, such as opening and reading a textbook, is a touchpoint. And, it is highly correlated with student success in learning. In today’s world of e-resources and e-texts, the purchase and use of learning materials is measurable and easily known down to the time of access and duration of reading.

Getting beyond such a touchpoint to making meaning out of the data requires more analysis. Recognizing the touchpoint, however, is key to starting.

A touchpoint on the horizon in the attract phase is the use of micro-credentials as a part of the college admission process. There are several experiments including one medical school using a mini-med school MOOC to attract and recruit students. Such tools generate numerous touchpoints from the first point of contact with the student through perhaps an internet click to registration to frequency of interaction. 

Micro credentials embedded in a MOOC can be designed with a view towards a progression in the student journey of success. Such a design can include elements of student attitude and perspective, which are often crucial in a holistic admission process.

How many touchpoints?

The question will arise about how many touchpoints are enough to chart likely student success. 

Most institutions serve multiple types of students, which could lead to a plethora of touchpoints across the student lifecycle. In addition, web technologies have led to a greater number of touchpoints by potential students and those who later become students. Making sense out of the expanded number of touchpoints across changing student types is critical to success. 

The power of CRM technologies may suggest that every data interaction is a touchpoint. Using the blunt force of CRM technology, however, may end up falling into the “measure everything that moves” trap rather than measuring what matters. A different approach is needed.

Our approach rests on finding touchpoints that will inform the quality of the student lifecycle journey, allow for comparative data and benchmarking, and provide degrees of freedom for institutional agility. This approach helps align successful students with institutional strategy. 

From touchpoints to data points

The next step after defining touchpoints is to find how to measure, track, and trend data that informs student success. The EduPartner.Solutions approach to this important step is the subject of the next Student360.Report.

Leave a comment

2021:2

Concerns about a lack of quality in higher education persist.

Most solutions are misguided.

Students are reported as unready for the world of work. Professors are thought to teach irrelevant subjects. Institutions are seen as focused on touting student success only as far as recruiting.

The challenge comes from a misguided focus. Quality is not represented by teaching and testing. The idea of teaching excellence, although better than a singular focus on research, misses the mark by focusing on faculty rather than students. Spotlighting a few students’ exceptional achievements as a recruiting tool seems almost cynical.  

In reality, academic quality is a paradox. Students are the key raw material of quality. Their success is the product of academic quality. Students are also a key element of the means of production for academic quality.

A fresh approach is needed to recognize this central role of students. Our suggestion is to focus across the student lifecycle rather than just one part such as recruiting, classroom learning, teaching performance, or success in the job market. We call our approach Student360.Design. The idea is to design around the entire student lifecycle.

As we see it, a student lifecycle can be divided into five segments from the first awareness of and attraction to an institution and then through recruitment, onboarding, learning, and finally to advancing through career success. At each step of the way, there are touchpoints where the institution can reach out to the student. These can be early contacts of interest or the rich set of touchpoints in the learning phase. Touchpoints all generate data, which can be used to develop insights about student success.

Touchpoints all generate data, which can be used to develop insights about student success. Our approach looks at how touchpoint data influences key metrics of student success such as persistence, retention, learning, and career success. 

Defining academic quality by the quality of interactions across the student lifecycle is very different from traditional approaches in higher education. The usual approach is to silo data and knowledge into functions such as recruitment, academic assessment, and alumni relations. The traditional approach misses the mark by building data silos and then looking at the data from the optic of the institution rather than the student.

Future issues of the Student360.Report should appear every week and will always have a free of charge component for readers. We will work with our client organizations on more in-depth and specific analysis and topics.

Share Student360.Report

Welcome to Student360.Report by Dr. Doug Gilbert, founder of EduPartner.Solutions. I am an international higher education expert with a long career in higher education. I am also an enthusiast of mountain lifestyle in Switzerland and Colorado.

EduPartner.Solutions is a Data Analytics as a Service (DAaaS) provider focused on student success in higher education through the use of analysis of key touchpoints across the student journey.

Topics will focus on improving student success, which translates into higher student retention and persistence for higher education institutions.

Sign up now so you don’t miss the next issue.

Subscribe now

In the meantime, tell your friends!