Designing Learning Analytics for Effective Learning Through Problem-Based Learning

This paper describes the development of a learning analytic course for a post-graduate programme at the authors’ university for adult learning. The analytics course is very timely, given the proliferation of Big Data and the need for industries to embrace analytics to examine and solve a range of business problems using data analytics. This is an educational pedagogy and practice paper to design and develop Learning Analytics using Problem-Based Learning (PBL). The contribution of this paper is to help educators in the community who are keen to design and develop similar courses to have a better understanding of data generation, assessment writing, team formation, and challenges faced. The paper will examine and describe the use of the PBL approach to teach learning analytics. The PBL is adopted as it demonstrates a learning design that integrates industry experience with practice-based learning within the classroom. As an approach, it equips students with the ability to solve complex problems through context-driven inquiries. The course has been well-received by students and this paper will discuss the strengths of this approach.

recommendations for other approaches which could enhance and complement the PBL strategy in this course.

Literature Review 2.1 Integration of Learning Analytics within the Master of Adult Learning Curricular Programme
The adult learning programme was developed as a step by step approach. The curriculum is not just about the content of programmes, but about the overall educational design which guides students through a set of teaching, learning and assessment experiences toward the achievement of intended and unintended educational outcomes (Ornstein & Hunkins 2018). Hence, in designing the adult learning education programme, the followings are taken into consideration for all courses:  Determining needs of the industry gathered from needs analyses to inform curricular choices regarding graduate attributes they are looking out for, the required learning outcomes, materials, tasks and activities, and assessments that best address the needs of learners.  Setting and Assessing Outcomes focuses on how to write learning outcome statements and how to assess those outcomes.  Determining the pedagogical approach to be used and this is followed by the sequencing of tasks which address identified learning needs, and learning outcomes in instructional activities. Previous survey needs of the industry determined the choice selection of the various courses within the curriculum; this including the learning analytics course. The curriculum is structured around subject content which is adult teaching and learning. As with all programme development, the course objectives and learning activities have to be identified (Tyler 2013). Using the learning outcomes as a base, the course is then framed by the pedagogical approach that has to be identified for actionable learning. Following Tyler's framework for curricular development (Tyler 2013) the development of the curricular and the subsequent courses was an iterative process which involved a constant review of the various components within the curricular to ensure that the outcomes were aligned with content, learning and teaching processes and assessment. Table 1 shows the course curriculum design process. In Section 3, the authors would examine the course objective, followed by an explanation of how the course is framed around PBL. In terms of curricular integration, the content of the different disciplines is sequenced in certain ways for students to discover implicit linkages through teaching and learning principles. The learning analytics course is pitched at the intermediate level as students need to have an overview of learning principles and applications before examining the use of data for educational improvements. It supports capstone projects culminating in the final year of the Master's programme as shown below.
Foundational Courses Learning Analytic Course Capstone/Research Projects To illustrate this further, the principles and approaches of learning and assessment through the following foundational courses: Engaging Adults for Learning, Approaches to Blended Learning, and Assessment to Adult Learning. The Learning Analytic course is introduced after these foundational courses. As an extension of the courses, the Learning Analytic course connects and builds on what was taught earlier through the data science and AI perspectives. It, therefore, supports the capstone and research courses, which requires the application of data analytical skillsets for their empirical work. In terms of designing the curriculum, the adult learning programme is based on the following principles which are applied across all courses:  Instead of mere acquisition of subject content, the curriculum is largely designed around inquiry-based activities,  The experiences of workplace processes of problem solving and inquiry are highly integrated into the student learning activities.  The division of roles between lecturer and student is minimized; the scope for two-way interactions between research and teaching is deliberately exploited. There has been much discussion about the nature of key skills and how they might be developed within the curriculum. A key decision is whether skills development should be addressed separately within the curriculum or incorporated into courses. For the curriculum, the skills are incorporated within courses rather than being addressed separately. Given the above, the degree programme has taken a largely experiential learning route given the need for applied learning and acquisition of skillsets. Throughout all courses, experiential learning, such as those involving PBL, begin virtually the first day the student enters the professional program and continues until graduation. The final year of the curriculum involves active participation and in-depth experiences to acquire practice skills and judgment to develop the level of confidence and responsibility needed for independent and collaborative work within the profession. As such, the learning analytic course plays a crucial role within the curriculum.
The learning analytic course is one of the many courses offered in the Master of Adult Learning programme which is offered at the authors' university. As the only course which examines educational data, it holds a strategic position in that it prepares students for futuristic teaching and learning through introducing computational techniques drawn from AI and data science. It offers a means to expand course content taught earlier to incorporate new AI and data science technologies and topics about teaching and learning. Concurrent with this curricular integration effort was a consensus is the need for students to be functionally literate in AI and data science concepts so that they could use them appropriately in subsequent courses such as capstone or research projects. Different teaching approaches were applied across the different courses in the Master of Adult Learning, depending on the nature of the course. (Shin, Haynes & Johnston 1993) compared the students from two medical schools named McMaster University, and the University of Toronto. Students at McMaster University are based on a problem-based undergraduate curriculum and the latter is based on a traditional curriculum. A questionnaire was mailed to the participants in private primary care practices and for self-completion on the chosen subject, "management of hypertension". The result showed that the graduates from PBL were more up-to-date in the knowledge of the chosen subject than those graduates from a traditional teaching school. (Barrows 1996) wrote a book to explain the educational objectives of problem-based learning (PBL) and encouraged the management to give the faculty full support if they would like to change the established curriculum to PBL to see what students can learn and think independently. (DeGrave, Boshuizen & Schmidt 1996) researched to understand the ongoing cognitive and metacognitive processes during an important phase of PBL-problem analysis. They analysed the verbal communication in the group and students' thinking process. The stimulated recall of the thinking process provided more and unique information when dealing with anomalous data than verbal interaction.

Problem-based Learning
(Hmelo-Silver 2004) suggested that in PBL, students were engaged in self-directed learning and worked in collaborative groups to solve a complex problem that has more than one solution. This paper focused on how PBL helped less-skilled learners to develop effective collaboration skills and lifelong learning skills.
The objective of (Graaff & Kolmos 2003) research paper is to identify the essential characteristics that lead to the success of PBL. They are curriculum structure, the learning process, and assessment. This paper also highlighted the Dutch approach of PBL through problem analysis and project-organised learning. They compared the PBL model with the Aalborg model of project work and stated the main difference was found in the type of assignments, assessment methods, and organisation of group work. The PBL required a high degree of students' involvement in study activities and hence, acquired a deeper understanding of knowledge. However, students may miss out on the broader perspective, and thus students need to fill in the potential gaps in the subject areas. (Sungur & Tekkaya 2006) investigated the effectiveness of PBL based on students' self-regulated learning, motivation, and learning strategies. The questionnaire was given to 61 tenth-grade students from 2 classes taught by the same biology teacher. The control group used the traditional teaching method focused on a teacher-centric approach, using a textbook while the other experiential group with PBL in which ill-structured problems were given. The result showed that PBL students had higher levels of intrinsic goal, critical thinking, and peerlearning skills as compared to the control group. (Loyens, Magda & Rikers, 2008) investigated the role of selfdirected learning (SDL) and how SDL relates to self-regulated learning (SRL) in PBL. A review of empirical studies on SDL and SRL was conducted and it showed that PBL can foster SDL. The study also aimed to establish conceptual clarity between SDL and SRL based on a literature review of peer-reviewed journals and edited books.
In a book written by (Evensen & Hmelo-Silver 2012), the authors discussed how PBL could help researchers to develop research questions using the methodology. It is also targeted to the practitioners who deal with students' issues of teaching and learning and information on curriculum development and guidance is given.
Based on the literature review, the authors summarise that PBL encourages students to learn course material by relating it to everyday activities. The approach enriches their knowledge and understanding by engaging them in the discussion forum and collaborative research and teamwork, instead of learning from lecturing. The main disadvantage of PBL is it is more time-consuming for the lecturers to be actively involved in group discussion Journal of Education and Practice www.iiste.org ISSN 2222-1735 (Paper) ISSN 2222-288X (Online) Vol.11, No.18, 2020 and to assess if the students are learning and applying solutions to the given problem correctly, as compared to lecturer based learning (LBL) instructions. The lecturers must also have relevant skillsets to ask questions and solve complex problems. In the next section, the authors will examine how PBL is incorporated within the design of a new course -"Learning Analytics" for the Master of Adult Learning programme.

Course Development and Activities for Learning Analytics 3.1 Course objective
Most of the organisations today are flooded with data, and using data for decision making has become a norm and a competitive advantage to survive. In particular, the recent advent of Big Data and analytics has magnified the questions and creates both opportunity and conundrum for organisations and institutions, including educators. With such massive amounts of data collected and huge potential value that is found in these data for marketing, operations, risk management, and personalised learning and many others. Thus, the demand for analytics professionals to make sense of these data is increasing. A knowledge society presents unique and fundamental questions of what data means, and how to use it responsibly and made a well-informed decision.
The skills and knowledge that are needed for analytics professionals in education would require a wide array of skills such as data management, analysis, mathematics and statistics, business and not forgetting intimate knowledge of the education domain. The course will enable students to understand how raw data could and should be handled so that useful information can be extracted from the data. Students will have an overview of fundamental issues governing the use and analysis of data, with emphasis on education-related issues. Increasingly, educators have access to students' data through the institution's student management system. This course critically examines the place and use of educational analytics within the context of educational administrative and academic decision-making. Given this, the learning outcomes are divided into two components, i.e. knowledge and skills components. The learning outcomes are guided by Bloom's taxonomy (Bloom 1956).
The learning outcomes for the course are to prepare and organise data derived from an identified field of specialisation, to assess the suitability of different data analysis techniques, discuss the key principles of data analysis, examine the use of educational analytics, evaluate various approaches to best present findings and select appropriate methods in data analysis for different scenarios.
The target audience is the adult learners who already have a few years of working experience in the field of education or related field. As there is advancement in the use of technology in the education landscapes, dissemination of information through online learning portal to students has become very easy. The authors expect students to move from dependence to self-directed learning. To fulfill the learning outcome, the following topics are covered in the course using a combination of lecturer-based learning (LBL) and problem-based learning (PBL) approach. The topics for this course include (a) introduction to data analysis, (b) exploratory data analysis (EDA) on data given, (c) derive insights, (d) data analytics applications in education such as early alert system and intervention, course recommendation using analytics, (e) develop various predictive analytics models, and not forgetting (f) data privacy and governance and finally (g) data visualisation and performance metrics.

Incorporating PBL teaching in Learning Analytics
The Problem-Based Learning (PBL) process was first developed by Barrows and Tamblyn in the 1960s for the medical school students at McMaster University in Canada, to allow them to apply relevant skills to future job roles. It has subsequently been expanded and adopted in other education programs for all ages.
PBL is a student-centric teaching pedagogy in which students will apply what they have learned with the involvement of lecturers to solve an open-ended problem, which generally allows for the development of other desirable skills such as communication, collaboration, and professional attitudes with teamwork. PBL includes the original problem that can be solved in many different ways and may have more than one solution. It has many advantages over the traditional approach, as it stimulates students' interest and promotes self-directed learning, allows students to enhance their knowledge by taking responsibility for their interest regarding what they need to learn and retains the knowledge better.
The planning and implementation of PBL in the learning analytics course mainly include: (i) designing a problem, (ii) forming a study group and engaging students in class discussion, (iii) assessing students' performance. The detail for each step will be further illustrated using the course development.
(i) Designing a problem The authors designed a problem that mimicked the real-world situation and data was generated to follow some business rules. Design is a key essential stage of PBL learning. One of the problems that were given to students in the course was to use data analytics to identify a group of students with the universities who were at-risk in terms of their academic performance. The dataset included 2000 records of students' data and the input fields were Student ID, gender, age, student attendance rate, number of courses suspended, number of courses withdrawn, number of courses taken, total courses required for graduation, quarterly GPA, Cumulative GPA, and Journal of Education and Practice www.iiste.org ISSN 2222-1735 (Paper) ISSN 2222-288X (Online) Vol.11, No.18, 2020 the target variable was a flag if the student was at-risk. If the student is at-risk, the flag is 1 Otherwise, if the student is not at-risk, the flag would be zero. The authors generated data based on their understanding of academic institutions. Most of the universities, there was a policy that if the students' cumulative GPA falls below 2.5, they were considered at-risk, and students needed to go for academic consultation with the director of the program. However, there were also other indicators such as student attendance rate, the number of courses suspended/withdrawn, and quarterly GPA which were important indicators to predict the students' at-risk status.
The authors generated the data based on their experiences as educators in the university. In this example, students took 1 to 40 courses, the portion of male to female ratio in the university was 40% to 60%. The number of courses withdrawn and suspended should be less than the number of courses taken and the students could only withdraw or suspend at most five courses. The authors also generated a random number between 0 to 5 which followed certain probability distribution such that 60% of the students would have zero courses withdrawn, 15% would have one-course withdrawal, 10% would have two courses withdrawn, and finally, 5% would each have three, four and five courses withdrawn. For GPA generation, a normal distribution was used to generate the GPA, with a mean 2.75 and a standard deviation of 0.75. The minimum CGPA is 0.5 and the maximum GPA is 4. Since students' academic data are highly sensitive and the authors need to protect their privacy and confidentiality, they are not able to share the real-world data. However, it was important for the students to use the realistic synthesized data for learning and the authors have embarked on the journey of data generation using this method. They validated that the data generated is very similar to the real-world data. The data generated do not have a loss of generality and the authors can share with students for the class exercises and assignment. For the learning analytics course, the authors generated more than ten datasets so that students could use it for their hand-on lab sessions and assessments.
(ii) Forming a study group and engaging students in class discussion Students were asked to form a study group or project group at the first lesson of the course. The authors would need to create groups before the first lesson and gave detailed instructions to students to join the groups. The authors also limited the number of people in the group to have a maximum of three members and each member of the team should contribute equally. If a big class size of fifty students, the lecturers could increase the number of people to four to five students in a group. Having more numbers in a group would tremendously reduce, participation from each member. For each lesson, there would be class activities for further research and discussion. An example of such a class activity is as follows: Question: A university received thousands of applicants for its undergraduate degree program each year. Identify the data required to develop a predictive model to admit or reject students' application, which kind of predictive models would be developed and finally, how do you measure the performance of the predictive models.
Students would be given thirty minutes to work on this activity. Students worked in their small groups and engaged in the class discussion, and the lecturer would walk around during those activities to give guidance and clarify students' doubts. At the end of thirty minutes, based on the class size and the number of groups available, each team or selected teams would present their answers to the class and then the lecturer would ask a question and summarise them to form the final result. For the above activities, some of the group's answers were applicant's demographic information, past academic results, work experience, first three choices of their intended program of study, extra curriculum achievement (ECA), personal statement, financial information, and face-to-face interview result. Students also explained how to derive some data input fields such as financial information based on postal code and accommodation type. Through this process, students enthusiastically learned and improved their ability to solve the problem by proposing data mining methods, such as logistics regression, decision tree, or neural network those they have learned before to develop the predictive models. IBM SPSS modeler is the software that the students have learned to analyse the data, apply the data analytics method, and interpret the outcomes. Throughout the process, the lecturer is not the only one delivering and gaining knowledge, but the students have also acquired knowledge from their peers and through active research of topics given. The lecturers' role is to guide the students through the discussion to achieve the intended learning outcomes and to urge them to apply the knowledge in a collaborative environment. Finally, students needed to identify the performance indicators as an accuracy rate and the true positive rate for the predictive models. The lecturer could further expand the scope of discussion to implementation issues such as business process change management and organisation culture to adopt data-focused decision makings within the organisation.
(iii) Assessing students' performance The assessment was broken down into two components, one of them was the continuous assessment and the other was the examinable assessment and the weightage for each is 50% respectively. The continuous assessment consisted of a pre-class quiz and an assignment. The assignment constituted three questions and each question was based on a given dataset to solve certain business-related problems in education. Students would need to apply their data analytics skillsets and problem-solving skills to discuss data issues, shared insights from Journal of Education and Practice www.iiste.org ISSN 2222-1735 (Paper) ISSN 2222-288X (Online) Vol.11, No.18, 2020 32 the data. They would further develop and evaluate various data mining approaches and choose the best models. Finally, they would use the most suitable data visualisation methods to present actionable insights. The marked assignment would be returned to the students to provide them with feedback to prepare them for the exams. The examinable component would focus more on open-ended questions with options to explore multiple approaches to solve the problems. All these were aimed to develop students as independent learners by allowing them to explore multi-faceted approaches to solve problems of their interest and to further develop skills to tackle and solve the real-world problems. This is a major advantage of PBL in that students are responsible for their learning and engaged in self-directed learning, as stated by (Barrows 1996).

Course delivery and students' feedback
The authors' university focuses on blended learning. The course materials such as lecturer notes, e-textbook, and other reading resources are available six weeks before the start of the course. Students need to go through the materials before attending face-to-face classes. There are six four-hour face-to-face lessons where students and lecturers meet in the seminar room. Students were encouraged to read the material beforehand and came to the class prepared. They were strongly encouraged to participate actively in class discussions and activities. As it was a hand-on and practical lesson where most of the lectures involved lab, thus students were asked to bring their own laptop and the lecturer would go through in detail how to build an analytics model using tools like IBM SPSS Modeler and Spreadsheets Excel. The step-by-step guide for the lab was also uploaded before the class where students can practice before attending the class. Since it is a master level course, the authors want students to cultivate self-directed learning and lifelong learning skills using online resources.
The course obtained 4.5 out of 5 ratings in end-of-term student evaluation. There is a scale of 1 to 5, 5 is for strongly agree, and 1 is for strongly disagree. Table 2 shows part of the student's evaluation form. "The hands-on practice was very useful in helping to learn how to use the Excel formula and create dashboards." "This course assumes that the learner possesses sufficient knowledge of two fronts: Fundamental statistics and advanced Microsoft excel commands. Learners who do not possess both will suffer an extreme disadvantage. In my case, as I have very limited statistical knowledge (not to mention SPSS), I very much hope that more time could be spent building this up instead." These are also some of the suggestions to improve the course: "The pdf lab guides are useful, but a video would have been better." "Please inform students beforehand that two programs are mandatory for this course; SPSS modeler and at least a Microsoft Office 2013, for both assignments." Based on the students' evaluation, the authors proposed a few improvements for the next delivery. For each lab session, there would be a supplementary video to illustrate how to do the lab and it would be uploaded before the class together with a step-by-step lab guide document. The software requirements for this course would also be denoted in the course design document distributed to students even before the course has started, to inform students.

Conclusion
In this paper, the authors have shown how they developed a learning analytics course in the Master of Adult Learning course. PBL focuses on solving real-world problems by students guided by the lecturers. Through the process, students improved their interpersonal skills, teamwork, and leadership quality, which are undeniably important skillsets in the twenty-first century. In addition, PBL breaks the boundaries of the disciplines and improves the ability of students to synthesise knowledge. The authors applied the PBL pedagogical approach to developing the course and these include content development, data generation, and writing a business problem that mimics the real-world situation. Finally, they share with the readers the end-of-term course evaluation from students and steps to be taken forward for the next course delivery.
Besides incorporating students' suggestions (e.g. developing course videos) for the analytics course, there is potential in research on the impact of the PBL on students' learning. The future research direction could evaluate what phase of Problem Based learning is most impactful in students' learning. There are several steps to conduct this kind of research. Firstly, there is a need to properly define the phases of PBL and this includes designing problem, forming teams, and assessing students' performance. Next, a questionnaire would be sent out to students who have experienced PBL and to ask them to rank which phase of PBL affects their learning the most followed by some qualitative comments to understand their various learning styles. Lastly, the important keywords for PBL could be analysed to improve students learning.