Visualizing Student Data in a Real Teaching Context by Means of Concept Maps Urko Rueda, Mikel Larrañaga, Mikel Kerejeta, Jon A. Elorriaga, Ana Arruarte (University of the Basque Country jibrumou, jiplaolm, jibqulam, elorriaga, arruarte{@si.ehu.es}) Abstract: This paper describes the process followed for visualizing, using DynMap, the student’s evolution through the learning process in a Computer Security course at the University. DynMap is a system based on Concept Maps developed with the aim of providing the education community with a tool that facilitates the inspection student data. After a brief introduction, the paper describes the problem to face up. Then, the main characteristics, functionality and components of DynMap are described. Next, the summary of two studies conducted to validate DynMap are presented. Finally, some conclusions and the future perspective are pointed out. Keywords: Concept Mapping, Open Student Models, Empirical Studies, Academic Marks Management. 1 Introduction Information Visualization (InfoVis) deals with the display of abstract data. It is defined as the use of computer-supported, interactive, visual representations of abstract data to amplify cognition [Card, 99]. Applications of information visualization arise in domains such as information and software systems, biological sciences, artificial intelligence and machine learning, and the analysis of financial information. Inside the Artificial Intelligence in the Education area, and up to now, several authors have considered visualization and inspection in Computer Supported Learning Systems (CSLSs) components. Most effort has been oriented to consider visualization and inspection of student models [Bull, 03] [Dimitrova, 02] [Mazza, 03] [ZapataRivera, 00]. The student model component collects the learning characteristics of the student and his/her evolution during the learning process. Open Learning Modelling can be viewed as an attempt to directly involve learners in the diagnosis process and, as a result, infer their knowledge about the target teaching/learning domain. Commonly, systems built following the approach of open learner modelling allow to externalise the student model and, in some cases, provide mechanism for both teachers and learners, to change the content of the models. The efforts range from simply visualizing the model, to actively involving the student in the modelling process through negotiation or collaborative construction of the model. In recent years it has become apparent to many educational researchers that representing knowledge in a visual format allows one to better recognize and/or understand incoming information. It is easier for the brain to make meaning when information is presented in visual formats; visual symbols are quickly and easily recognized. Concept Mapping is a technique that allows information visualization. A Concept Map (CM) is a graphical way of representing and organizing knowledge. It is comprised of nodes and links, arranged in some order to reflect the information of the domain being represented. Nodes represent concepts, and links represent the relationship between concepts; both concepts and links are labelled and may be categorized. Although concept mapping has become an increasingly popular cognitive tool, originally it was defined as a method to present knowledge and information in a graphical way [Novak, 77]. CMs allow to see all the basic information in one page and make it easier for the user to see information in different ways, from different viewpoints. Concept mapping technique began as paper-based and evolved to computer-based tools with the innovation of visual design software. 2 Context The study here presented has been conducted in the context of a Computer Security subject at the University of the Basque Country. The syllabus of the subject is organized into three modules with twelve units. The teaching of this subject has an innovative approach in order to meet the following objectives [Elorriaga, 98]: (1) to encourage the implication of the students in the learning process, (2) to promote collaborative working, (3) to implement a continuous way of evaluating the educational benefit of each student, (4) to train the capabilities of preparing, writing and presenting reports, (5) to train the student in general working skills, such as analysis, criticism, argumentation and public presentation, (6) to leave open the possibility for each student to adapt the learning process to her or his areas of interest (7) to implement positive feedback mechanisms for teachers and students. The teacher should be able to detect the motivational state of the class, and the student should know that her/his work is acknowledged. The subject includes lectures and a wide set of didactic activities carried out individually or in groups. These activities include practical lab sessions, essays, presentations, tests, etc. In order to meet the educational goals above mentioned, the assessment of this subject needs a complex continuous procedure. The student has to get nine credits in order to pass the subject. The student and the teacher choose the practical works (learning activities) to get the credits. Each work is evaluated and if the mark is higher that the established threshold the student gets the corresponding credits. Each activity is related to one or more of the units of the subject and has a different amount of credits depending on the type and quality. In order to carry out continuous assessment, the teachers gather information on the student performance in their learning activities. The student’s final mark is the average of all the obtained credits. Due to the complexity and dynamism of the assessment system, the students need to check often their marks along the term in an understandable way. In addition the teacher needs to have a comprehensive and complete view of the work performed by the student in order to assess the works, to propose new ones and to guide the learning process. Therefore, both type of users, students and teachers, shares similar needs to visualize the information used in the assessment process. The marks of the students are managed by means of a spreadsheet that contains four sheets and shown in the web page organized into two-dimensional tables. However, this complex model of assessment uses and produces different data that is hard to visualize just by means of tables because of the cross-relations between data. Some of the problems identified in this visualization way are due to the difficulties in: • finding the mark of an assignment and the corresponding assignment/lab of a mark, • detecting when a student focuses the work excessively in a few areas, • detecting lacks of knowledge or excellent results in concrete areas, • detecting problems or excellent performance in concrete type of activities. The reason of these problems is that the users (student and teacher) have no general view of the student data. In order to overcome these limitations, it is necessary a comprehensive graphical view of the student data that allow the user to inspect the information at different levels of detail and a way to summarize the information. The solution that is proposed in this work is the use of CMs. 3 DynMap DynMap is a tool that shows graphically the knowledge of the students through the learning process to both the student and the teacher. The graphical representation is based on CMs. The tool uses CM-ED, a concept map editor developed in Java that can be adapted for different purposes [Rueda, 04]. 3.1 Visualization of Student Models using DynMap DynMap allows the user to inspect the assessment data of the students going through a CM that represent the knowledge of the student in the subject (Student Concept Map: SCM). The SCM includes two types of nodes: concepts of the subject and learning activities related to these concepts. The user can inspect the static view of the current state of the student – Static Student Model – and also can display the student evolution during the learning process – Dynamic Student Model –. In both cases, student marks are organized in two ways: according to the domain structure or according to the learning activities the student has done (skills of the students). Therefore, the CM that represents the student data has two views: Domain View and Activity View. In the Domain View, Computer Security domain has been organized in a map of concepts enriched with the didactic activities of the subject. In this view DynMap represents the knowledge of the student in a CM as a subset of the whole domain. DynMap represents the student model based on the CM of the subject. Therefore, the student model is a subset of the CM of the domain that includes those concepts that the student has learnt or is learning. The CM of the domain is drawn by the teacher. This CM is the backbone to organize and show the activities that the student has perform on each concept. In the Activity View, the student data is shown organized by the type of activities s/he has done, i.e. the skills s/he has worked. In this view, the nodes are not fixed by the teacher because the set of possible activities is very wide. Figure 1 shows the domain view of the static student model of one student. As it is usual in CMs the concepts are organised around a central node that represent the Computer Security subject. Taking into account that in some subjects the SCM may be large the tool offers the functionality for contracting parts of the SCM. So, the user can navigate from the central node expanding and contracting the nodes. It is also possible to apply a filter and show just a concrete type of activity. In order to show different circumstances in the student’s knowledge about the domain contents DynMap uses a set of graphic resources in the nodes of the CM that represent them. Each resource symbolizes a particular aspect of the knowledge that the student has about that content. The following resources have been chosen after considering the results of the two experiments that are explained below. • Thickness: a thick border represents the level of knowledge of the student about the concept. In order to determine the thickness the number of activities and their marks are taken into account. • Continuous / Dashed Line: a dashed border represents that the concept is not completely achieved, i.e. the performance of the student has shown when working with the activities related to the concept has gone beyond a threshold, otherwise the line will be continuous. • Flag: this graphic resource is used to mark when the concept has attached • • • learning activities (for example, lectures , labs , assignments ). Form: type of the node. Shadowed line: shows whether the node is contracted (shadowed) or not. Rectangle: a green rectangle is used to highlight new nodes in the SCM. Figure 1. Domain view of a Static Student Model. The leaves of the SCM represents the activities (labs and assignments) of the student that the teacher assesses. In addition, DynMap propagates those marks to the upper nodes. Therefore, as it is shown in figure 1 the graphical resources are also used in the nodes that represent the concepts. In this way, it is possible to have a general view of the student’s knowledge although some nodes are contracted. Taking into account that the student knowledge changes along the learning process it is very interesting that the student module reflects this evolution. Unlike most of the revised open student models that only represent the current knowledge stage, DynMap is able not only to keep the current state of the student knowledge but also to record the evolution of the student through the learning sessions. This feature is used to show dynamically the sequence of states of the student model. Using a set of video-like buttons of a tool bar the user can move through the learning process of the students. The evolution of the SCM can be observed at two levels of detail: jumping between each basic change in a session or between each session in the learning process. 3.2 Internal Representation of Student Models The DynMap Repository manages the storage of the dynamic evolution of all the SCMs. Due to performance issues, it is not appropriate to save the whole map after every small change. The chosen solution is to store the complete state of the SCM after each learning session and the changes performed on it while the session is carrying out in the Operations Log file. An Operation Log contains the sequence of operations performed in a SCM and the time when each one was done. Therefore, the Dynamic Student Concept Map (DynSCM) is composed of a sequence of SCM and Operation Logs: the initial SCM, the Operation Log of the first session, the SCM after the first session and so on. Using the storing facilities of CM-ED the DynSCMs are represented in XML. 3.3 Using DynMap The process to use DynMap is split in three phases: Translation, DynMap Building and Visualization /Inspection (see Figure 2). Domain CM Student Data Translation XML DynSCM Building DynSCM Visualization / Inspection Figure 2. Process for using DynMap The translation is the process for connecting the data source about the student with the inspection tools. The data source can be different: an Intelligent Tutoring Systems, an Open Learning System, a tool for managing marks of students in classic teaching, etc. Therefore, the translator is a piece of code integrated into the data source written in a programming language appropriate for that data source. It translates the information from the student data source to an intermediate XML representation that can be processed by DynMap. In the study here presented, the data source is a spreadsheet that stores the data of the activities performed by the student in a Computer Security course together with the evaluation of the activities when applicable. The translation process is composed of two steps. First, the data is standardized by means of a set of macros in a new worksheet with a fixed set of columns representing known attributes (name of the student, subject unit, activity type, activity description, date, session number, weight in the subject and mark). Next, this data is converted to a XML file. In the second step, DynSCM Building, the XML file containing the data of the student is read and the SCM that represent it is created. It uses the CM of the domain as the base to generate the student model. Concretely, it converts the data of the file to CM-ED operations and stores them in the Operations Log. In the last step, the user (teacher or student) can Visualize / Inspect the current state of the student model (Static student model) and the evolution of the student’s knowledge (Dynamic student model) as explained above. 4 Experiments A medium where a student model is open for inspection should meet some criteria: understability, effective inspection and reducing the cognitive load [Dimitriova, 02]. Focusing on these criteria and in order to evaluate the validity of the graphical approach chosen for dynamically visualising the Student Model two experiments have been conducted. Next, their results are briefly described. 4.1 Study 1: evaluation of graphical resources The aim of the first study was to check whether the graphical resources selected to show different characteristics of the student model were adequate and easily understandable by teachers and students. The graphical resources included in the study together with their initial meaning are the following. • Size: Degree of knowledge about the content. • Thickness: The content is enough worked or not. • Line: The student has acquired the required level or not. • Colour: The student worked in that content recently or not Two groups of people took part in the experiment: a group of 22 students from the Computer Science Faculty of the University of the Basque Country enrolled in the Computer Security course in 2003-2004 and a group of 22 teachers from the same college. During the experiment students answered a questionnaire composed of 9 multiple choice questions and 5 open questions. The questionnaire included four parts. The aim of the first part was to investigate what the changes produced in each graphical resource (e.g. different node size) suggested to the participants. In the second part five questions about a concrete state of a student model were asked. In part three, participants were asked to establish their own criteria for graphically represent the four characteristic mentioned above. Finally, in part 4 the participants had the opportunity to contribute with any suggestion. Regarding the obtained results, deeply explained in [Rueda, 03], and considering the participants’ answers and suggestions some conclusions can be pointed out: • Meanwhile Size, Thickness and Line seems to be resources quite intuitive and easy to understand, Colour does not suggest anything • Size and Thickness are so close that it is difficult to understand the difference between them. • • • 4.2 It is difficult to select any resource for represent the time when the content was worked; some participants suggest the use of flags or numbers. The combination of graphical resources add an additional difficulty to the student model understanding. Although some resources are easy to recognise and understand a previous explanation of each one is welcomed. Study 2: evaluating DynMap in a real use context The aim of this study was to evaluate DynMap as a mechanism to visualise the evolution of the students through the learning sessions of a Computer Security course, in a real use context. The study aimed to reveal potential usability problems of DynMap and evaluated the next issues. • Accessibility: Accessibility of the information that DynMap offers. • Organization: Adequacy of the organization of the SCM. • Information: Suitability of the information that DynMap provides. • Individual and group data: Interest and privacy issues in accessing individual and group data. • Other uses: Exploring new uses of CMs in the teaching of a subject. • Suggestions: Suggestions of improvements in DynMap. One group of 32 students from the Computer Science Faculty at the University of the Basque Country enrolled in the Computer Security course in 2003-2004 took part in the experiment. First, all students were asked to search for some data in the SCM (e.g. mark of the lab entitled netbus, mark of one of the student’s personal work, average mark of student’s personal works and the total number of credits already obtained in the subject). It may be pointed out that participants did not receive any help in using the tool before and during the experiment. After carrying out these searches each student answered a questionnaire composed of 17 multiple choice questions and 6 open questions. Some conclusions extracted from this study are: • Taking into account that no help in using DynMap was provided to the participants, they did not have mayor difficulties in using the tool. • Being realistic, the main aim of students using DynMap is to know the marks and number of credits obtained in each activity and in the whole subject. • The use of some graphical resources for highlighting some circumstances on the student model will be welcomed. • The study shown that DynMap can be also useful in management tasks. Considering the teacher, he said that the tool can help in the assessment decisions and also that it is useful as a medium to communicate the marks to the students. In addition, teacher added the next issues: • The graphical view of the student model allows the teacher to analyse the distribution of the students’ activities among the units of the subject. This is useful to identify weakness and strengths in the knowledge of the student and also to detect learners that are focussing exclusively in some parts. • It is interesting to observe the evolution of the student through the learning process due to the continuous assessment of the subject. • The teacher was convinced about the utility of having group models. Again, this feature would be interesting to detect weakness and strengths in a group and also to identify most popular contents. 5 Conclusions Along this paper the authors have presented the potential of DynMap for visualizing students’ data. DynMap is able to visualize the evolution of the student through the learning process of a particular subject using Concept Maps. Concretely, the paper showed the process followed for dynamically visualizing the behaviour of the students of Computer Security course at the University of the Basque Country. The complexity and dynamism of the assessment system together with the aim of carrying out a continuous assessment method led authors to analyse the behaviour of DynMap for graphically represent the students’ marks along the term. Two studies were conduced in this real context with the aim of validating both the use of a set of graphical resources for showing different circumstances of the student in a Concept Map and DynMap as a mechanism to visualize the students’ evolution through the learning process. The obtained results confirmed that users are able to read, manipulate and communicate with concept maps so, a system based on concept maps like DynMap can be use inside the education community as a tool that facilitate the visualization of student data. As a future work some improvements are being considered: the representation of groups of students, the improvement of the communication protocols between teachers and students and, the functionality for managing the student activities directly on the tool. Acknowledgements This work is funded by the University of the Basque Country (UPV00141.226-T15948/2004), the CICYT (TIC2002-03141) and the Gipuzkoa Council in an European Union program. References [Bull, 03] Bull, S., McEvoy, A.T. & Reid, E.: Learner Models to Promote Reflection in Combined Desktop PC/Mobile Intelligent Learning Environments. In: Aleven, V., Hoppe, U., Kay, J., Mizoguchi, R., Pain, H., Verdejo, F., Yacef, K. (eds): AIED2003 Supplementary Proceedings (2003) 199-208. [Card, 99] Card, S., Mackinlay, J., Shneiderman, B.: Readings in Information Visualization. Using Vision to Think. Morgan Kaufmann Publishers, San Francisco (1999). [Dimitrova, 02] Dimitrova, V., Brna, P. and Self, J.: The Design and Implementation of a Graphical Communication Medium for Interactive Learner Modelling. In Cerri, S. Gouardères and Paraguaçu (eds.) Proceedings of International Conference on Intelligent Tutoring Systems ITS’2002 (2002) 432-441. [Elorriaga, 98] Elorriaga, J.A., Gutiérrez, J., Ibáñez, J. And Usandizaga, I.: A Proposal for a Computer Security Course. ACM SIGCSE Bulletin (1998) 42-47. [Hartley, 02] Hartley, D. and Mitrovic, A.: Supporting Learning by Opening the Student Model. In Cerri, S. Gouardères and Paraguaçu (eds.) Proceedings of International Conference on Intelligent Tutoring Systems ITS’2002 (2002) 453-462. [Kay, 01] Kay, J.: Learner Control. User Modelling and User-Adapted Interaction, Vol. 11 (2001) 111-127. [Mazza, 03] Mazza, R. and Dimitrova, V.: CourseVis: Externalising Student Information to Facilitate Instructors in Distance Learning. In: Hoppe, U., Verdejo, F. and Kay, J. (eds.): Proceedings of Artificial Intelligence in Education AIED2003 (2003) 279-286. [Novak, 77] Novak, J.D.: A theory of education. Cornell University, Ithaca, NY (1977) [Rueda, 03] Rueda, U., Larrañaga, M., Arruarte, A., Elorriaga, J.A.: Applications of a Concept Mapping Tool. In: Cañas, A.J., Novak, J.D., González, F. (eds): Proceedings opf the First International Conference on Concept Mapping (2004) 545-553. [Rueda, 04] Rueda, U., Larrañaga, M., Arruarte, A., Elorriaga, J.A.: [Zapata-Rivera, 00] Zapata-Rivera, J. and Greer J.: Inspecting and Visualizing Distributed Bayesian Student Models. In: Gauthier, G., Frasson, C. and VanLehn, K. (eds.): Proceedings of International Conference of Intelligent Tutoring Systems 2000 (2000) 544-553.