Effect of the student learning approach on social annotation activity in Diigo

Abstract The use of social annotation tools in higher education has attracted the attention of researchers with many studies having looked at student perceptions and attitudes to them. This paper investigates students’ activity and perceptions of the social annotation tool Diigo along with the extent to which they correlate with their approaches to learning. Quantitative analysis was conducted with year 1 Administration students as part of cross-curricular cooperation between the courses Basic Informatics and English for Specific Purposes. The revised two-factor study process questionnaire and a questionnaire about students’ attitudes to collaboration in Diigo were used, together with qualitative analysis of a learning achievement post-test. Spearman’s correlation coefficients, the Kruskal-Wallis H test and Dunn’s Test were used for statistical analysis. The empirical results reveal a positive attitude to collaboration in Diigo, and a connection between a deep learning approach and quantity of activity as well as a connection between a deep learning approach and positive perceptions of the experience. The results suggest that collaborative tasks in Diigo can be useful in cross-curricular cooperation, and that considering students’ different study approaches while forming groups can improve student activity and motivation.


Introduction
The use of social annotation tools (SAT) has seen an increase in the last couple of decades.This trend is attributed to the affordances of these tools such as supporting collaboration, enabling ABOUT THE AUTHORS Damijana Keržič, PhD in Social Informatics, BSc in Mathematics, MSc in Computer Science, is a senior lecturer in Informatics in Public Administration at the Faculty of Administration, University of Ljubljana.She was a member of the faculty team for developing the e-learning system and now supports and develops a system to meet the faculty's needs.Her research interests include e-learning, educational data mining and educational technology in higher education.Vida Zorko, MA EdTech & TESOL, is a lecturer in English for specific purposes at the Faculty of Administration, University of Ljubljana.Her research interests include ESP methodology for students of public administration, English for academic purposes, e-learning and collaborative e-learning.shared access to and annotation of resources, which are useful for community building and project work (Novak et al., 2012, p. 40).SAT allow users to annotate online texts in a collaborative way by highlighting text, adding sticky notes and writing comments, whereby the work of one user can be seen by all participants.
Among the many ready-made tools educators most commonly use Hylighter, Marginalia and Diigo, or their own tools (Ghadirian et al., 2018, p. 139).
Several studies have explored student perceptions and attitudes to SAT in higher education contexts.In the early 2000s, Nokelainen et al. (2005) researched the use of SAT among master's and postgraduate university students who used the EDUCOSM system for annotations.They concluded that collaboratively created annotations supported learning.A few years later, Su et al. (2010, pp. 761-763) studied university students' attitudes to use of the Personalised Annotation Management System 2.0 (PAMS 2.0).They found that students held positive attitudes as regards "perceived usefulness, perceived ease of use, learning satisfaction, and willingness for future use", that group work promoted learning, and that students who posted more annotations had better learning achievements.However, some students may have contributed less because they could easily obtain annotations of their peers.Another study on university students' use of PAMS 2.0 by Yang et al. (2011) showed that it increased the sharing of knowledge, improved the reading process and promoted better understanding of the study materials.A 2013 study by Gao (2013) explored preservice teachers' use of and interaction in Diigo and their perceptions of the experience.She established that Diigo supported learning and helped students in noticing specific information but was not so helpful for obtaining a sense for the text's complete meaning.Zhao et al. (2018) used Zoho Docs for translation assignments with Chinese university students of English translation.They found that online collaboration enhanced student motivation and promoted learning through opportunities for self-reflection and co-creation of knowledge.
Other authors have explored how SAT help scaffold reading comprehension.Tseng et al. (2015), for example, were not only interested in general understanding of text but in different levels of reading comprehension as well.Their results show that students had better surface-based comprehension, which was ascribed to their use of highlighting vocabulary and adding explanations to it.Highlighting words, adding explanations in the mother tongue and inserting summaries were efficient for improving the text-based and situation-based levels of comprehension.This is because these annotations provided scaffolding and helped students see the overall structure of the text.Therefore, highlighting information or words, and adding explanations, summaries etc. improves text comprehension.Pereira Nunes et al. (2012, pp. 51-68) examined the reading comprehension of postgraduate students by collecting quantitative data on use of the highlighting tool and some qualitative data from the feedback in a questionnaire.They determined that when students practised active reading by annotating the text, this helped them to understand the text and retain the information.Novak et al. (2012) reviewed the literature on SAT and established that they improved learning; namely, critical thinking skills, meta-cognition and reading comprehension.Students liked using SAT and were more motivated to do the reading accompanied with more positive emotions and less negative emotions, and felt that SAT supported their learning.SAT also proved to be useful in information-search tasks.The only negative aspect identified was the extra time the students needed in order to become used to the tool, meaning their performance was initially lower.Their research also revealed that while paper-based annotations hold value for the annotator and are more frequent, online annotations are intended more for sharing with others than for individual learning, and are shorter.All in all, Novak et al. confirmed that SAT can promote student participation, engagement, communication, organisation, reading and peer-critique skills, the gaining of ideas, seeing the perspectives of other students, and help improve instruction.Some years later, Ghadirian et al. (2018) carried out similar research that reviewed 71 papers on SAT used in higher education to explore how and why students interacted with the learning materials and with their peers.Their study demonstrated that SAT were increasingly used in higher education to support learning, chiefly in education and computer technology disciplines.An important reason for their use was their perceived usability (mainly interactivity and helpfulness).The study also showed that SAT made explicit what the students thought, enabled reflection and collective thinking, while also helped promote the development of metacognitive skills.Another advantage was improved learning due to the social annotation activities combining group work and individual work.The authors identified a few problems with the use of SAT, such as textformatting issues, problems with discussions on annotations when the number of students is high due to the limited space of the margins, and the lack of tangibility and sense of location.
Research on SAT suggests they are able to enhance higher education study in a number of ways since student perceptions of them are positive, they aid in motivation to work, and can improve learning.The present study aims to contribute to this line of research by exploring how students perceive collaborative activity in SAT, and whether their perceptions and activity are related to their study approach.Diigo was selected as SAT since it was found to be particularly suitable for improving university students' skills in searching, managing, analysing and categorising information, for sharing and organising study resources, and for collaborative group research projects where students write their annotations (Estellés et al., 2010).The following research questions were thus posed: • What are students' perceptions of the study experience in Diigo?
• Is there a relationship between the students' learning approach and their annotations in Diigo?
• Is there a relationship between the students' learning approach and their perceptions of the study experience in Diigo?

Research design and data collection
To examine students' attitudes to collaboration in Diigo, the relationship between the learning approach and Diigo activity, as well as the relationship between the learning approach and perceptions of the study experience in Diigo, a case study was conducted with a cohort of 166 year 1 students (divided into four study groups) of the Higher Education Professional Study Programme in Administration, 1st cycle, at the Faculty of Public Administration, University of Ljubljana.The case study was performed as part of cross-curricular cooperation between the core courses Basic Informatics and English for Specific Purposes.The study entailed four steps: (1) An online collaborative task in Diigo: For each study group, a Diigo group containing an academic article on the topic of informatics related to the course Basics of Informatics was created.Each group was given a collaborative task that involved reading the article and collaboratively annotating it in Diigo, and thereby practising what they had learned in the English class, that is, management of a new vocabulary, writing definitions of terms, writing the gist, analysing paragraphs, and writing English or Slovenian summaries.Students were invited to select one or more important terms and write their English definitions, Slovenian translations, English collocations or synonyms.They could also write English or Slovenian summaries of parts of the article.They were given one week to complete the task.In this time, both teachers encouraged them and provided support.They informed them of the task objective, the advantages of group annotations.They also told them that their contributions would not be assessed nor affect their final grade, and that the results of their work would be anonymised and used for research purposes only.
(2) An online questionnaire to assess the students' approaches to learning: To evaluate the learning environment, all students were invited to complete an online questionnaire.The questionnaire was a translation of Biggs et al.'s (2001, p. 137) R-SPQ-2F study process questionnaire containing 20 statements that measure students' approaches to learning.Students responded using a 5-point Likert-type scale ranging from 1 ("never true of me") to 5 ("always true of me").The R-SPQ-2F questionnaire was chosen because has proved to be a valid tool and has been widely used for over 20 years (Leiva-Brondo et al., 2020).
(3) An online questionnaire to explore the students' perceptions of the experience: After completing the online collaborative task, students were invited to fill in a questionnaire about their attitudes to the collaborative use of Diigo.The questionnaire was an adaptation of Su et al. (2010) and Nokelainen et al.'s (2005) questionnaires and examined students' perceived usefulness, perceived ease of use, learning satisfaction, as well as positive impact and willingness for future use (see Appendix).Students responded using a 5-point Likert-type scale ranging from 1 ("strongly disagree") to 5 ("strongly agree").
(4) An online test to assess student comprehension of the academic article: An online post-test was conducted to assess the students' learning achievement.Students had to answer two open questions in Slovenian: (1) explain a key phrase in the article; and (2) write the gist of the article.

Data and methodology
For the purposes of quantitative analysis, student annotations in Diigo were described by three variables: number of written definitions (Def), number of given translations (Trans) and number of suggested synonyms (Synon).The collocations and summaries are not included because there were no annotations with them.A segment of one of the articles with student activity in Diigo is presented in Figure 1 to illustrate student annotations in the text.
The R-SPQ-2F questionnaire includes 20 items that are divided into two main scales (Biggs et al., 2001): (1) deep approach (DA) and surface approach (SA).Each scale is divided into two subscales; deep approach includes deep motive (DM) and deep strategy (DS) while surface approach (SA) includes surface motive (SM) and surface strategy (SS).To obtain the main scales' and subscales' scale scores, the item scores are summed.
The questionnaire measuring students' perceptions of collaboration in Diigo was divided into four dimensions: perceived usefulness (PU), perceived ease of use (PEU), learning satisfaction (LS) and positive impact (PI).Each dimension was described by the average of all student responses within the corresponding dimension.
In the online test which tested the comprehension of the academic article, the students had to explain in Slovenian a key phrase (KP) that appears in the article, and write the Slovenian gist of the article (GA).The responses were coded as follows: • KP: 0-not appropriate, 1-appropriate, • GA: 0-not appropriate, 1-partly appropriate, 2-appropriate A (pseudo)anonymization process using students' ID was used to connect the data.Therefore, each student was described with an array of 15 features (Def, Trans, Synon, DA, DM, DS, SA, SM, SS, KP, GA, PU, PEU, LS, PI).We calculated some basic descriptive statistics.Spearman's correlation coefficients were calculated to detect any link between students' learning approach and their perceptions of the study experience in Diigo.We intended to explore group dynamics in relation to Diigo and to understanding of the article.Since we had four independent categorical variables (i.e.groups) and ordinal or scale dependent variables, a Kruskal-Wallis H test was performed to find any statistically significant differences between the four groups in the Diigo annotations and also concerning the students' comprehension of the article.Given that the Kruskal-Wallis H test cannot reveal which group is different from another, we carried out Dunn's post hoc test to identify significant pairs.Significant values were adjusted with a Bonferroni correction for multiple tests.

Findings
Of 166 students who attended the classes, 59 completed the R-SPQ-2F questionnaire, 84 signed into Diigo, 56 annotated the text in Diigo (Table 1), and 55 completed the questionnaire about attitudes to Diigo.The reason the response rate was relatively low (only 51%) may be that student participation was voluntary and that the task was done in the week before the New Year's holidays, which is close to the end of the semester when student engagement in activities tends to decrease.
As regards student collaboration in Diigo, the share of active students is almost identical (about 70%) in groups A, B and C. Yet, group D stands out with only half (52%) of the students signed into Diigo actively collaborating.In all groups, the most frequent annotations were translations of terms, while synonyms were the least frequent with no synonyms recorded in group C.There were no annotations containing collocations of terms and no summaries of text.It is interesting that most annotations with synonyms were added in group D, which had the smallest share of active students in Diigo.As for the share of annotations added per student, group A is strongly prominent with the smallest share; students in the other three groups had relatively the same share.

Students' perceptions of the study experience in Diigo
Students' experience of collaborating in Diigo was generally positive.In Table 2, we report the mean values and standard deviations for each group, and show the distribution of the responses of the groups with a box plot (Figure 2).All groups perceived the activities and Diigo services as useful for sharing knowledge in the group with mean values ranging from 3.53 to 4.00.Their perceptions of ease of use is slightly lower, with mean values between 2.89 and 3.21.In terms of learning satisfaction, most groups felt positively about the group work and interaction in Diigo with mean values ranging from 3.24 to 3.80, with the smallest variance (shortest boxplot) for all groups.Slightly lower mean values were recorded for students' perceptions of the positive impact of Diigo and their willingness to use it in the future (from 2.74 to 3.52).Group C had an average lowest variance in all questions, the highest median (horizontal line in the box) and mean (X in the middle of the box).The highest variance is observed in groups A and B.

Learning approach and annotations in Diigo
All four groups show a greater mean value of the deep approach compared with the surface approach.In all groups, the mean values of the deep approach increase if students who did not actively participate in Diigo but only observed the work of their peers are excluded from the calculations (Table 3, groups marked with an asterisk).Group A demonstrates the lowest mean value of deep motivation (DM = 15.00) and for deep approach (DA = 30.06)(Table 1).
Group D has the highest mean value for surface motivation (SM = 11.61) and surface approach (SS = 14.67).It is interesting that this was the group with the smallest share of active students in Diigo.Nevertheless, if we only look at those students who were active in Diigo, their mean value for deep approach is higher.This is true for all four groups: when we limit the calculations to active students, deep motivation becomes one of the highest values.Hence, passive students in the group represent students with lower values for deep approach.
To test the correlations between learning approaches and number of activities in Diigo, we calculated Spearman's correlation coefficients for all three types of annotation (Def, Trans, Synon) and students' approaches to learning (DM, DS, SM, SS, DA, SA).The test revealed only two significant positive correlations, namely a weak correlation between deep motivation and translation (r = 0.298, p = 0.021) and between deep approach and translation (r = 0.284, p = 0.028) (Table 4).In particular, group B showed the highest deep motivation and provided the highest number of translations per student.

Learning approach and students' perceptions of the study experience in Diigo
46 students completed both questionnaires: the R-SPQ-2F questionnaire and the questionnaire of the attitudes to the collaborative study of the academic article using Diigo.Table 5 shows Spearman's correlation coefficients and corresponding significances (Sig) between four dimensions of student attitudes to collaboration in Diigo and students' approaches to learning for both scales and all subscales.The strongest positive significant correlation was discovered between deep approach (DA, r = 0.432, p = 0.003) and positive impact of work in Diigo (PI), indicating that students with a deep approach in group collaboration in Diigo felt a positive impact and wish to have more similar experiences also in the future.We also find that both deep motivation and deep   strategy have a positive significant correlation with the positive impact of Diigo, the former with a stronger correlation (DM, r = 0.421, p = 0.004) than the latter (DS, r = 0.336, p = 0.022).Deep motivation also shows a weak correlation with perceived ease of use (PEU, r = 0.318, p = 0.031), while a deep strategy has a weak correlation with perceived usefulness.All three surface dimensions (SS, SM, SA) do not show any significant correlation with the four measured dimensions of students' perceptions and attitudes to using Diigo (PU, PEU, LS, PI).

Differences between the groups in annotations, comprehension of the text, and perceptions of the experience in Diigo
To test the differences between the groups in Diigo annotations and student comprehension of the article, we used the Kruskal-Wallis H test since we have four independent categorical variables (i.e.groups) and an ordinal or scale dependent variable.This test revealed evidence of a difference in the mean ranks for five tested variables (bold marked p-values in Table 6).The test revealed statistically significant differences between the groups in variables: KP (p = 0.013), GA (p = 0.015), LS (p = 0.029), Trans (p = 0.027) and Synon (p = 0.042).These differences were therefore manifested in understanding the meaning of the article, learning satisfaction and in two of the Diigo activities.
A pairwise independent groups post-hoc Dunn test with Bonferroni adjustments was carried out to investigate which pairs of groups showed a statistically significant difference between the mean ranks.The test revealed statistically significant difference in the mean ranks in the interpretation of a key phrase between groups A and B (p = 0.010), as already manifested by the biggest difference between means (Table 6).While observing the written gist of the article, we detected two pairs with significant differences, namely groups B and D (p = 0.026) and groups A and D (p = 0.048).In both pairs, group D revealed the biggest mean value; therefore, the students here performed better than in groups A and B. In the case of learning satisfaction, a significant difference is detected between groups D and B (p = 0.044).In the case of transcriptions, groups A and B show a significant difference in the mean ranks (p = 0.044).Although the Kruskal-Wallis H test indicated a statistical difference in the case of writing synonyms, no pair of groups had a statistically significant difference (p < 0.05).Nevertheless, only group C did not differ significantly since it was not included in any significant pairs.

Discussion
This study examined student annotation activity in Diigo in an attempt to answer three research questions.The first question concerned students' attitudes to Diigo.It was found that they perceived the activity and the Diigo tool as positive.The most positive attitudes were expressed with respect to the usefulness of and learning satisfaction with the group activity in Diigo.These results echo Nokelainen et al. (2005), Su et al. (2010) and Gao (2013) who found that students felt the use of SAT added value to their learning.
The second question aimed to reveal any correlations between students' approaches to learning and their activity in Diigo.The results show that a smaller amount of activity is correlated with surface motivation and a surface approach to learning.Since students with a surface approach often try to complete the required task with the least effort possible, and wait for their peers to do the group work (Fryer & Bovee, 2016;Zheng & Guo, 2019), this could be the cause of this result.Our finding that passive students' study approach is less deep is in harmony with the results of a study by Zheng and Guo (2019) who concluded that the differences in the behaviours of deep and surface learners are reflected in their activity: deep learners have a higher frequency of viewing content and writing comments than their peers who are surface learners.Namely, students with a deep learning approach are strongly motivated for learning and will tackle a problem or task even though they might find it difficult.
It was also confirmed that students with deep motivation contributed more translations of terms.This is in line with the findings of Nokelainen et al. (2005) and Chan (2016) that students who evaluated themselves as more motivated were more active.The co-construction of group motivation was also shown by Dörnyei (2002, pp. 153-157) who established that peers with lower motivation will be more motivated to work if they collaborate with peers who have a better attitude to the task.
The third question concerned the relationship between students' learning approaches and their perceptions of the activity.The results indicate four correlations.First, students with a deep study approach felt a positive impact of the activity in Diigo and would be willing to collaborate in Diigo in the future.Second, the two sub-components of a deep study approach-deep motive and deep strategy-were also correlated with the perception of a positive impact of the collaboration in Diigo.Third, the sub-component of deep motivation was correlated with perceived ease of use of Diigo.Fourth, the sub-component of deep strategy was correlated with the perceived usefulness of Diigo.This suggests that students with a deep study approach welcome new forms of learning and prefer a diversity of learning tasks because that presents them with a challenge.Similarly, Taher et al. (2011) found that openness to a new experience is correlated with a deep approach to learning.For such students, the effort made to become acquainted with a new learning environment is not a problem.This is due to them being genuinely interested in learning, which was noted by Zheng and Guo (2019, p. 95) as one of the behavioural differences between deep and surface learners.Our results are also in line with the findings of Ellis et al. (2009) and Ellis et al. (2016) that more positive student perceptions are associated with deeper study approaches.We thus agree with Ellis et al. (2009) that to improve the student experience teachers must raise student awareness of the purpose and value of the activity.The students may thereby become more motivated to engage in it.
Differences between the groups were detected in terms of their activity in Diigo, comprehension of the text, and perceptions of the experience, while no differences were found between them with regard to perceived usefulness and positive impact of the collaborative activity.Further research in the form of in-depth interviews is necessary to uncover whether these differences are the result of different group dynamics such as the level of collaboration between students.It would also be interesting to determine how well students know each other, how much they communicate beyond the classroom, and whether this is reflected in their collaboration in Diigo.

Conclusion
The results of the examination of students' collaboration in Diigo may inform teachers' design of group annotation activities in three ways.Firstly, the study of students' perceptions to collaboration in Diigo revealed that students had a positive attitude to the annotation activity.This finding may encourage the teacher to use Diigo for collaborative projects.
Secondly, the study uncovered a correlation between the students' learning approach and their annotations in Diigo, which revealed a greater passivity of students with a surface approach and surface motivation.This implies that understanding the relationship between the students' learning approach and their online activity may help the teacher better address the students' intrinsic interests.One way could entail organising collaborative projects by balancing groups in a way that includes both types of students, those with deep as well as those with surface motivation.In this way, the activity of the former will be motivating for the latter.
Thirdly, the research confirmed the correlation between students' deep learning approach and perceived positive impact of the collaborative activity, ease of use and usefulness of Diigo.This could inspire the teacher to help increase the students' intrinsic motivation by effectively explaining the value of collaborative activities.
This study has a few limitations that should be noted.First, since participation in the collaborative activity and questionnaires was voluntary, the sample only included students who opted in.The second limitation is the study approach questionnaire involving self-evaluation, which affects the reliability.There is also the issue of the transferability of results because the subjects were year 1 students and the collaborative task was done in the first semester when they were still being introduced to the courses, and when they may still have not been fully active.Despite these limitations, the study suggests some directions for further research on group dynamics and the relationship between student activity and their learning approaches.

Figure 2 .
Figure 2. Distribution of the responses on students' attitudes to the collaborative use of Diigo for the groups.

Table 2 . Mean and standard deviation (SD) of students' perceptions of Diigo across the groups
*Number and percentage of responses of students who signed into Diigo.

Table 4 . Spearman's correlation coefficients between types of annotation and learning approaches
*Correlation is significant at the 0.05 level.

Table 5 . Spearman's correlation coefficients between student study experience and learning approaches
**Correlation is significant at the 0.01 level.*Correlation is significant at the 0.05 level.

Table 6 . Variables with statistically significant differences between the four groups using the Kruskal-Wallis H test Group mean value
*The significance level is 0.05.