‘Writing Analytics‘ is seen as a potentially useful technique that uses textual features to provide formative feedback on students’ writing. However, for this feedback to be effective, it is important that it is aligned to pedagogic contexts. Such efficient integration of technology in pedagogy could be supported by developing writing analytics literacy. The proposed workshop aims to build this capacity by mapping technical constructs to a broader educational sense for pragmatic applications. It provides a hands-on experience for participants to work with text analytics and discuss its implications for writing feedback. Participants will work with a set of text analysis code to extract features using the Text Analytics Pipeline (TAP) and map them to writing feedback using two case studies. Participants with extended technical knowledge will also be given the opportunity to develop rules based on extracted text features to write feedback for their own pedagogic contexts with additional support.

 

Across educational contexts students’ written communication is a fundamental concern (National Commission On Writing, 2003; OECD, 2013). Research and commercial tools built on Natural Language Processing (NLP) technologies have gained some traction in supporting the analysis of this writing for educational purposes (e.g., McNamara, Graesser, McCarthy, & Cai, 2014; Shermis & Burstein, 2013). In learning analytics, the sub-domain of writing analytics has emerged to support students in their writing practices through the provision of formative feedback that provides insights to educators and students. Two previous workshops (Buckingham Shum et al., 2016; Knight, Allen, Gibson, McNamara, & Buckingham Shum, 2017) have focused on this topic, with community building events focusing on critical perspectives on writing analytics (Buckingham Shum et al., 2016), and the development of greater ‘literacy’ for writing analytics – building capacity for the use of analytics through its alignment with pedagogic ends (Knight et al., 2017).

This proposed workshop, which would be the third in the series, will build on these previous events by introducing participants to a tool: the Text Analytics Pipeline (TAP). The workshop will build a shared approach to mapping low level textual features to rules that are based on empirical and theoretical work, which can be translated into feedback for students and educators. The LAK17 workshop focused on a need to go beyond simple text analytics, to think about how educators might make effective use of the power of NLP in their teaching. In that workshop the focus was on how a set of existing tools, submitted by participants, map to particular problems in writing, and how each of those tools provide feedback. In the current workshop, participants will work hands-on with text features by generating them using sample code in notebooks, and mapping these features to pedagogic contexts alongside writing useful feedback drawing on the features. At a simple level this might involve, for example, understanding how basic named entity recognition features can be used to develop rules to provide feedback to students: “You’ve included x, y, and z, but isn’t v also an important researcher in this area?”. Discussion will focus on technical concerns, and the pedagogic, drawing on research in the pedagogy of teaching writing, and the potential and pitfalls of NLP in addressing that pedagogy. This aligns closely with the theme of user-centered analytics by involving different stakeholders in the design of writing analytics feedback.

For educators to make effective use of writing analytics tools for impact on learning, tools must be integrated into teaching and learning contexts where they guide action, connecting theory, pedagogy, and assessment (Clow, 2012; Knight, Buckingham Shum, & Littleton, 2014; Wise & Shaffer, 2015; Wise, Vytasek, Hausknecht, & Zhao, 2016). Thus, the third workshop is intended to:

1. Build synergy between writing analytics literacy and writing assessment literacy – that is, build understanding both of the potential of writing analytics and of how to assess writing (both using, and without, analytics tools)
2. Build practitioner capacity for research on their students writing, through developing understanding of how data from writing analytics might provide insights on that writing
3. Build student writing analytics literacy as a means to develop their writing via their – critical – interaction with text features that contribute to good writing

 

 

This half day workshop will take a participatory approach, blending workshop, tutorial, and hackathon to consider the potential of text analytics tools for supporting writing, and how through use of openly available tools and a data carpentry approach, novices to text analytics – including educators, learning technologists, and others – can be inducted into its potential. The tentative schedule is given below:

9-9.15am – Introduction: Introducing the objectives, organizers and technical set-up for participants.

9.15- 9.35 am – Introduction to the notebook and TAP: The basics of using Jupyter notebooks and the Text Analytics Pipeline (TAP).

9.35- 10.30 am – Tutorial and discussion Part I: Case Study1: Hands-on engagement with text analysis using Jupyter notebooks – extracting features from text and turning them into feedback.

10.30-11 am – Coffee Break

11am – 12pm – Tutorial and discussion Part II: Case Study 2,3: Hands-on engagement with text analytics – turning features into feedback.

12pm – 12.15 pm – Open discussion: Discussion about other text features that can be used in pedagogic applications, and where to next.

12.15 – 12.30pm – Closing remarks: Brief summary and wrap-up comments.

 

Participation will be ‘open’ (i.e., any interested delegate may register to attend). This workshop will be of interest to a wide range of LAK delegates including students and researchers actively engaged in writing research, text analytics or writing analytics specifically; educators in schools, universities and businesses; leaders and policymakers; and companies active or potentially active in the field. An invitation will be extended to participants of previous Writing Analytics workshops to bring different perspectives on the textual features that can be identified and the kinds of feedback that can be provided about the textual features to help students improve their writing. Participants are encouraged to bring their own devices (laptops best or tablets with keyboards) with a modern web browser (e.g. Chrome, Firefox, Safari, Microsoft Edge or IE10+). Some coding skills although not mandatory, might be useful.

 

Shibani Antonette is a Doctoral Student in Learning Analytics at the Connected Intelligence Centre, University of Technology Sydney, supervised by Prof. Simon Buckingham Shum and Dr. Simon Knight. Her research work is focused on developing Writing analytics tools that are integrated into the classroom for pedagogic use with learning design. She uses text analytics for developing automated feedback and for analyzing writing and revision behaviors, and studies the augmentation of automated feedback with peer discussion. Her previous work on text analytics has been presented at LAK and other conferences. She is also the co-chair of the Writing Analytics workshop at ALASI2017.

Sophie Abel is a Doctoral student at the Connected Intelligence Centre, UTS. Her research is focused on how writing analytic tools can be used to help Higher Degree Research (HDR) students improve their research writing. She is particularly interested in how ‘good’ features of academic writing can be instantiated as text-analytic features in writing analytic tools.  She is co-organising a Writing Analytics workshop at ALASI2017.

Andrew Gibson is a Research Fellow in Reflective Writing Analytics at the Connected Intelligence Centre (CIC), UTS. He has developed the open source text analytics software (TAP) that this workshop is based on, and delivered workshops and talks locally and internationally on Text Analytics and in particular Writing Analytics for Learning including a Writing Analytics workshop at LASI 2017 and ALASI2017. Andrew holds a PhD from QUT on Reflective Writing Analytics and makes regular research contributions to the international Learning Analytics community. He is an active member of the Society of Learning Analytics Research.

Simon Knight is a Lecturer at the Connected Intelligence Centre, University of Technology Sydney. His research focuses on the relationship of analytics to epistemology, pedagogy and assessment, discourse analytics, and epistemic cognition, particularly around information seeking, work which has been presented at LAK and ICLS. He co-chaired the ICLS14 Workshop on Learning Analytics for Learning and Becoming in Practice and LAK15 Workshop on Temporal Analyses of Learning Data. He has also chaired Writing analytics workshops in LAK16 and LAK17.

 

Links to the previous Writing Analytics workshop websites here:

LAK16: Critical Perspectives on Writing Analytics

LAK17: Writing Analytics Literacy – Bridging from Research to Practice

 

 

 

 

The Jupyter notebook was made available for all participants during the workshop session. It is available in Github as a open-source resource:

Jupyter Notebook in Python: https://github.com/heta-io/tap-jupyter-notebooks-python

To install your own Jupyter notebook: http://jupyter.org/install

TAP: https://github.com/heta-io/tap

Consider contributing to the TAP repository with YOUR notebooks to share text analytics knowledge within the LAK community and beyond.

Find more about the Connected Intelligence Centre (CIC)’s work at https://cic.uts.edu.au/