LAK16: Critical Perspectives on Writing Analytics

Introduction

Edinburgh, Mon 25 April, 2016

Broadly defined, writing analytics involves the measurement and analysis of written texts for the purpose of understanding writing processes and products, in their educational contexts. Writing analytics are ultimately aimed at improving the educational contexts in which writing is most prominent. The principal goal of writing analytics is to move beyond assessment of texts divorced from contexts, transitioning instead to a more nuanced investigation of how analytics may be effectively deployed in different writing contexts. Writing analytics thus aims to employ learning analytics to develop a deeper understanding of writing skills. Thus, workshop discussions will focus on writing from a number of different perspectives. In particular, we will discuss analytics that can help to better understand both the writing process as well as the final product, as well as their interactions with task demands, such as essay genre and voice.

An additional focus of this workshop will be on the pedagogical context in which writing analytics should take place. Our aim is not simply to focus on the automated scoring of written essays. Rather, we aim to discuss writing analytics that can be meaningfully embedded within a pedagogical context. These discussions can relate to a number of issues, such as the delivery of feedback and adaptive instruction.

Photo credit: Edinburgh by Oscar F. Hevia, under a CC-BY-ND-NC-SA license

Rationale

Writing as a window into the mind

Effective writing is not only central to education and the workplace, but also a lifelong citizenship skill necessary for effectively engaging with society. A large majority of academic disciplines focus on the development of learners’ skills in critical review, conceptual synthesis, reasoning, and disciplinary/professional reflection. In these subjects, writing arguably serves as the primary window into the mind of the learner. Huge effort is invested in literacy from the earliest schooling, extending into higher education. Yet educators and employers alike recognize the challenge of cultivating this ability in graduates, with poor written communication skills a common cause of complaint.

Extending beyond scholarly academic writing, many educators also have a keen interest in disciplined, autobiographical reflective writing as a way for students to review and consolidate their learning, thus providing a means for assessing the deepest kinds of shifts that can occur in learner agency and epistemology. Such approaches are also common in the training and development of professional reflective practitioners.

Writing is, however, time consuming, labor-intensive to assess, difficult for students to learn, and not something that all educators can coach well, or even consider their job to coach. It is in addressing these systemic limitations that Writing Analytics is attracting significant educational interest and commercial investment.

Harnessing NLP

Natural language processing (NLP) techniques have been proposed as one of the most effective methods for analyzing writing. In particular, NLP can provide detailed information about the properties of students’ writing at multiple levels of the text. For instance, NLP tools have been developed to provide information about the words in the text, such as their imageability and frequency in the English language. Additionally, NLP tools have been developed to calculate various other aspects of text, such as the difficulty of the sentences, and the presence of cohesion at multiple different levels of the text (e.g., between sentences, paragraphs, etc.).

As NLP moves beyond research labs and into mainstream products, the Learning Analytics community has the opportunity and challenge of harnessing language technologies, and delivering them in effective ways that enhance learning.

NLP approaches are, of course, the key enabling capability of these technologies, but they are just one piece of the puzzle for an effective learning analytics solution: these approaches need to be tuned by theories of how writing and learning shape each other, the scholarship of teaching writing, appropriate pedagogical practices and user interface design, and evidence from empirical evaluation of the total system, not just algorithmic metrics.

The LAK community should be in a position to guide educators and students on the evidence of impact in this new space. What questions should be asked before buying a new product or trialling a new research prototype? What are the options for evaluating such tools? What staff competencies are required to ensure that such tools have the maximum chances of success? Do students need orientation or training? What pedagogical contexts can this tool be applied to? These are the often-ignored constraints around a potentially disruptive technology.

Promises and pitfalls

Ultimately, for the tools to be successful, educators and students must trust them, and the effort of learning these new tools must pay back. Computational assessment of writing elicits strong reactions from many educators. For skeptics, handing the task of assigning feedback or grading essays to a machine crosses a boundary line marking the limits of artificial intelligence (AI). An important research question is whether or not such skepticism is justified.

Writing Analytics have in common similar potential and pitfalls to other learning analytics applications. At the most optimistic level, the promise of writing analytics is the kind of 24/7 personalized feedback that is currently only available to a privileged minority via detailed, timely feedback from educators as they draft texts. However, this workshop will take a systemic perspective, problematizing the contexts that Writing Analytics are deployed within, and partially constituted by. Evaluation of Writing Analytics is thus framed as a design problem, raising questions about conventional metrics (such as precision, recall), alongside: socio-technical concerns; pedagogic and assessment contexts; and ethical issues.

Critical, systemic perspectives

Learning Analytics as a field sits at the confluence of existing research tributaries. The LAK Discourse-Centric Learning Analytics (DCLA) workshops forged connections with CSCL discourse researchers to ensure that DCLA built on existing work. DCLA workshops have had a couple of papers on extended student writing, which we now argue merits its own workshop.

This workshop thus seeks to build similar bridges to existing research communities. There is a decade’s tradition of Workshops on Innovative Use of NLP for Building Educational Applications, operating within the computational linguistics research paradigm, with evaluation based on information retrieval (IR) metrics, but applied specifically to educational texts. The Computer Assisted Assessment community has a tradition of research into student writing, and has a strong educational researcher presence. Research in Computer-Supported Collaborative Learning has a primary focus on student discourse.

The workshop aims to reflect the distinctive contribution that SoLAR and LAK bring — namely, a holistic perspective in which the definition of “the system” and “success” is not restricted to IR metrics such as precision and recall, but recognizes the many wider issues that aid or obstruct analytics adoption in educational settings, such as theoretical and pedagogical grounding, usability, user experience, stakeholder design engagement, practitioner development, organizational infrastructure, policy and ethics.

Program

Contributors:

Yigal Attali (Educational Testing Services)

Phil Winne (Simon Fraser U.)

Chris Brooks (U. Michigan)

Ágnes Sándor (Xerox)

Denise Whitelock (Open U)

Andrew Gibson (University of Technology Sydney)

Noureddine Elouazizi (U. British Columbia)

8:30-9 Welcome and Introduction to the Workshop

Opening thoughts: Simon & Danielle

9-10 Case-Based Writing Analytics Groups

9-9:30 Group session – ‘case based’ discussions, using a dataset vignette

9:30-10 – Groups feed back

10-10:30 Break

10:30-12 Computational Perspectives on Writing Analytics

10:30-11:10: 4 speakers as a panel: Yigal, Agnes, Andrew

Panelists are asked to respond to 2 key questions (listed below) including “what keeps you up at night”, and pose one critical question for us to consider.

What are the limits of your approach that keep you awake at night?

Audience

- - Who can make sense of your tool’s analytics and how do you know?
  - How do you engage stakeholders and writing educators in the design of analytics?

Theory

- - In what way is your work grounded in, or contributing to “theory”? If not, does it matter? Can writing analytics tools be completely ‘data-driven’?

Research and Business

- - How should academics and vendors be working better together? What role does open source play?

Organisational Embedding

- - How have you engaged with the social/organisational factors for embedding writing analytics into everyday practice?
  - To what degree does WA solves the administrator’s problems and not the learner’s?

“Do No Harm”

- - Are there any risks of impeding writing skills through automated approaches to writing instruction and assessment?

11:10-11:35: Specific questions on the panel contributions

11:35-12: Thematic discussion on computational approaches.

12-1 Lunch

1-2:30 Social & Process Perspectives on Writing Analytics

1-1:40: 4 more panellists: Phil, Chris, Denise, and Noureddine

1:40-2:05: Specific questions on the panel contributions

2:05-2:30 Thematic discussion on social and process issues

2:30-3 Break

3-3:45: Designing the LAK17 hackathon

How would a hackathon around common datasets address the challenges of writing analytics? (small groups w/feedback)

Types of data

Types of analyses

Genre of writing

Types of user experience, feedback, and stakeholder interactions

Types of tools (and the role of open v closed systems)

3:45-4:30: Wrap Up (Group Discussion): Envisioning #WA2025

In 2025, these barriers have been overcome… these new challenges have arisen…

Is there still an active research field?

6:30: Optional workshop dinner

Speakers

Invited Speaker profiles…

Yigal Attali (Educational Testing Service)

Bio: Yigal Attali is a principal research scientist in the Research & Development Division at Educational Testing Service in Princeton, NJ. He earned his Ph.D. in cognitive psychology at the Hebrew University of Jerusalem in 2001, and joined ETS in 2002. Yigal is interested in the development and evaluation of innovative assessments. He is co-inventor of the automated essay scoring system e-rater® V.2 and has done extensive research in this area. Yigal is also interested in feedback mechanisms in interactive assessments and in judgment and decision making processes in the context of testing and assessment.

Type of writing data: Academic writing in the context of timed prompt-based essay writing assessments, mostly GRE and TOEFL writing, but also Criterion online writing evaluation service.

Tools used: The e-rater essay scoring system (Attali & Burstein, 2006).

Writing features analysed: Wide range of features, including content evaluation (Attali & Burstein, 2006; Attali, 2011).

Target writers (e.g. K12, undergrad, etc.): Writing of K12 students was the focus of a developmental writing scale based on automated evaluation (Attali & Powers, 2009). Writing of undergraduate and graduate students (both native speakers of English and English language learners) is the focus of most other papers listed below.

Evaluation measures used to validate your tool/approach: For a general overview of the subject, see Attali (2013). E-rater was evaluated through a wide range of criteria (many of the papers below address this subject). Comparisons with human ratings include correlational analyses, bias in scores across various types of classifications (prompt, genre, gender, ethnicity, country of origin, first language), and factor analysis.

Investigations of construct validity include factor analyses across a wide range of developmental levels and writing tasks and comparisons of human and automated essay scores with other student measures (such as verbal ability scores).

Reliability analyses were performed across a similarly wide range of situations and for both overall scores and sub-scores.

Audience for the analysis (educators, students, administrators, researchers): All four audience groups would be interested to understand more about automated essay scoring, its strengths and weaknesses.

Links to key projects/papers

Most papers are available through ResearchGate.

Journal papers and chapter focused on automated scoring

Buzick, H. M., Oliveri, M. E., Attali, Y., & Flor, M. (2016, in press). Comparing human and automated essay scoring for prospective graduate students with learning disabilities and/or ADHD. Applied Measurement in Education.

Attali, Y. (2015). Reliability-based feature weighting for automated essay scoring. Applied Psychological Measurement, 39, 303-313.

Attali, Y. (2013). Validity and reliability of automated essay scoring. In M.D. Shermis & J.C. Burstein (Eds.), Handbook of automated essay evaluation: Current applications and new directions (pp. 181-198). New York, NY: Routledge.

Bridgeman, B., Trapani, P., & Attali, Y. (2012). Comparison of human and machine scoring essays: Differences by gender, ethnicity, and country. Applied Measurement in Education, 25, 27-40.

Attali, Y., Bridgeman, B., & Trapani, P. (2010). Performance of a generic approach in automated essay scoring. Journal of Technology, Learning, and Assessment, 10(3). Available from http://www.jtla.org

Attali, Y., & Powers, D. (2009). Validity of scores for a developmental writing scale based on automated scoring. Educational and Psychological Measurement, 69, 978-993.

Shermis, M. D., Shneyderman, A., & Attali, Y. (2008). How important is content in the ratings of essay assessments? Assessment in Education: Principles, Policy & Practice, 15(1), 91-105.

Attali, Y., & Burstein, J. (2006). Automated essay scoring with e-rater® V.2. Journal of Technology, Learning, and Assessment, 4(3). Available from http://www.jtla.org

Higgins, D., Burstein, J., & Attali, Y. (2006). Identifying off-topic student essays without topic-specific training data. Natural Language Engineering, 12(2), 145-159.

Journal papers focused on the intersection of human and automated scoring

Attali, Y. (2016). A comparison of newly-trained and experienced raters on a standardized writing assessment. Language Testing, 33, 99-115.

Attali, Y. (2014). A ranking method for evaluating constructed responses. Educational and Psychological Measurement, 74, 795-808.

Attali, Y., Lewis, W., & Steier, M. (2013). Scoring with the computer: Alternative procedures for improving the reliability of holistic essay scoring. Language Testing, 30, 125-141.

Research reports

Attali, Y., & Sinharay, S. (2015). Automated Trait Scores for TOEFL® Writing Tasks (ETS Research Report, ETS RR-15-14). Educational Testing Service: Princeton, NJ.

Attali, Y., & Sinharay, S. (2015). Automated Trait Scores for GRE® Writing Tasks (ETS Research Report, ETS RR-15-15). Educational Testing Service: Princeton, NJ.

Breyer, F. J., Attali, Y., Williamson, D. M., Ridolfi‐McCulla, L., Ramineni, C., Duchnowski, M., & Harris, A. (2014). A Study of the Use of the e‐rater® Scoring Engine for the Analytical Writing Measure of the GRE® revised General Test (ETS RR-14-24). Educational Testing Service: Princeton, NJ.

Attali, Y. (2011). A differential word use measure for content analysis in automated essay scoring (ETS RR-11-36). Educational Testing Service: Princeton, NJ.

Attali, Y. (2011). Automated subscores for TOEFL iBT Independent essays (ETS RR-11-39). Educational Testing Service: Princeton, NJ.

Attali, Y., & Powers, D. (2008). A developmental writing scale (ETS RR-08-19). Educational Testing Service: Princeton, NJ.

Attali, Y. (2007). Construct Validity of e-rater in Scoring TOEFL Essays (ETS RR-07-21). Educational Testing Service: Princeton, NJ.

Attali, Y. (2007). On-the-fly customization of automated essay scoring (ETS RR-07-42). Educational Testing Service: Princeton, NJ.

Chris Brooks (U.Michigan)

Bio: Christopher is a Research Assistant Professor in the School of Information, as well as Director of Learning Analytics and Research within the Digital Education and Innovation unit at the University of Michigan. His research focuses on applying machine learning methods to traces of learner data in order to understand how learner behaviors relate to achievement, motivation, and success in digital environments. He is interested in both traditional higher education (where traditional includes computer mediated tools such as discussion forums, content management systems, lecture capture, etc.) as well as lifelong learning and low stakes learning environments (such as MOOCs).

Type of writing data: I’m interested in three kinds of data:

Discourse data (broadly). I am interested in how the text of the data relates to the non-textual interactions around the data (e.g. reading of messages). This includes more traditional social network analysis as well.
Interaction data related to writing. Largely in how people form messages and revise their messages in learning contexts. Merging keystroke level data across to the semantics and syntax of the writing itself, and attempting to form more holistic models of learning.
Short focused writing responses. Writing data that would be used within a classroom context, e.g. during a lecture.

Tools used: I largely use tools such as NLTK, NetworkX, and Weka.

Writing features analysed: My work to date has been on non-linguistic features of discourse, such as reading of messages and the social networks that are formed. Currently I’m looking at merging linguistic features (e.g. topic analysis, clustering, and sentiment analysis) of short student written works in active learning situations with other characteristics of students (from LMS behaviors to SIS measures) to form small group matching. This is being done through the Michigan Educational Text Analysis (META) project.

Target writers (e.g. K12, undergrad, etc.): Large undergraduate classrooms (often STEM).

Noureddine Elouazizi (University of British Columbia)

Bio: I am an interdisciplinary researcher with expertise in educational technologies design and in theoretical (formal) and computational linguistics (NLP). I integrate the use of language technologies and learning technologies and their related methodologies to contribute to the science behind science education. In my current role of the strategist for learning technologies at the UBC’s Faculty of Science/Skylight-Dean’s office, I lead the strategic management, sustainability and evaluation of the learning technologies ecosystem at the Faculty of Science. Part of my role is to conduct and lead research projects to examine and document the evidence of impact of using technology for learning and teaching in the context of science education. Currently, I am the research lead and principal investigator on two UBC’s Faculty of Science projects. The first project is about the pedagogy of peer review and argumentation in science education. The second project aims to develop an NLP-informed learning analytics technology to enable the extraction and measuring of aspects of argumentation in written text.

Type of writing data: The writing data that we work with comes from students’ generated essays (including drafts and final versions). This includes: (a) argumentation essays and (b) peer review essays. We approach these written data sets with the assumption that language is the direct by-product of cognition and through examining written language structure and usages, we can gain insight into how students argue and communicate science.

Tools used: My research projects team and I are developing a tool/technology to automatically extract and analyze different features of argumentation from written text/essays. The goal of this technology is to track and measure students’ argumentation in writing intensive courses and in ways that make it easy for faculty, students and scholars to use. We are exploring the integration of text mining algorithms, machine learning algorithms and NLP-domain specific algorithms to extract aspects of argumentation. For the architecture of the system, we are building on the accumulated knowledge in the field for the design of argumentation systems (e.g. Araucaria (Reed and Rowe 2004), Compendium (Buckingham Shum et al. 2006), Rationale (Van Gelder 2007), CoChemEx (Tsovaltzi et al. 2010), a.o.)) and we are adding novel NLP-informed design, computations and architecture. Some of the tools we are using for different tasks of computational analysis of language/text data includes: R modules such as R(tm), R(RTextTools), R(TextmineR), WordSmith, NLTK, LightSide, RSTTool, OpenNLP.

Writing features analyzed: We are examining different language/linguistic features (covering both form and interpretation) that convey different aspects of argumentation strategies, including but not limited to: (a) subjectivity, (b) stance taking, (c) mitigation strategies, (d) assertion strategies, (e) indexicality and flow of argumentation, (f) paraphrasing, (g) propositional attitudes and hedging, etc.

Target writers (e.g. K12, undergrad, etc.): Undergraduate Science students.

Evaluation measures used to validate your tool/approach: We compare methods for automatic scoring of essays and human scoring of essays. We are currently developing argumentation rubrics to guide the construction of argumentation (by students), deconstruction of argumentation (by peer reviewing students and instructors).

Audience for the analysis (educators, students, administrators, researchers): The audience for the analysis includes: education researchers (in STEM and SoTL), administrators, instructors, students, computational linguists with interest in applying formal linguistic analysis for building NLP-informed educational applications and learning technologists.

Links to key projects/papers

Project site http://ubc.science-nlpla.ca is under construction. Projects information and documentation will be shared there.

Some selected publications

Elouazizi, Noureddine. (2015). Critical factors in data governance for eLearning Analytics. Journal of Learning Analytics, 1(3), 211-222. Published by the Society for Learning Analytics Research (SoLAR), University Of Technology, Sydney, Australia.

Elouazizi, Noureddine. (2014). Point-of-View Mining and Cognitive Presence in MOOCs: A (Computational) Linguistics Perspective. In the Proceedings of the 2014 Conference on Empirical Methods In Natural Language Processing. Modeling Large Scale Social Interaction in Massively Open Online Courses. Eds., Carolyn Penstein Rosé and George Siemens. Published by The Association for Computational Linguistics.

Elouazizi, Noureddine and Bachvarova, Yulia. (2004). On Cognitive Relevance in Automatic Multimodal Artificial Systems. In Proceedings of IEEE Sixth International Symposium on Multimedia Software Engineering. Published by The Institute for Electrical and Electronics Engineering (IEEE) and The American Computer Society. Pp. 418-427.

Elouazizi, Noureddine. (2004). Modeling Referential Resolution for Dialog Systems. In Beitrage zur 7. Konferenz zur Verarbeintung naturlicher Sprache (KOVENS/ Advanced Topics in Modelling Natural Language Dialog. Published by the Austrian Association for Artificial Intelligence, Vienna. Pp. 01-10.

Andrew Gibson (University of Technology Sydney)

Bio: Andrew is a Research Fellow in Reflective Writing Analytics (RWA) at the University of Technology Sydney (UTS). His doctoral research involved the development of a conceptual model of RWA, a mode of reasoning for transepistemic work called Transepistemic Abduction (TeA), and the development of software for the collection and analysis of reflective writing. Andrew’s current research interest is in the dynamic realtime provision of actionable formative feedback to students for reflective writing improvement.

Type of writing data: Reflective Writing Data. Short diary-style reflective writing, and longer essay-style reflective writing.

Tools used: Write own software using Scala based on Scala and Java NLP frameworks such as CoreNLP, Factorie, and OpenNLP. Latest interests include Deep Learning and Word2Vec on Apache Spark.

Writing features analysed: Any features, but particularly interested in combining rules based analysis of lexical features with probabilistic and neural modelling of features through document and corpus analysis.

Target writers (e.g. K12, undergrad, etc.): Current focus is Higher Ed students – mix of undergrad and postgrad.

Evaluation measures used to validate your tool/approach: Currently developing a new evaluation method based around a user feedback loop.

Audience for the analysis (educators, students, administrators, researchers): Students and educators.

Links to key projects/papers

https://cic.uts.edu.au/projects/uts-projects/authentic-assessment-analytics-from-reflection/

http://awa.uts.edu.au/

http://alasi.nlytx.io/

https://github.com/andrewresearch

http://eprints.qut.edu.au/view/person/Gibson,_Andrew.html

Ágnes Sándor (Xerox Research Centre Europe, Grenoble)

Bio: PhD in pragmatics at the University Lyon 2. Research scientist at XRCE since 1996. Research interest: identifying and modeling rhetorical and discourse indicators of mainly academic genres and implementing them in a grammar engine for automated processing. I have also studied other types of texts, like financial reports and technical social media forums.

Type of writing data: any academic genre: research article, student essay, project proposal

Tools used: Xerox Incremental Parser

Writing features analysed: genre-specific rhetorical, conceptual

Target writers (e.g. K12, undergrad, etc.): university level

Evaluation measures used to validate your tool/approach: precision, recall compared to human annotation

Audience for the analysis (educators, students, administrators, researchers): potentially all

Links to key projects/papers:

Lisacek, F., Chichester, C., Kaplan, A. & Sándor, Á. (2005). Discovering paradigm shift patterns in biomedical abstracts: application to neurodegenerative diseases. First International Symposium on Semantic Mining in Biomedicine, Cambridge, UK, April 11-13, 2005.

Sándor, Á. (2007). Modeling metadiscourse conveying the author’s rhetorical strategy in biomedical research abstracts. Revue Française de Linguistique Appliquée 200(2), pp. 97–109.

De Liddo, A., Sándor, Á. and Buckingham Shum, S. (2012). Contested Collective Intelligence: rationale, technologies, and a human-machine annotation study. Computer Supported Cooperative Work (CSCW), 21(4-5), pp.

Denise Whitelock (Open University UK)

Bio: Denise has over twenty years experience in designing, researching and evaluating online and computer-based learning in Higher Education. She is a Professor of Technology Enhanced Assessment and Learning in the Open University’s Institute of Educational Technology. She has just completed directing the SAFeSEA project. http://www.open.ac.uk/researchprojects/safesea/ The aim of this research was to provide an effective automated interactive feedback system that yields an acceptable level of support for university students writing essays in a distance or e-learning context. She has edited four special journal issues on e-Assessment for BJET, LMTECI, IJCELL and IJEA. She chaired the International Computer Assisted Assessment Conference from 2010 until 2013. Her work has received international recognition as she holds visiting chairs at the Autonoma University, Barcelona and the British University in Dubai. She is also a serving member of the governing council for the Society for Research into Higher Education.

Type of writing data: We have been using OpenEssayist to give feedback on draft undergraduate essays for the Masters in Open and Distance Education (MAODE) at the Open University. We are now exploring using OpenEssayist with undergraduate draft essays at Maastricht University. I have also been involved with using the Xerox Incremental Parser (XIP) to see if its analytical features match those that tutors use to ascribe marks to students essays.

Tools used: OpenEssayist was built with Oxford University and funded by the EPSRC. Please see SAFeSEA project website http://www.open.ac.uk/researchprojects/safesea/ It is a Natural Language analytics engine to provide feedback to students when preparing an essay for summative assessment. I have also worked with the Xerox Incremental Parser (XIP). See Reference 2.

Writing features analysed: OpenEssayist processes open-text essays and offers feedback through key phrase extraction and extractive summarization. Each essay is automatically pre-processed using modules from the Natural Language Processing Toolkit (A). These include several tokenizers, a lemmatizer, a part-of-speech tagger and a list of stop words. Key phrase extraction identifies which phrases are most suggestive of the content and extractive summarization identifies key sentences. These constitute the basis for providing feedback.

The system presents users with several kinds of feedback, including: identification of an essay’s most prominent words, with graphical illustrations of their use across the essay; identification of the most representative sentences, with hints encouraging users to reflect on whether they express, in their view, the central ideas; and graphical illustrations of the essay’s internal structure.

(A). Bird, S., Klein, E. and Loper, E. (2009). Natural Language processing with Python. Cambridge, MA: O’Reilly.

Target writers (e.g. K12, undergrad, etc.): Higher Education students; Undergraduate and Masters’ level students

Evaluation measures used to validate your tool/approach: Significant positive correlations between the number of drafts submitted to OpenEssayist and the grades awarded for the essay. The student cohort that had access to OpenEssayist had significantly higher overall grades than the cohort that had no access to OpenEssayist.

Audience for the analysis (educators, students, administrators, researchers): NLP researchers, learning technologists, tutors, students, Pro-Vice Chancellors responsible for Learning & Teaching.

Links to key projects/papers

Whitelock, D., Twiner, A., Richardson, J.T.E., Field, D. & Pulman, S. (2015). OpenEssayist: A supply and demand learning analytics tool for drafting academic essays. The 5th International Learning Analytics and Knowledge (LAK) Conference, Poughkeepsie, New York, USA. 16-20 March 2015. ISBN 978-1-4503-3417-4. doi 10.1145/2723576.2723599
Simsek, D., Sandor, A., Buckingham Shum, S., Ferguson, R., De Liddo, A. & Whitelock, D. (2015). Correlations between automated Rhetorical Analysis and Tutors’ Grades on Student Essays. The 5th International Learning Analytics and Knowledge (LAK) Conference, Poughkeepsie, New York, USA. 16-20 March 2015. ISBN 978-1-4503-3417-4. http://dx.doi.org/10.1145/2723576.2723603
Whitelock, D., Field, D., Pulman, S., Richardson, J.T.E. & Van Labeke, N. (2014) Designing and Testing Visual Representations of Draft Essays for Higher Education Students. The 2nd International Workshop on Discourse-Centric Learning Analytics, 4th Conference on Learning Analytics and Knowledge (LAK 2014), Indianapolis, Indiana, USA. http://oro.open.ac.uk/41845/
Alden Rivers, Bethany; Whitelock, Denise; Richardson, John T.E.; Field, Debora and Pulman, Stephen (2014). Functional, frustrating and full of potential: learners’ experiences of a prototype for automated essay feedback. In: Kalz, Marco and Ras, Eric eds. Computer Assisted Assessment: Research into E-Assessment. Communications in Computer and Information Science (439). Cham, Switzerland: Springer, pp. 40–52. http://oro.open.ac.uk/40421/
Whitelock, D., Twiner, A., Richardson, J.T.E., Field, D. & Pulman, S. (2015). Feedback on academic essay writing through pre-emptive hints: Moving towards ‘advice for action’. Winner of Best Research Paper Award. Special Issue of European Journal of Open, Distance and E-Learning, Best of EDEN RW8, 8th EDEN Research Workshop (eds. U. Bernath and A. Szucs). Published by European Distance and E-Learning Network, pp.1-15, ISSN 1027 5207. http://oro.open.ac.uk/43207/
Alden, B., Van Labeke, N., Field, D., Pulman, S., Richardson, J.T.E. & Whitelock, D. (2014) Using student experience as a model for designing an automatic feedback system for short essays. International Journal of e-Assessment, 1(1) ISSN: 2045-9432. http://oro.open.ac.uk/40517/
Field, D, Richardson, J.T.E., Pulman, S., Van Labeke, N. & Whitelock, D. (2014) An exploration of the features of graded student essays using domain- independent natural language techniques. International Journal of e-Assessment, 1(1) ISSN: 2045-9432. http://oro.open.ac.uk/40516/

Phil Winne (Simon Fraser University)

Bio: I’m a learning scientist focusing my research on self-regulated learning, metacognition and technology tools that generate ambient big data about cognitive and motivational processes in these arenas. My current project lies in the area of authentic learning projects that call for round-trip information problem solving – defining and setting parameter values for information needed to complete the learning project, searching for and filtering information sources (websites, documents) judged to meet needs, analyzing and extracting information from filtered sources, organizing that content, designing and drafting a report, and evaluating and revising the draft to produce a polished final version.

Type of writing data: text selected, notes generated, concept maps designed, and conversations amongst peers in early phases of IPS, features of drafts (e.g., concepts used, coverage of conceptual spaces in drafts relative to source documents)

Tools used: My team and I are developing nStudy, an extension to the Chrome web browser designed to to support online information problem solving (IPS) in learning projects that gathers ambient data (data “collected as a matter of course;” Pistilli, et al., 2014, p. 85) to track IPS events. In nStudy, learners can search for information, highlight and tag content, annotate texts and videos, create terms to build a glossary, map concepts, chat about content and tasks, share resources (e.g., notes), organize resources, draft essays, and more. We are exploring text and data mining algorithms, graph theory and other tools that can exploit nStudy’s fine-grained, time-stamped trace data linking learners’ work in the pre-drafting stage if IPS to drafts composed and edited for the final product of the learning project.

Writing features analysed: We plan to analyze concept acquisition, relations between concepts operated on in early phases if IPS and a used in drafts, clustering of concepts in studying and in drafts, cohesion, form(s) of argumentation and explanation

Target writers (e.g. K12, undergrad, etc.): upper-level school through post-graduate

Evaluation measures used to validate your tool/approach: human scoring of essays, graph theoretic indexes reflecting qualities of information surveyed and used in essays

Audience for the analysis (educators, students, administrators, researchers): students, instructors, learning scientists, methodologists working in the field of IPS

Links to key projects/papers:

http://www.sfu.ca/edpsychlab/nstudy.html

https://www.youtube.com/watch?v=qa9OT8Atdts

Organiser profiles

Laura Allen (Arizona State University)

Brief bio para: Laura Allen is a Doctoral Student in the Psychology Department (Cognitive Science) at Arizona State University. The overarching aim of her research is to better understand the cognitive processes involved in language comprehension, writing, knowledge acquisition, and conceptual change, and to apply that understanding to educational practice by developing and testing educational technologies. Her research has been presented at LAK15 and other conferences related to writing analytics.

Type of writing data: Argumentative essays (“5- paragraph essays”), Source-based essays, Self-explanations and Think-alouds during text comprehension.

Tools used:

Natural language processing tools: Coh-Metrix, Writing Assessment Tool (WAT), TAALES, TAACO, SEANCE
Tutoring Systems: Writing Pal (W-Pal), iSTART
Keystroke and Screen logging: InputLog

Writing features analysed

Linguistic features: lexical sophistication, syntactic complexity, cohesion, writing style, rhetorical features, sentiment
Process/Behavioral features: Keystrokes, Affective judgments

Target writers (e.g. K12, undergrad, etc.): High school and undergraduate students

Evaluation measures used to validate your tool/approach: Expert human ratings, data mining, machine learning algorithms

Audience for the analysis (educators, students, administrators, researchers): Researchers, teachers

Links to key projects/papers: http://soletlab.com

Duygu Bektik (The Open University, UK)

Brief bio para: PhD student in discourse-centric learning analytics at the Open University, UK. My research investigates whether computational techniques can automatically identify the attributes of good academic writing in undergraduate student essays within different disciplines; and if this proves possible, how best to feedback actionable analytics to support educators in their essay assessment processes.

Type of writing data: Student academic writing: Undergraduate student essays across genres, mostly argumentatively written student essays that require students to acknowledge the literature, the debate between researchers and then build on these ideas with a critical eye.

Tools used: Xerox Incremental Parser (XIP) implemented by Xerox Research Centre Europe (Ágnes Sándor).

Writing features analysed: From the XIP analysis, rhetorically salient sentences that convey the meaning as background, contrast, and summary.

Target writers (e.g. K12, undergrad, etc.): Undergraduate students

Evaluation measures used to validate your tool/approach: Evaluation has been done both quantitatively and qualitatively. Measures used are both educators, their marking guidelines, assessment regimes and awarded essay mark by educators.

Audience for the analysis (educators, students, administrators, researchers): Educators

Links to key projects/papers

Simsek, D., Buckingham Shum, S, Sándor, Á, De Liddo, A., and Ferguson, R. (2013) XIP Dashboard: Visual Analytics from Automated Rhetorical Parsing of Scientific Metadiscourse. 1st International Workshop on Discourse-Centric Learning Analytics, at 3rd International Conference on Learning Analytics & Knowledge. Leuven, BE (Apr. 8-12, 2013).

Simsek, D., Sandor, Á., Buckingham Shum, S., Ferguson, R., De Liddo, A. and Whitelock, D. (2015). Correlations between automated rhetorical analysis and tutors’ grades on student essays. In: 5th International Learning Analytics & Knowledge Conference (LAK15) 16-20 March 2015, Poughkeepsie, NY, USA ACM, pp. 355-359.

Simon Buckingham Shum (Univ. Technology Sydney)

Bio: I’m Professor of Learning Informatics at the University of Technology Sydney, where I direct the Connected Intelligence Centre (CIC). Around 2010 I became active in the new field of Learning Analytics. I’m particularly interested in whether analytics offer new ways to provide rapid feedback on learner qualities that have always been important — but which have until now been hard to track and visualise in a timely way. The focus of CIC’s work is particularly around analytics for graduate attributes, which includes the ability to make analytical thinking or personal reflection clearly visible in writing.

Type of writing data: We are currently working with academics in Pharmacy, Accounting, Management, Data Science, Engineering and Ecology. Following briefings on academic writing through our Graduate Research School, PhD students also have access to our prototype tool. The texts we are concentrating on currently are student assignments, falling broadly into two genres of academic writing: reflective [1] and analytical [2-4]. In previous work I collaborated with colleagues on student discourse-centric learning analytics [5-6], and on applied research project reports [7].

Tools used: We have worked in collaboration with Xerox Research Centre Europe (Ágnes Sándor) developing a text analytics tool called AWA (Academic Writing Analytics) which leverages the NLP capabilities of the Xerox Incremental Parser (XIP). We have developed the user interface layer over this service. We’re now investigating additional approaches including TAACO, topic modelling and machine learning.

Writing features analysed: From XIP, the key development is to detect different kinds of rhetorical move that signify higher order thinking within a given genre, at the sentence level.

Target writers (e.g. K12, undergrad, etc.): University students from Undergraduate, through Masters and PhD. Possibly high schools in the future.

Evaluation measures used to validate your tool/approach: Over time we are aiming to develop a holistic approach. Currently this includes conventional F metrics comparing machine classification of sentences to human; writing researcher feedback on the parser; informal educator feedback on the pedagogical potential of the tool; structured educator feedback on parser output; think-aloud educator feedback on the user experience; quantitative and qualitative student feedback on their user experience.

Audience for the analysis (educators, students, administrators, researchers): Educators, students, researchers

Links to key projects/papers

Buckingham Shum, S., Á. Sándor, R. Goldsmith, X. Wang, R. Bass and M. McWilliams (2016). Reflecting on Reflective Writing Analytics: Assessment Challenges and Iterative Evaluation of a Prototype Tool. 6th International Learning Analytics & Knowledge Conference (LAK16), Edinburgh, UK, April 25 – 29 2016, ACM, New York, NY. http://dx.doi.org/10.1145/2883851.2883955 Preprint: http://bit.ly/LAK16paper
Simsek, D., Á. Sándor, S. Buckingham Shum, R. Ferguson, A. D. Liddo and D. Whitelock (2015). Correlations between automated rhetorical analysis and tutors’ grades on student essays. Proceedings of the Fifth International Conference on Learning Analytics And Knowledge, Poughkeepsie, New York, ACM. http://dx.doi.org/10.1145/2723576.2723603
Simsek, D., S. Buckingham Shum, Á. Sándor, A. D. Liddo and R. Ferguson (2013). XIP Dashboard: Visual Analytics from Automated Rhetorical Parsing of Scientific Metadiscourse. 1st International Workshop on Discourse-Centric Learning Analytics, 3rd International Conference on Learning Analytics & Knowledge, Leuven, BE (Apr. 8-12, 2013). . Open Access Eprint: http://oro.open.ac.uk/37391
Ferguson, R., Z. Wei, Y. He and S. Buckingham Shum (2013). An Evaluation of Learning Analytics to Identify Exploratory Dialogue in Online Discussions. Proc. 3rd International Conference on Learning Analytics & Knowledge, Leuven, BE, ACM. Open Access Eprint: http://oro.open.ac.uk/36664
Ferguson, R. and S. Buckingham Shum (2011). Learning analytics to identify exploratory dialogue within synchronous text chat. 1st International Conference on Learning Analytics and Knowledge, Banff, Canada, ACM. Open Access Eprint http://oro.open.ac.uk/28955
De Liddo, A., S. Buckingham Shum, I. Quinto, M. Bachler and L. Cannavacciuolo (2011). Discourse-Centric Learning Analytics. Proc. 1st International Conference on Learning Analytics & Knowledge, Banff, CA, ACM: New York. DOI: http://dx.doi.org/10.1145/2090116.2090120. Open Access Eprint: http://oro.open.ac.uk/25829
De Liddo, A., Á. Sándor and S. Buckingham Shum (2012). Contested Collective Intelligence: Rationale, Technologies, and a Human-Machine Annotation Study. Computer Supported Cooperative Work 21(4-5): 417-448. Open Access Eprint: http://oro.open.ac.uk/31052

Scott Crossley (Georgia State University)

Brief bio para: Scott Crossley is an Associate Professor of Applied Linguistics at Georgia State University. Professor Crossley’s primary research focus is on natural language processing and the application of computational tools and machine learning algorithms in language learning, writing, and text comprehensibility. His main interest area is the development and use of natural language processing tools in assessing writing quality and text difficulty.

Type of writing data: Independent and integrated essays. Genre and discipline specific writing samples. Constructed responses. Paraphrases and summaries

Tools used: Natural language processing tools to include Coh-Metrix, the Writing Assessment Tool, TAALES, TAACO, TAASSC, SEANCE

Writing features analysed: Linguistic features including lexical sophistication, syntactic complexity, cohesion, discourse features, grammar, rhetorical features, affective features

Target writers (e.g. K12, undergrad, etc.): K-12, college level writers, second language (L2) writers

Evaluation measures used to validate your tool/approach: Human ratings, machine learning algorithms, data mining

Audience for the analysis (educators, students, administrators, researchers): Researchers, administrators, students, teachers

Links to key projects/papers

http://soletlab.com/

http://alsl.gsu.edu/profile/crossley-scott/
http://www.kristopherkyle.com/tools.html

Organisers

Simon Buckingham Shum is Professor of Learning Informatics at the University of Technology Sydney, where he directs the Connected Intelligence Centre. His research focuses on learning analytics for higher order competencies such as academic writing, argumentation and learning-to-learn. He served as LAK12 Program Co-Chair, and co-chaired the LAK13/14 workshops on Discourse-Centred Learning Analytics.

Simon Knight is a Research Fellow in Writing Analytics at the Connected Intelligence Centre, University of Technology Sydney. His research focuses on the relationship of analytics to epistemology, pedagogy and assessment, discourse analytics, and epistemic cognition, particularly around information seeking, work which has been presented at LAK and ICLS. He co-chaired the ICLS14 Workshop on Learning Analytics for Learning and Becoming in Practice and LAK15 Workshop on Temporal Analyses of Learning Data.

Danielle McNamara is a Professor of Psychology at Arizona State University, where she directs the Science of Learning and Educational Technology Lab. Her research focuses on discovering new methods to improve students’ ability to understand text, learn new information, and convey their thoughts in writing. Her work integrates various approaches and methodologies including the development of intelligent tutoring systems and the development of natural language processing tools.

Laura Allen is a Doctoral Student in the Psychology Department at Arizona State University. The overarching aim of her research is to better understand the cognitive processes involved in language comprehension, writing, knowledge acquisition, and conceptual change, and to apply that understanding to educational practice by developing and testing educational technologies. Her research has been presented at LAK15 and other conferences related to writing analytics.

Duygu Bektik is a Doctoral Student at the Knowledge Media Institute, Open University UK. Her research investigates whether computational techniques can automatically identify the attributes of good academic writing in undergraduate student essays within different disciplines; and if this proves possible, how best to feedback actionable analytics to support educators in their essay assessment processes, which has been presented at LAK14/15.

Scott Crossley is Associate Professor of Applied Linguistics at Georgia State University. His primary research focus is on natural language processing and the application of computational tools and machine learning algorithms in language learning, writing, and text comprehensibility. His main interest area is the development and use of natural language processing tools in assessing writing quality and text difficulty. Professor Crossley works as a senior researcher on Writing Pal, an intelligent tutoring system under development at Arizona State University.
Collected references

The following represent the chairs’ research in this topic.

Allen, L., Jacovina, M. and McNamara, D. in press. Computer-based writing instruction. In C. A. MacArthur, S. Graham, & J. Fitzgerald (Eds.), Handbook of Writing Research.

Allen, L., Snow, E. and McNamara, D. S. 2015. Are you reading my mind? Modeling students’ reading comprehension skills with Natural Language Processing techniques. In: 5th International Learning Analytics & Knowledge Conference (LAK15) 16-20 March 2015, Poughkeepsie, NY, USA ACM, pp. 246-254.

Buckingham Shum, S., Ágnes Sándor, Rosalie Goldsmith, Xiaolong Wang, Randall Bass and Mindy McWilliams (2016, In Press). Reflecting on Reflective Writing Analytics: Assessment Challenges and Iterative Evaluation of a Prototype Tool. 6th International Learning Analytics & Knowledge Conference (LAK16). Edinburgh, UK. ACM Press. http://dx.doi.org/10.1145/2883851.2883955 Preprint: http://bit.ly/LAK16paper

Knight, S. and K. Littleton (2015). Discourse-centric learning analytics: mapping the terrain. Journal of Learning Analytics, 2 (1), pp. 185-209.

Knight, S., S. Buckingham Shum and K. Littleton (2014). Epistemology, assessment, pedagogy: where learning meets analytics in the middle space. Journal of Learning Analytics, 1, (2), pp. 23-47.

McNamara, D., Crossley, S., Roscoe, R., Allen, L. and Dai, J. 2015. Hierarchical classification approach to automated essay scoring. Assessing Writing, 23, 35-59.

McNamara, D., Graesser, A., McCarthy, P. and Cai, Z. (2014). Automated evaluation of text and discourse with Coh-Metrix. Cambridge: Cambridge University Press.

Roscoe, R.D., Varner, L.K., Crossley, S.A. and McNamara, D.S. (2013). Developing pedagogically-guided algorithms for intelligent writing feedback. International Journal of Learning Technology, 8, 362-381.

Roscoe, R., Varner, L., Weston, J., Crossley, S. and McNamara, D. (2014). The Writing Pal Intelligent Tutoring System: Usability Testing and Development. Computers and Composition, 34, 39-59.