Hi there, I’m currently reviewing some Learning Analytics papers and wondering if anyone of you knows about an evaluation framework / review guidelines for Learning Analytics? I’m asking that because one of the objectives of Learning Analytics is the personalization of learning, but in order to measure what are promising steps towards this personalization we need to define benchmarks and evaluation criteria in form of a framework. Otherwise I can hardly compare the outcomes of one LA application / paper with another one. Developing such an evaluation framework is not a very easy task, on the other hand many other related research fields achieved a kind of evaluation standard like in data mining and recommender system with the TREC conference for example. They focus on performance, accuracy, precision and recall measures that are rather technical but useful to show the effects of a certain technologies on a specific dataset.   Learning Analytics is not only about technical measures. At the end we want to support teachers and students in their learning process and make the educational system more transparent. But measuring an increase in the effectiveness (learning outcomes) and efficiency (study time) of the learning process takes most of the time much longer than testing the technical measures. The best well known educational evaluation measure is from Kirkpatrick (http://www.businessballs.com/kirkpatricklearningevaluationmodel.htm) but that requires several pre- and post test in a longer timeperiod.     So any ideas about an evaluation framework for Learning Analytics? By the way the ACM Recommender System conference applies the following evaluation criteria to their papers. Algorithms A good RecSys algorithms paper will: • describe the recommender/ranking/prediction algorithm in sufficient detail that someone else could implement it • articulate the important new idea(s) that the algorithm instantiates, in comparison to previously known algorithms • demonstrate that performance is better on some well-defined metric, than some baseline algorithm.   Applications A good RecSys paper reporting on a case study of an application deployment will: • Identify a novel type of item to be recommended or decision process to be influenced, in comparison to previously reported targets of recommender systems. • Identify unusual properties of the new item type that created special problems or opportunities • Explain any non-trivial mappings of known techniques to the new domain • Report on challenges and how they were overcome • Articulate lessons that might be relevant to others deploying Recommender Systems in similar or related contexts   Presentation Techniques A good RecSys paper about a new way of using recommendations/prediction/ranking to enhance the user experience will: • Clearly explain the presentation technique • Articulate what is novel about it, in comparison to existing techniques • Demonstrate that it has desirable properties for users, through anecdotes or data from lab studies or field deployment   Recommender Inputs A good RecSys paper about a new source of information to be used as an input to recommender algorithms will: • Clearly explain how the information will be gathered or elicited • Articulate what is novel about the information source, in comparison to other sources • Provide some evidence that it can be used to make good recommendations