Fischer, C., Pardos, Z. A., Baker, R. S., Williams, J. J., Smyth, P., Yu, R., Slater, S., Baker, R., & Warschauer, M. (2020). Mining Big Data in Education: Affordances and Challenges. Review of Research in Education, 44(1), 130–160. https://doi.org/10.3102/0091732X20903304
Friedman, C. P., & Flynn, A. J. (2019). Computable Knowledge: An Imperative for Learning Health Systems. Learning Health Systems, 3(4), e10203. https://doi.org/10.1002/lrh2.10203
Friedman, C. P., Rubin, J. C., & Sullivan, K. J. (2017). Toward an Information Infrastructure for Global Health Improvement. Yearbook of Medical Informatics, 26(01), 16–23. https://doi.org/10.15265/IY-2017-004
Geertz, C. (1973). Thick Description: Toward an Interpretive Theory of Culture. In Interpretation of Cultures: Selected Essays (pp. 3–30). Basic Books.
George, B. C., Bohnen, J. D., Williams, R. G., Meyerson, S. L., Schuller, M. C., Clark, M. J., Meier, A. H., Torbeck, L., Mandell, S. P., Mullen, J. T., Smink, D. S., Scully, R. E., Chipman, J. G., Auyang, E. D., Terhune, K. P., Wise, P. E., Choi, J. N., Foley, E. F., Dimick, J. B., … Collaborative (PLSC), on behalf of the P. L. and S. (2017). Readiness of US General Surgery Residents for Independent Practice. Annals of Surgery, 266(4), 582–594. https://doi.org/10.1097/SLA.0000000000002414
Due to the increasing amount of available published evidence and the continual need to apply and update evidence in practice, we propose a shift in the way evidence generated by learning health systems can be integrated into more traditional evidence reviews. This paper discusses two main mechanisms to close the evidence-to-practice gap: (1) integrating Learning Health System (LHS) results with existing systematic review evidence and (2) providing this combined evidence in a standardized, computable data format. We believe these efforts will better inform practice, thereby improving individual and population health.
Harrison, M. I., Koppel, R., & Bar-Lev, S. (2007). Unintended Consequences of Information Technologies in Health Care—An Interactive Sociotechnical Analysis. Journal of the American Medical Informatics Association, 14(5), 542–549. https://doi.org/10.1197/jamia.M2384
Hawn Nelson, A., Jenkins, D., Zanti, S., Katz, M., Berkowitz, E., Burnett, T. C., & Culhane, D. (2020). A Toolkit for Centering Racial Equity Throughout Data Integration – Actionable Intelligence for Social Policy. Actionable Intelligence for Social Policy, University of Pennsylvania. https://aisp.upenn.edu/resource-article/a-toolkit-for-centering-racial-equity-throughout-data-integration/
Hershkovitz, A., de Baker, R. S. J., Gobert, J., Wixon, M., & Pedro, M. S. (2013). Discovery With Models: A Case Study on Carelessness in Computer-Based Science Inquiry. American Behavioral Scientist, 57(10), 1480–1499. https://doi.org/10.1177/0002764213479365
Hollands, F., & Bakir, I. (2015). Efficiency of Automated Detectors of Learner Engagement and Affect Compared with Traditional Observation Methods [Working Paper]. Center for Benefit-Cost Studies of Education, Teachers College, Columbia University. https://repository.upenn.edu/cbcse/4
Huberth, M., Chen, P., Tritz, J., & McKay, T. A. (2015). Computer-Tailored Student Support in Introductory Physics. PLOS ONE, 10(9), e0137001. https://doi.org/10.1371/journal.pone.0137001
James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An Introduction to Statistical Learning: With Applications in R. Springer Science & Business Media.
Kerr, D., Andrews, J. J., & Mislevy, R. J. (2016). The In-Task Assessment Framework for Behavioral Data. In The Wiley Handbook of Cognition and Assessment (pp. 472–507). John Wiley & Sons, Ltd. https://doi.org/10.1002/9781118956588.ch20
Knowles, J. E. (2015). Of Needles and Haystacks: Building an Accurate Statewide Dropout Early Warning System in Wisconsin. Journal of Educational Data Mining, 7(3), Article 3. https://doi.org/10.5281/zenodo.3554725
Baker, R. S., & Koedinger, K. R. (2018). Towards Demonstrating the Value of Learning Analytics for K–12 Education. In D. Niemi, R. D. Pea, B. Saxberg, & R. E. Clark (Eds.), Learning Analytics in Education (pp. 49–62). IAP.
Koedinger, K. R., Baker, R. S. J. d, Cunningham, K., Skogsholm, A., Leber, B., & Stamper, and J. (2010). A Data Repository for the EDM Community: The PSLC DataShop. In C. Romero, S. Ventura, M. Pechenizkiy, & R. S. J. d. Baker (Eds.), Handbook of Educational Data Mining (pp. 43–56). CRC Press.
Koedinger, K. R., & Corbett, A. (2005). Cognitive Tutors: Technology Bringing Learning Sciences to the Classroom. In R. K. Sawyer (Ed.), The Cambridge Handbook of the Learning Sciences (pp. 61–78). Cambridge University Press. https://doi.org/10.1017/CBO9780511816833.006
Koedinger, K. R., D’Mello, S., McLaughlin, E. A., Pardos, Z. A., & Rosé, C. P. (2015). Data Mining and Education. WIREs Cognitive Science, 6(4), 333–353. https://doi.org/10.1002/wcs.1350
Koedinger, K. R., Stamper, J. C., McLaughlin, E. A., & Nixon, T. (2013). Using Data-Driven Discovery of Better Student Models to Improve Student Learning. In H. C. Lane, K. Yacef, J. Mostow, & P. Pavlik (Eds.), Artificial Intelligence in Education (pp. 421–430). Springer. https://doi.org/10.1007/978-3-642-39112-5_43
Kreuter, F., & Peng, R. D. (2014). Extracting Information from Big Data: Issues of Measurement, Inference and Linkage. In H. Nissenbaum, J. Lane, S. Bender, & V. Stodden (Eds.), Privacy, Big Data, and the Public Good: Frameworks for Engagement (pp. 257–275). Cambridge University Press. https://doi.org/10.1017/CBO9781107590205.016
Krumm, A. E., & Beattie, R. (2017). Strategies for Making Digital Learning System Data Usable: A Design Workshop Approach. AERA Anual Meeting, San Antonio, TX.
Krumm, A. E., Yeager, D. S., & Yamada, H. (2019). Explanatory and Predictive Modeling Within Improvement Science Projects. AERA Annual Meeting, Toronto, ON. http://tinyurl.com/yaxj9my4
This paper describes a partnership-driven approach for developing measures of learning behaviors using event data from a digital learning environment is used in all grades and subjects within a charter management organization. The approach that we developed and followed included involved (1) gathering leaders’, teachers’, and students’ perspectives on learning behaviors; (2) collaboratively analyzing data; (3) using exploratory factor analyses to generate an single score; and (4) conducting explicit model-based tests that assessed the degree to which the single score was correlated with outcomes that were important to all members of the partnership.
Krumm, A. E., Beattie, R., Takahashi, S., D’Angelo, C., Feng, M., & Cheng, B. (2016). Practical Measurement and Productive Persistence: Strategies for Using Digital Learning System Data to Drive Improvement. Journal of Learning Analytics, 3(2), Article 2. https://doi.org/10.18608/jla.2016.32.6
Krumm, A. E., Boyce, J., & Everson, H. T. (2021). A Collaborative Approach to Sharing Learner Event Data. Journal of Learning Analytics, 8(2), Article 2. https://doi.org/10.18608/jla.2021.7375
Liu, R., & Koedinger, K. R. (2017). Closing the Loop: Automated Data-Driven Cognitive Model Discoveries Lead to Improved Instruction and Learning Gains. Journal of Educational Data Mining, 9(1), Article 1. https://doi.org/10.5281/zenodo.3554625
Liu, R., Stamper, J. C., & Davenport, J. (2018). A Novel Method for the In-Depth Multimodal Analysis of Student Learning Trajectories in Intelligent Tutoring Systems. Journal of Learning Analytics, 5(1), Article 1. https://doi.org/10.18608/jla.2018.51.4
Means, B. (2018). Tinkering Toward a Learning Utopia: Implementing Learning Engineering. In C. Dede, J. Richards, & B. Saxberg (Eds.), Learning Engineering for Online Education. Routledge.
Meyer, A., Grunow, A., & Krumm, A. E. (2017). Are these Changes an Improvement? Using Measures to Inform Homework Practices. AERA Annual Meeting, San Antonio, TX.
Dede, C. (2015). Data-Intensive Research in Education: Current Work and Next Steps. Computing Research Association. https://cra.org/wp-content/uploads/2015/10/CRAEducationReport2015.pdf
Nelson, I. A., London, R. A., & Strobel, K. R. (2015). Reinventing the Role of the University Researcher. Educational Researcher, 44(1), 17–26. https://doi.org/10.3102/0013189X15570387
Paquette, L., Ocumpaugh, J., Li, Z., Andres, A., & Baker, R. (2020). Who’s Learning? Using Demographics in EDM Research. Journal of Educational Data Mining, 12(3), Article 3. https://doi.org/10.5281/zenodo.4143612
Piety, P. J. (2019). Components, Infrastructures, and Capacity: The Quest for the Impact of Actionable Data Use on P–20 Educator Practice. Review of Research in Education, 43(1), 394–421. https://doi.org/10.3102/0091732X18821116
Piety, P. J., Hickey, D. T., & Bishop, M. J. (2014). Educational data sciences: Framing emergent practices for analytics of learning, organizations, and systems. Proceedings of the Fourth International Conference on Learning Analytics And Knowledge, 193–202. https://doi.org/10.1145/2567574.2567582
Piety, P. J., & Pea, R. D. (2018). Understanding Learning Analytics Across Practices. In D. Niemi, R. D. Pea, B. Saxberg, & R. E. Clark (Eds.), Learning Analytics in Education (pp. 215–232). IAP.
Reardon, S. F. (2019). Educational Opportunity in Early and Middle Childhood: Using Full Population Administrative Data to Study Variation by Place and Age. RSF: The Russell Sage Foundation Journal of the Social Sciences, 5(2), 40–68. https://doi.org/10.7758/rsf.2019.5.2.03
Roschelle, J., Knudsen, J., & Hegedus, S. (2010). From New Technological Infrastructures to Curricular Activity Systems: Advanced Designs for Teaching and Learning. In M. J. Jacobson & P. Reimann (Eds.), Designs for Learning Environments of the Future: International Perspectives from the Learning Sciences (pp. 233–262). Springer US. https://doi.org/10.1007/978-0-387-88279-6_9
Luckin, R., Hansen, C., Wasson, B., Clark, W., Avramides, K., Hunter, J., & Oliver, M. (2015). Teacher Inquiry into Students’ Learning: Researching Pedagogical Innovations. In P. Reimann, S. Bull, M. Kickmeier-Rust, R. Vatrapu, & B. Wasson (Eds.), Measuring and Visualizing Learning in the Information-Rich Classroom. Routledge.
Scoville, R., & Little, K. (2014). Comparing Lean and Quality Improvement [White Paper]. Institute for Healthcare Improvement (IHI). https://www.ihi.org:443/resources/Pages/IHIWhitePapers/ComparingLeanandQualityImprovement.aspx
Sendak, M. P., Balu, S., & Schulman, K. A. (2017). Barriers to Achieving Economies of Scale in Analysis of EHR Data. Applied Clinical Informatics, 08(03), 826–831. https://doi.org/10.4338/ACI-2017-03-CR-0046
Shah, N. H., Milstein, A., & Bagley, P., Steven C. (2019). Making Machine Learning Models Clinically Useful. JAMA, 322(14), 1351–1352. https://doi.org/10.1001/jama.2019.10306
Stamper, J. C., & Koedinger, K. R. (2011). Human-Machine Student Model Discovery and Improvement Using DataShop. In G. Biswas, S. Bull, J. Kay, & A. Mitrovic (Eds.), Artificial Intelligence in Education (pp. 353–360). Springer. https://doi.org/10.1007/978-3-642-21869-9_46
Star, S. L. (2010). This is Not a Boundary Object: Reflections on the Origin of a Concept. Science, Technology, & Human Values, 35(5), 601–617. https://doi.org/10.1177/0162243910377624