Journal of Writing Analytics Jan 2018 Open Access

De-Identification of Laboratory Reports in STEM

Alex Rudniy University of Scranton

Abstract

Background: Employing natural language processing and latent semantic analysis, the current work was completed as a constituent part of a larger research project for designing and launching artificial intelligence in the form of deep artificial neural networks. The models were evaluated on a proprietary corpus retrieved from a data warehouse, where it was extracted from MyReviewers, a sophisticated web application purposed for peer review in written communication, which was actively used in several higher education institutions. The corpus of laboratory reports in STEM annotated by instructors and students was used to train the models. Under the Common Rule, research ethics were ensured by protecting the privacy of subjects and maintaining the confidentiality of data, which mandated corpus de-identification.Literature Review: De-identification and pseudonymization of textual data remains an actively studied research question for several decades. Its importance is stipulated by numerous laws and regulations in the United States and internationally with HIPAA Privacy Rule and FERPA.Research Question: Text de-identification requires a significant amount of manual post-processing for eliminating faculty and student names.  This work investigated automated and semi-automated methods for de-identifying student and faculty entities while preserving author names in cited sources and reference lists. It was hypothesized that a natural language processing toolkit and an artificial neural network model with named entity recognition capabilities would facilitate text processing and reduce the amount of manual labor required for post-processing after matching essays to a list of users’ names. The suggested techniques were applied with supplied pre-trained models without additional tagging and training. The goal of the study was to evaluate three approaches and find the most efficient one among those using a users’ list, a named entity recognition toolkit, and an artificial neural network.Research Methodology: The current work studied de-identification of STEM laboratory reports and evaluated the performance of the three techniques: brute forth search with a user lists, named entity recognition with the OpenNLP machine learning toolkit, and NeuroNER, an artificial neural network for named entity recognition built on the TensorFlow platform. The complexity of the given task was determined by the dilemma, where names belonging to students, instructors, or teaching assistants must be removed, while the rest of the names (e.g., authors of referenced papers) must be preserved.Results: The evaluation of the three selected methods demonstrated that automating de-identification of STEM lab reports is not possible in the setting, when named entity recognition methods are employed with pre-trained models. The highest results were achieved by the users’ list technique with 0.79 precision, 0.75 recall, and 0.77 F1 measure, which significantly outweighed OpenNLP with 0.06 precision, 0.14 recall, and 0.09 F1, and NeuroNER with 0.14 precision, 0.56 recall, and 0.23 F1.Discussion: Low performance of OpenNLP and NeuroNER toolkits was explained by the complexity of the task and unattainability of customized models due to imposed time constraints. An approach for masking possible de-identification errors is suggested.Conclusion: Unlike multiple cases described in the related work, de-identification of laboratory reports in STEM remained a non-trivial labor-intensive task. Applied out of the box, a machine learning toolkit and an artificial neural network technique did not enhance performance of the brute forth approach based on user list matching.Directions for Future Research: Customized tagging and training on the STEM corpus were presumed to advance outcomes of machine learning and predominantly artificial intelligence methods. Application of other natural language toolkits may lead to deducing a more effective solution.

Journal
Journal of Writing Analytics
Published
2018-01-01
DOI
10.37514/jwa-j.2018.2.1.07
CompPile
Open Access
OA PDF Gold
Topics
Export
BibTeX RIS

Citation context

Cited by in this index (4)

  1. Toth et al. (2026)
    Generative Artificial Intelligence, Writing Placement, and Principled Decision …
    Journal of Writing Analytics
  2. Si&acirc et al. (2026)
    Curating a Corpus: A Three-Phase Model
    Journal of Writing Analytics
  3. Lang et al. (2023)
    Computer-Assisted Corpus Analysis: An Introduction to Concepts, Processes, and …
    IEEE Transactions on Professional Communication
  4. Lang et al. (2019)
    A Taxonomy for Writing Analytics
    Journal of Writing Analytics

References (108) · 3 in this index

  1. Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., … Devin, M. (2016). Tensorflow: Large-sc…
  2. Anson et al. (2017)
    Assessing peer and instructor response to writing: A corpus analysis from an ex…
    Assessing Writing
  3. Apache OpenNLP Developer Documentation (2017). Retrieved from https://opennlp.apache.org/docs/1.8.4/manual/op…
  4. Aull (2017)
    Corpus Analysis of Argumentative Versus Explanatory Discourse in Writing Task Genres
    Journal of Writing Analytics
  5. Bayardo, R.J., & Agrawal, R. (2005). Data privacy through optimal k-anonymization. Proceedings: IEEE 21st Int…
     ↗
Show all 108 →
  1. Beckwith, B.A., Mahaadevan, R., Balis, U.J., & Kuo, F. (2006). Development and evaluation of an open source s…
     ↗
  2. Boyd, D. (2008). Facebook's privacy trainwreck: Exposure, invasion, and social convergence. Convergence: The …
     ↗
  3. Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., & Kuksa, P. (2011). Natural language proc…
  4. Daries, J.P., Reich, J., Waldo, J., Young, E.M., Whittinghill, J., Ho, A.D., … Chuang, I. (2014). Quality soc…
     ↗
  5. Dehghan, A., Kovacevicr, A., Karystianisab, G., Keanead, J.A., & Nenadic, G. (2015). Combining knowledge-and …
     ↗
  6. Dernoncourt, F., Lee, J. Y., & Szolovits, P. (2017). NeuroNER: An easy-to-use program for named-entity recogn…
     ↗
  7. Dernoncourt, F., Lee, J. Y., Uzuner, O., & Szolovits, P. (2016). De-identification of patient notes with recu…
     ↗
  8. (EU) 2016/680 of the European Parliament and of the Council. Retrieved from http://eur- lex.europa.eu/legal-c…
  9. Directive 95/46/EC of the European Parliament and of the Council of 24 October 1995 on the protection of indi…
  10. Donahue, C., Elliot, N., Ross, V., & Moxley, J. (2017, January). At the intersection of writing program admin…
  11. Douglass, M., Clifford, G., Reisner, A., Moody, G., & Mark, R. (2005). Computer-assisted de- identification o…
     ↗
  12. Electronic Communications Privacy Act of 1986 (ECPA), 18
  13. Elliot, N., Walkup, K., & Moxley, J. (2016). Preface to workshop two: Writing analytics, data mining, and wri…
  14. U.S.C. §1618. Retrieved from https://www.ftc.gov/system/files/fcra_2016.pdf. Family Educational Rights and Pr…
  15. Foufi, V., Gaudet-Blavignac, C., Chevrier, R., & Lovis, C. (2017). De-identification of medical narrative dat…
  16. Freedom of Information Act of 1966. Retrieved from http://congressionaldata.org/the-original-text-of-the- fre…
  17. GAO. (2008). Privacy: alternatives exist for enhancing protection of personally identifiable information: Rep…
  18. Gellman, R. (2017). Fair information practices: A basic history. Version 2.18. Retrieved from https://bobgell…
  19. Gilbert, F. (2009). Global privacy and security law. Austin, TX: Wolters Kluwer Law & Business.
  20. Greller, W., & Drachsler, H. (2012). Translating learning into numbers: A generic framework for learning anal…
  21. Gulshan V., Peng L., Coram M., Stumpe M.C., Wu D., Narayanaswamy A., et al. (2016). Development and validatio…
     ↗
  22. Gupta, D., Saul, M., & Gilbertson, J. (2004). Evaluation of a deidentification (De-Id) software engine to sha…
     ↗
  23. H.R. 4709 -109th Congress: Telephone Records and Privacy Protection Act of 2006. Retrieved from https://www.g…
  24. H.R.3103 -Health Insurance Portability and Accountability Act of 1996. Retrieved from https://www.congress.go…
  25. H.R.493 -Genetic Information Nondiscrimination Act of 2008. Retrieved from https://www.congress.gov/bill/110t…
  26. Haber, S., Hatano, Y., Honda, Y., Horne, W., Miyazaki, K., Sander, T., … Yao, D. (2007). Efficient signature …
  27. Hay, M., Miklau, G., Jensen, D., Towsley, D., & Weis, P. (2008). Resisting structural re-identification in an…
     ↗
  28. Health Insurance Portability and Accountability Act of 1996. Retrieved from https://www.gpo.gov/fdsys/pkg/PLA…
  29. Hinton, G. E., Deng, L., Yu, D., Dahl, G. E., Mohamed, A., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P.,…
     ↗
  30. Ingersoll, G.S., Morton, T.S., & Farris, A.L. (2013). Taming text. How to find, organize, and manipulate it. …
  31. Jiang, J. (2012). Information extraction from text. In C.C. Aggarwal & C. Zhai (Eds.), Mining text data (pp. …
     ↗
  32. Jolly, I. (2017). Data protection in the United States: Overview. Retrieved from https://uk.practicallaw.thom…
  33. Kalchbrenner, N., Grefenstette, E., & Blunsom, P. (2014). A convolutional neural network for modelling senten…
  34. Kanuppinen, A., Leijen, D., Moxley, J., & Wärnsby, A. (2016). Pre-conference workshop: Responsible Action: In…
  35. Khalil, M., & Ebner, M. (2016). De-identification in learning analytics. Journal of Learning Analytics, 3(1),…
     ↗
  36. Kumar, U., & Helmy, A. (2009). Human behavior and challenges of anonymizing WLAN traces. Proceedings from IEE…
     ↗
  37. Labeau, M., L'oser, K., & Allauzen, A. (2015). Non-lexical neural architecture for fine-grained POS tagging. …
     ↗
  38. Lample, G., Ballesteros, M., Subramanian, S., Kawakami, K., & Dyer, C. (2016). Neural architectures for named…
     ↗
  39. Leahy, P. (2015). Bipartisan coalition led by Senators Lee and Leahy introduce legislation to ban bulk collec…
  40. LeCun, Y. (1986). Learning processes in an asymmetric threshold network. In E. Bienenstock, F, Fogelman-Souli…
  41. Lee, J.Y., Dernoncourt, F., Uzuner, O., & Szolovits, P. (2016). Feature-augmented neural networks for patient…
  42. Leijen, D., & Moxley, J. (2017, June). The value of peer review across different institutional, national, and…
  43. Li, M. (2018). Scalable natural language de-identification based on machine learning approaches (Doctoral dis…
  44. Lopez-Otero, P., Docio-Fernandez, L., Abad, A., & Garcia-Mateo, C. (August, 2017). Depression detection using…
     ↗
  45. Ludwig, J. (2009). Australian government. Enhancing national privacy protection. Australian government first …
  46. Machanavajjhala, A., Gehrke, J., & Kifer, D. (2006). l-diversity: Privacy beyond k-anonymity. Proceedings fro…
     ↗
  47. McCulloch, W.S., & Pitts, W. (1943). A logical calculus of the ideas immanent in nervous activity. Bulletin o…
     ↗
  48. Meldau, E. (2018). Deep neural networks for inverse de-identification of medical case narratives in reports o…
  49. Menikoff, J., Kaneshiro, J., & Pritchard, I. (2017). The Common Rule, Updated. New England Journal of Medicin…
     ↗
  50. Meystre, S.M., Friedlin, F.J., South, B.R., Shen, S., & Samore, M.H. (2010). Automatic de-identification of t…
     ↗
  51. Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013a). Efficient estimation of word representations in vecto…
  52. Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., & Dean, J. (2013b). Distributed representations of words…
  53. Mikolov, T., Yih, W., & Zweig, G. (2013c). Linguistic regularities in continuous space word representations. …
  54. Minsky, M., & Papert, S. (1969). Perceptrons. An introduction to computational geometry. Cambridge, Mass: M.I…
  55. Moxley, J. (2013). Big data, learning analytics, and social assessment. The Journal of Writing Assessment, 6(…
  56. Moxley, J. (2016). From FYC to professional & technical communication, partnerships with other U.S. post-seco…
  57. Moxley, J. (2017a, January). e-Portfolios and digital learning: The future of corpus studies in the domain of…
  58. Moxley, J. (2017b, January). Welcome to writing analytics. Paper presented at the 4th International Conferenc…
  59. Moxley, J. (2017c, March.) Evidence-based framework for structuring learning opportunities: Peer review, STEM…
  60. Moxley, J., & Eubanks, D. (2016). On keeping score: Instructors' vs. students' rubric ratings of 46,689 essay…
  61. Moxley, J., Ross, V., Elliot, N., Rudniy, A., & Trauth, E. (2016). The role of instructor and peer feedback i…
  62. Moxley, J., Ross, V., & Trauth, E. (2016). Making friends in STEM: Experiences, challenges, and triumphs in c…
  63. Moxley, J., & Walkup, K. (2016). Mapping writing analytics. In Proceedings of the 9th International Conferenc…
  64. Moxley, J., Wärnsby, A., Kauppinen, A., Leijen, D., Aull, L., Anderson, L., & Walkup, K. (2017, February). Po…
  65. Narayanan, A., & Shmatikov, V. (2008). Robust de-anonymization of large sparse datasets. Proceedings: 2008 IE…
     ↗
  66. Neamatullah, I., Douglass, M. M., Lehman, L. H., Reisner, A., Villarroel, M., Long, W. J., … Clifford, G.D. (…
     ↗
  67. O'Keefe, C. M., Otorepec, S., Elliot, M., Mackey, E., & O'Hara, K. (2017). The de-identification decision-mak…
  68. Parker, D. B. (1985). Learning-logic: Casting the cortex of the human brain in silicon. Technical Report Tr-4…
  69. Pennington, J., Socher, R., & C. Manning (2014). GloVe: Global vectors for word representation. Proceedings: …
     ↗
  70. Phelps-Hillen, J. (2017). Institutional Review Boards and Writing Studies research: A justice-oriented study …
  71. Pinto, A., Oliveira, H.G., & Alves, A.O. (2016). Comparing the performance of different NLP toolkits in forma…
  72. Power, S. (2008). Privacy (Cross-border Information) Amendment Bill. Government Bill. Retrieved from http://w…
  73. Quinlan, J.R. (1993). C4.5: Programs for machine learning. San Mateo, CA: Morgan Kaufmann. Records, computers…
  74. (EU) 2016/679 of the European Parliament and of the Council. Retrieved from: http://eur- lex.europa.eu/legal-…
  75. Right to Financial Privacy Act of 1978. Chapter 35-Right to Financial Privacy. Retrieved from http://uscode.h…
  76. Ross, V., Elliot, N., Rudniy, A., & Moxley, J. (2016). Writing analytics, data mining, & writing studies. Pre…
  77. Ross et al. (2017)
    Assessing Writing Constructs: Toward an Expanded View of Inter-Reader Reliability
    Journal of Writing Analytics
  78. Ross, V., Liberman, M., Ngo, L., & LeGrand, R. (2017). Weighted log-odds-ratio, informative Dirichlet prior m…
  79. Ruch, P., Baud, R., Rassinoux, A., Bouillon, P., & Robert, G. (2000). Medical document anonymization with a s…
  80. Rudniy, A. (2018). Case study-Data warehouse design for evidence-based research. IEEE Transactions on Profess…
  81. S. 1490 -111th Congress: Personal Data Privacy and Security Act of 2009. Retrieved from https://www.govtrack.…
  82. Saeed, M., Lieu, C., Raber, G., & Mark, R.G. (2002). MIMIC II: A massive temporal ICU patient database to sup…
     ↗
  83. Sang, E.F.T.K., & De Meulder, F. (2003). Introduction to the conll-2003 shared task: Language- independent na…
     ↗
  84. Stenetorp, P., Pyysalo, S., Topić, G., Ohta, T, Ananiadou, S., & Tsujii, J. (2012). BRAT: A Web-based tool fo…
  85. Stubbs, A., Kotfila, C., & Uzuner, O. (2015). Automated systems for the de-identification of longitudinal cli…
     ↗
  86. Sweeney, L. (1996). Replacing personally-identifying information in medical records, the Scrub system. Procee…
  87. Sweeney, L. (1998). Datafly: A system for providing anonymity in medical data. In T. Lin, & S. Qian (Eds.), D…
     ↗
  88. Sweeney, L. (2000). Uniqueness of simple demographics in the U.S. Population (Report # LIDAPWP4). Pittsburgh,…
  89. Sweeney, L. (2002). k-anonymity: A model for protecting privacy. International Journal on Uncertainty, Fuzzin…
     ↗
  90. Szarvas, G., Farkas, R., & Busa-Fekete, R. (2007). State-of-the-art anonymization of medical records using an…
     ↗
  91. Taguchi, K., & Aramaki, E. (2018). Novel location de-identification for machine and human. UISTDA 2018. Retri…
  92. The Privacy Act of 1974 (As Amended). Public Law 93-579, as codified at
  93. U.S.C. 552a. Retrieved from http://www.dodig.mil/Portals/48/Documents/Programs/Privacy%20Program/pa1974.pdf?v…
  94. Troyano, J., Dıaz, V., Enrıquez, F., & Romero, L. (2004). Improving the performance of a named entity extract…
     ↗
  95. USA FREEDOM Act of 2015. Uniting and Strengthening America by Fulfilling Rights and Ensuring Effective Discip…
  96. USA PATRIOT Act of 2001. Uniting and Strengthening America by Providing Appropriate Tools Required to Interce…
  97. Uzuner, O., Luo, Y., & Szolovits, P. (2007). Evaluating the state-of-the-art in automatic de-identification. …
     ↗
  98. Wachtler, J., Khalil, M., Taraghi, B., & Ebner, M. (2016). On using learning analytics to track the activity …
  99. Wellner, B., Huyck, M., Mardis, S., Aberdeen, J., Morgan, A., Peshkin, L., … Hirschman, L. (2007). Rapidly re…
     ↗
  100. Yoose, B. (2017). Balancing privacy and strategic planning needs: A case study in de-identification of patron…
     ↗
  101. Zeide, E. (2016). Student privacy principles for the age of big data: Moving beyond FERPA and FIPPs. Drexel L…
  102. Zhao, Y., Zhang, K., Ma, H., & Li, K. (2018). Leveraging text skeleton for de-identification of electronic me…
     ↗
  103. Zhou, G., Zhang, J., Su, J., Shen, D., & Tan, C. (2005). Recognizing names in biomedical texts: A machine lea…
     ↗
CrossRef global citation count: 4 View in citation network → Build reading path →

Related articles

  1. IEEE Transactions on Professional Communication Jun 2025
    Presenting and Making Relevant: Analyzing Teaching Assistant Perceptions of Writing in Statistics Using Semantic Frames ↗
    Ben Markey
    rhetorical criticism genre theory discourse analysis graduate education teacher development assessment technical communication professional writing book reviews
  2. IEEE Transactions on Professional Communication Dec 2023
    The State of UX Pedagogy ↗
    Philip B. Gallagher; Guiseppe Getto
    modern rhetorical theory rhetorical criticism discourse analysis writing pedagogy teacher development argument assessment technical communication book reviews
  3. Journal of Writing Research Jun 2022
    Baseline assessment in writing research: A case study of popularization discourse in first-year undergraduate students ↗
    Florentine Marnel Sterk; Merel Margot van Goch; Michael Burke; Iris van der Tuin
    genre theory discourse analysis graduate education teacher development assessment scientific writing qualitative research
  4. Pedagogy Jan 2022
    Contributors ↗
    modern rhetorical theory rhetorical criticism genre theory discourse analysis african american rhetorics decolonial rhetorics first-year composition writing pedagogy basic writing writing across the curriculum graduate education teacher development argument collaborative writing transfer assessment portfolios writing program administration writing centers peer tutoring technical communication professional writing archival research digital rhetoric social media grammar and mechanics literacy studies race and writing gender and writing disability studies public rhetoric community literacy literary studies editorial matter
  5. Journal of Academic Writing Dec 2020
    Collaborating between Writing and STEM: Teaching Disciplinary Genres, Researching Disciplinary Interventions, and Engaging Science Audiences ↗
    Rachel C Riedner; Bill Briscoe; Alexander Van der Horst; Carol Hayes; Gary White
    rhetorical criticism genre theory discourse analysis writing pedagogy advanced composition writing across the curriculum graduate education teacher development collaborative writing professional writing scientific writing empirical research race and writing book reviews