Review of Detecting Text generated by ChatGPT Using Machine and Deep-Learning Models: A Tools and Methods Analysis
DOI:
https://doi.org/10.24237/djes.2025.18102Keywords:
Text Detection, Machine learning, Deep Learning, NLP, Transform Coding, ChatGPTAbstract
Recently, generative models, such as ChatGPT, have gained considerable attention because of their capacity to generate text almost identical to that produced by humans. However, ChatGPT raises several concerns, particularly regarding the integrity of academic work, the protection of personal information and security, the reliance on artificial intelligence (AI), the evaluation of learning, and the precision of information. Distinguishing between writing generated by machines and text that humans wrote is one of the most critical issues at present. The purpose of this literature review is to provide a comprehensive, up-to-date analysis of the most recent methods for identifying text that ChatGPT created. It examines more than 60 academic papers, especially research articles published after the model’s release in 2022, and analyzes state-of-the-art machine learning, deep learning, and hybrid approaches for detecting AI-generated text. The review categorizes detection methods into statistical models, transformer-based architectures, perplexity-based techniques, and human-assisted evaluation. The findings indicate that deep learning models, particularly the Robustly Optimized BERT Pretraining Approach (RoBERTa) and Cross-lingual Language Model with RoBERTa Architecture, have high detection accuracy (up to 99%), whereas traditional statistical methods exhibit limitations in distinguishing complex AI-generated content. This work recommends the use of machine and deep learning techniques and human reviewers in ongoing efforts to distinguish between AI-generated and human-written text. However, given the increasing sophistication and complexity of models, such as ChatGPT, detection techniques have to be continuously improved and innovated to ensure reliability and maintain the integrity of content across various sectors.
Downloads
References
[1] A. Aliwy, A. Abbas, and A. Alkhayyat, "NERWS: Towards improving information retrieval of digital library management system using named entity recognition and word sense," Big Data and Cognitive Computing, vol. 5, no. 4, p. 59, 2021, doi: 10.3390/bdcc5040059.
[2] J. Qadir, "Engineering education in the era of ChatGPT: Promise and pitfalls of generative AI for education," in 2023 IEEE Global Engineering Education Conference (EDUCON), 2023: IEEE, pp. 1-9, doi: 10.1109/educon54358.2023.10125121.
[3] A. Vaswani, "Attention is all you need," in Advances in Neural Information Processing Systems, Long Beach, CA, USA., 2017.
[4] A. J. Yousif and M. H. Al-Jammas, "Real-time Arabic Video Captioning Using CNN and Transformer Networks Based on Parallel Implementation," Diyala Journal of Engineering Sciences vol. 17, no. 1, pp. 84-93, 2024, doi: 10.24237/djes.xxxx.13301
[5] D. Bahdanau, "Neural machine translation by jointly learning to align and translate," Preprint 2014, doi: 10.48550/arXiv.1409.0473.
[6] M. H. Al-Tai, B. M. Nema, and A. Al-Sherbaz, "Deep learning for fake news detection: Literature review," Al-Mustansiriyah Journal of Science, vol. 34, no. 2, pp. 70-81, 2023, doi: http://doi.org/10.23851/mjs.v34i2.1292.
[7] D. Jacob, C. Ming-Wei, L. Kenton, and T. Kristina, "Bert: Pre-training of deep bidirectional transformers for language understanding," in Proceedings of naacL-HLT, 2019, vol. 1: Minneapolis, Minnesota, p. 2.
[8] T. Brown et al., "Language models are few-shot learners," Advances in neural information processing systems, vol. 33, pp. 1877-1901, 2020.
[9] E. Strubell, A. Ganesh, and A. McCallum, "Energy and policy considerations for modern deep learning research," in Proceedings of the AAAI conference on artificial intelligence, 2020, vol. 34, no. 09, pp. 13693-13696.
[10] J. Li, A. Dada, B. Puladi, J. Kleesiek, and J. Egger, "ChatGPT in healthcare: a taxonomy and systematic review," Computer Methods and Programs in Biomedicine, vol. 245, p. 108013, 2024.
[11] M. Imran and N. Almusharraf, "Analyzing the role of ChatGPT as a writing assistant at higher education level: A systematic review of the literature," Contemporary Educational Technology, vol. 15, no. 4, p. ep464, 2023.
[12] L. Uzun, "ChatGPT and academic integrity concerns: Detecting artificial intelligence generated content," Language Education and Technology, vol. 3, no. 1, 2023.
[13] M. T. Younis, N. M. Hussien, Y. M. Mohialden, K. Raisian, P. Singh, and K. Joshi, "Enhancement of ChatGPT using API wrappers techniques," Al-Mustansiriyah Journal of Science, vol. 34, no. 2, pp. 82-86, 2023.
[14] J. Huang and M. Tan, "The role of ChatGPT in scientific communication: writing better scientific review articles," American journal of cancer research, vol. 13, no. 4, p. 1148, 2023.
[15] Y. Ma et al., "AI vs. Human--Differentiation Analysis of Scientific Content Generation," arXiv preprint arXiv:2301.10416, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2301.10416.
[16] A. Pegoraro, K. Kumari, H. Fereidooni, and A.-R. Sadeghi, "To ChatGPT, or not to ChatGPT: That is the question!," arXiv preprint arXiv:2304.01487, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2304.01487.
[17] C. Chaka, "Detecting AI content in responses generated by ChatGPT, YouChat, and Chatsonic: The case of five AI content detection tools," Journal of Applied Learning and Teaching, vol. 6, no. 2, 2023, doi: https://doi.org/10.37074/jalt.2023.6.2.12.
[18] C. Chen, J. Fu, and L. Lyu, "A pathway towards responsible ai generated content," arXiv preprint arXiv:2303.01325, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2303.01325.
[19] Hackernoon, "AI-Generated vs. Human-Written Text: A Technical Analysis," 2023. [Online]. Available: https://hackernoon.com/ai-generated-vs-human-written-text-technical-analysis.
[20] P. S. University. "Q&A: Increasing Difficulty Detecting AI vs Human Writing." https://www.psu.edu/news/information-sciences-and-technology/story/qa-increasing-difficulty-detecting-ai-versus-human (accessed 10-Feb-2025.
[21] Originality.ai. "How to Identify AI-Generated Text." https://originality.ai/blog/identify-ai-generated-text (accessed 10-Feb-2025.
[22] Reddit. "How Do You Distinguish Whether an Article Is AI-Generated or Human-Written?" https://www.reddit.com/r/Futurology/comments/zn7qb9/how_do_you_distinguish_whether_an_article_is (accessed 10-Feb-2025.
[23] G. Jawahar, M. Abdul-Mageed, and L. V. Lakshmanan, "Automatic detection of machine generated text: A critical survey," arXiv preprint arXiv:2011.01314, 2020. [Online]. Available: https://doi.org/10.48550/arXiv.2011.01314.
[24] E. N. Crothers, N. Japkowicz, and H. L. Viktor, "Machine-generated text: A comprehensive survey of threat models and detection methods," IEEE Access, vol. 11, pp. 70977-71002, 2023, doi: 10.1109/ACCESS.2023.3294090.
[25] C. Vasilatos, M. Alam, T. Rahwan, Y. Zaki, and M. Maniatakos, "Howkgpt: Investigating the detection of chatgpt-generated university student homework through context-aware perplexity analysis," arXiv preprint arXiv:2305.18226, 2023.
[26] B. K. Al-Windi, A. H. Abbas, and M. S. Mahmood, "Using Texture Analyses and Statistical Classification for Detection Plant Leaf Diseases," Al-Mustansiriyah Journal of Science, vol. 32, no. 5, pp. 1-4, 2021, doi: http://doi.org/10.23851/mjs.v32i5.1115.
[27] H. A. Alatabi and A. R. Abbas, "Sentiment analysis in social media using machine learning techniques," Iraqi Journal of Science, vol. 61, no. 1, pp. 193-201, 2020, doi: 10.24996/ijs.2020.61.1.22.
[28] R. Shijaku and E. Canhasi, "ChatGPT generated text detection," Publisher: Unpublished, 2023, doi: 10.13140/RG.2.2.21317.52960.
[29] T. Chen and C. Guestrin, "Xgboost: A scalable tree boosting system," in Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, 2016, pp. 785-794, doi: http://dx.doi.org/10.1145/2939672.2939785.
[30] A. J. Yousif and M. H. Al-Jammas, "A Lightweight Visual Understanding System for Enhanced Assistance to the Visually Impaired Using an Embedded Platform," Diyala Journal of Engineering Sciences, pp. 146-162, 2024, doi: https://djes.info/index.php/djes/article/view/1377.
[31] Y. Liu, "Roberta: A robustly optimized bert pretraining approach," arXiv preprint arXiv:1907.11692, 2019.
[32] I. Katib, F. Y. Assiri, H. A. Abdushkour, D. Hamed, and M. Ragab, "Differentiating chat generative pretrained transformer from humans: detecting ChatGPT-generated text and human text using machine learning," Mathematics, vol. 11, no. 15, p. 3400, 2023, doi: https://doi.org/10.3390/math11153400.
[33] H. Alshammari, A. El-Sayed, and K. Elleithy, "Ai-generated text detector for arabic language using encoder-based transformer architecture," Big Data and Cognitive Computing, vol. 8, no. 3, p. 32, 2024, doi: https://doi.org/10.3390/bdcc8030032.
[34] M. M. D. Oghaz, K. Dhame, G. Singaram, and L. B. Saheer, "Detection and Classification of ChatGPT Generated Contents Using Deep Transformer Models," Authorea Preprints, vol. 11, 2023, doi: 10.22541/au.167702907.35890747.
[35] N. M. Tien and C. Labbé, "Detecting automatically generated sentences with grammatical structure similarity," Scientometrics, vol. 116, no. 2, pp. 1247-1271, 2018.
[36] C. Labbé, D. Labbé, and F. Portet, "Detection of computer-generated papers in scientific literature," Creativity and universality in language, pp. 123-141, 2016.
[37] D. I. Adelani, H. Mai, F. Fang, H. H. Nguyen, J. Yamagishi, and I. Echizen, "Generating sentiment-preserving fake online reviews using neural language models and their human-and machine-based detection," in Advanced information networking and applications: Proceedings of the 34th international conference on advanced information networking and applications (AINA-2020), 2020: Springer, pp. 1341-1354.
[38] H. Stiff and F. Johansson, "Detecting computer-generated disinformation," International Journal of Data Science and Analytics, vol. 13, no. 4, pp. 363-383, 2022.
[39] D. Beresneva, "Computer-generated text detection using machine learning: A systematic review," in Natural Language Processing and Information Systems: 21st International Conference on Applications of Natural Language to Information Systems, NLDB 2016, Salford, UK, June 22-24, 2016, Proceedings 21, 2016: Springer, pp. 421-426.
[40] W. Antoun, V. Mouilleron, B. Sagot, and D. Seddah, "Towards a Robust Detection of Language Model Generated Text: Is ChatGPT that Easy to Detect?," arXiv preprint arXiv:2306.05871, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2306.05871.
[41] K. Clark, "Electra: Pre-training text encoders as discriminators rather than generators," arXiv preprint arXiv:2003.10555, 2020.
[42] S. Gehrmann, H. Strobelt, and A. M. Rush, "Gltr: Statistical detection and visualization of generated text," in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, Florence, Italy, 2019, vol. , pp. 111–116.
[43] B. Guo et al., "How close is chatgpt to human experts? comparison corpus, evaluation, and detection," arXiv preprint arXiv:2301.07597, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2301.07597.
[44] H.-Q. Nguyen-Son and I. Echizen, "Detecting computer-generated text using fluency and noise features," in Computational Linguistics: 15th International Conference of the Pacific Association for Computational Linguistics, PACLING 2017, Yangon, Myanmar, August 16–18, 2017, Revised Selected Papers 15, 2018: Springer, pp. 288-300.
[45] H.-Q. Nguyen-Son, N.-D. T. Tieu, H. H. Nguyen, J. Yamagishi, and I. E. Zen, "Identifying computer-generated text using statistical analysis," in 2017 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), 2017: IEEE, pp. 1504-1511, doi: https://doi.org/10.1109/APSIPA.2017.8282270.
[46] M. Dhaini, W. Poelman, and E. Erdogan, "Detecting chatgpt: A survey of the state of detecting chatgpt-generated text," arXiv preprint arXiv:2309.07689, 2023, doi: https://doi.org/10.48550/arXiv.2309.07689.
[47] R. Safi and A. J. Naini, "The Work of Students and ChatGPT Compared: Using Machine Learning to Detect and Characterize AI-Generated Text," in Twenty-ninth Americas Conference on Information Systems, Panama, 2023.
[48] C. Chaka, "Reviewing the performance of AI detection tools in differentiating between AI-generated and human-written texts: A literature and integrative hybrid review," Journal of Applied Learning and Teaching, vol. 7, no. 1, 2024, doi: https://doi.org/10.37074/jalt.2024.7.1.14.
[49] G. P. Georgiou, "Differentiating between human-written and AI-generated texts using linguistic features automatically extracted from an online computational tool," 2024.
[50] C. Opara, "StyloAI: Distinguishing AI-generated content with stylometric analysis," in International conference on artificial intelligence in education, 2024: Springer, pp. 105-114, doi: https://doi.org/10.1007/978-3-031-64312-5_13.
[51] T. T. Nguyen, A. Hatua, and A. H. Sung, "How to Detect AI-Generated Texts?," in 2023 IEEE 14th Annual Ubiquitous Computing, Electronics & Mobile Communication Conference (UEMCON), 2023: IEEE, pp. 0464-0471.
[52] I. J. Goodfellow, J. Shlens, and C. Szegedy, "Explaining and harnessing adversarial examples," arXiv preprint arXiv:1412.6572, 2014.
[53] L. Yang, F. Jiang, and H. Li, "Is chatgpt involved in texts? measure the polish ratio to detect chatgpt-generated text," APSIPA Transactions on Signal and Information Processing, vol. 13, no. 2, 2023, doi: 10.1561/116.00000250.
[54] H. M. Fadhil, Z. O. Dawood, and A. Al Mhdawi, "Enhancing Intrusion Detection Systems Using Metaheuristic Algorithms," Diyala Journal of Engineering Sciences, pp. 15-31, 2024.
[55] X. He, X. Shen, Z. Chen, M. Backes, and Y. Zhang, "Mgtbench: Benchmarking machine-generated text detection," in Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, 2024, pp. 2251-2265, doi: https://doi.org/10.1145/3658644.3670344.
[56] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, "Language models are unsupervised multitask learners," OpenAI blog, vol. 1, no. 8, p. 9, 2019.
[57] R. J. Kolaib and J. Waleed, "Crime Activity Detection in Surveillance Videos Based on Developed Deep Learning Approach," Diyala Journal of Engineering Sciences, pp. 98-114, 2024.
[58] B. Mann et al., "Language models are few-shot learners," arXiv preprint arXiv:2005.14165, vol. 1, 2020.
[59] Y. Xie, A. Rawal, Y. Cen, D. Zhao, S. K. Narang, and S. Sushmita, "MUGC: Machine Generated versus User Generated Content Detection," arXiv preprint arXiv:2403.19725, 2024.
[60] L. Dugan, D. Ippolito, A. Kirubarajan, S. Shi, and C. Callison-Burch, "Real or fake text?: Investigating human ability to detect boundaries between human-written and machine-generated text," in Proceedings of the AAAI Conference on Artificial Intelligence, 2023, vol. 37, no. 11, pp. 12763-12771.
[61] N. Islam, D. Sutradhar, H. Noor, J. Raya, M. Maisha, and D. Farid, "Distinguishing Human Generated Text from ChatGPT Generated Text Using Machine Learning. arXiv," arXiv preprint arXiv:2306.01761, 2023.
[62] Y. Wang et al., "M4: Multi-generator, multi-domain, and multi-lingual black-box machine-generated text detection," arXiv preprint arXiv:2305.14902, 2023.
[63] C. A. Gao et al., "Comparing scientific abstracts generated by ChatGPT to real abstracts with detectors and blinded human reviewers," NPJ Digital Medicine, vol. 6, no. 1, p. 75, 2023, doi: https://doi.org/10.1038/s41746-023-00819-6.
[64] Y. Zhang et al., "Enhancing Text Authenticity: A Novel Hybrid Approach for AI-Generated Text Detection," arXiv preprint arXiv:2406.06558, 2024.
[65] N. Prova, "Detecting AI Generated Text Based on NLP and Machine Learning Approaches," arXiv preprint arXiv:2404.10032, 2024.
[66] B. Alhijawi, R. Jarrar, A. AbuAlRub, and A. Bader, "Deep Learning Detection Method for Large Language Models-Generated Scientific Content," arXiv preprint arXiv:2403.00828, 2024.
[67] Y. Hui, "Using generative adversarial network to improve the accuracy of detecting AI-generated tweets," Scientific Reports, vol. 14, no. 1, p. 29322, 2024.
[68] H. Wang, J. Li, and Z. Li, "AI-Generated Text Detection and Classification Based on BERT Deep Learning Algorithm," arXiv preprint arXiv:2405.16422, 2024.
Downloads
Published
Issue
Section
License
Copyright (c) 2025 Shaymaa Dhyaa Aldeen, Thekra Abbas, Ayad Rodhan Abbas
This work is licensed under a Creative Commons Attribution 4.0 International License.
