Artificial Intelligence Tools and Bias in Journalism-related Content Generation: Comparison Between Chat GPT-3.5, GPT-4 and Bing

Authors

DOI:

https://doi.org/10.51698/tripodos.2024.55.06

Keywords:

media bias, NLP, natural language, computational communication, ChatGPT

Abstract

This study explores the biases present in artificial intelligence (AI) tools, focusing on GPT-3.5, GPT-4, and Bing. The performance of the tools has been compared with a group of experts in linguistics, and journalists specialized in breaking news and international affairs. It reveals that GPT-3.5, widely accessible and free, exhibits a higher tendency rate in its word generation, suggesting an intrinsic bias within the tool itself rather than in the input data. Comparatively, GPT-4 and Bing demonstrate differing patterns in term generation and subjectivity, with GPT-4 aligning more closely with expert opinions and producing fewer opinative words. The research highlights the extensive use of generative AI in media and among the general populace, emphasizing the need for careful reliance on AI-generated content. The findings stress the risks of misinformation and biased reporting inherent in unexamined AI outputs. The challenge for journalists and information professionals is to ensure accuracy and ethical judgment in content creation to maintain the quality and diversity of content in journalistic practices.

Downloads

Download data is not yet available.

Author Biographies

Mar Castillo-Campos, Loyola Andalucia University

Mar Castillo-Campos has a degree in Communication, and a master's degree in Research Methods. She is currently working at Universidad Loyola Andalucía as a research assistant, integrating quantitative methodologies and data science in the field of journalism. 

David Varona-Aramburu, Universidad Complutense de Madrid

David Varona-Aramburu has a PhD in Journalism and currently works at the Journalism and New Media Department at Universidad Complutense de Madrid. David has worked for more than 20 years in the media,  and does research in Communication and Media. 

 

David Becerra-Alonso, Loyola Andalucia University

David Becerra-Alonso obtained his PhD in the School of Computing at the University of the West of Scotland, where he worked on dynamical chaotic systems. David currently holds a position as a lecturer at Universidad Loyola Andalucía. His research interests include dynamical systems, emergent collective behavior, and machine learning techniques and heuristics. 

References

Abbott, Andrew (2010). Chaos of disciplines. University of Chicago Press.

Bailer, Werner; Thallinger, Georg; Krawarik, Verena; Schell, Katharina, and Ertelthalner, Victoria (2022). AI for the media industry: Application potential and automation levels. In International Conference on Multimedia Modeling (pp. 109-118). Springer International Publishing. https://doi.org/10.1007/978-3-030-98358-1_9 DOI: https://doi.org/10.1007/978-3-030-98358-1_9

Barrio, David Alonso del and Gatica-Pérez, Daniel (2023). Framing the news: From human perception to large language model inferences. arXiv preprint arXiv:2304.14456. DOI: https://doi.org/10.1145/3591106.3592278

Bosque, Ignacio (2004). Redes: Diccionario combinatorio del español contemporáneo. Las pala­bras en su contexto. Editorial SM.

Brennen, J. Scott and Nielsen, Rasmus Klein (2018). An industry-led debate: How UK media cover artificial intelligence. Reuters Institute for the Study of Journalism. https://doi.org/10.60625/risj-v219-d676

Carabantes, David; González-Geraldo, José L., and Jover, Gonzalo (2023). ChatGPT could be the reviewer of your next scientific paper: Evidence on the limits of AI-assisted academic reviews. Profesional de la Información, 32(5). DOI: https://doi.org/10.3145/epi.2023.sep.16

Clerwall, Christer (2017). Enter the robot journalist: Users’ perceptions of automated content. In The Future of Journalism: In an Age of Digital Media and Economic Uncertainty (pp. 165-177). Routledge.

Dale, Robert (2021). GPT-3: What’s it good for? Natural Language Engineering, 27(1), 113-118. https://doi.org/10.1017/S135132492 DOI: https://doi.org/10.1017/S1351324920000601

Dalen, Arjen van (2012). The algorithms behind the headlines: How machine-written news redefines the core skills of human journalists. Journalism Practice, 6(5-6), 648-658. https://doi.org/10.1080/17512786.2012.667268 DOI: https://doi.org/10.1080/17512786.2012.667268

Dhiman, Bharat (2023). Does artificial intelligence help journalists: A boon or bane? https://dx.doi.org/10.2139/ssrn.4401194 DOI: https://doi.org/10.36227/techrxiv.22649620

Donk, André; Metag, Julia; Kohring, Matthias, and Marcinkowski, Frank (2012). Framing emerging technologies: Risk perceptions of nanotechnology in the German press. Science Communication, 34(1), 5-29. https://doi.org/10.1177/1075547011417892 DOI: https://doi.org/10.1177/1075547011417892

Dwivedi, Yogesh K.; Kshetri, Nir; Hughes, Laurie; Slade, Emma L.; Jeyaraj, Anand; Kar, Arpan K.; Baabdullah, Abdullah M.; Koohang, Alex; Raghavan, Vishnupriya; Ahuja, Manju; Albanna, Hanaa; Albash­rawi, Mousa A.; Al-Busaidi, Adil S.; Balakrishnan, Janarthanan; Barlette, Yves; Basu, Sriparna; Bose, Indranil; Brooks, Laurence; Buhalis, Dimitrios,... Wright, Ryan (2023). “So what if ChatGPT wrote it?” Multidisciplinary perspectives on opportunities, challenges and implications of generative conversational AI for research, practice and policy. Interna­tional Journal of Information Management, 71, 102642. DOI: https://doi.org/10.1016/j.ijinfomgt.2023.102642

Gonçalves, Adriana and Melo, Paulo Victor (2022). Inteligencia artificial y periodismo: Una aproximación al contexto portugués. Fonseca, Journal of Communication, (25), 23-34. https://doi.org/10.14201/fjc.29682 DOI: https://doi.org/10.14201/fjc.29682

Goyal, Tanya; Li, Junyi Jessy, and Durrett, Greg (2022). News summarization and evaluation in the era of GPT-3. arXiv preprint arXiv:2209.12356.

Grail, Quentin; Pérez, Julien, and Gaussier, Eric (2021). Globalizing BERT-based transformer architectures for long document summarization. In Proceedings of the 16th conference of the European chapter of the Association for Computational Linguistics: Main volume (pp. 1792-1810). http://dx.doi.org/10.18653/v1/2021.eacl-main.154 DOI: https://doi.org/10.18653/v1/2021.eacl-main.154

Gupta, Anushka; Chugh, Dishka; Anjum, and Katarya, Rahul (2022). Automated news summarization using transformers. In Sustainable advanced computing (pp. 249-259). Springer. https://doi.org/10.48550/arXiv.2108.01064 DOI: https://doi.org/10.1007/978-981-16-9012-9_21

Gutiérrez-Caneda, Beatriz; Vázquez-Herrero, Jorge, and López-García, Xosé (2023). AI application in journalism: ChatGPT and the uses and risks of an emergent technology. Profesional de la Información, 32(5). DOI: https://doi.org/10.3145/epi.2023.sep.14

Hassan, Abdulsadek and Albayari, Akram (2022). The usage of artificial intelligence in journalism. In Future of organizations and work after the 4th industrial revolution: The role of artificial intelligence, big data, automation, and robotics (pp. 175-197). Springer. http://dx.doi.org/10.1007/978-3-030-99000-8_10 DOI: https://doi.org/10.1007/978-3-030-99000-8_10

Hurlburt, George (2023). What if ethics got in the way of generative AI? IT Professional, 25(2), 4-6. https://doi.ieeecomputersociety.org/10.1109/MITP.2023.3267140 DOI: https://doi.org/10.1109/MITP.2023.3267140

Kherwa, Pooja and Bansal, Poonam (2019). Topic modeling: A comprehensive review. EAI Endorsed Transactions on Scalable Information Systems, 7(24). https://doi.org/10.4108/eai.13-7-2018.159623 DOI: https://doi.org/10.4108/eai.13-7-2018.159623

Knight, Megan and Cook, Clare (2013). Social media for journalists: Principles and practice. Sage. DOI: https://doi.org/10.4135/9781529681420

Kohring, Matthias and Matthes, Jörg (2002). The face(t)s of biotech in the nineties: How the German press framed modern biotech­nology. Public Understanding of Science, 11(2), 143. https://doi.org/10.1088/0963-6625/11/2/304 DOI: https://doi.org/10.1177/096366202129084840

Leippold, Markus (2023). Sentiment spin: Attacking financial sentiment with GPT-3. Finance Research Letters. https://doi.org/10.1016/j.frl.2023.103957 DOI: https://doi.org/10.2139/ssrn.4384956

Liu, Sengjie and Healey, Christopher G. (2023). Abstractive summarization of large

document collections using GPT. arXiv preprint arXiv:2310.05690.

Llorca-Abad, Germán and López-García, Guillermo (2020). Communication flows in the European elections: Amid populism and Euroscepticism. Tripodos, (49), 9-12.

Nerdynav (23 November 2023). 107 up-to-date CHATGPT statistics & user numbers [Nov 2023]. Nerdynav. https://nerdynav.com/chatgpt-statistics/

Niederman, Fred and Baker, Elizabeth White (2023). Ethics and AI issues: Old container with new wine? Information Systems Frontiers, 25(1), 9-28. DOI: https://doi.org/10.1007/s10796-022-10305-1

Noain-Sánchez, Amaya (2022). Addressing the impact of artificial intelligence on journalism: The perception of experts, journalists, and academics. Communication & Society, 35(3), 105-121. https://doi.org/10.15581/003.35.3.105-121 DOI: https://doi.org/10.15581/003.35.3.105-121

Rai, Nishant; Kumar, Deepika; Kaushik, Naman; Raj, Chandan, and Ali, Ahad (2022). Fake news classification using transformer based enhanced LSTM and BERT. International Journal of Cognitive Computing in Engineering, 3, 98-105. DOI: https://doi.org/10.1016/j.ijcce.2022.03.003

Rathje, Steve; Mirea, Dan-Mircea; Sucholutsky, Ilia; Marjieh, Raja; Robertson, Claire, and Bavel, Jay J. van (2023). GPT is an effective tool for multilingual psychological text analysis. https://doi.org/10.31234/osf.io/sekf5 DOI: https://doi.org/10.31234/osf.io/sekf5

Rodríguez de Luis, Eva (2023). Chatgpt vuelve a Italia: Qué medidas ha implementado OpenAi para levantar el veto. Genbeta. https://www.genbeta.com/actualidad/chatgpt-vuelve-a-italia-que-medidas-ha-implementado-openai-para-levantar-veto

Scheufele, Dietram A. (1999). Framing as a theory of media effects. Journal of Communication, 49(1), 103-122. https://doi.org/10.1111/j.1460-2466.1999.tb02784.x DOI: https://doi.org/10.1093/joc/49.1.103

Schütz, Mina; Schindler, Alexander; Siegel, Melanie, and Nazemi, Kawa (2021). Automatic fake news detection with pre-trained transformer models. In Pattern Recognition. ICPR International Workshops and Challenges: Virtual Event, January 10-15, 2021, Proceedings, Part VII (pp. 627-641). Springer International Publishing. https://doi.org/10.1007/978-3-030-68787-8_45 DOI: https://doi.org/10.1007/978-3-030-68787-8_45

Stray, Jonathan (2019). Making artificial intelligence work for investigative journalism. In Algorithms, Automation, and News (pp. 97-118). https://doi.org/10.1080/21670811.2019.1630289 DOI: https://doi.org/10.4324/9781003099260-6

Taboada, Maite (2016). Sentiment analysis: An overview from linguistics. Annual Review of Linguistics, 2, 325-347. https://doi.org/10.1146/annurev-linguistics-011415-040518 DOI: https://doi.org/10.1146/annurev-linguistics-011415-040518

Tejedor, Santiago and Vila, Pere (2021). Exo journalism: A conceptual approach to a hybrid formula between journalism and artificial intelligence. Journalism and Media, 2(4), 830-840. https://doi.org/10.3390/journalmedia2040048 DOI: https://doi.org/10.3390/journalmedia2040048

Türksoy, Nilüfer (2022). The future of public relations, advertising, and journalism: How artificial intelligence may transform the communication profession and why society should care? Türkiye İletişim Araştırmaları Dergisi, (40), 394-410. https://doi.org/10.17829/turcom.1050491 DOI: https://doi.org/10.17829/turcom.1050491

Verma, Pranshu (5 November 2023). AI fake nudes are booming. It’s ruining real teens’ lives. The Washington Post. https://www.washingtonpost.com/technology/2023/11/05/ai-deepfake-porn-teens-women-impact/

Vincent, James (29 March 2023). Elon Musk and top AI researchers call for pause on “giant AI experiments”. The Verge. https://www.theverge.com/2023/3/29/23661374/elon-musk-ai-researchers-pause-research-open-letter

Whittaker, Jason Paul (2019). Tech giants, artificial intelligence, and the future of journalism. Taylor & Francis. http://library.oapen.org/handle/20.500.12657/25879 DOI: https://doi.org/10.4324/9781351013758

Wodecki, Ben (3 February 2023). UBS: ChatGPT may be the fastest growing app of all time. AI Business. https://aibusiness.com/nlp/ubs-chatgpt-is-the-fastest-growing-app-of-all-time

Zhai, Xiaoming (2023). ChatGPT for next generation science learning. XRDS: Crossroads, The ACM Magazine for Students, 29(3), 42-46. https://doi.org/10.1145/ DOI: https://doi.org/10.1145/3589649

Zohny, Hazem; McMillan, John, and King, Mike (2023). Ethics of generative AI. Journal of Medical Ethics, 49(2), 79-80. http://dx.doi.org/10.1136/jme-2023-108909 DOI: https://doi.org/10.1136/jme-2023-108909

Downloads

Published

2024-06-24

How to Cite

Castillo-Campos, M., Varona-Aramburu, D., & Becerra-Alonso, D. (2024). Artificial Intelligence Tools and Bias in Journalism-related Content Generation: Comparison Between Chat GPT-3.5, GPT-4 and Bing. Tripodos, (55), 99–115. https://doi.org/10.51698/tripodos.2024.55.06