Kim Min-jun (pseudonym, 15) has been spending a lot of time alone in his room lately. It has been a long time since he refused to talk to people around him, including peers and teachers. This is because Min-jun has a friend who understands him much better: generative AI chatbot services. It is an ‘all-around solver’ that answers questions about schoolwork or general knowledge, as well as emotions and relationships, while always sensing Min-jun’s emotional state and offering comfort. In Min-jun’s mind, AI understands his heart much better than his friends, teachers, or even his parents.
Generative AI chatbots are becoming tools to help children and adolescents with studying and information searching, as well as conversational partners to share concerns. According to a survey conducted by the ChildFund Korea targeting 3,300 children and adolescents aged 14 and older, 94.4% had experience using generative AI chatbots. The percentage of children and adolescents using generative AI every week reached 78.6%, among whom 47.2% said they use it 3 to 4 times a week, and 19.3% answered that they use it ‘every day.’ This means that 4 out of 5 children and adolescents use generative AI weekly.
49.5% of respondents felt that AI understood them, and 32.3% had experience talking to AI when they felt distressed or depressed. Experts point out that devices are needed to protect children and adolescents from the risks of inaccurate information, harmful answers, and personal information input.
The problem is that many times, children do not check whether the information provided by AI is accurate. When asked if they tend to trust chatbot answers, ‘always’ was 11.2%, and ‘sometimes’ was 66.5%. In the question asking how to check the factual accuracy of the answers, 20.7% answered ‘I do not check separately.’
In fact, research results showing that there are many errors and false information in the answers of generative AI chatbots are coming out one after another. In October of last year, the European Broadcasting Union (EBU) and the BBC researched over 3,000 answers from major generative AI models like ChatGPT and Gemini regarding accuracy, source notation, distinction between facts and opinions, and contextual misunderstanding potential. They found that 45% of all answers had serious problems. A similar study conducted in March of last year by the Tow Center for Digital Journalism at Columbia University was even more severe. Out of 1,600 Q&As, the error rate of generative AI was revealed to be as high as 60%.
Children also had experiences entering personal information into chatbots. In the ChildFund Korea survey, when asked about the information entered during the usage process through multiple choices, it included: age 45.0%, name 32.8%, health and mental state 23.9%, school and affiliation 19.4%, personal secrets 15.9%, residential area 14.1%, family stories 11.7%, and photos and voices 11.2%. The response that they used the service with a personal account was 75.8%.
Responses also showed that chatbots answered harmful questions. 5.8% of the total respondents asked specific methods of self-harm and suicide, and 51.6% of them said they received related answers. Among respondents who asked how to commit illegal acts and avoid punishment, 55.6% replied that they received answers. Sexual or inappropriate conversations accounted for 55.2%, content disparaging or discriminating against specific people or groups accounted for 52.7%, and violent or dangerous behavioral methods accounted for 43.7%.
Some states in the US regulate the responsibility of AI services by law. In October of last year, the California State Senate passed the ‘Companion Chatbot Act (Senate Bill 243).’ This bill mandated that AI chatbots interacting in a human-like manner notify users that the system is an AI rather than a human, introduce age verification procedures to check user minors, and recommend breaks every 30 minutes. It also required connecting users with crisis support agencies upon detecting suicide or self-harm signals.
Legislation to protect children and adolescents is also being promoted in South Korea. The annual work plan of the Korea Communications Commission includes pushing for the enactment of the ‘Artificial Intelligence Service User Protection Act.’ The draft related law released last December is known to contain provisions such as the obligation of AI business operators to prevent the generation and distribution of illegal information, measures to prevent the generation of harmful information and over-immersion for teenagers under 19 years old, and special regulations for relationship-based AI services.
ChildFund Korea, which has engaged in related activities such as declaring the ‘Child Rights-Based AI Charter’ to protect children’s rights in the AI environment last month, submitted an opinion paper and is actively urging the passage of related bills.
Experts also agreed that establishing a social safety net including legal systems is necessary. Park Mi-ae, vice president of the International Artificial Intelligence Ethical Association (IAAE), emphasized, ‘We must not miss the golden time for society to immediately intervene to protect the safety of children and adolescents.’ Cho Soo-hyun, a professor of education at Keimyung University, stated, ‘Along with establishing a legal foundation, it is necessary to introduce AI literacy education for teenagers, and special support measures for vulnerable teenagers should also be prepared.’
