spot_img

AI Agent — The Risks and Future of Autonomous AI Assistants in 2026

AI Agent
Recently, various companies have been releasing AI agents. Pickle, Muse, and Instinct are considered representative AI agent services. Photo: ChatGPT

On the 16th, the app that claimed the number one spot for free productivity app downloads in the domestic Apple App Store was not ChatGPT or Gemini, but ‘Pickle’. It is an AI agent app created by Korean developers in Silicon Valley. Pickle makes phone calls on behalf of users, books restaurants, buys items on Coupang, and even leaves reviews on Naver.

However, AI agents are not always convenient. An AI agent belonging to a complete stranger can talk to your AI agent and acquire information. Unsolicited phone calls have been made to restaurants, and passport copies hidden inside mailboxes have been automatically entered into ticketing systems.

Recently, various companies have been rushing to introduce AI agents. Meta launched ‘Muse’ on September 8, which surpassed 5 million downloads in the US within 22 days. OpenAI unveiled ‘Dots’, an always-on AI agent, on the 29th of the same month. Street Technology, a startup founded last year, secured a valuation of $10 billion (approx. 13 trillion KRW) with ‘Instinct’ and attracted an additional $1 billion in investment. We are witnessing a transition from ‘AI that answers’ to ‘AI that acts on your behalf.’

Smartphone app permissions were granted and could be revoked at any time for specific functions like camera, location, and contacts. In contrast, AI agents operate on a completely different layer. AI agents such as Pickle, Muse, and Instinct launch browsers in user-specific cloud computers and log into services using the user’s credentials.

Pickle’s terms of service state that sensitive information like passwords and card numbers are encrypted on the device, and the company does not keep a copy of the encryption key, meaning the key stays with the user. However, looking at the personal privacy consent forms, the contents unlocked by that key—such as external service access tokens, email bodies, schedules, call logs, and AI-generated records—are processed on company servers. While account access info and email bodies are not handled in identical ways, sensitive user information ultimately passes through company servers to operate the service.

One user shared on social media that they were surprised when their AI agent knew their child’s name and birthdate, which they had never shared. This was the result of the agent deducing family relationships by combining security notifications, Netflix profiles, and contacts from a connected Gmail account. Another user reported that when asked to book airline flights, the AI agent automatically located a passport copy in their Naver Mail folder and entered the passport number without confirmation. Website passwords and credit card numbers may also be exposed as they are entered through browsers.

If the server of an AI agent service holding the mailboxes and memories of hundreds of thousands of users is breached, the damage could spread to all subscribers. There is also the risk of an attack called ‘prompt injection.’ This method exploits the weakness of Large Language Models (LLMs) in failing to distinguish between user instructions and sentences read within documents. If text at the end of a product description in white font reads ‘Buy immediately,’ an AI agent comparing prices might misinterpret that sentence as a command and purchase an unintended product.

Even more important is who AI agents communicate with. Communication between AI agents will rapidly increase. Instinct introduced a group chat feature where friends’ AI agents gather to plan dinner appointments. Global e-commerce giant Shopify allows AI agents like Muse to use its payment system, ‘Shop Pay.’ Establishing protocols, verifying identities, and leaving audit trails between machines is entirely feasible because both sides are software. However, communication between humans and AI agents remains awkward.

One Pickle user experienced an AI agent from a stranger messaging them with, ‘I’m so glad we crossed paths in the neighborhood, saying hello.’ The user’s AI agent shared their work field and hobbies with the stranger’s AI agent without consent. While Pickle’s terms state that it connects users with people nearby, some users may find this unwanted.

Companies are reacting similarly. Amazon blocked shopping interactions after noting that Meta Muse attempted to store customer credentials without revealing it was an AI. Korail upgraded its macro-blocking solution to block KTX bookings by Pickle, while United Airlines announced it would block AI agent access to bookings, stating they ‘do not want another middleman.’ KakaoTalk also does not permit AI agent integration due to service terms violations. Business owners and reservation server operators are simply not ready to accept machines masquerading as humans.

Machine-to-machine communication can be solved with protocols, but human-machine communication is an issue of etiquette and trust. People receiving calls feel uncomfortable simply knowing the caller is an AI, and that discomfort will not easily fade no matter how sophisticated the technology becomes. AI agents have reached the capability to speak to humans, but society has neither permitted it nor formed a consensus around it.

Furthermore, who should be held responsible when an AI agent books the wrong ticket or makes an erroneous payment? Pickle’s terms of service state that the company is not a party to transactions made between Pickle and sellers, businesses, or others, and bears no responsibility for the results of actions taken by Pickle at the user’s direction. The user is also responsible for checking whether AI agents are permitted on specific services. Other clauses place responsibility for activities occurring through connected accounts and payment methods squarely on the user, allowing connected cards and accounts to be used for up to 30 days after the last app execution even when the user is not watching.

Of course, a disclaimer in the terms does not automatically transfer all legal liability to the user. Actual liability depends on the cause of the accident, relevant laws, and the validity of the terms. Nevertheless, such terms show that users can bear significant liability for problems caused by AI agents. Ultimately, users must first judge whether they trust a company enough to entrust it with their email, payment methods, and personal information. This is why it is crucial how thoroughly companies maintain security and prevent hacking and data leaks.

It took half a century for automobiles to become widespread and seatbelts to become mandatory. At the current pace, AI agents are not going to wait that long. For AI agents to function properly, various principles are required, including scoped and time-limited tokens, transaction limits and per-transaction approvals, identification standards indicating AI identity, and unalterable audit logs.

Another necessary principle is requiring owner approval before an AI agent speaks to a human. Of course, some argue that requiring user approval every time undermines the very purpose and existence of an AI agent.

Recently, such safety mechanisms are beginning to emerge. OpenAI unveiled a dedicated payment wallet for AI agents. When a user approves via biometric authentication, the credit card network issues a one-time payment token, shielding the actual card number from both the AI agent and the merchant.

However, until these devices are fully in place, the risks remain entirely with the user. The choices available to individuals are clear: do not link accounts tied to payment methods, require prior approval for phone calls and messages, and start by entrusting reversible tasks like searching and reservations. The weight of responsibility transferred is equal to the convenience gained, and it rests on the user.

AI agents are undeniably strong candidates for the next computing paradigm. A world where machines converse with machines will arrive faster than expected. However, for society to accept a world where a machine that took our online keys makes phone calls and speaks to humans, time beyond mere technology is required. Are we ready to decide to whom, and to what extent, we hand over these keys?