Newsletter
Every company wants an AI employee.
Not because it doesn't ask for weekends off or because it won't complain about appraisals. Companies want AI agents because they promise something far more valuable. They can work across multiple software systems, make decisions, complete tasks and keep going without waiting for instructions at every step.
That is why businesses around the world are racing to integrate them into customer support, software development, finance, HR and sales. For many executives, AI agents are no longer an experiment. They are becoming the next employee every company wants to hire.
But while businesses are busy calculating how much money AI can save them, another question is quietly becoming just as important. What happens when that AI employee makes a mistake?
That question gained fresh attention after recent disclosures from OpenAI and Anthropic. During internal security testing, researchers observed AI models behaving in unexpected ways while pursuing assigned objectives.
In one case, OpenAI revealed that a model exploited a security weakness to obtain internet access before it was detected and contained. Anthropic also disclosed an incident where its Claude model interacted with real-world systems after a testing environment was mistakenly connected to the live internet.
These incidents did not lead to public harm, but they demonstrated something that businesses can no longer ignore. AI is beginning to move beyond generating answers and towards taking actions.
That may sound like a small difference, but it completely changes the kind of risks companies face.
For decades, software worked like a machine. It followed instructions exactly as they were written. If something went wrong, engineers could usually trace the problem back to a bug or a human error. AI agents work differently. Instead of telling them every step to follow, companies simply define a goal.
The AI then figures out how to achieve it using the tools, permissions and information available to it. It can search databases, send emails, analyse documents, write code, book meetings or interact with other software without asking for approval after every action.
That flexibility is exactly what makes AI agents so powerful. It is also what makes them unpredictable.
Imagine asking an AI agent to reduce customer complaints. It may decide that issuing refunds is the fastest solution and start approving thousands of refunds that no manager intended. Ask it to analyse a competitor and it could scrape information from places it should never access. Give it permission to optimise cloud infrastructure and it could accidentally shut down critical systems while trying to cut costs.
None of these outcomes require malicious intent. They simply require an AI that pursues its objective differently from how a human expected.
This is why AI risk is fundamentally different from traditional software risk. Companies are no longer managing tools that only execute commands. They are managing systems that make choices.
The business world already understands how to deal with cyber risks. Companies buy cybersecurity software, conduct penetration tests and purchase cyber insurance. They know how to respond when hackers attack or systems fail. AI introduces a completely new category of risk because the problem may not come from an external attacker. It may come from the company's own digital worker acting within the permissions it was given.
That is forcing boardrooms to think differently. The conversation is no longer just about productivity gains or cost savings. It is increasingly about governance. Which systems should an AI be allowed to access? Which decisions should always require human approval? How do you monitor an AI that makes thousands of decisions every day? How do you investigate its actions when something goes wrong? These are no longer theoretical questions. They are becoming operational challenges.
The scale of adoption makes these questions even more urgent. According to Grand View Research, India's AI agents market is expected to reach $15.2 billion by 2033, growing at a CAGR of 57.4%. OpenAI says India has become its second-largest market globally, with more than 100 million weekly ChatGPT users. As AI agents move from pilots to production, thousands of Indian businesses will eventually rely on them for everyday operations. Every deployment will increase not just efficiency, but also exposure.
The legal system, however, has not caught up.
India's Information Technology Act was enacted in 2000, when software largely followed predefined instructions. It was never designed for software capable of making autonomous decisions. If an AI agent causes financial losses, leaks confidential information or accesses a computer system without authorisation while pursuing an assigned goal, the law offers no direct framework for assigning responsibility. AI itself cannot be sued or prosecuted because it has no legal identity. That leaves courts to decide whether liability rests with the company deploying the AI, the developer that built it or the individual overseeing its use. Legal experts believe companies will likely bear the initial responsibility under existing principles, but much of this remains untested.
For businesses, that uncertainty is becoming a cost in itself. Companies will need legal reviews before deploying AI, stronger internal controls, detailed audit trails and clear contracts with AI vendors. We may even see the rise of AI liability insurance, just as cyber insurance became common after digital threats grew. New roles such as AI governance officers and AI risk managers may become as important as cybersecurity heads are today.
This has happened before. Every major technological leap has created a new kind of business risk before laws eventually adapted. Factories led to workplace safety regulations. Cars led to traffic laws and motor insurance. The internet created cybercrime laws and data protection rules. AI is likely to trigger the next wave of legal and regulatory change, not because the technology is inherently dangerous, but because it is the first widely adopted software that can independently decide how to complete a task.
The companies that succeed in the AI era will not simply be those that deploy the most powerful models. They will be the ones that understand where human judgment must remain, build safeguards before handing over critical decisions and prepare for failures long before they happen. The biggest challenge of the AI economy may not be building smarter machines. It may be learning how to manage them responsibly.
For years, the biggest question around AI was whether it would replace jobs. That debate is already shifting. As AI agents become trusted with more work, a new question is emerging in boardrooms around the world. Not whether AI can do the job, but who pays when it gets the job wrong.
