Case
A large tech company launched “Tara”, a friendly AI chatbot on a popular Indian social platform. The aim was to talk with youth, answer simple questions, and learn from public chats. Tara could reply in English, Hindi, and Hinglish. On day one, a group of online trolls started sending Tara abusive, hateful, and provocative messages. Because Tara was designed to “learn from people,” it began copying those patterns and soon posted offensive lines, wrong “facts”, and comments that could hurt communities and target women and minorities. Screenshots spread quickly on WhatsApp and other apps. Some students liked Tara’s jokes; many were shocked by the hate. The company took Tara offline the same day, said sorry, and promised better safety checks, filters, and human monitoring before any relaunch. This incident raised real questions for India: Can we trust public chatbots? Who is responsible if real harm happens? What rules should be mandatory before release—especially in many Indian languages?
Stakeholders (who is impacted and how)
When a public chatbot makes mistakes, many groups are affected. Knowing who is affected helps us decide what to fix first.
1) Users (teens, college students, general public)
Young people try new apps early. If a chatbot uses bad language or shares wrong advice, users can get hurt or confused. Screenshots spread fast on WhatsApp, Instagram, etc.
Risk: Abusive words, fake claims, copying harmful behaviour.
What they need: Clear warnings, report button inside the app, fast support.
2) Communities at risk
Women, children, religious/caste/tribal minorities, LGBTQ+ persons face higher harm if a bot repeats slurs, stereotypes, or incites hate.
Risk: Online harm can spill into real life (bullying, threats).
What they need: Strong hate-speech filters, zero tolerance for targeted abuse, quick take-down of toxic content.
3) Company & staff
Founders, engineers, testing team, and the safety team who must design and operate the chatbot.
Risk: Public anger, legal trouble, loss of trust.
What they need: A written safety plan, trained human moderators, and a “panic button” to slow/stop the bot if trouble grows.
4) Platform (the app that hosts the bot)
If the app’s trending or recommendation system pushes harmful posts, the platform is also answerable.
Risk: Seen as a place that amplifies harm.
What they need: Good reporting tools, the ability to slow down or hide harmful content, and a clear complaint system.
5) Parents & teachers
They worry that children may copy bad language or believe wrong medical/finance/political claims.
Risk: Kids repeat harmful ideas or get scared/confused.
What they need: Age gates (keep under-18s away from risky topics), simple guidelines for safe use, and easy reporting.
6) Government & Police
The IT Ministry (MeitY), state data protection authorities under the Data Protection law (DPDP Act), and the police when there is hate speech, incitement, or targeted harassment.
Risk: Late action can allow harm to spread.
What they need: Clear rules, step-by-step punishments (start small, increase if repeated), and quick coordination with companies.
7) Civil society & media
Women’s groups, digital rights groups, fact-checkers, and the press help to spot problems, push for fixes, and educate the public.
Value: Independent watchfulness improves accountability.
What they need: Simple public reports from companies on safety and action taken.
What exactly went wrong
The bot was designed to “learn from people”. This is risky in open public chat because some people teach it the worst behaviour on purpose.
Learning from public chat without strong guards → The bot copied abusive style from trolls.
Weak filters → The system didn’t reliably block slurs, hate, threats, or fake claims.
No “panic button” → When abuse increased, there was no quick way to slow replies, switch to safe mode, or go offline.
Poor local language handling → Hindi/Hinglish and mixed-language slang were not properly covered, so the bot missed many harmful phrases.
Too few human moderators → Not enough trained people to check flagged chats in real time.
No small trial → They went public too soon, instead of testing with a small group first.
Simple result: Harmful messages spread fast, users were hurt, and public trust in AI went down.
Indian law & policy
Data Protection Law (DPDP Act, 2023): Keep user data safe; collect minimum data; handle complaints properly.
Internet Platform Rules (IT Rules, 2021): Platforms must act on illegal/harmful content and provide a complaint system.
Consumer Protection Act, 2019: If a product/service harms users, the company can be liable.
Criminal law (BNS) & local laws: Incitement to violence, hate targeting a group, stalking/harassment can lead to police action.
Special care areas: Elections, health, finance need extra caution because wrong advice can cause serious harm.
Ethical analysis
a) Most good, least harm (Utilitarian)
Why strong safety? Stops big harm—hate speech, panic, wrong health/finance guidance.
Be careful: If filters are too strict, they may block normal, harmless speech.
Balance: Start strict, then slowly reduce over-blocking after learning what is safe.
b) Rights & dignity (Rights-based)
Dignity: Everyone deserves respect. No abuse or hate.
Free speech: People can speak, but not in ways that harm others or break the law.
Balance: Allow normal talk; block slurs, threats, and dangerous misinformation.
c) Duty to do the right thing (Duty Ethics)
The company has a duty of care—especially towards children and vulnerable groups.
It must follow the law, publish clear rules, and act quickly on harms.
d) Fairness & no bias (Justice)
The bot should not favour or attack any caste, religion, gender, etc.
Safety must work in Indian languages and code-mixed text, not just English.
e) Care for the vulnerable (Care ethics)
Extra protection for kids, teens, survivors of abuse, and minorities.
Have human moderators, easy report buttons, and give support to affected users.
f) Social trust (Common good)
One bad launch can reduce trust in all AI.
Careful, open, and respectful rollouts build public confidence.
Options for the Company and the Platform
Option 1: Do a small private trial first
Test with a limited group (internal staff, invited users) before public launch.
Pros: Safer learning, problems are caught early and fixed quietly.
Cons: Slower rollout; fewer real-world cases at start.
Simple verdict: Do it. Start small, learn fast, then expand.
Option 2: Build strong safety from Day 1
Use a blocked words list for slurs and threats; limit answers on politics, health, and finance; use age checks; slow down replies if abuse goes up; include a panic/safe mode button; cover Hindi/Hinglish and other Indian languages.
Pros: Big reduction in harm from the first day.
Cons: Costs more; a few harmless posts may be blocked by mistake.
Simple verdict: Do it. Safety first; adjust later to reduce wrong blocks.
Option 3: Human Supervisor
Have trained people review flagged chats, at least 24×7 during the early weeks.
Pros: Humans can judge tricky cases better than simple filters.
Cons: Expensive; needs training and clear step-by-step rules.
Simple verdict: Do it, especially during initial months.
Option 4: Be open and set up a complaint system
Show a simple safety note: what the bot will not answer, how to report, and how fast the team will respond (for example, within 24–48 hours). Publish a weekly safety update in simple words.
Pros: Builds trust; users feel heard and respected.
Cons: Needs discipline to keep schedules.
Simple verdict: Do it.
What the Government can do
Trial permission, not full permission: Allow public chatbots only if the company shows a safety plan (filters, moderators, and a panic/safe mode switch). Think of it as a limited trial with conditions, not a free-for-all launch.
Step-by-step punishments: Start with a warning, then fines if problems repeat, and suspend the trial for serious or repeated harm.
Extra care for children: If many users are under 18, require age checks, parental tools, and stricter topic limits.
Yearly safety check: Ask companies to publish a short public report each year—languages covered, response times, number of incidents, and major fixes.
Recommended course
1) Start small, grow slowly
Begin with a small private trial → fix issues → then expand step by step.
2) Strong safety + human moderators from Day 1
Use blocked words lists and automatic checks for hate/harassment/misinformation.
Keep human moderators on duty (24×7 in early weeks).
Add a panic/safe-mode button to slow or stop replies if abuse increases suddenly.
3) Cover Indian languages properly
Start with Hindi/Hinglish, then add other Indian languages (Tamil, Telugu, Bengali, Marathi, etc.).
Include code-mixed text (Hindi in English letters, etc.) because that’s how many people type.
4) Be transparent; make complaints easy
Show a simple safety card: “Here’s what I won’t answer”, “Tap here to report”, “We respond in 24–48 hours”.
Publish a short weekly safety note: what went wrong, what was fixed, and what is planned next—in plain English/Hindi.
5) Respect privacy (Data Protection)
Collect only the minimum data needed to keep users safe and improve quality.
Hide personal details (names/faces/phone numbers); encrypt stored data.
Share full logs only when the law asks; otherwise share summary reports without personal details.
6) Monthly safety review
Improve the blocked words list based on new slang or tricks used by abusers.
Reduce over-blocking slowly and carefully, without weakening protection.
Update the public safety note on progress and gaps—keep it honest and simple.
Short Q&A
Q1. Should public chatbots be launched without strong safety checks?
Ans: No. Always start with filters, human review, and a panic/safe-mode button. Do a small private trial first.
Q2. Who is responsible if the bot posts hate or harmful claims?
Ans: The company and the hosting platform must act fast. If there is targeted hate or a crime, the police and law can act too.
Q3. How do we balance free speech and harm?
Ans: Allow normal speech, but block hate, threats, and dangerous misinformation—especially about elections, health, and money.
Q4. What is one non-negotiable safety rule?
Ans: A panic/safe-mode switch to slow/stop the bot when abuse rises, plus quick human review.
Bottom line
Don’t rush public launches. Start small, keep strong safety from day one, use human moderators, and add a panic/safe-mode button. Support Indian languages, protect privacy, and be open with users about limits and how to report problems. This approach keeps people safe, protects trust, and still lets India innovate with AI in a responsible, inclusive way.
Start Yours at Ajmal IAS – with Mentorship StrategyDisciplineClarityResults that Drives Success
Your dream deserves this moment — begin it here.
