Users of ChatGPT, Gemini, and other AI chat systems should be worried about their privacy, according to a new study released this week. The report’s lead author, Jennifer King, did not mince words about the threat. The warning lands as millions rely on AI assistants for work, school, and health questions, raising urgent concerns about how personal data is collected, stored, and shared.
“Absolutely yes,” says Jennifer King, lead author of a new study about chatbot privacy concerns.
The study’s timing matters. Policymakers are weighing new rules, companies are updating data controls, and past incidents show how chat data can leak. The core question is simple: how much do AI companies need to know about users to make these tools work—and how much do they keep?
Why The Warning Matters
AI chat systems learn from huge amounts of text. Some services also use user prompts and replies to improve models. That can expose sensitive details, from medical worries to legal issues. Many users assume chats are private. They are not always private by default.
In 2023, Italy’s data protection authority temporarily blocked access to one major chatbot over age checks and transparency concerns. That suspension was lifted after changes, but it showed how quickly regulators can act when data risks appear.
Also in 2023, OpenAI reported a bug that briefly exposed some users’ chat titles and partial billing information. The company said the issue was fixed. The event still highlighted how small technical flaws can reveal personal details at scale.
How Chatbots Handle Your Data
Policies differ across providers. Some services retain prompts for a period and may use them to improve models. Others offer settings to limit training on user chats. Enterprise plans often have stricter protections, with data separated from model training.
Google’s Gemini includes controls that let users manage activity and storage tied to the app. OpenAI offers a setting to turn off chat history and model training for most consumer accounts, and it has separate data terms for enterprise and education plans. Microsoft touts “commercial data protection” for business users of Copilot, limiting data flow outside the tenant. Anthropic has described default retention periods and offers deletion requests for certain data.
Even with controls, there are trade-offs. Turning off data use can reduce personalization or lead to less context across sessions. Leaving it on can build detailed profiles that raise privacy risks if accessed or shared.
Industry And Regulator Responses
Companies say they anonymize or aggregate data to protect users. But privacy experts warn that re-identification can be easier than it seems when combined with enough details. The study led by King argues that clear notices and simple controls are still lacking for many users.
Regulators are moving as well. The European Union’s AI Act is set to introduce obligations on transparency and risk management. In the United States, the Federal Trade Commission has signaled scrutiny of deceptive claims and data security lapses. State privacy laws, such as those in California, Colorado, and Virginia, already require clear disclosures and certain user rights.
These moves could force product changes. They may also push companies to separate consumer and enterprise offerings, with different default settings and retention rules.
What Users Can Do Now
Experts recommend practical steps while rules catch up and companies refine safeguards:
- Use privacy settings to limit training on your chats.
- Avoid sharing health, financial, or legal details in prompts.
- Prefer enterprise or education plans if your employer or school offers them.
- Review retention periods and delete past chats you no longer need.
- Use local or on-device features when available for sensitive tasks.
Security basics still matter. Strong account passwords and two-factor authentication reduce the chance of account takeover, which could expose entire chat histories.
What Comes Next
The next year will test whether industry safeguards match the scale of AI use. Clearer notices, stronger defaults, and shorter retention windows would address many concerns raised by King and other researchers.
There is also a growing push for third-party audits of AI systems. Independent checks on data handling could build public trust, especially after past incidents. Insurance and legal contracts may begin to require such audits for high-risk use cases.
For now, the message is direct. AI chats can be helpful, but they are not private diaries. Users should treat prompts like any other data they would not post online. As regulations tighten and companies adjust, strong privacy by default will be the standard to watch.
King’s warning frames the stakes. People type their lives into these tools. The winners in AI may be the firms that prove they can keep that data safe—and say exactly how they do it.
