OpenAI employs hundreds of contract workers to read real ChatGPT conversations and rate them. The stated goal is to reduce the model's tendency to be overly agreeable. The problem is that the setting that allows this review is on by default for every account.
Prompts are anonymized, but anonymization doesn't remove the risk: a conversation can contain details about clients, suppliers, internal strategies or personal data that stay readable even without a name at the top of the text.
If your company operates in a regulated sector (healthcare, finance, legal, HR) or handles data covered by confidentiality agreements, check this setting today. It's not a technical issue: it's a matter of compliance and of trust with your clients.
What really happens when you send a prompt to ChatGPT
According to The Decoder, OpenAI has hundreds of contractors (contract workers, not employees) who read real user conversations and rate them on a scale from one to seven. The work is meant to train the model to be less sycophantic and less "human" in the wrong sense of the word.
One detail matters for anyone using ChatGPT at work: the "Improve the model for everyone" setting is on by default. Until you turn it off, your conversations can be read by real people.
Anonymized isn't enough: the real risk for SMEs
OpenAI anonymizes prompts before passing them to reviewers. But anonymizing doesn't make a text harmless. A conversation where you ask to summarize a client contract, analyze revenue data or draft a reply to a complaint contains sensitive information even without the person's name at the top of the page.
Who is really exposed:
- Law firms and consultants who use ChatGPT to draft legal documents or opinions
- HR departments working on texts about performance reviews, dismissals and salaries
- Healthcare or pharmaceutical companies with patient or clinical trial data
- Accountants and CFOs who paste in financial statement excerpts or financial projections
- Anyone who has signed NDAs with clients and uses ChatGPT to work on those projects
The topic isn't new: we've already looked at what ChatGPT contractors really read and how the review process works. What's new today is confirmation of the scale: not one or ten reviewers, but hundreds.
How to turn off human review: three steps
The procedure is simple. The problem is that almost nobody knows about it.
- Open the settings of your ChatGPT account (icon at the bottom left on desktop, or the profile menu on mobile)
- Go to "Data controls" in the settings section
- Turn off "Improve the model for everyone": the toggle should turn gray
Once you've done this, your conversations are no longer used to train the model and are not assigned to human reviewers. Note: this setting must be checked on each company account separately. If your company has multiple ChatGPT users, every person needs to do this on their own profile.
Anyone using ChatGPT Enterprise, the paid version for businesses, already has this protection in place by contract. But anyone on the free account or the individual Plus plan doesn't.
The bigger picture: AI governance and company data
This story is part of a more structural problem. Companies adopt AI tools fast, often without a clear policy on which data can be shared with external services. The risk isn't only regulatory (GDPR, NIS2, industry rules) but also contractual and reputational.
It's not an isolated case. We've seen how Meta offers 95% discounts in exchange for sharing data for training: different models, same principle. The data you produce using a free or semi-free AI tool has value, and that value gets extracted in some form.
When it comes to governing AI agents, the systems that act autonomously on the user's behalf, the problem grows even bigger. We covered it by analyzing the concrete risks for anyone using AI APIs and what changes for governance when agents operate outside direct control.
The topic of AI regulation, privacy and governance is becoming operational, not just theoretical. It's no longer about waiting for European guidelines: it's about deciding today which data goes into which tools.
Mistakes to avoid in the coming days
Turning off the setting is the first step, but it isn't enough on its own. These are the most common mistakes SHM Studio sees in companies tackling this issue:
- Thinking once is enough: if you add new users or accounts, the setting goes back to on by default
- Confusing ChatGPT with ChatGPT Enterprise: they're different products with different terms of service
- Not updating internal policies: turning off the setting without telling your colleagues doesn't fix the cultural problem
- Ignoring the other AI tools in use: ChatGPT isn't the only service with similar settings; it's worth running the same check on Copilot, Gemini and any other tools your company has adopted
The question to ask your team this week is simple: do we know which company data is going into which AI tools, and with what settings?
Related articles
Discover more articles exploring similar topics, selected to offer you a more complete and stimulating perspective. Each piece of content is carefully chosen to enrich your experience.
