Introduction: AI Is Convenient. But "Careless" Accidents Happen Easily
Generative AI (such as ChatGPT) is active across a wide range, from writing, research, and summarization to coding support. On the other hand, risks such as hallucination (plausible lies), information leakage, and bias directly lead to real trouble depending on how you use it.
In this article, we break down difficult topics as much as possible and bring countermeasures useful in the field down to a "you can do it from today" level.
Risk 1: Hallucination (AI's "Plausible Misinformation")
What kind of problem occurs?
Hallucination is the phenomenon of AI confidently saying wrong things. It is especially likely in the following situations.
- Latest information (outside the model's training range, very recent news)
- Specialized domains (law, medicine, finance, security)
- When a source is required (papers, statistics, systems, terms)
- Proper nouns (people, companies, product names, clause numbers, model numbers)
Common "accident examples"
- Presenting nonexistent papers or URLs as "references"
- Asserting laws or systems while still in old content
- Misleading by arbitrarily and "plausibly" filling in internal rules
Countermeasure: Make AI a "draft craftsman," not a "responder"
Hallucination countermeasures are, roughly speaking, "do not make the AI's answer the final answer." Specifically, the following work.
1) Ask for the basis (source) as a set
Put "always include the basis" and "write uncertain points as uncertain" into the prompt.
Example: "Answer in the order conclusion then basis then how to verify. Always attach a source URL or primary information (the official document name). For unclear points, don't guess; write 'unknown.'"
2) Have it convert into a "form that is easy to verify"
Rather than having it assert in prose, having it turn things into a checklist of verification items increases safety.
- Decompose claims into bullets
- Attach a "needs verification" flag to each claim
- Have it propose where to verify (official sites, terms, primary sources)
3) Lean toward RAG (internal data search) or citation-based operations
If you use it for work, a mechanism that goes and fetches the correct information source is important rather than dumping everything on the model. For example, RAG (Retrieval-Augmented Generation) searches internal documents and a knowledge base and builds answers using those results as the basis.
Concrete options used include Azure AI Search, Amazon Kendra, Elasticsearch, OpenSearch, or search infrastructure integrated with Notion/Confluence/Google Drive.
4) Build human review in as a "process step"
What is strong in operations is to make review criteria explicit.
- Numbers, dates, and proper nouns are checked in principle
- Content touching terms, laws, or contracts must reference primary information
- Externally published materials pass a "pre-publication check"
Risk 2: Information Leakage (It Can Be Over the Moment You Input It)
What kind of leakage occurs?
Information leakage with generative AI broadly divides into the following two kinds.
- Input leakage: an employee pastes confidential information (customer data, unpublished materials, source code) into a prompt
- Output leakage: the AI infers from the conversation context and outputs information that should not be shown (surfaces when permission management is weak)
Countermeasure: First "don't put it in," "separate it," "don't keep it"
Before difficult countermeasures, there is a basic template that reduces accidents.
1) Concretely decide what information must not be input
The more fuzzily internal rules are written, the less they are obeyed. Concretizing with examples works in the field.
- Personal information (name, address, phone, email, employee number)
- Customer information (contract content, transaction history, inquiry content)
- Unpublished information (pre-earnings numbers, M&A, roadmaps)
- Credentials (API keys, passwords, tokens)
- Source code (especially your company's unique core parts)
2) If it is truly necessary, "anonymize, summarize, or dummy it"
In practice, there are situations where information cannot be reduced to zero. In that case, instead of pasting the original data as-is, convert it into a low-risk form.
- Personal name then "Customer A"
- Email address then masked (such as example.com)
- Raw sales numbers then ranges or indexed (e.g., year-over-year difference, growth rate)
- Logs then extract only the necessary lines
3) Use enterprise settings/plans and check the handling of logs/training
Generative AI services may differ in the handling of conversation log retention and use for training depending on plan and settings. Don't decide this "somehow"; confirm it via the contract conditions and the management console settings.
Also, choosing an environment that supports SSO, admin policy settings, audit logs, and data retention period control makes operations considerably easier.
4) Reduce "careless pasting" with DLP (Data Loss Prevention)
People make mistakes. So protecting with mechanisms is realistic. For example, with DLP you can run operations that detect and warn on email addresses, My Number format, or API-key-like strings.
It is smooth to introduce when connected with existing security infrastructure such as Microsoft Purview, Google's information protection, and various CASB/SASE products.
Risk 3: Bias (Fairness and Validity Break Down)
What kind of problem occurs?
AI is strongly influenced by training data. Therefore, in areas that "affect people or opportunities" such as hiring, screening, evaluation, and ad delivery, bias easily leads to backlash and legal risk.
- Expressions/judgments disadvantageous to specific attributes (gender, nationality, age, etc.) get mixed in
- Stereotyped explanations lead to brand damage or discriminatory expression
- It is weak on minority cases and treats outliers unfairly
Countermeasure: Decide evaluation metrics and a "prohibited line"
1) Draw a "line for the use case" before using it
Generative AI is not omnipotent. In particular, for the following uses, it is safer to have a human make the final judgment in principle, or not to use it.
- Automated decisions on hiring eligibility, credit, insurance premiums
- Definitive medical or legal advice
- Attribute inference (deciding gender or nationality from appearance or name, etc.)
2) State "neutrality" explicitly in the prompt
What works surprisingly well is binding it in the prompt.
Example: "Do not infer based on attributes (gender, age, nationality, etc.). Unify into neutral phrasing. When judgment is needed, list multiple viewpoints and avoid assertions."
3) Create test cases and do a "regular checkup"
Bias cannot be prevented by willpower. So periodically run test inputs with the same conditions and check for bias in the output.
- Identical-content inquiries with only the attribute changed (A/B)
- Whether the politeness of complaint-handling text changes by attribute
- Whether expressions that exclude a specific attribute appear in job postings
An "Operational Template" Useful in the Field: Guidelines + Mechanisms + Education
Summarizing the discussion so far into a form that works in the field, the following set of three is strong.
1) An AI usage guideline understandable on one page
- OK examples: summarizing publicly available information, drafts of text, ideation
- Caution: consultations involving customer information, contracts/legal, automatic generation of final answers
- NG: pasting personal information, credentials, or unpublished information
2) Put checkpoints into the workflow
- Before input: whether confidential information is included (DLP / visual check)
- After output: basis verification (links, primary sources, internal documents)
- Before publication: review (PR, legal, security)
3) Conduct "short training" regularly
Rather than long training, 30 minutes once a month sticks better. The best material is the mistakes that actually tend to happen.
- Real examples of hallucination and how to spot them
- Concrete examples of information that must not be pasted
- Practice rephrasing biased expressions
Summary: AI Is Strong When You Design "Convenience" and "Safety" as a Set
Used correctly, generative AI greatly raises productivity. However, hallucination, information leakage, and bias will eventually break down if you rely only on individuals' attentiveness.
The recommendation is to embed (1) input restriction, (2) basis verification, and (3) regular testing as a minimal set into the team's workflow. Even just this gets you much closer to "AI you can use with peace of mind."
