Lawyer burned by fake AI tax cases - Don’t be next!

Artificial Intelligence (AI) tools like ChatGPT, Grok, Claude, and Perplexity are all the rage today, and people are using them for a wide range of tasks, including legal research.

But caution: AI frequently gives wrong answers, euphemistically called “hallucinations” by AI developers. Hallucinations are particularly common when you ask AI tools tax questions.

The duped lawyer

One hapless lawyer found this out when he attempted to defend a client in Tax Court from a nearly $2.3 million IRS notice of deficiency for failure to report restaurant business income. The attorney saw that the notice contained only the name of a technical services territory manager followed by initials.

The attorney apparently asked an AI tool whether a “wet” signature by an IRS employee is needed for an IRS notice of deficiency to be valid. If so, his client would be home free.

The AI tool apparently gave the attorney the answer he wanted: It cited Cacchillo v. Commissioner, 130 T.C. 132(2008), for the proposition that the Tax Court lacks jurisdiction where a notice of deficiency does not contain a valid manual signature. The AI tool also stated that Cacchillo overturned two prior cases, Miller v. Commissioner, 57 T.C 440 (1971), and Tefel v. Commissioner, 118 T.C. 324 (2002). The attorney cited these cases in briefs submitted to the Tax Court.

There was only one problem: The citations were all fake.

Neither Cacchillo v. Commissioner nor Tefel v. Commissioner exist. There is a case called Miller v. Commissioner, but it has nothing to do with the deficiency notices.

Sad fact. There are multiple real cases that show that a manual signature is not required on a notice of deficiency. Indeed, a signature is not required at all on a notice of deficiency for it to be valid. This has been the rule since 1932.

The Tax Court caught these errors and was not amused, declaring that the attorney’s argument collapses like an overmixed soufflé when one looks at the citations used to prop it up. The court declined to sanction the attorney involved, finding that the embarrassment he suffered upon being caught was sufficient punishment.

But it stressed that lawyers who appear before the Tax Court must follow the Model Rules of Professional Conduct, which ban knowingly making a false statement of law and require a lawyer to correct a previously made false statement.

This was a first for the Tax Court, but lawyers relying on AI have been caught submitting fake or misleading case citations and other false authorities in cases across the country. There is even an online database documenting dozens of these instances.

Lawyers have been fined thousands of dollars for submitting fake citations derived from AI. In extreme cases, they could be suspended from the practice of law.

Using AI

AI tools can be useful if their limitations are understood.

Hallucinations are common. One survey of general-purpose AI tools found that they hallucinate 58 percent to 82 percent of the time on legal queries.

What about AI tools specially designed for legal research? These tools are less prone to hallucinations because they use retrieval-augmented generation (RAG), which sources material only from a defined legal database. However, even these tools still hallucinate 17 percent to 34 percent of the time.

Pattern matching without reasoning

Although labeled “artificial intelligence,” AI is not intelligent, creative, or sentient. An AI tool doesn’t “think” like a human.

Rather, it produces new content by analyze vast amounts of prior works (training data), including texts of all types. Generative AI (GAI) large language models (LLMs) use “machine learning” to identify the underlying patterns and structures in the training data and then make successful predictions based on probability.

Complex math and enormous computing power are required for GAI tools to function, but they are still nothing more than prediction models that respond to prompts with statistically well-informed guesses about what the next word should be.

For example, if you give an AI tool the prompt “Is a manual signature from an IRS employee required on an IRS notice of deficiency?”, it will analyze its dataset of texts about tax, break them down into data points, and generate a response based on the statistical patterns it extracts.

In short, AI tools predict words, not truth. They do not perform legal analysis as humans do and have no built-in fact-checker. For this reason, AI tools may generate different responses to the same question if asked more than once.

Training data limitations

AI tools are only as good as their training data. General-use AI tools learn from vast libraries of texts of all types, including authoritative tax sources such as IRS publications and Tax Court decisions. But these sources may be underrepresented compared with informal discussions on websites like Reddit, outdated articles, or oversimplified summaries. The AI tool might have encountered 10 blog posts with wrong information for every accurate IRS publication.

Specialized AI tools just for tax research use higher-quality tax training data, so they generally provide more accurate answers. But they still are not always accurate,

Overconfidence and desire to please

AI is trained to provide a confident-sounding, coherent, and plausible answer, regardless of whether it’s drawing on solid training examples or essentially guessing. AI tools also tend to want to provide you with the answer you’re looking for. They want to make you happy.

Frequent tax law changes

Tax laws change frequently. An AI tool might confidently cite rules that have changed or cases that were overruled. AI tools don’t inherently know when their information is outdated.

Tax complexity

Tax law presents a particularly difficult change for AI tools because it is so complex. Tax authority is layered: a correct answer may require analyzing Internal Revenue Code section(s), Treasury regulations, revenue rulings, revenue procedures, IRS notices, court cases, and IRS publications and instructions. AI tools frequently cite the wrong layer or misread cross-references.

Moreover, if a question sounds like it should have a clean answer, AI tools are biased to give one-even if the underlying law is ambiguous, unsettled, or face-dependent.

Here are a few methods to help avoid AI hallucinations or at least discover them if they occur:

Tell the AI tool not to guess.
Instruct AI to provide a citation to the primary authority for every legal claim, and be sure to check it yourself.
Break complex questions into smaller pieces. Multi-part tax questions are where errors multiply.
Ask the AI tool to state a confidence level for its answer.

There is no law against using AI tools for legal research. But many courts are now requiring attorneys to disclose whether they used AI in court filings and to certify that any AI-generated document has been independently reviewed by a human.

The IRS has recently advised its employees not to become too dependent on AI tools. Instead, it emphasizes that these tools should be used to support and enhance their work, not to replace their own critical thinking and judgment.

FAQs

What is an “AI hallucination”?
When a generative AI confidently produces false or fabricated facts, citations, or legal authorities instead of accurate information.

Are AI tools ever reliable for legal research?
Specialized legal AI systems using retrieval-augmented generation (RAG) that search curated legal databases are more reliable than general tools but still hallucinate (the article cites rates of ~17–34% for specialized tools versus 58–82% for general tools in one survey).

Does a manual (wet) signature need to appear on an IRS notice of deficiency?
No. The long-standing rule is that a manual signature is not required for a notice of deficiency to be valid; real authorities say a signature isn’t necessary.

BergerCPAFirst, with over 30+ years of experience, offers comprehensive tax preparation services for individuals and businesses nationwide. Our commitment is to provide personalized attention while ensuring compliance and maximizing tax benefits. If you have any questions or would like to schedule a consultation, please call (201) 587-9200 or send us an inquiry.