QuickStats Healthan Ayom Healthcare venture
Biostatistics Made Simple

Can AI Do Your Thesis Statistics? Using ChatGPT Safely

What published studies show about ChatGPT and AI tools for statistics, methodology and references, what journals require, and how to use AI safely.

By the QuickStats Health editorial teamUpdated 10 November 20266 min readLevel: Beginner

Many students now ask ChatGPT or similar AI chatbots which statistical test to use, how to calculate a sample size, or how to word their methods. Used carefully, these tools can save time. Used alone, they can put errors into your thesis that you are still responsible for. This guide summarises what published studies and journal policies say, and how to use AI without risking your work.

What AI tools are good at

Large language models (LLMs) such as ChatGPT are good with words. They can:

  • Explain a concept in simpler language, such as what a confidence interval means.
  • Suggest questions to ask your guide or statistician.
  • Help you draft or polish text that you then check line by line.
  • Help with software syntax, for example an R or SPSS command, which you then test on your own data.

They are a reasonable starting point for learning. They are not a substitute for checking.

What published studies found

Statistics problems

Ignjatović and Stevanović gave 10 textbook biostatistics problems to ChatGPT. On the first attempt, GPT-3.5 solved 5 and GPT-4 solved 6. The errors included choosing the wrong formula, building an incorrect contingency table for a chi-square test, miscalculating means and expected frequencies, and simple arithmetic mistakes. The models often insisted their answers were correct and changed them only after being given hints. The authors concluded that students must be aware the tool "can be wrong" even when it shows its working.

The study was small, and newer models perform better, but the lesson holds: an answer that looks confident and detailed is not the same as a correct answer.

References

AI tools can also invent references that look real.

  • Bhattacharyya and colleagues checked 115 references produced by ChatGPT for medical articles: 47% were fabricated, 46% were real but contained errors, and only 7% were fully accurate.
  • Walters and Wilder found that about 55% of the works cited by GPT-3.5 and about 18% of those cited by GPT-4 did not exist. Many real citations also had errors.

A fabricated reference in a thesis or manuscript is treated as a serious problem, whoever produced it.

What journals and publication bodies say

The rules are now clear and consistent:

  • AI cannot be an author. The ICMJE, the Committee on Publication Ethics (COPE) and journals such as Nature all state that AI tools cannot be listed as authors, because they cannot take responsibility for the work.
  • You are responsible for everything. The ICMJE says humans are responsible for any material that used AI, and that AI output "can be incorrect, incomplete, or biased".
  • You must disclose it. COPE asks authors to state in the methods section which AI tool was used and how, including for data analysis. Many universities and journals now ask the same.

Check your own university's and journal's policy as well, because some are stricter.

Why AI alone is risky for a thesis

Risk What it looks like
Wrong test A t-test suggested for skewed data, or an unpaired test for paired data
Wrong numbers Arithmetic or formula errors presented with confidence
Missing context Advice that ignores your study design, sampling or variable types
Invented references Real-looking citations that do not exist
Privacy Patient data pasted into a public chatbot
Integrity Undisclosed AI use, or text you cannot explain at your viva

AI does not know your study. It answers the question you typed, not the question your data needs.

A safe way to use AI

  1. Use it to learn, not to decide. Ask it to explain options, then confirm the choice with your guide or a statistician.
  2. Never paste identifiable patient data into a public AI tool. Remove names, phone numbers and hospital registration numbers, or better, do not upload data at all.
  3. Run the analysis in proper software (SPSS, R, Stata or Jamovi) and check the output yourself.
  4. Check every number and every reference against the original source. Search PubMed or the journal website for each citation.
  5. Keep a record of what you used AI for, and disclose it as your university and journal require.
  6. Make sure you can explain every step in your own words. Your examiner and reviewers will ask.

When to get expert help

Ask a biostatistician or an experienced researcher in your field when:

  • you are choosing the study design, sample size or main statistical test,
  • the AI answer and your textbook or guide disagree,
  • your results look surprising, or
  • you are preparing the results and discussion for submission.

An expert looks at your actual study and data, and can explain the reasoning so that the work, and the authorship, remain yours.

Key takeaways

  • AI tools can explain and draft, but they can choose the wrong test, miscalculate and invent references.
  • Published studies found frequent errors in AI statistics answers and high rates of fabricated references.
  • Journals do not accept AI as an author; you are responsible for all content and must disclose AI use.
  • Never put identifiable patient data into a public AI tool.
  • Use AI to learn, then confirm your methods and analysis with an expert from the field.

References

  1. International Committee of Medical Journal Editors. Recommendations for the Conduct, Reporting, Editing, and Publication of Scholarly Work in Medical Journals: Artificial intelligence (AI)-assisted technology. icmje.org
  2. Committee on Publication Ethics. Authorship and AI tools: COPE position statement. 13 February 2023. publicationethics.org
  3. Tools such as ChatGPT threaten transparent science; here are our ground rules for their use. Nature. 2023;613(7945):612.
  4. Ignjatović A, Stevanović L. Efficacy and limitations of ChatGPT as a biostatistical problem-solving tool in medical education in Serbia: a descriptive study. J Educ Eval Health Prof. 2023;20:28. doi:10.3352/jeehp.2023.20.28
  5. Bhattacharyya M, Miller VM, Bhattacharyya D, Miller LE. High rates of fabricated and inaccurate references in ChatGPT-generated medical content. Cureus. 2023;15(5):e39238. doi:10.7759/cureus.39238
  6. Walters WH, Wilder EI. Fabrication and errors in the bibliographic citations generated by ChatGPT. Sci Rep. 2023;13:14045. doi:10.1038/s41598-023-41032-5

This article is for education. Always follow the requirements of your university, guide, ethics committee and target journal.