Key papers from the AI safety and evaluation community that inform our direction. Full credit to the original authors.
Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, Ting Liu
Categorizes hallucination in LLMs, analyzing causes and reviewing detection and mitigation strategies.
Stephanie Lin, Jacob Hilton, Owain Evans
Benchmark measuring whether language models generate truthful answers. Larger models can be less truthful.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al.
Training AI to be helpful, harmless, and honest using a set of principles, reducing the need for human feedback.
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, Dan Mane
Five practical AI safety problems: safe exploration, distributional shift, side effects, reward hacking, and oversight.
Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, Hannaneh Hajishirzi
Breaks LLM text into atomic facts and verifies each against a knowledge source for fine-grained factuality scoring.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen-tau Yih, Tim Rocktaschel, Sebastian Riedel, Douwe Kiela
The seminal RAG paper. Combines retrieval with generation to improve factual accuracy and reduce hallucinations.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al.
Comprehensive benchmark evaluating LLMs across accuracy, calibration, robustness, fairness, and efficiency.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.
Scaling enables strong few-shot performance, but highlights challenges in reliability, bias, and factual consistency.
Sebastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al.
GPT-4 capabilities and limitations: remarkable reasoning alongside persistent factual accuracy failures.
Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al.
Methods for red teaming LLMs to discover harmful outputs, and how model size affects the harms found.
Our Direction
These papers represent the ideas that motivate our work, from hallucination detection to systematic evaluation.
As we publish our own findings, they will appear here alongside the work that inspired them.
AI Safety
3 papers
Evaluation
4 papers
Hallucination
1 paper
RAG & Retrieval
1 paper
We connect with researchers working on AI safety, evaluation, and reliability.
Get in Touch
