OpenAI’s Latest Breakthrough: CriticGPT Revolutionizes Code Review

Friday, June 28, 2024, marked a significant milestone in the world of artificial intelligence as OpenAI unveiled CriticGPT, a groundbreaking model designed to identify errors in ChatGPT’s code output. This innovative tool is built on the GPT-4 family of large language models and is poised to revolutionize the way humans review and correct AI-generated code.

AI-assisted code review is the future

CriticGPT: The AI Alignment Game-Changer

CriticGPT is an AI tool that aids human trainers in reviewing the programming code generated by ChatGPT. By utilizing reinforcement learning from human feedback, CriticGPT helps humans detect mistakes that might otherwise be overlooked. This approach is a significant step towards achieving AI alignment, a process that ensures AI systems behave according to human expectations.

“We found that when people get help from CriticGPT to review ChatGPT code, they outperform those without help 60% of the time. This is a step towards being able to evaluate outputs from advanced AI systems that can be difficult for people to rate without better tools.” - OpenAI

Training CriticGPT: A Human-Centric Approach

To train CriticGPT, human developers were asked to edit the code generated by ChatGPT, intentionally introducing errors and offering sample feedback. This data was then utilized to train CriticGPT to detect both common and novel coding errors.

“We find that CriticGPT critiques are preferred by trainers over ChatGPT critiques in 63% of cases on naturally occurring bugs, in part because the new critic produces fewer ’nitpicks’ (small complaints that are unhelpful) and hallucinates problems less often.” - OpenAI

The Future of AI-Assisted Code Review

While CriticGPT is a significant breakthrough, OpenAI acknowledges that the model still produces hallucinations, which may lead trainers to make mistakes after seeing these hallucinations. However, the company remains committed to improving the model and pinpointing mistakes that can be identified in one specific place.

The future of code review is AI-assisted

In conclusion, CriticGPT is a game-changing innovation that has the potential to revolutionize the way we review and correct AI-generated code. As AI systems become increasingly advanced, tools like CriticGPT will play a vital role in ensuring that these systems behave according to human expectations.