OpenAI GPT 4o ranked as best AI model for writing Solidity smart contract code by IQ

SolidityBench by IQ has launched as the first leaderboard to evaluate LLMs in Solidity code generation. Available on Hugging Face, it introduces two innovative benchmarks, NaïveJudge and HumanEval for Solidity, designed to assess and rank the proficiency of AI models in generating smart contract code.

Developed by IQ’s BrainDAO as part of its forthcoming IQ Code suite, SolidityBench serves to refine their own EVMind LLMs and compare them against generalist and community-created models. IQ Code aims to offer AI models tailored for generating and auditing smart contract code, addressing the growing need for secure and efficient blockchain applications.

As IQ told CryptoSlate, NaïveJudge offers a novel approach by tasking LLMs with implementing smart contracts based on detailed specifications derived from audited OpenZeppelin contracts. These contracts provide a gold standard for correctness and efficiency. The generated code is evaluated against a reference implementation using criteria such as functional completeness, adherence to Solidity best practices and security standards, and optimization efficiency.

The evaluation process leverages advanced LLMs, including different versions of OpenAI’s GPT-4 and Claude 3.5 Sonnet as impartial code reviewers. They assess the code based on rigorous criteria, including implementing all key functionalities, handling edge cases, error management, proper syntax usage, and overall code structure and maintainability.

Optimization considerations such as gas efficiency and storage management are also evaluated. Scores range from 0 to 100, providing a comprehensive assessment across functionality, security, and efficiency, mirroring the complexities of professional smart contract development.

Which AI models are best for solidity smart contract development?

Benchmarking results showed that OpenAI’s GPT-4o model achieved the highest overall score of 80.05, with a NaïveJudge score of 72.18 and HumanEval for Solidity pass rates of…