AI Code Review: Comparing Metabob with Sonar & DeepSource

AI code review is becoming more important in software development, ensuring code quality, identifying bugs, and improving overall project maintainability. In recent years, several tools have emerged to automate and streamline the code review process. In this comparative blog, we will explore and compare Metabob, an AI code review tool, against popular static code review tools like Sonarlint, Sonarcloud, and DeepSource. We will also take a look at how some of these solutions utilize Generative AI for user interaction.

The bases of comparison are the level of complexity of identified issues and overall user experience. To conduct our comparison, buggy GitHub Repositories were singled out for extensive testing. These repos include:

• fiatrete/OpenDan
• opencodeiiita/PhotoStore
• fief-dev/fief
• TransformerOptimus/SuperAGI

Metabob stands out as a unique AI code review tool that leverages Graph Neural Networks to identify bugs and LLMs to provide automated code recommendations. It analyzes code snippets and offers suggestions for improvement, bug detection, and best coding practices. Sonarlint, developed by SonarSource, is a powerful static code analysis tool that integrates with popular integrated development environments (IDEs) like IntelliJ, Visual Studio Code, and Eclipse. Sonarcloud, also developed by SonarSource, takes static code analysis a step further by offering cloud-based continuous code inspection. It automatically scans code repositories, providing comprehensive reports on code quality, security vulnerabilities, and technical debt. DeepSource is another static code analysis platform that helps identify bugs, security vulnerabilities, and style violations.

Complexity

Simple, rule-based code reviews

Syntax errors, empty variable assignments, and unnecessary complexity are all examples of simple bugs that occur in scripts. In this regard, Sonarlint, SonarCloud, and DeepSource perform very well. They identify such errors quickly and highlight regions of code that are potentially buggy.

Metabob is also able to highlight very blatant errors like empty variable assignments, but it does not do well recommending stylistic syntax issues. Metabob glosses over these issues and deems them insignificant.

Complex, context-based code reviews

Using its attention mechanism from Graph Neural Networks to comprehend semantic and rational markers, Metabob performs much better than its competitors in this aspect. Metabob is able to detect complex, context-based errors, while the other solutions fail to do so.

The example below details a script that uses the HS256 algorithm to generate access tokens. Metabob was able to identify the usage of this algorithm and its weakness against certain attacks. Metabob then recommends the replacement of this algorithm with RS256 or ES256 algorithms to prevent such attacks.

Another example of complex errors is a race condition. Metabob is able to understand the objective of the function and what it is trying to achieve. It is also contextualizing this objective to the other functions present in the script. This combination of information helps it identify race conditions.

Without contextualizing code within and across scripts, SonarLint, SonarCloud, and DeepSource are not able to determine such issues.

User Experience

SonarCloud and DeepSource are web-based SaaS platforms that take some time to configure and use. They both feature a dashboard that displays statistics about each repository (security, performance, stylistic issues) while allowing users to navigate through different issues. This makes navigation within the platform very user-friendly and organized.\

SonarLint and Metabob are available as integrated extensions on VScode. They operate right out of the IDE, with no complex configurations required. Metabob then takes it a step further to incorporate generative AI into its user experience. Using LLMs, Metabob generates error explanations, answers users’ questions, and even recommends code fixes. This allows users to engage with the tool to better understand the identified issues and bugs.

Conclusion

While SonarLint, SonarCloud, and DeepSource are the go-to solution when it comes to simple error identification, Metabob's competitive advantage lies in identifying complex errors. This was evident in the repository analysis conducted, where Metabob could detect all the contextual-based errors while the other tools failed to do so. However, Metabob is only available for Python as a programming language for now, and support for more languages is being worked on.

Metabob utilizes generative AI to enhance its user experience, keeping the tool interactive. Its unique question-and-answer AI assistant allows users to clarify issues, and the LLM code fix recommendations elevate the coding standard to a higher level.

Metabob offers free-to-use individual developer versions and is currently available on VScode as an extension. The tool can be tested here: Metabob: Debug and Refactor with AI - Visual Studio Marketplace!