Sunday 26 January 2014

Static Code Analyzers - Checkstyle, PMD, FindBugs - Are those alternatives or not?

I hope, you have already read "Preface" of my static code analyzers series.

Majority of programmers take shortcuts while writing code, because shortcuts save time and usually go unnoticed … until something breaks badly in production. To avoid such situations, on-going java source code review activity must be part of development life cycle.  To keep code quality consistent in a project, you should rely on using java code analyzers instead of just manual code review performed by self or peer.  But which tools can ensure quality accredited standards in project?

If using Eclipse IDE for development, then commonly known java code audit analyzer plug-ins are: Checkstyle, PMD and FindBugs. If you are 2-3 years experience java developer, you should be already using any of these or similar tools. I know, next question would be - should use all three or any one of Checkstyle, PMD and FindBugs plug-in? I would try to answer this question in this post and you can always refer tool's documentation for more detail such as demo screenshots, installation steps, user guide, etc.

My Personal Feedback
  • When to use?
    • Are you looking for a tool to enforce coding conventions and standards in code? Then you can use Checkstyle. --- Its main focus is ensuring the coding style adheres to a set of conventions. It would catch things like missing/improper javadoc, naming conventions, placement of braces and parentheses, whitespace, line length, etc.
    • Do you want a tool to detect bad practices in code? Then you can use PMD. --- It scans source code and looks for bad code such as empty try/catch blocks, using .equals() instead of ‘==’, over-complicated expressions, unused variables and imports, unnecessary loops and if statements, enforce naming conventions, etc.
    • Do you need a tool to find potential bugs in code for you? Then you can use FindBugs. --- It can find problems like improper use of .equals() and .hashCode(), unsafe casts, possible ignored exceptions, possible stackoverflows, when something will always be null, etc.
    • PMD vs. FindBugs --- There is a lot of overlap between FindBugs and PMD. But the biggest difference between PMD and FindBugs is that FindBugs works on byte code, whereas PMD works on source code. So some of potential problems FindBugs can find at byte code level, which PMD won't be able to report by scanning source code. Also sometimes based on what type of software artifact (source code or compiled classes or jar file) is available for analysis, you can select PMD vs. FindBugs.
  • Initial learning curve and ease-of-use, setup requirements?
    • Eclipse plug-in is available for each of three tools; as well as other known IDEs are supported. Just install Eclipse plug-in and start using it. Pretty simple to use with default settings. They  ( Checkstyle Eclipse Plug-in, PMD Eclipse Plug-in,FindBugs Eclipse Plug-in) are actively upgraded to support latest stable Eclipse (Eclipse Kepler - 4.3 version as on 18-Jan-2014).
    • Also these tools integrate with Maven, SonarQube, etc. Please check website of each tool to know all available options to run the tool and setup requirements.
    • Important note - If you run tool with default settings, then you would get long list of review findings. Obviously all can't be fixed always considering the time factor. Hence it demands to prioritize project specific quality standards and then customize tool's configuration to match those, which requires good knowledge of code quality standards as well as tool specific rules.
  • Cost?
    • All three are open-source projects, so don't worry about licensing costs.
  • Challenges?
    • One of challenge is that sometimes developer requires detailed knowledge or manual analysis to distinguish a false positive finding (a bug that's not really a bug) reported by tool.
Practical Shot
  • I ran all three tools (by installing plug-ins in Eclipse Kepler) with default settings on java code of one of open source project, which I used in past (MockMock: a cross-platform SMTP server built on Java which allows you to mock email sending functionality and to see how emails look like). I don't aim to improve code by analyzing all reported problems for real vs. false warnings. Instead I would quickly show, what kind of issues are reported by each tool on the same codebase. Here is a summary of reported findings by each tool and my feedback.

  • Checkstyle - Found total 2228 problems as per below screenshot. You can see reported problems are mainly related to java code conventions and standards, for example - missing javadoc, related to code beautification, declaring field as final, etc. If we work rigorously to fix those, then it would greatly improve maintainability of code.

  • PMD - Found total 1321 violations as per below screenshot. The key focus of the tool is finding programming bad practices or code smells such as extra imports, related to variable declaration, avoid catching generic exception, avoid too many methods in class, etc. Sometime it requires in-depth analysis against reported violation, for example here Google Guava library is used in the code where it shows potential violation of LawOfDemeter rule. Also it has "Copy/Paste Detector (CPD)" add-on to find duplicate code. So we should refactor code to fix violations reported by PMD to reduce potential problems and improve overall maintainability of the code.

  • FindBugs - FindBugs reported total 2 potential bugs as per screenshot. It works on concept of finding bug patterns. FindBugs works by analyzing Java bytecode (compiled class files), so you don't even need the program's source code to use it.  Sometimes FindBugs can report false warnings (not real errors).  FindBugs says - "In practice, the rate of false warnings reported by it is less than 50%". We can see here, reported bug by FindBugs in Line 85 is not found by PMD and vice-versa is in Line 87.

My Final Thoughts
  • If you use Eclipse, do take benefit of available code analyzer plug-ins (instead of standalone tools) to boost confidence on quality of your delivered java code. You should check each tool's supported integration options against project's ecosystem for getting best benefits. For example , in given situation you may benefit by integrating code audit tools with Maven, Jenkins, SonarQube, etc.
  • Checkstyle, PMD, FindBugs --- There is a good amount of overlap among them, but each provides a unique service. Keeping ROI in mind, I recommend to use PMD at minimum for projects having just 3-4 months development phase with less than 5 developers team. Otherwise for best results, developers should use all three "Checkstyle + PMD + FindBugs" for code review by doing necessary customizations (audit rules configuration  changes) as per project requisite.
  • The next question can be --- Is there any single packaged tool having ability of "Checkstyle + PMD + FindBugs + more..."? Well, personally I prefer either "Codepro Analytix" or "SonarQube".


Disclaimer
I don't aim to exploit code of any open source project or sample application, while I share my evaluation feedback of given tool on selected publicly available code. Also I am not biased to particular free or commercial tools, rather my objective is about sharing my own experience on set of tools.

Also Refer

2 comments:

  1. Excellent blog, a lot of interesting things, thanks a lot for all knowledge sharing!!!

    ReplyDelete
  2. Nice post man

    ReplyDelete