Skip to main content

SAST tools speed comparison: Snyk Code vs SonarQube and LGTM

著者:
Frank Fischer

2021年7月22日

0 分で読めます

We’ve been asked to provide a comparison of scan times between Snyk Code and two common SAST tools: LGTM and SonarQube. For our research, we made several assumptions, but we’ve shared the details in order to be transparent.

TL;DR 

Static Application Security Testing (SAST) can only be developer-friendly when it provides near real-time feedback and does not delay your development processes. Snyk Code is up to 106 times faster than LGTM. On average, Snyk Code is 5x times faster than SonarQube or 14x times faster than LGTM. In summary, Snyk Code proves to be one of the fastest semantic scanning engines on the market.

Test field

We have selected 48 JavaScript open source repositories (listed below). The idea was to mimic typically modern developer code sets and JavaScript seemed a good common delimiter. We have chosen a random sample from top-rated repositories on GitHub to represent real-world challenges.

As scanners, we have the Community Edition of SonarQube which is a broadly used open source static analysis tool. It runs locally, so we needed to provide a quite decent PC. As previous tests using the free SonarCloud edition showed: SonarQube on a good PC is faster than free SonarCloud, so it is not unfair to use the local engine instead of the cloud version.  We have to take one of the existing developer machines (details below). The second contestant is LGTM which originates from a company called Semmle which was acquired by GitHub. LGTM uses a deep semantic code search based on CodeQL. We used the LGTM SaaS offering.

Finally, Snyk Code is Snyk’s SAST solution. It is based on the former DeepCode scan engine, now with several months of additional development time within Snyk under its belt. Aside from being developer-friendly and highly accurate, one of Snyk Code’s design goals is to be extremely fast. It uses a proprietary constraint engine to achieve this. We selected this field as (1) the licenses allow us to run and compare, (2) these are semantic engines and not linters (like ESlint), (3) these are common and widely used engines, and (4) we have both a locally running and a SaaS in the field.

Speed is essential when a SAST solution wants to be developer-friendly. It’s widely known that developers are most efficient when security issues are identified during the development process so they can be addressed before the code gets checked in. The code is fresh in their minds. Snyk Code provides IDE plugins that embed seamlessly into the developer workflow. On top of that, Snyk Code provides easy to understand data flow diagrams and extensive explanations, including examples of fixes used in open source libraries with the same context. But for the purposes of this test, we focused on speed.

The way we tested is we ran a scan for each of the repositories — SonarQube locally, LGTM and Snyk Code as SaaS. As mentioned above, this led to our repo selection as we did not want to make scan times dependent on network bandwidth.

We also selected real-life open source repositories and not benchmarks as the quest here is to simulate what real developers do. Using benchmarks to compare SAST tools has its own problems, but in this case it would not help our cause.

Test results

Below are the general statistical values (rounded) in seconds of scan time. (n=48, rounded to integer):

Average

Max

Min

5-percentile

95-percentile

Median

sd

SonarQube

110

895

63

63

257

77

127

LGTM

312

1835

2

141

1046

189

352

Snyk Code

22

162

5

5

72

12

28

Here are the results as a scatterplot (lower is better, logarithmic scale, done with R and ggplot2):

wordpress-sync/blog-sast-speed-comp-scatter

The logarithmic axis over-emphasizes the spread in the lower field. The standard deviation of Snyk Code is actually much smaller than the one of SonarQube for example.

Below is a box plot of the values (lower is better, we used a logarithmic scale to fit all values in the graph, done with R and ggplot2, median and average are shown):

wordpress-sync/blog-sast-speed-comp-box

The plot clearly shows that in the majority of cases, Snyk is dramatically faster than either of the other two. The median of Snyk is 6.7x (SonarQube) up to 16.4x (LGTM) times faster, which shows that the results do not rely on some extremely good outliers but instead are general ones.

In the above diagram, the spread of values in the LGTM column is noteworthy. It roughly lays between 2 minutes (somehow ok) and more than 17 minutes (not acceptable). Developers will not wait for the results taking several minutes during their workday. Luckily, the median of LGTM is around 3 minutes.

In comparison to the first speed test we did (that time as DeepCode, now Snyk Code) we noticed a lot of performance improvement because we optimized loading to the cloud service significantly in the past months. Also, this time, SonarQube was faster, probably because we used a stronger PC with higher grades of parallelism compared to the first test.

Another interesting observation is that SaaS versus locally-installed engines are not actually that different from a speed perspective.

Assumptions and limitations

  • We have chosen SonarQube (Community Edition 8.9.1) and LGTM as the license allows us these comparisons and they are broadly used.

  • We have chosen 48 medium JavaScript repositories as this test field seems to reflect what typical developers work on. We sampled randomly from the top ~200k GitHub repos by stars. Code size was not a criterion.

  • We forced a full scan and did not use a differential scanning mechanism. For example, in the case of Snyk Code in IDE plugins, after an initial scan the IDE plugin does differential scans and saves bandwidth.

  • We used an Intel Xeon @ 2GHz with 16 cores and 64GB RAM

Raw data

We also want to provide the raw data so you can apply your own statistics. (Times in seconds)

Repository

SonarQube

LGTM

Snyk Code

neumino/chateau

101

259

9

ecomfe/etpl

80

237

24

ustwo/ustwo.com-frontend

77

278

20

node-inspector/node-inspector

157

373

21

q-nick/npm-gui

71

227

8

dojo/dojox

317

662

53

majimboo/node-benchmarks

63

141

5

prettier/prettier

84

277

45

d3/d3.github.com

895

1104

67

adobe/node-smb-server

79

194

13

noble/bleno

66

141

5

wagtail/wagtail

115

1462

56

meteor/blaze

83

194

12

dwyl/hapi-socketio-redis-chat-example

68

141

5

chancancode/hn-reader

64

165

20

shipshapecode/ember-shepherd

82

152

30

draptik/angulardemorestful

76

163

6

icebob/vue-express-mongo-boilerplate

77

422

9

ngryman/ribs

64

193

7

rjrodger/seneca-examples

70

176

12

telepat-io/telepat-api

68

173

8

sematext/logagent-js

68

162

8

cookpad/elasticfox-ec2tag

87

173

11

polonel/trudesk

170

1835

77

saintedlama/passport-local-mongoose

72

173

7

cholalabs/passport-localapikey

63

3

5

mozilla/openbadger

79

184

20

inbasic/turbo-download-manager

94

206

31

pgherveou/gulp-awspublish

65

152

5

swagger-api/swagger-socket

93

746

7

tactivos/jquery-sew

69

225

5

johansatge/jpeg-autorotate

63

152

6

mqttjs/mqtt-packet

65

173

12

fulcrum-agile/fulcrum

80

152

18

microweber/microweber

300

937

162

adamhalasz/uniqid

68

151

5

componentjs/builder2.js

71

142

8

frankyghost/projekktor

92

195

9

nielsutrecht/jwt-angular-spring

88

288

8

petereigenschink/steganography.js

69

141

5

mixteam/mixsln

75

205

48

borismus/webvr-boilerplate

70

213

42

arrix/node-readability

77

152

19

iamcal/js-emoji

72

412

6

jbavari/ionic-socket.io-redis-chat

149

384

19

cheeaun/steepless

69

142

5

joewalker/devtools.html

177

2

74

gaearon/react-hot-loader

80

162

15