Could Clang displace GCC generally? Part II: Performance of PostgreSQL binaries

This is the second in a two-part series on Clang. If you haven't already, you'll want to read my original post on the topic, Could Clang displace GCC among PostgreSQL developers? Part I: Intro and compile times.

So, what about the performance of PostgreSQL binaries themselves when built with each compiler? I had heard contradictory reports of the performance of binaries built with Clang. In Belgium, Chris Lattner said that Clang built binaries could perform better, but a number of independent benchmarks suggested that Clang was generally behind, with some notable exceptions. I asked 2ndQuadrant colleague and PostgreSQL performance expert Greg Smith to suggest a useful benchmark to serve as a good starting point for comparing Postgres performance when built with Clang to performance when built with GCC. He suggested that I apply Jeff Janes' recent patch for pgbench that he'd reviewed. It stresses the executor, and therefore the CPU quite effectively, rather than table locks or IPC mechanisms. The results of this benchmark were very interesting.



Greg provided me with shell access to a beefy server, the same server that he used in his review of Jeff’s patch, which added the -P option: http://archives.postgresql.org/message-id/4DFE788F.5020704@2ndQuadrant.com . I hacked together a shell script to run pgbench for this purpose. Binaries were built using GCC and Clang, each with exactly the same flags - Clang accepts the same flags as GCC. To smooth the results out, and to get a conclusive outcome, I decided on 16 10 minute -P runs with 4 connections, that alternated between using each set of binaries, lasting a total of 3 hours. Here’s a summary of the results:

1) GCC test:
tps = 34.242839 (including connections establishing)
2) Clang test:
tps = 34.370732 (including connections establishing)
3) GCC test:
tps = 34.186687 (including connections establishing)
4) Clang test:
tps = 34.922954 (including connections establishing)
5) GCC test:
tps = 32.393383 (including connections establishing)
6) Clang test:
tps = 34.994233 (including connections establishing)
7) GCC test:
tps = 33.019546 (including connections establishing)
8) Clang test:
tps = 34.234937 (including connections establishing)
9) GCC test:
tps = 33.233653 (including connections establishing)
10) Clang test:
tps = 35.233373 (including connections establishing)
11) GCC test:
tps = 33.962637 (including connections establishing)
12) Clang test:
tps = 33.869868 (including connections establishing)
13) GCC test:
tps = 33.488347 (including connections establishing)
14) Clang test:
tps = 33.005470 (including connections establishing)
15) GCC test:
tps = 33.600023 (including connections establishing)
16) Clang test:
tps = 34.770840 (including connections establishing)

The total transactions per second with the Clang binaries was marginally ahead of the GCC binaries. While further analysis is certainly needed, it is a remarkable achievement for Clang to have been able to hold its own against, or even slightly outperform a compiler as mature and popular as GCC here.

So, is it ready for prime-time? Well, not quite. Even my bleeding edge Fedora 15 system only comes with Clang 2.8, and only Clang 2.9 is listed as supported in 9.1. These build time figures are only obtainable on very recent revisions of Clang. While I’m extremely encouraged by our pgbench benchmark results, I hesitate to recommend the use of Clang for building production PostgreSQL binaries just yet. The benchmark is quite synthetic, and may not be a great proxy for general performance, although it certainly wasn’t cherry-picked.  I encourage others to independently reproduce my work here, and to suggest alernative benchmarks.

I can heartily recommend Clang for hacking on PostgreSQL today. I was also impressed with how accessible the Clang developers are if you have a problem. Sometimes you have to be a bit persistent, but in general the Clang community are quite responsive to end-users concerns.

Watch this space.