[benchmarks] Add option not to report on the Souffle binary