Enhanced Benchmarks: SPEC invalidates over 2,500 hits using Intel CPUs

Standard Performance Evaluation Corporation (SPEC), known for its benchmarks for evaluating processor performance, invalidated more than 2,500 results with Intel CPUs, according to media reports. It is said that a specially optimized compiler has been used to improve the performance of the processors.

That reports Tom's devices Refers to articles previously published by Voronix And Serving the country, which writes 1 out of over 2,500 and 1 out of over 2,600 benchmark results affected for the SPEC CPU2017 benchmark. The software is mainly used to compare the performance of server platforms.

When one of these results is called, the SPEC note quoted below now appears:

SPEC ruled that the compiler used for this result was performing a compilation that specifically optimized the performance of the 523.xalancbmk_r / 623.xalancbmk_s benchmarks by using prior knowledge of the SPEC code and dataset to perform a transformation that had narrow applicability.

In order to encourage improvements that have broad applicability (see Rule 1.4 https://www.spec.org/cpu2017/Docs/runrules.html#rule_1.4), SPEC will no longer publish results using this improvement.

This result is left in the SPEC results database for historical reference.

Accordingly, the company behind the SPEC standards complains that optimization has been used on the part of the compiler, “Which specifically improves the performance of the 523.xalancbmk_r and 623.xalancbmk_s benchmarks by using prior knowledge of the SPEC code and dataset to perform a transformation that only limitsIt does not apply in general.

“To promote improvements with broad applicability, SPEC will no longer publish results using this improvement“, it continues. However, the disputed results should be viewed as “Historical referenceremain in databases.

Sapphire Rapids is affected

Voronix He writes that the Intel oneAPI 2022.0 to 2023.0 compilers use the optimization in question. Older (before 2022) and more recent (after 2023) results are not affected. high Serving the country The results are primarily affected by server CPUs from the Sapphire Rapids family known as the 4th Generation Scalable Xeon. For the upcoming Intel Emerald Rapids, the latest version is 2023.2.3 without proper optimization.

Improvement effects

As Phoronix has since learned, it can be assumed that targeted optimization improves “SPECint Speed” results by about 9 percent and “SPECint Rate” results by about 4 percent. also Serving the country He talks about improvements in the single-digit percentage range.

Since the improvements were already removed in version 2023.2.3 of the compiler, which was published in December, this is evidence that they must have been known for a long time. “It seems to me that something was found and discussed, and then everyone moved on. The result is that we now only find clues in over 2,500 resultsCliff Robinson writes for Serving the country.