The Firepear computing stack is currently all Ryzen. It’s simple cost-benefit analysis: I’m interested in crunching as much data as possible per unit time, at a reasonable cost (in both money and electricity). Right now, that’s Ryzen.

I’m upgrading all my machines to the new R9 3900X CPU. This provides an opportunity to compare all three generations of the Ryzen family.

3900X vs 3950X

I want to throw as many cores as I can at the problems that I volunteer compute time for, but I also have a finite budget. The 3950X will be beyond my reach at launch, so my strategy is to upgrade to 3900X starting in early July.

Extrapolating from the price history of the 1X00 and 2X00 processors, between 6 months and a year after release, prices on the 3X00 CPUs will start to slide downward. When the 3950X reaches a price that works for me, I’ll do a second round of upgrades – assuming, of course, that the pattern holds!

Ryzen by the numbers

Here’s how every Ryzen I’ve ever had stacks up, by specs and metrics. Temperature data for 3X00 is not yet available, due to lack of initial system support.

1. LoadClk is the observed, all-core frequency under 100% load
2. Scale is how one CPU compares to another, with the 1600 being defined as the baseline (1.00)
   • Formula: 1 * LoadClkΔ% * IPCΔ% * ThreadΔ%
3. LoadP is the observed power usage under 100% load, minus the system idle load of 24.5W
4. Tdie temps were measured by lm-sensors, at 100% load, with stock coolers

• The notional 2700L model represents values obtained by clocking the 2700 to 3200MHz and setting Vcore offset to -100mV
• The notional 3900L model represents values obtained by underclocking the 3900X for lower power usage (details below)
• There are no metrics for the 1700 because I no longer have any of them

All CPUs were tested using ASRock B450 mini-ITX boards, NVMe SSDs, and 450W STX PSUs. There are memory speed variances between the machines: initially there was a 2400MHz/2666MHz split, later the split was 3000MHZ/3200MHz.

Underclocking the 3900X

NB: I am leaving this section intact for historical purposes, but it was written before Zen 2’s clock stretching issues were known. For current advice and accurate numbers, please see this update

1. LoadP is the observed power usage under 100% load, minus the system idle load of 24.5W
2. Tdie temps were measured by lm-sensors, at 100% load, with the stock cooler

• Test 1 (and all following tests) turnsoff AMD CBS and changes the Vcore offset to -100mV
• Test 4 is where the 3900X is running at the same clockspeed as the 2700L. Equal clocks, but the 3900X still has 15% IPC uplift
• Test 6 is where the 3900X should be doing the same amount of work per clock as the 2700L. 15% lower clock results in equivalent performance.

Undervolting the 3900X

NB: I am leaving this section intact for historical purposes, but it was written before Zen 2’s clock stretching issues were known. For current advice and accurate numbers, please see this update

My goal is to find the intersection of high performance and low power. My initial goal, with respect to power, was to meet or beat 80W (the draw of a 2700 at 3.3GHz). At stock voltages, that would require dropping below 3GHz, which was throwing away too much performance. Here’s the path I took to defining the 3700L configuration.

Vcore offset was -100mV. All tests were performed with a full load of WCG workunits running.

World Community Grid: WUs in 24h

Now it’s time to look at how the CPUs compare when running actual scientific computing workloads.

These tables shows the number of workunits crunched in 24 hours for each processor type, for several World Community Grid subprojects. During these periods, the CPUs were only working on one type of WU.

These are the WU counts for 24 hours, with SMT disabled. There are no non-SMT counts for the 3900X/L because – spoiler alert – there was no point in spending three days gathering that data. SMT is good for all these applications.

There is information on the performance of the 3900 with varying numbers of threads in the Stockfish and OpenFOAM sections of this document.

Note: I do not have an explanation for the 2700 underperforming the 2700L when crunching Zika workunits in this instance. It had a slight performance lead in all other head-to-head tests, including Zika with all threads enabled. I double-checked the logs: that machine crunched 237 Zika WUs and zero other WUs in 24 hours.

These are the counts for 24h in the stock configuration (SMT enabled; SMT uplift in parens). There is no OpenZika data for 3X00 CPUs yet because there are currently no WUs available. 3900X data will be available later.

World Community Grid: WU timings

There is no OpenZika data for 3X00 CPUs yet because there are currently no WUs available. 3900X testing will be done later.

WCG OpenZika WU timings: min/max and by quintile

OpenZika is problematic for benchmarking, as not all Zika WUs contain similar amounts of work. Therefore, in addition to min/max numbers, here is a table of WU runtimes, bucketed by quintile.

Quintiles are not equal across models, but split the span from minimum to maximum time for each CPU. This is bad for pure statistics, but 1:1 comparisons are impossible here due to the many variables in play. Treat these as simply more datapoints about each CPU’s performance under real-world conditions.

WCG Fight Aids @ Home WU timings

Despite having a an 8% higher maximum runtime, the 3900L is 10% faster on average.

WCG Mapping Cancer Markers WU timings

Again we see the 3900L with a higher maximum runtime but a lower minimum and average. This time it’s 14% faster on average.

Microbiome Immunity Project Workunit timings

MIP1 uses the Rosetta suite of molecular dynamics tools, which wants 4MB of L3 cache per running thread. Even the 3900, with its 70MB of L3, would be oversubscribed if 24 instances of MIP1 were running. MIP1 performance suffers when L3 misses are common and the working data needs to be reloaded from system memory.

Therefore, instead of running only MIP WUs on each CPU, this chart shows the times for MIP1 WUs which ran concurrently with other types of workunits over a period of 144h. The low end was (probably) when only 1 or 2 MIP1 WUs were running concurrently. The upper end was (probably) when more were in-flight.

This is also why there is no count of MIP1 WUs completed in a 24h window; the CPUs were crunching an effectively random mix of subproject WUs during the sample period, so counts would be meaningless.

Here the enormous L3 cache of the 3900 comes into play. Its average WU runtimes are half that of the other CPUs, and its maximum runtime is close to three times faster.

Stockfish chess engine

These numbers were generated by the version of Stockfish used widely for benchmarking CPUs, which you can find here. The command used was:

./asmFishL_2017-05-22_popcnt bench 1024 [THREADS] 26

• Scale is performance in multiples of the 1600:12
• SMT Uplift is all-threads performance of a CPU in multiples of its own all-cores performance

During the 24 thread test, the 3900X was showing sustained boosts of 4.01GHz, with a power draw of 155.5W.

Note that the real-world differential between the 1600 and the 3900X, in this test, was 2.47X. The by-the-numbers theoretical differential was 2.78X. That’s pretty close to on-paper performance.

OpenFOAM CFD

These numbers were generated with the current Docker image (of_v1906) of the OpenFOAM computational fluid dynamics package, using the motorBike simpleFOAM tutorial, which is the standard for benchmarking. The algorithm was set to scotch, end time set to 100, and the tutorial’s Allrun script was used for execution. The command used to extract the performance data was:

grep Execution log.simpleFoam | tail -n 1 | cut -d " " -f 3

All CPUs were tested with 2 and 4 processes, to provide a baseline for comparison. Then each was tested with processes equal to its actual number of cores and threads, to show maximum performance.

Incredible numbers from the 3900: 1.7X the performance of the 1600 at 12 processes. And again we see an isolated case where an underclocked/undervolted Ryzen (barely) outperforms one running at stock configuration.

There is very little ILU parallelism in this test, but I suppose that makes sense given the nature of fluid dynamics simulations. Jumping from all-cores to all-threads resulted in a tiny (3.5%) speedup on the 1600, and tiny regressions on the 2700 and 3900X (4.2% and 1%, respectively).

This test was also a good chance to get a look at frequency response and power usage in the 3900X under varying amounts of load.

• LoadFreq is the sustained clock on loaded cores
• IdleFreq is the clock on unloaded cores
• Power was sampled during the actual simpleFOAM run portion of the test, which was the most demanding. The 24.5W system idle power has been subtracted from this value.

Conclusions

The 3900X is…

• A monster that can churn through incredible amounts of work
• HEDT performance for $500
• Absolutely worth it, if you’ve got the work to do
• But hot and power-hungry in a stock configuration

You probably don’t want to disable SMT

The technique known as Symmetric Multithreading (SMT) dates back to 1982, but it didn’t appear in consumer-grade CPUs until 2002, when Intel introduced it in the Pentium 4 (calling it HyperThreading). Intel’s initial implementation – coupled with the terrible design of the P4 – meant that some workloads, in some cases, would see performance degragations with HT enabled.

There are examples in gaming, even today, of many cores impacting performance. But this seems to have more to do with the nature of game engines, which are coded to be as responsive as possible to the senses of the human playing the game. Concerns like accuracy, provability, and/or reproducibility, are distant seconds at best. It is an entirely separate regime of programming from the tasks this review is concerned with. (To be clear, that regime is neither better or worse, or any more or less legitimate: simply very, very different.)

In these tests, I saw only one instance of a performance regression due to threads exceeding the number of cores: the OpenFOAM runs. Given the small scale of these regressions, my hunch is that the issue was that there are no “extra” FPU/vector units in the Zen or Zen 2 cores, so oversubscribing execution threads led to scheduling hold-ups.

I don’t believe there was an actual performance penalty due to SMT being enabled. 2X the threads running in 1.01X the time sounds like scheduling overhead to me. It is a clear sign, however, that if you know your workload is extremely FPU/vector heavy, you should test to be sure of the response you’ll get.

Generally speaking, disabling SMT is throwing away between 15% and 45% of your CPU’s total possible performance.

Make sure your RAM is fast enough

During the initial rounds of testing, my stock 2700 was slightly but consistently slower than my clock-locked and undervolted 2700 (the 2700L, above). This made absolutely no sense, as I could see that the 2700’s clock speeds were a minimum of 100MHz faster than the 2700L’s at all times, and the systems were otherwise identical. At least, that’s what I thought.

It turned out that the 2700’s BIOS was clocking its memory at 2133MHz, rather than its rated 3000MHz. Changing this setting resulted in a 5% speedup for the 2700. The lesson here is that if you have many cores to feed, make sure you have memory fast enough to keep them fed.

Given that 3200MHz/CL16 RAM is currently ~$70 for 16GB, do yourself a favor and run decently fast RAM.

You can save power without sacrificing much performance

I was worried that the performance impacts would be severe, but you’ve seen the results with real-world testing. For the 2700/2700L:

• Performance delta between 0% and 5%
• Temperatures 11% cooler
• Using 16% less power

And for the 3900X/3900L it’s even better:

• Performance delta between 0% and 11% (about 9% on average)
• Temperatures ??% cooler
• Using 50% less power