About those Mac Studio test comparisons

I’m not comfortable with recent speed test comparisons that suggest the M1 Ultra chip inside Mac Studio doesn’t actually beat Nvidia’s RTX 3090 in GPU performance. I’ve a feeling that while there may be some elements of truth to these claims, they may not turn out to be completely vindicated.

What’s the story?

When it introduced the new chip Apple claimed the processor would deliver the same level of GPU performance as the highest end GPU.

The Verge decided to test M1 Ultra against Nvidia RTX 3090 using Geekbench 5 graphics chips and claims it found that at full power the Nvidia chip beats Apple’s. Apple delivered around half the performance of the Nvidia, it said.

This wasn’t a total rout. The Mac Studio still beat the Mac Pro, MacBook Pro 16 and snaps at the heels of Threadsnapper.

But there are several reasons the test may not be completely appropriate.

1. Geekbench 5

Geekbench 5 will likely need to be optimized for use with Apple Silicon. It also seems worth noting that earlier Geekbench tests seemed to prove Apple’s performance claims.

2. What tests?

Apple has failed itself a little here in that while it made claims in favor of its chip it was at best opaque concerning what tests it ran, and how it ran them. This seems an unnecessary omission. It is also interesting to note that other people’s tests seem to be generating different data – to what extent where the chosen tests cherry-picked to deliver these results? This would also apply to Apple’s tests, of course.

3. Open CL

The Verge notes it used Geekbench 5’s Compute test to generate its results. What it doesn’t seem to have mentioned is that those results aren’t precisely comparative. While Macs still run OpenCL, Apple has deprecated it. No termination of such support has been announced, but Apple does recommend developers transition to Metal in their applications. That may mean nothing, it may mean something, but in terms of these latest results this should probably have been disclosed. Here’s the current Mac Studio CPU and Computetests on Geekbench for comparison.

[Also read: 6 (new) things Apple said about Mac Studio]

4. The power thing

Apple’s whole Mac Studio pitch pushed the message that you’ll get more performance per watt from one of these Macs. What the new tests suggest is that while you can push other systems harder by injecting them with more energy to boost performance, Apple’s own chip still sits in a leading position at a fraction of the energy costs.

5. The comparison

Apple claimed the M1 Ultra has 90% higher CPU speed at 60-watts than a 16-core Core i9-12900K at the same power consumption. It also claimed the graphics to be equally as fast as Nvidia’s RTX 3090 but at 200-watts less power.

Not to neglect that The Verge failed to mention that its use of Tomb Raider as a test also meant running the game under Rosetta, as Mike Wuerthele noted.

6. The form

It’s reasonable I think to believe The Verge tested its Nvidia GPU on a desktop PC. That’s fair enough, given Mac Studio is being sold as a desktop, but it probably skips one big difference: the Studio sits inside a smaller box in which most of the internal space is dedicated to thermal management. At least one reviewer has notedthat they think the M1 Ultra could deliver even more performance, and we all know Apple has popped a hefty lump of copper in the Mac to help with thermal management. The PC test system is likely hosted in a bigger box, which means it’s chips can run at higher energy and hotter than the Mac Studio can support. It seems reasonable to think that the true comparison test between a desktop PC running Intel/Nvidia and M1 Ultra won’t really take place until Apple ships its high-performance Mac Pro.

7. Real world results

While these new criticisms of Mac Studio performance may prove to be verified to some extent, they may not mean what they seem. That’s because real world performance tests are showing consistent improvements across multiple tests and applications. One area in which they perform less well is in compression tests in 7-Zip and Geekbench. Might the new data have selected that?

Conclusion

The bottom line is that there is no way you could put an Intel Core-i9-12900K and Geforce RTX 3090 inside a system of the same dimensions as Mac Studio and be able to run them at maximum speed without the whole system melting down. They would need to be throttled, which would inevitably deliver different results.

This will not be a limitation in the future Mac Pro.

That high-end system will likely possess much more internal space for thermal management, may conceivably add additional processors, and should be able to push maximal performance out of Apple’s M1 Ultra chips.

I’ll be quite surprised if this latest set of speed comparisons will hold up over time, particularly with an Apple processor in a like for like chassis. I’ve a feeling things weren’t quite right.

Please follow me on Twitter, or join me in the AppleHolic’s bar & grill and Apple Discussions groups on MeWe.