Onshape Benchmark (UNOFFICIAL) - Fill out to see your results!

MichaelPascoe

Onshape Benchmark Sheet (UNOFFICIAL)

This is the Unofficial Onshape Benchmark. This was created so that we can compare our systems and see which ones are the very best for Onshape. There are several types of results; the "Open Division" is for everyone to participate in, the "Closed Division" will be strict tests where only one variable is changing at a time in order to obtain pure data. I would love to add more entries to the open division is anyone is willing to participate. 


Your benchmark sheet includes a live chart which is linked to the Open Division records.
If you send me your Benchmark sheet, I will add it to the records!

Results

Up to date results for the Open Division can be found here. This will be updated as more records are posted in this discussion. 
Open Division - Onshape Benchmark - Results
Closed Division - Onshape Benchmark GPU - Results

Special thanks to:

@alnis for providing the assembly studio
@Jason_S for walking me through Onshape technical info
@JD for letting me use those graphics cards for the GPU tests

How to use the Onshape Benchmark Sheet (UNOFFICIAL)

https://www.youtube.com/watch?v=SzhkG_qmItc

MichaelPascoe

Let me know if you can think of more user controlled criteria.
Here are some closed division tests that would be good to do:

• Graphics card
• RAM
• Internet download speed
• Internet upload speed
• Ping / latency
• Processor
• Screen resolution

MichaelPascoe

New results are available for download speeds. This was a closed division test, maintaining steady upload speed and latency for each test. See the link below for more details:
Download Speed - Closed Division - Onshape Benchmark - Results

Edit: The table in the results sheet, below the chart, also shows the AWS-3 server speeds at the time of the test.

eric_pesty

I would think the server load at the time you run the test might have more of an impact than the the local resources (for a given available bandwidth) as all that is done locally is loading a webpage and displaying a bunch of triangles on screen, which should be a small part of the overall process (except for the slowest devices).

It's interesting to see that there is almost no benefit to more bandwidth beyond 10Mbps

MichaelPascoe

Good point, there are so many variables; I have eliminated as many as I can think of so that only 1 variable is changing at a time. These test are really only for things that the user can control and potentially upgrade or fix.

The table in the results sheet, below the chart, also shows the AWS-3 server speeds at the time of the test. In that same table, you can click on the link to the data sheet where the benchmark sheet of each of those recordings.

Upload and latency test coming soon..

christian_petty

Here is my benchmark:
https://docs.google.com/spreadsheets/d/1R0fZ4pzEVtq6vMES8ey6lCFOfNff0uFTXzOCO1MzNnE/edit?usp=sharing

I have a 4K screen and if that's of any interest for closed testing.

MichaelPascoe

Thanks for the info @christian_petty
Your results have been added to the Open Division records.

Congrats! Your setup is currently the fastest Onshape pc on record. 

christian_petty

@MichaelPascoe no problem. I hope someone can topple the laptop at some point.

Why do copy operations take so long in this test? Is information passed through the client and back to the servers or is that still a time intensive operation server to server?

MichaelPascoe

When the document is copied, it forces the server to recalculate a portion of the document. I think.. Regardless, most of the time I am not copying a document, so I didn't emphasize that number too much.

It did seem like it could be the most repeatable number, if it really does force the server to recalculate everything. I don't fully understand though, because copying a document still doesn't re-calculate some of the features. If it did, it would take much longer. I would like a way to hard refresh the entire tree and every feature. This isn't possible on our end unless we change a value in each feature.

nick_papageorge073

Here are my results for the lowest end MacBook Air M1 chip. 8 CPU cores, 7 GPU cores, 8GB RAM (Shared).

https://docs.google.com/spreadsheets/d/1dY77PFQ1471b3CsT2P4IcwoD3HfYkmPyuRfoH9VgNUI/edit?usp=sharing

For the new M1 Macs, there should probably be a row that states how many cores are in the GPU. All the new M1 GPU's are called the same thing and none of the tests identify how many cores there are.

nick_papageorge073

Here is the lowest spec M1 Ultra. 20 CPU cores, 48 GPU cores, 64GB RAM (Shared).

https://docs.google.com/spreadsheets/d/1JgzNhp8p556kGoAzgdCcRw7jkxKXZ8aW2QpyN0V6RZ4/edit?usp=sharing

nick_papageorge073

Here is a Win PC I homebuilt in 2020 to run Creo before I switched to OS. 9th gen i7, 5700xt GPU w/8GB RAM, 32GB RAM.

https://docs.google.com/spreadsheets/d/1-Pzc5G-p6zw25Zyj_JzcYiUCRxR24qp4LG2Vgk2gAxI/edit?usp=sharing

MichaelPascoe

Legit! Thanks @nick_papageorge073

Your results have been added to the Open Division records.

nick_papageorge073

I have a new system. I tested my dayjob coworker's desktop computers that are used for machine learning. They are custom built Linux machines, 11th gen i7, and a 3080Ti GPU with 12GB GPU RAM. I believe this is the strongest GPU across all the prior entries. However, the results are nowhere near the strongest. I'm confused... Here is the result:

https://docs.google.com/spreadsheets/d/1_Z5dkQ_MFDYtDzS0xabfaOyuR_ysvkMynXmbp-P_azI/edit?usp=sharing

MichaelPascoe

Added. That's a sweet system.

Keep in mind this test is only looking at document load times. I would imagine having a better card may help with dragging things around in an assembly, "dragability". I'm not sure how to test dragability in a reliable way. Any ideas?

Perhaps I could record a mouse macro that drags at a consistent speed then monitor the fps. That might work for a closed division run.

nick_papageorge073

Yeah, I guess that's the main question. What is your test procedure actually measuring when we are using OS on a daily basis? I think on my work, I have two bottlenecks that are local. One is the GPU. I see its usage go to 100% and that is when my browser freezes. The second is system RAM. The more I have, the more OS browser windows I can have open, which I do often to see old versions of what I'm modifying, see multiple assemblies, etc. I'm almost never using only one browser window.

MichaelPascoe

Update!

The Comparison Tool will help you compare any two data sheets from the Open Division records.

kevin_o_toole_1

Nice tool!

I figured I'd throw in a comparison from an Onshape developer's machine. Here's results from our standard issue Macbook Pro, tested from our Seaport office:
https://docs.google.com/spreadsheets/d/1P12CM2Ih2fT6ya91LhlfPNUemnZllRojamcMVRYsISg/edit#gid=0

All in all not far from the average! That's some validation in not showing hyper-fitting to our own setup. In theory, once basic needs are met, things like copy and refresh times in Onshape's architecture are going to be mostly independent of client hardware.

MichaelPascoe

Valuable info, thanks @kevin_o_toole_1! Added your sheet to the records.

MichaelPascoe

In case anyone was unable to view the google sheet in the comment above, here is a snip of the internet download speed benchmark. Looks like results are all about the same until I throttled the connection to below 10 Mbps.

Note that this is testing the generic download speed with speedtest.net. AWS speeds, upload, and ping are also listed in the table, but are not the focus of this specific closed division test.

Download Speed - Closed Division - Onshape Benchmark - Results

MichaelPascoe

Here is a custom feature Regen Sim. Use responsibly.

By changing the size of the sphere and the magnitude (quantity), we get the following results:

Note that regeneration times vary greatly depending on if you have simple spheres vs a complex part with many shapes. More difficult shapes seem to affect regen time grater, even if they have less triangles than a simple large sphere.

Summary:

Keep your models efficient. Once your document starts to slow due to inefficient modeling, it may slow exponentially.
Don't be a noob, follow each of the Self Paced Learning Pathways so that you are modeling like an Onshape pro.

S1mon

In my experience with the Parasolid kernel in Solidworks and Onshape, regen time goes up a lot for features which require a lot of intersections. A wall or a rib is no big deal, but a complex shape with a ton of ribs and bosses will slow way down as you add more and more detail. 

NeilCooke

MichaelPascoe said:

Here is a custom feature Regen Sim. Use responsibly.

I'm not sure the Onshape servers will thank you for this.

MichaelPascoe

If you like @NeilCooke, I can remove it. However, I figured that not many people would even care to use it, but that it would be nice for the few who were interested.

MichaelPascoe

Thanks for the info @kdee122! I've added your results to the Open Division - Onshape Benchmark - Results.