Register

MLCommons Inference 3.1 Results (4.0+ results will be added later)

Platform Submitter System Device Devices per node FrameworkPerformance
Platform Submitter System Device Devices per node FrameworkPerformance
rigel-nvidia_original-gpu-tensorrt-vdefault-default_configCTuningOne Stop Systems Rigel Edge SupercomputerNVIDIA HGX A100 SXM4Nvidia inference implementation with CM API, TensorRT v8.6.1.6173501
R750xa_A100_PCIE_80GBx4_TRTDellDell PowerEdge R750xa (4x A100-PCIe-80GB, TensorRT)NVIDIA A100-PCIe-80GB4TensorRT 9.0.0, CUDA 12.2158285
r282_z93_q5QualcommGIGABYTE R282-Z93 (5x QAIC100 Pro)QUALCOMM Cloud AI 100 PCIe/HHHL Pro5QUALCOMM Cloud AI SDK v1.9.1125950
R760xa_L40x4_edge_TRTDellDell PowerEdge R760xa (4x L40, TensorRT)NVIDIA L404TensorRT 9.0.0, CUDA 12.2115291
r282_z93_q5eQualcommGIGABYTE R282-Z93 (5x QAIC100 Pro, EE)QUALCOMM Cloud AI 100 PCIe/HHHL Pro5QUALCOMM Cloud AI SDK v1.9.1106349
se450_q4_stdLenovoLenovo ThinkEdge SE450 Edge Server (4x QAIC100 Standard)QUALCOMM Cloud AI 100 PCIe/HHHL Standard4QUALCOMM Cloud AI SDK v1.9.180092.7
e920d_q4_stdHPEHPE ProLiant e920d (4x QAIC100 Standard)QUALCOMM Cloud AI 100 PCIe/HHHL Standard4QUALCOMM Cloud AI SDK v1.9.179948.9
NX_3155G_G8_A100_PCIe_80GBx2_TRTNutanixNX_3155G_G8_A100_PCIe_80GBx2NVIDIA A100-PCIe-80GB2TensorRT 8.6.0, CUDA 12.074652.6
Lenovo_L40_PCIe_48GBx2_TRTLenovoLenovo ThinkEdge SE450 Edge Server (2x NVIDIA L40 PCIe 48GB)NVIDIA L40-PCIE-48GB2TensorRT 9.0.0, CUDA 12.252623.4
amd_zen4_workstation-nvidia_original-gpu-tensorrt-vdefault-default_configCTuningPCSPECIALIST AMD AM5 PC with Nvidia RTX 4090NVIDIA GeForce RTX 4090 (Ada Lovelace)1Nvidia inference implementation with CM API, TensorRT v8.6.1.645772.1
xr4520c_q2_stdDellDell PowerEdge XR4520c (2x QAIC100 Standard)QUALCOMM Cloud AI 100 PCIe/HHHL Standard2QUALCOMM Cloud AI SDK v1.9.140362.5
amd_ryzen_workstation_triton-nvidia_original-gpu-tensorrt-vdefault-using_tritonCTuningPCSPECIALIST AMD AM5 PC (Nvidia Triton server)NVIDIA GeForce RTX 4090 (Ada Lovelace)1Nvidia inference implementation with CM API, TensorRT v8.6.1.637842.2
R5300G6_L40x1_TRTH3CH3C UniServer R5300 G6 (1x L40, TensorRT)NVIDIA L401TensorRT 9.0.0, CUDA 12.227970.4
NF5468M6_A40x1_TRTIEINF5468M6 (1x A40, TensorRT)NVIDIA A401TensorRT 9.0.0, CUDA 12.219391.2
L4x1_TRTNVIDIAASROCKRACK 1U1G-MILAN (1x L4, TensorRT)NVIDIA L41TensorRT 9.0.0, CUDA 12.212881.7
XR7620_L4x1_TRTDellDell PowerEdge XR7620 (1x L4, TensorRT)NVIDIA L41TensorRT 9.0.0, CUDA 12.212839.4
XR4520c_L4x1_TRTDellDell PowerEdge XR4520c (1x L4, TensorRT)NVIDIA L41TensorRT 9.0.0, CUDA 12.212836.1
gcp_g2.standard.4-nvidia_original-gpu-tensorrt-vdefault-default_configCTuningGoogle Cloud Platform (g2.standard.4)NVIDIA L41Nvidia inference implementation with CM API, TensorRT v8.6.1.612557.8
XR5610_L4x1_TRT_MaxQDellDell PowerEdge XR5610 (1x L4, MaxQ, TensorRT)NVIDIA L41TensorRT 9.0.0, CUDA 12.2,12432.5
KR580S1-nvidia_original-gpu-tensorrt-vdefault-default_configTTAKR580S1NVIDIA T42tensorrt 8.6.1, 10639.8
gloria_highendQualcommFoxconn Gloria (Highend)QUALCOMM Cloud AI 100 DM.21QUALCOMM Cloud AI SDK v1.9.19808.25
Orin_TRTNVIDIANVIDIA Jetson AGX Orin Developer Kit 64G (TensorRT)NVIDIA Jetson AGX Orin 64G1Jetpack 5.1.1, TensorRT 8.5.2, CUDA 11.46423.63
nvidia_orin_64k_default_jetpack-nvidia_original-gpu-tensorrt-vdefault-default_configCTuningSiliconhighway NVIDIA Jetson AGX Orin (64K page size)NVIDIA Jetson AGX Orin 32G1Jetpack 5.1.1 with TensorRT 8.5.2, cuDNN 8.6.0, CUDA 11.4.196341.65
nvidia_orin_maxn_pagesize.4k-nvidia_original-gpu-tensorrt-vdefault-default_configCTuningSiliconhighway NVIDIA Jetson AGX Orin (4K page size)NVIDIA Jetson AGX Orin 32G1Jetpack 5.1.1 with TensorRT 8.5.2, cuDNN 8.6.0, CUDA 11.4.196341.65
aws_g4dn.xlarge-nvidia_original-gpu-tensorrt-vdefault-default_configCTuningAWS cloud instance g4dn.xlargeNVIDIA T41Nvidia inference implementation with CM API, TensorRT v8.6.1.66049.73
aws_t4-nvidia_original-gpu-tensorrt-vdefault-default_configCTuningAWS cloud instance g4dn.xlargeNVIDIA T41Nvidia inference implementation with CM API, TensorRT v8.6.1.64601.81
gcp_t4-nvidia_original-gpu-tensorrt-vdefault-default_configCTuningGoogle Cloud Platform (n1-standard-4)Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz2Nvidia inference implementation with CM API, TensorRT v8.6.1.64575.76
amd_ryzen_workstation-cpp-gpu-onnxruntime-vdefault-default_configCTuningPCSPECIALIST AMD AM5 PCNVIDIA GeForce RTX 4090 (Ada Lovelace)1cTuning/MLCommons C++ Modular Inference Library (MILT), Onnxruntime v1.15.14007.25
amd_ryzen_workstation-reference-gpu-onnxruntime-v1.15.1-default_configCTuningPCSPECIALIST AMD AM5 PC with Nvidia RTX 4090NVIDIA GeForce RTX 4090 (Ada Lovelace)1MLCommons reference implementation with CM API, Onnxruntime v1.15.13857.76
Orin_TRT_MaxQNVIDIANVIDIA Jetson AGX Orin Developer Kit 64G (MaxQ, TensorRT)NVIDIA Jetson AGX Orin 64G1Jetpack 5.1.1, TensorRT 8.5.2, CUDA 11.43526.29
orin-nvidia_original-gpu-tensorrt-vdefault-default_configCTuningSiliconhighway NVIDIA Jetson AGX Orin (non MaxN image)NVIDIA Jetson AGX Orin 32G1Nvidia inference implementation with CM API, Jetpack 5.1.1 with TensorRT 8.5.2, cuDNN 8.6.0, CUDA 11.4.193129.69
davinci_hhhlSiMaMLSoC HHHLDaVinci1SiMa SDK Version 0.72952.58
Orin_NX_TRTNVIDIANVIDIA Orin NX 16G (TensorRT)NVIDIA Orin NX 16G1Jetpack 5.1.1, TensorRT 8.5.2, CUDA 11.42640.51
Orin_NX_TRTConnectTechIncNVIDIA Orin NX 16G (TensorRT) + CTI Hadron Carrier (NGX012)NVIDIA Orin NX 16G1Jetpack 5.1.1, TensorRT 8.5.2, CUDA 11.42565.64
amd_ryzen_workstation-reference-gpu-tf-v2.13.0-default_configCTuningPCSPECIALIST AMD AM5 PC with Nvidia RTX 4090NVIDIA GeForce RTX 4090 (Ada Lovelace)1MLCommons reference implementation with CM API, Tensorflow v2.13.02359.98
amd_ryzen_workstation-reference-gpu-tf-v2.12.0-default_configCTuningPCSPECIALIST AMD AM5 PC with Nvidia RTX 4090NVIDIA GeForce RTX 4090 (Ada Lovelace)1MLCommons reference implementation with CM API, Tensorflow v2.12.02131.38
Orin_NX_TRT_MaxQNVIDIANVIDIA Orin NX 16G (MaxQ, TensorRT)NVIDIA Orin NX 16G1Jetpack 5.1.1, TensorRT 8.5.2, CUDA 11.41681.87
aws_g4dn.xlarge-reference-gpu-onnxruntime-v1.15.1-default_configCTuningAWS cloud instance g4dn.xlargeNVIDIA T41MLCommons reference implementation with CM API, Onnxruntime v1.15.1 (with gpu support)528.194
aws_g4dn.xlarge-reference-gpu-tf-v2.13.0-default_configCTuningAWS cloud instance g4dn.xlargeNVIDIA T41MLCommons reference implementation with CM API, Tensorflow v2.13.0342.669
macbook_pro_m1_2-reference-cpu-tf-v2.13.0-default_configCTuningApple MacBook Pro M1undefined10MLCommons reference implementation with CM API, Tensorflow v2.13.0 with tensorflow-metal307.693
51535c5ae107-reference-gpu-onnxruntime-v1.15.1-default_configCTuningCollabora AMD ThreadRipperAMD Ryzen Threadripper 1920X 12-Core Processor12MLCommons reference implementation with CM API, Onnxruntime v1.15.1305.092
7920t-kilt-onnxruntime_gpuKraiDell Precision 7920 Tower (1x NVIDIA RTX A5000 GPU)NVIDIA RTX A5000 GPU1KRAI Inference Library Technology (KILT) with ONNX Runtime GPU support272.245
amd_ryzen_workstation-reference-gpu-onnxruntime-v1.14.0-default_configCTuningPCSPECIALIST AMD AM5 PC with Nvidia RTX 4090NVIDIA GeForce RTX 4090 (Ada Lovelace)1MLCommons reference implementation with CM API, Onnxruntime v1.14.0 (with GPU support)255.537
amd_ryzen_workstation-reference-cpu-onnxruntime-v1.15.1-default_configCTuningPCSPECIALIST AMD AM5 PCAMD Ryzen 9 7950X 16-Core Processor16MLCommons reference implementation with CM API, Onnxruntime v1.15.1253.917
amd_zen4_workstation-reference-cpu-onnxruntime-v1.14.0-default_configCTuningPCSPECIALIST AMD AM5 PCAMD Ryzen 9 7950X 16-Core Processor16MLCommons reference implementation with CM API, Onnxruntime v1.14.0252.609
amd_zen4_workstation-reference-cpu-tf-v2.12.0-default_configCTuningPCSPECIALIST AMD AM5 PCAMD Ryzen 9 7950X 16-Core Processor16MLCommons reference implementation with CM API, Tensorflow v2.12.0252.406
amd_ryzen_workstation-reference-cpu-tf-v2.13.0-default_configCTuningPCSPECIALIST AMD AM5 PCAMD Ryzen 9 7950X 16-Core Processor16MLCommons reference implementation with CM API, Tensorflow v2.13.0252.083
amd_ryzen_workstation-reference-cpu-tf-v2.12.0-default_configCTuningPCSPECIALIST AMD AM5 PCAMD Ryzen 9 7950X 16-Core Processor16MLCommons reference implementation with CM API, Tensorflow v2.12.0248.944
amd_ryzen_workstation-cpp-cpu-onnxruntime-vdefault-default_configCTuningPCSPECIALIST AMD AM5 PCAMD Ryzen 9 7950X 16-Core Processor16cTuning/MLCommons C++ Modular Inference Library (MILT), Onnxruntime v1.15.1234.449
7920t-kilt-onnxruntime_cpuKraiDell Precision 7920 TowerIntel(R) Xeon(R) Gold 6240R CPU @ 2.40GHz24KRAI Inference Library Technology (KILT) with ONNX Runtime CPU support134.187
eb6-kilt-snpe_aipKraiThundercomm TurboX EB6 Edge AI BoxQUALCOMM NPU 230 AIP1KRAI Inference Library Technology (KILT) with Snapdragon AIP support128.469
eb6-kilt-snpe_dspKraiThundercomm TurboX EB6 Edge AI BoxQUALCOMM Hexagon 698 DSP1KRAI Inference Library Technology (KILT) with Snapdragon DSP support105.985
amd_zen4_workstation-reference-cpu-ncnn-vdefault-default_configCTuningPCSPECIALIST AMD AM5 PCAMD Ryzen 9 7950X 16-Core Processor16MLCommons reference implementation with CM API, NCNN vdefault92.4211
macbook_pro_m1_2-reference-cpu-onnxruntime-v1.15.1-default_configCTuningApple MacBook Pro M1Apple M1 Pro10MLCommons reference implementation with CM API, Onnxruntime v1.15.169.2985
gcp.c3_standard_8-reference-cpu-tf-v2.13.0-default_configCTuningGoogle Cloud Platform (c3.standard.8)Intel(R) Xeon(R) Platinum 8481C CPU @ 2.70GHz4MLCommons reference implementation with CM API, Tensorflow v2.13.066.8676
gcp.c3_standard_8-cpp-cpu-onnxruntime-vdefault-default_configCTuningGoogle Cloud Platform (c3.standard.8)Intel(R) Xeon(R) Platinum 8481C CPU @ 2.70GHz4cTuning/MLCommons C++ Modular Inference Library (MILT), Onnxruntime v1.15.165.7066
gcp.c3_standard_8-reference-cpu-onnxruntime-v1.15.1-default_configCTuningGoogle Cloud Platform (c3.standard.8)Intel(R) Xeon(R) Platinum 8481C CPU @ 2.70GHz4MLCommons reference implementation with CM API, Onnxruntime v1.15.164.5665
nvidia_orin-reference-cpu-tf-v2.13.0-default_configCTuningSiliconhighway NVIDIA Jetson AGX OrinARMv8 Processor rev 1 (v8l)12MLCommons reference implementation with CM API, Tensorflow v2.13.041.6048
nvidia_orin-reference-gpu-tf-v2.13.0-default_configCTuningSiliconhighway NVIDIA Jetson AGX OrinARMv8 Processor rev 1 (v8l)12MLCommons reference implementation with CM API, Tensorflow v2.13.041.1616
nvidia_orin-reference-cpu-onnxruntime-v1.15.1-default_configCTuningSiliconhighway NVIDIA Jetson AGX OrinARMv8 Processor rev 1 (v8l)12MLCommons reference implementation with CM API, Onnxruntime v1.15.132.3828
aws_g4dn.xlarge-reference-cpu-tf-v2.13.0-default_configCTuningAWS cloud instance g4dn.xlargeIntel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz2MLCommons reference implementation with CM API, Tensorflow v2.13.031.8977
nvidia_orin_maxq-reference-cpu-onnxruntime-v1.15.1-default_configCTuningSiliconhighway NVIDIA Jetson AGX Orin (non MaxN image)ARMv8 Processor rev 1 (v8l)12MLCommons reference implementation with CM API, Onnxruntime v1.15.128.9732
aws_g4dn.xlarge-reference-cpu-onnxruntime-v1.15.1-default_configCTuningAWS cloud instance g4dn.xlargeIntel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz2MLCommons reference implementation with CM API, Onnxruntime v1.15.127.9115
aws_inf2.xlarge-reference-cpu-onnxruntime-v1.15.1-default_configCTuningAWS cloud instance inf2.xlargeAMD EPYC 7R13 Processor2MLCommons reference implementation with CM API, Onnxruntime v1.15.124.3065
eb6-kilt-snpe_gpuKraiThundercomm TurboX EB6 Edge AI BoxQUALCOMM Adreno 650 GPU1KRAI Inference Library Technology (KILT) with Snapdragon GPU support21.0084
amd_ryzen_workstation-tflite_cpp-cpu-tflite-vmaster-default_configCTuningPCSPECIALIST AMD AM5 PCAMD Ryzen 9 7950X 16-Core Processor16cTuning/MLCommons TFLite inference implementation, tflite vmaster20.0564
amd_ryzen_workstation-tflite_cpp-cpu-tflite-vmaster-with_live_preprocessingCTuningPCSPECIALIST AMD AM5 PCAMD Ryzen 9 7950X 16-Core Processor16cTuning/MLCommons TFLite inference implementation, tflite vmaster (with live preprocessing)19.4557
51535c5ae107-reference-cpu-ncnn-vdefault-default_configCTuningCollabora AMD ThreadRipperAMD Ryzen Threadripper 1920X 12-Core Processor12MLCommons reference implementation with CM API, NCNN vdefault18.4072
aws_t4-reference-cpu-tf-v2.13.0-default_configCTuningAWS cloud instance g4dn.xlargeAMD EPYC 7R322MLCommons reference implementation with CM API, Tensorflow v2.13.017.4295
gcp.c3_standard_8-tflite_cpp-cpu-tflite-vmaster-default_configCTuningGoogle Cloud Platform (c3.standard.8)Intel(R) Xeon(R) Platinum 8481C CPU @ 2.70GHz4cTuning/MLCommons TFLite inference implementation, tflite vmaster10.8284
gcp.c3_standard_8-tflite_cpp-cpu-tflite-vmaster-with_live_preprocessingCTuningGoogle Cloud Platform (c3.standard.8)Intel(R) Xeon(R) Platinum 8481C CPU @ 2.70GHz4cTuning/MLCommons TFLite inference implementation, tflite vmaster10.7615
aws_g4dn.xlarge-tflite_cpp-cpu-tflite-vmaster-default_configCTuningAWS cloud instance g4dn.xlargeIntel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz2cTuning/MLCommons TFLite inference implementation, tflite vmaster8.54644
gcp_t2a.standard.2-reference-cpu-tf-v2.13.0-default_configCTuningGoogle Cloud Platform (t2a-standard-2)Neoverse-N12MLCommons reference implementation with CM API, Tensorflow v2.13.08.20758
gcp_t2a.standard.2-reference-cpu-onnxruntime-v1.15.1-default_configCTuningGoogle Cloud Platform (t2a-standard-2)Neoverse-N12MLCommons reference implementation with CM API, Onnxruntime v1.15.16.85995
eb6-kilt-snpe_cpuKraiThundercomm TurboX EB6 Edge AI BoxQUALCOMM Snapdragon 865 (QRB5165)8KRAI Inference Library Technology (KILT) with Snapdragon CPU support6.3871
nvidia_orin-tflite_cpp-cpu-tflite-vmaster-default_configCTuningSiliconhighway NVIDIA Jetson AGX OrinARMv8 Processor rev 1 (v8l)12cTuning/MLCommons TFLite inference implementation, tflite vmaster3.99455
firefly_rk3399-reference-cpu-ncnn-vdefault-default_configCTuningCollabora firefly_rk3399Cortex-A536MLCommons reference implementation with CM API, NCNN vdefault2.74346
firefly_rk3399-tflite_cpp-cpu-tflite-vmaster-default_configCTuningCollabora firefly_rk3399Cortex-A536cTuning/MLCommons TFLite inference implementation, tflite vmaster0.873957

Select the options for the results table

Submission Version
Submission Category
Submission Division
Model
Scenario
Metric
Filter Platforms
Filter Devices
Voting & Accepting an answer helps improve the community
Welcome to Collective Knowledge Q&A, where you can ask questions and receive answers from other members of the community.
Follow Us
.