Here is the output of running the shark-bench tool on the microsoft/MiniLM-L12-H384 model. SHARK Runtime on M1MAX GPU: 1.5x faster than TF-metal 2.8.0 SHARK on Apple M1MAX GPU We have also run the same benchmarks on a 16″ MacBook Pro and notice the same performance and both don’t thermally throttle during our benchmarks. So if you love PyTorch and want to use those 32 GPU cores in your new Apple Silicon Macbook Pro read on.įor our experiment we will utilize a 14″ MacBook Pro with the Apple M1 Max with 64GB RAM. Today we demonstrate SHARK targeting Apple’s 32 Core GPU in the M1Max with PyTorch Models for BERT Inference and Training.

In the past we demonstrated better codegen than Intel MKL and Apache/OctoML TVM on Intel Alderlake CPUs and outperforming Nvidia’s cuDNN/cuBLAS/CUTLASS used by ML frameworks such as Onnxruntime, Pytorch/Torchscript and Tensorflow/XLA. Nod.ai has added AMD GPU support to be able to retarget the code generation for AMD MI100/MI200 class devices and move machine learning workloads from Nvidia V100/A100 to AMD MI100 seamlessly. Since SHARK generates kernels on the fly for each workload you can port to new architectures like the M1Max without the vendor provided handwritten / hand tuned library. SHARK is built on MLIR and IREE and can target various hardware seamlessly. Though Apple has released GPU support for Tensorflow via the now deprecated tensorflow-macos plugin and the newer tensorflow-metal plugin the most popular machine learning framework PyTorch lacks GPU support on Apple Silicon. In this blog we demonstrate PyTorch Training and Inference on the Apple M1Max GPU with SHARK with only a few lines of additional code and outperforming Apple’s Tensorflow-metal plugin. SHARK is a portable High Performance Machine Learning Runtime for PyTorch.

Gpu with metal support