AI Formats May Ease Neural Jitters

SAN JOSE, Calif. — A group of mainly chip vendors released a draft standard that aims to act as an interface between software frameworks for creating neural network models and the hardware accelerators that run them. It shares goals with a separate effort started as an open-source project earlier this year by Facebook and Microsoft.

The Khronos Group is seeking industry feedback on a preliminary version of its Neural Network Exchange Format. NNEF initially aims to be a single file format to describe any trained neural network model to any chip performing inference tasks with it.

“We have dozens of training frameworks and potentially hundreds of inference engines on the way,” said Neil Trevett, president of Khronos. “That’s a horrible fragmentation.”

The working group that created the draft consists of more than 30 mainly semiconductor companies including AMD, ARM, Intel, Imagination, Qualcomm, and Samsung. The chip vendors see NNEF as a way to share the effort of creating a single software target for their chips, something many are already doing internally.

Web giants such as Amazon, Google, and others each develop their own software frameworks for creating neural net models. They see them as strategic tools to get an edge in efficiency and attract developers.

To jumpstart their support, Khronos created open-source versions of programs that can export NNEF files from Caffe and Google’s TensorFlow, two popular frameworks.

“We will need a bunch more exporters, but we have those two available now … we will do some paid RFQ-based projects with partners to develop more exporters,” said Trevett.

So far, the web giants seem to be coalescing around their own effort called the Open Neural Network Exchange format (ONNX). The open-source project had a version 1.0 release earlier this month and now has support from Amazon as well as a handful of hardware companies such as AMD, ARM, Huawei, IBM, Intel, and Qualcomm.

ONNX aims to translate models created with any of a dozen competing software frameworks into a graph representation. Trevett said that Khronos is open to collaborating with the effort but pointed out that NNEF is different in two key ways that are important to chip vendors.

Technically, ONNX is a flat representation of operations as a graph. NNEF can do that too, but the Khronos approach also supports compound operations that fuse nodes in a graph. Packing and unpacking operations in this way is one approach that chip vendors plan to use to execute operations efficiently, he said.

Perhaps more importantly, Khronos has had bad experiences with open-source projects that can change rapidly, sometimes breaking hardware compatibility.

For example, changes in the open-source LLVL intermediate representation for processor compilers twice broke compatibility with the group’s OpenCL spec. “People had invested in hardware, and road maps were being broken — this was extremely painful” for chip vendors, said Trevett.

Khronos created its own spec, Spir-V, in place of LLVM, updating it periodically to keep pace with the open-source software. “Our experience with LLVM is that we need a stable spec with multi-company governance,” said Trevett, noting that the group “is up for” the work of keeping that spec current with software shifts.

NNEF requires an interface on the hardware side as well. The initial prototype uses Khronos’ OpenVX interface for computer vision and its recently released neural network extensions.

To use NNEF on smartphones, engineers will have to develop interfaces to Apple’s CoreML or Google’s Android Neural Network API. However, both are openly published formats, so anyone can do that work, said Trevett.

“Once there’s a bunch of trained nets in NNEF, anyone benefits from importing them to their hardware,” he said.

Interestingly, Nvidia is one of the few chip vendors not participating in the NNEF work so far.

“That might change, we haven’t made decisions yet,” said Trevett, who is also vice president of developer ecosystems at Nvidia. The company “has a lot of internal projects that do this sort of thing and is heads-down in solutions for customers.”

To ease the path for chip makers, Khronos also launched an open-source NNEF syntax parser and validator. It lets chip vendors make sure that NNEF files are properly formed and converted for their hardware.

Khronos hopes to stay neutral in any AI battles, giving vendors room to pick the frameworks, APIs, and file formats such as ONNX if they prefer. “NNEF support doesn’t negate support for other things,” he said.

Ultimately, the group hopes that NNEF can also be used as a file format for hardware accelerators used for training and as a way to exchange models among software frameworks.

“We see a need for the authoring interchange, but we haven’t prototyped it, so we will probably have to add stuff … fundamentally, there’s a lot of research in AI, and new neural net topologies will appear, so NNEF will have to evolve,” said Trevett.

— Rick Merritt, Silicon Valley Bureau Chief, EE Times