Technical Articles

AI by Vision App

Artificial intelligence as app for industrial cameras

The upcoming vision app-based industrial camera series IDS NXT rio and rome are already AI-Ready! An AI-vision app developed by IDS turns them into energy-efficient but powerful inference cameras with integrated AI accelerators. With existing neural networks, a variety of AI-based image processing tasks can be realized in a short time.

Machine vision today makes it possible to seamlessly monitor manufacturing processes. Camera technology and image processing identify optical features in captured image data making them available for subsequent steps in the production chain. Once programmed, the classic image processing system always operates in the same way - but only as long as the features to be identified are unique and have been pre-programmed. In order to lower error rates continuously, production problems must be detected and avoided at an early stage. AI-based analysis provides new approaches for cases where it is almost impossible to completely predict the complete variety of possible deviations and errors. Artificial neural networks (ANN) nowadays interpret complex image contents with unexpected accuracy and offer solutions, which could not be realized by manual programming with basic algorithms even with a lot of effort. In addition to quality assurance or preemptive maintenance in industrial production, medical diagnosis or warehouse inspections in retail and logistics are also viable applications.

The ANN diversity is made available to users through a wide range of open source frameworks, high-level software and services. Several published ANN architectures already cover different requirements for complexity, accuracy or inference times. Due to the automation and monitoring of industrial plants, more and more training data is available for these architectures. However, the use of ANNs formerly called for expensive and performance-hungry hardware.

Special embedded accelerators, i.e. hardware chips with high computing power at low power consumption, can help. Such accelerators should be integrated directly into the camera, enabling decentralized image analysis and avoiding bandwidth bottlenecks during transmission. As a result, the user has the choice to execute the artificial intelligence classically on a PC, in the cloud or on an embedded vision camera. In addition to providing image data, it also directly takes care of its analysis. The dissemination and networking of such "cyberphysical building blocks" will make process data available that will sustainably enhance the automation and processing speed of industrial manufacturing processes.

The ANN structure and working methods have shown that proven hardware has to be reviewed in order to accelerate them. Although they interpret images with simple arithmetic operations, such as additions and multiplications, billions of them have to be performed to check the image data against all trained features. In order to ensure high inference rates at low latency and in real-time, a large amount of parallel processing is necessary. The selection of the right technology (e.g. GPUs, DSPs or FPGAs) for AI acceleration is therefore another important variable when designing a complete image processing system, the implementation of which depends on requirements such as cost, size, performance, quality and hardware compatibility.

Industrial camera and embedded AI platform brought together in one device

With the IDS NXT rio and rome, IDS brings AI on the "edge ": These new vision app-based industrial cameras are more than just image providers. The user is able to extend the standard camera functions by image processing tasks with so-called vision apps. On the hardware side, the CPU is assisted by a parallel working FPGA that can be programmed at runtime, making the complete data path flexible to use. An IDS developed AI vision app transforms the integrated FPGA into an AI processor capable of accelerating many known architectures of neural networks.

Equipped with this AI-based embedded system, the user can conveniently deploy his own neural networks in the inference camera to perform various tasks: anomaly detection, fruit classification, surface inspection, PCB and assembly verification, etc. The flexible IDS NXT platform facilitates integration into an existing system and adaptation to different markets.

IDS NXT industrial cameras become universal inference cameras including ANN accelerators through the AI vision app. Several pre-trained networks of common architectures can be loaded and switched in a few milliseconds at runtime.
IDS NXT industrial cameras become universal inference cameras including ANN accelerators through the AI vision app. Several pre-trained networks of common architectures can be loaded and switched in a few milliseconds at runtime.

A special interpreter ensures the preparation of the ANNs together with their trained weights and defined outputs for use with the IDS NXT AI processor. If necessary, they can be optimized ("pruned"), which additionally improves the ANN speed for the desired application. FPGA-based AI acceleration enables inference times of a few milliseconds with common architectures.  So IDS NXT cameras can compete with modern desktop CPUs in terms of the accuracy and speed of AI tasks, while using much less space and energy.

The reprogrammable ANN accelerator offers advantages in terms of a future-proof solution, low recurring costs and time-to-market. AI technology is evolving so rapidly that new frameworks and architectures appear every month and can now be implemented without changing the hardware platform. Due to the fast reconfiguration of the dedicated processor, it is possible to switch between several loaded ANNs within a few milliseconds at runtime. This enables the sequential execution of different classifications using the same image data within the same application.


Artificial neural networks have already proven their benefits for the modern machine vision world. Machine object recognition and classification are two of the most important new capabilities that will enhance automation in industry and many applications in other markets. The manufacturers and system integrators of image processing components must therefore quickly demonstrate a controllable way in which this technology can be used simply and yet efficiently without expert knowledge.

At the upcoming "Vision", the leading trade show for industrial image processing in Stuttgart, IDS will present a functional prototype of an AI-based object recognition system that runs completely autonomously on an IDS NXT industrial camera. With the flexible vision app-based platform, users will be able to quickly and easily bring their prepared neural networks to the machine as a complete AI image processing system.