“Game changer” IDS NXT
Interview with an engineer about an image processing application with AI - easy to use and understandable
“I am an engineer no software developer and I can say now that I have done image processing with AI”, says Marco Ullrich from Kempten University of Applied Sciences. As research associate at the Institute for Production and Informatics, he supervised a group of students who worked on the development of a mill-playing robot with AI in the course of their master's degree.
Image analysis was realised by IDS NXT - the all-in-one system for the use of intelligent cameras. In this interview you will learn more about their challenges, experiences and the final solution.
What was the exact task of the project?
The project's goal was to develop robots that could play the German board game "Mühle" (Mill) geographically independent. This means that one robot cell in Kempten (Germany) and another in Salzburg (Austria) could play against each other. We achieved this by using a vision system to detect changes on the game board. The intelligent IDS NXT camera was an essential component of this system.
What challenges did your team face during the project work?
The initial situation was to replace an existing Raspberry Pi camera with an industrial camera. We also needed to select a suitable image processing algorithm, lighting, and camera installation position. Additionally, we had to find a method to equalise the image for an HMI (Human Machine Interface that allows users to communicate with machines), as we installed the camera at an angle but required a top-down view for the user interface. Overall, the integration of the camera into the existing robot system was a significant challenge.
What was the specific task of the vision system in your application?
The vision system's task was to detect changes on the game board reliably and safely. As engineers and not software developers, we also intended a system that is user-friendly and easy to use. The camera needed to be suitable for industrial use and ideally innovative or state-of-the-art. In addition, we wanted the image processing to be done using artificial intelligence. However, since our target group was not necessarily composed of AI experts, we needed to make sure also the AI application was easy to handle and understandable.
Why did you choose IDS camera technology for the project?
The IDS NXT system was ideal for our project because it met all our requirements. The first and most important factor was the IDS focus on reducing the barrier of using artificial intelligence in real applications, making it user-friendly and accessible to engineers like us who are not software developers. Furthermore, we did not require additional computing resources for image processing, making it easy to integrate into our existing system.
Everything runs on the camera itself, so no additional processing unit is required.
We want to offer our users the best vision experience. This is why we focus on easy to use camera technology with the motto “it's so easy”. Could you explain why the IDS NXT system was easy to use?
As I previously mentioned, I am an engineer, not a software developer. However, I was able to work with AI and neural networks for image processing through the IDS NXT system. It does not require any large investments in computing resources for image processing since everything happens on the camera device itself, eliminating the need for any additional processing unit. This made it incredibly easy to integrate the system into our existing setup.
Can you tell us about the interface used by the IDS camera to communicate with your project environment?
We have explored several options to communicate with the IDS camera, including the REP interface and the cockpit software. However, we ultimately decided to leverage the OPC UA protocol because it is the standard communication method in our system. We connected the camera to our PLC via an Ethernet cable through a switch.
How did you get on with the app approach of the IDS NXT cameras? What was your experience?
We tested two vision apps, the "Classifier“ and the „Object Detector". With both vision apps, we were able to achieve success rates of (almost) 100 percent with enough images. The Object Detector had a slightly lower success rate and was a bit more complex than the Classifier, but I do not see this as a negative point. Overall, the app approach is very intuitive, especially if you are not a programmer and are not familiar with AI algorithms.
One of your resolutions was that a better analysis of requirement case beforehand would have saved development time or would have led you to use other methods right away. What do you think was the reason for that? Is working with AI based image processing that different?
That resolution had nothing to do with AI in our use case. The reason we would do things differently today is that we underestimated the importance of the right approach or way to start the project. It is important to note that there is much more to a vision system than just the camera and an algorithm. We were not really experts, so we did not realise how crucial it is to evaluate the right camera position, lighting, and other factors, which also have a significant influence on performance.
In our case, the camera position was a complex task because the camera system we used before was right above the game field. The IDS camera is angled for various reasons, which brings a lot of challenges.
Has the implementation of the IDS vision system resulted in an advantage for token detection and process monitoring through its AI evaluation capabilities?
In our experience, it has provided an advantage. You have to remember that before the IDS camera, we utilised a Raspberry Pi camera with image processing set up in Python, which had limited usability. Therefore, the IDS camera's benefit lies in its seamless integration and ease of use without requiring extensive high-level language expertise.
You had mentioned earlier that the code reading function and multi-step image evaluation would be valuable enhancements to the application. Well, I have some exciting news. The IDS NXT system now supports both functions with the block-based editor. Would you use these tools to plan the application differently if you were doing the project again today?
That's really great news because the suggestion for those features actually came from the students. It is impressive to see that IDS was able to incorporate our suggestions so quickly. Yes, if we were to do the project again today, we would definitely do things differently since many things have changed.
Currently, we are planning to integrate the same IDS camera into our industrial plant at the institute in Sonthofen, and during that process, I plan to test the new functions and features. Furthermore, we intend to continue collaborating with your team on various research projects. I would be happy to keep working together on exciting topics in the future.
We have reached the end of this interview. Marco, thank you for sharing your valuable insights and future plans with us. We look forward to continuing our collaboration with you and wish you further success in your research endeavours.
Further information
If you want to learn more, you can find this interview and other videos about cameras and image processing technology on the IDS Vision Channel or on YouTube.