Edge AI architecture: The edge is the place to be for AI

AI seems to be part of every technology-related conversation lately. Although for a time AI seemed to be more hype than reality, we are probably now at an important tipping point. To cite one example of AI's ubiquity, in September 2023, I had the pleasure of attending the "AI Hardware and Edge AI" summit. Andrew Ng, one of the knowledge leaders in the AI space, opened his talk by describing AI as the "new Electricity.

According to the MIT Management Sloan School AI can increase the productivity of highly skilled workers with 40% compared to those who do not use it. That's another promising idea. Historically, we would already consider a gain of a few percent a huge improvement, but leaps like this are truly revolutionary. But how then? Why is there suddenly so much momentum and enthusiasm around AI? AI is not new; the core concepts have been around since the 1950s. Let's look at the factors driving AI's growth.

What is driving the growth of AI?

First, significant progress has been made in the field of Deep Neural Networks. The introduction of the'Transformer Model' by Ashish Vaswani in 2017 is considered a turning point in this area. The transformer-based model is more accurate than other models because it can understand the relationship between sequential elements that are far apart. Moreover, it is fast because it pays more attention to the most important parts of a sequence. These challenges hampered previous models. The transformational model has played a key role in today's powerful language models. In addition, a huge number of pre-trained models have been developed in recent years. These can be used freely. This has dramatically reduced the amount of work required to create a custom model for a specific usage scenario.

Next is the abundance of data. An incredible amount of data is produced at the edge, for example, in modern factories. In its keynote at the 2024 CES estimates Dr. Roland Busch, CEO of Siemens, that a highly automated factory now generates about 2,000 TB (equivalent to the data volume of 500,000 films) of data each month. However, only a very small portion of this data is actually used. That means there is enormous potential in harnessing this data and creating actionable insights from it.

Another important factor is the increasing computing power that AI workloads can handle. In addition to the more traditional growth of CPU processing power according to Moore's law, we are seeing the emergence of specific so-called "accelerator architectures." Examples include Neural Processing Units (NPUs) that are much more effective and powerful in processing neural networks. They are the basis for many AI models.

AI becomes known

ChatGPT's OpenAI release to the public in November 2022 is an important milestone when it comes to AI. Why? Because it gave everyone, not just researchers or professionals in technology, the opportunity to see what AI can do. It helped, in a way that is hard to overestimate, to increase awareness, interest and understanding of the concepts and potential of AI in a wide audience. This, in turn, has given many people ideas to use AI.

Companies are increasingly realizing the potential benefits of using AI across all functions of a business. At OnLogic, for example, we have identified more than 100 possible use cases across company functions where AI may be able to promote efficiency and effectiveness within the organization. That's a huge amount of opportunity, and it's important to remember that this is just the tip of the AI iceberg. Companies see a real risk of being left behind if they do not use AI to increase their competitiveness in their markets. Consequently, many companies are asking themselves whether they can afford not to invest in AI. According to Forbes 64% of companies expect AI to increase productivity. From research shows however, that only 35% of companies were using AI in 2023. Once again, it is clear that frothy heads are emerging on the AI wave.

As a result, AI is expected to have a compound annual growth rate of 37.3% from 2023 to 2030. Let's put that into perspective. Suppose you have €100 in your retirement savings plan at the beginning of 2023 and that you don't deposit anything more. If your balance grows by 37.3% per year, it will be €1,262,886 by the end of 2030, more than a factor of 12 in growth from the beginning of 2023. That's a huge growth rate! This staggering projected growth of AI means a lot of work needs to be done in terms of developing and implementing these AI solutions and the devices and infrastructure to support them.

What and where is the edge?

Okay, so we've established that AI has potential, but how does it relate to edge computing? First, let's make sure we're on the same page where edge is concerned. Below is an overview developed by the Linux Foundation and that shows the "edge-to-cloud" continuum.

Our focus is on the "user edge," which describes the hardware on factory floors. This can include anything from end-point devices such as PLCs that control production equipment, to hardware on AGVs (Automated Guided Vehicles) that move products or parts on a factory floor. The user edge also includes any industrial PCs used on the factory floor that can perform, for example, derived workloads for a machine vision system. The diagram below shows manufacturing operations with two factories. Each factory can house many edge devices that perform inference operations, for example. The user edge usually also includes a localized data center that can be used for data storage, training models, data analysis and more.

Why edge AI architecture?

Now we get to the heart of the question - why is it beneficial in many cases to run AI workloads at the edge of the network rather than relying on the cloud?

For starters, the edge is where data is created. It is where all the Operational Technology (OT) of a plant is located. It is expensive to send all data to the cloud. Remember it? In a highly automated factory, the equivalent data volume of 500,000 films is generated monthly. It would be prohibitively expensive to send all this data to the cloud and store it there. In addition to the cost of sending and storing data, there is also the cost of using computer resources in the cloud. The cloud is in high demand these days, and the cost of computing resources in the cloud is significant. And conversely, if data processing is performed at the edge, on the customer's hardware, then the only costs are the cost of the hardware and the cost of operation and maintenance. These can be further reduced by using the right industrial hardware.

In addition to the cost aspect, latency is also an important consideration. This can be problematic and limiting, depending on the usage scenario. After all, decisions often need to be made quickly on the shop floor or at the edge in general. For example, if just-produced connectors are racing down the production line at high speed, it is critical that the decision of which connector will and will not pass quality inspection be made almost immediately. If a production line relies on decisions made solely in the cloud, a disconnect is also likely to lead to production line downtime, costing money, time and valuable production throughput.

Another big advantage of an edge computing architecture is security. Many companies are reluctant when it comes to sending confidential data about their business to the cloud. They prefer to store this data on-premises to reduce the risks of a cyber attack. The merger of IT and OT increases the potential attack surface and it is important for companies to strengthen their edge computing nodes to protect their data. Many techniques, including Secure Access Service Edge (SASE) and Zero Trust, are increasingly being deployed to improve data security. However, many enterprises still prefer edge computing architectures.

Environmental considerations are an additional incentive for the use of edge computing. For example, using the edge versus the cloud reduces the overall energy profile. Data centers consume large amounts of energy, contributing to greenhouse gas emissions. In addition, energy is required to send data back and forth between the cloud and the edge. Businesses can reduce their energy costs by moving AI workloads from the cloud to the edge.

Let me give one more proof that Edge AI is becoming a reality. Let's take a look at the 2023 Gartner Hype cycle™. The image below was released in July 2023 and shows how Edge AI reaches the "Plateau of productivity" - that is, the state in which it is fully ready for implementation - within two years from that time. That said, let's take a look at how we implement AI at the edge.

Implementing AI at the edge

Suppose you've considered all the options (and the information above) and decided that an edge AI implementation is the right choice for your business. Where should you start? Basically, AI is software, but to implement that software you need the right hardware. Here are a few points to consider.

Surroundings

With edge computing, the installation environment often differs greatly between a data center or air-conditioned office. It may be hot, cold or humid, and located outside or, for example, in a blazing hot steel mill. The air may be full of dust particles, ranging from saw dust at a cabinet manufacturer to powdered aromatics on a potato chip production line. The edge consists of many places where a standard computer would not survive for long. To hold up in such harsh environments, you need industrial or ruggedized computers. These types of systems are built for harsh environments and offer operators reliability and peace of mind.

Workloads

For inference operations, computing power needs depend on the exact usage scenario, how much and how fast data needs to be processed. It may surprise some, but for many AI use scenarios, the CPU's built-in computing power is sometimes already sufficient. Processor manufacturers such as Intel® are not only increasing CPU power from generation to generation, but also adding more integrated GPU compute to the processor package. The latest generations of processors sometimes include Neural Processing Units (NPUs). As a result, more and more inference applications can be handled without an additional accelerator. If the application requires more power than the CPU package can provide, for example multiple high FPS camera feeds, industrial PCs with graphics cards or dedicated accelerators are required.

Model training

In addition to the inference workload, the models in most applications require continuous training to keep learning. This training usually requires a larger amount of computing power. This training can be handled by an edge server, and it can be located on the production floor or in a dedicated server room. Edge servers can then also be used to monitor and process data drift or perform any analysis needed for the business. See AI acceleration at the Edge for more information. This is a session, held at the AWS re:invent 2023, that focuses on various aspects of edge deployments.

Software Requirements

To deploy this type of edge computing architecture requires a number of software layers beyond the operating system layer. Edge deployments often require a large number of gateways, edge servers, inference devices, and so on. The ability to seamlessly integrate, deploy and efficiently update devices is essential. This is especially important because there are often little to no on-site technical resources. Companies with multiple locations can experience problems when system updates or deployments are out of sync. This is usually handled by edge orchestration and remote management software. AI workloads are also often run as containers. For that, you need the right software. And on top of all these layers, the AI application software for the specific use case is required.

Cloud repatriation

So we looked at how to successfully implement AI at the edge of the network. Yet companies often start with an AI solution implemented in the cloud. More and more companies are realizing the value of edge computing and are working on cloud repatriation to move computing resources back to the edge. According to ComputerWeekly.com 71% of respondents in a recent IDC survey indicated that they plan to partially or fully repatriate. The most common reasons for this intended move are cost reduction, improved performance and better security. Sound familiar?

Every business is unique, and the optimal location of computing resources depends on exactly what you are trying to do with your data. The final implementation in some cases will not be as black and white as cloud or edge, but rather a hybrid solution. Many software packages can run workloads both in the cloud and at the edge and are not tailored to the specific hardware being used. This gives users a lot of flexibility to take a "best of both worlds" approach; the computing power benefits of the cloud and the physical control of the edge.

Conclusions

Let me close with some more numbers: According to Gartner, 70% of large enterprises will have a documented strategy for edge computing by the end of 2026. In 2023, this was less than 10%. According to the same study, 50% of edge computing deployments will include machine learning by 2026, up from 5% in 2022. All of this means that the edge will continue to be the right place for businesses of all sizes in the coming years, and those who are knowledgeable about it will be the best beneficiaries.