EHNINGEN, GERMANY – Data Center
Getty Images
A decade ago, artificial intelligence was still something we mostly thought of as science fiction. First, there was machine learning, with its rudimentary propositions of data fitting, non-deterministic outcomes and recognizable algorithms. Before that, there was the cloud.
Cloud computing became a buzzword in the era of big data, before computers learned to think, when having larger data sets already allowed us to start mining for what was often then called “business intelligence.” You could make workflows more efficient, store giant volumes of data, and get dynamic, on-demand services though the Internet. The cloud was born.
Now, as AI drives rapid change, as neural nets are doing more of our thinking for us, leadership teams are … pulling workloads back out of the cloud? It’s enough to give the average company a bit of whiplash. Just a while ago, teams are adapting business processes to “cloud-native” designs. Now those cloud-native designs are many times being updated to “AI-native” systems that are on-premises, if not hybrid.
Why is this happening, and what does it mean?
Solving the Latency Problem
As engineers design AI services, they are, once again, obsessed with speed. How long will it take for a signal to get through an entire stack and make its way to the user? Well, that has to do with how the hardware is set up.
In pre-AI systems, there was, assumedly, less concern about making sure that data was fired at the user in the quickest possible way. Of course, speed has always been a metric, and engineers were always trying to figure out how to make the cloud work faster. But latency hides in many places, as you can see from this roundup posted on Medium.
Basically speaking, though, you can cut a bunch of latency by co-locating the processing of data near where it originates. This strategy has become known as “edge computing,” and – you guessed it – the anti-cloud.
In so many cases, edge design erases significant amounts of latency. But it’s not a silver bullet; you still have to consider other sources.
Hybrid Cloud
So it’s time to take everything out of the cloud? Not so fast. There’s often an appeal to a more balanced approach, as in this take from Tushar Panthari at Dev:
“By 2025, Edge AI vs Cloud AI is no longer a battle of superiority, it’s about fit,” Panthari writes. “The numbers make it clear: edge is exploding, cloud remains dominant, and hybrid is the rising default. Decision makers need to stop asking, ‘Which is better?’ and instead ask, ‘Which is better for this workload?’ If 2020–2024 was about proving AI works, the next decade will be about deploying it in the right place, at the right scale, with the right governance.”
In other words, the cloud might work better for some things, and the edge for others. But edge computing is often popular for setups where crunching the numbers close to their source of aggregation can make the system go a lot faster.
Conquering Latency
Innovative design is important in verticals like retail and manufacturing: it’s arguably even more important in systems that are mission-critical, for example, defense applications.
“The future of AI in defense lies in making intelligence capabilities truly distributed and adaptive,” Jags Kandasamy says in an interview published at Defense Daily. “We’re moving from centralized, cloud-dependent AI toward edge-native intelligence that can operate independently in contested environments.”
His company, Latent AI, helps to reduce latency for systems, with an emphasis on edge computing. With over 200,000 hours of testing runs and 12 terabytes of data, the firm has been able to really study enterprise adoption trends and their results.
In a presentation at Stanford last year, Kandasamy elaborated on some of these concepts.
“Edge AI matters because some decisions need to be made in milliseconds,” he explained. “Would you be okay if your Tesla car when it’s in FSD mode, (is) sending all the vision data back to a cloud to process, and break about seven seconds later? it’s not acceptable.”
He spoke to the principles behind considering a “re-homing” of workloads, from the cloud to the edge.
“How do you have continuity and trust?” he said. “You want things to be local. You want to be able to trust the system and be able to respond to your requests.”
Another part of Kandasamy’s appeal was to the capabilities of our handheld and wearable devices.
“Our phone in the pocket is 10 times more powerful than the computer that sent men to the moon, right?” he said. “So why can’t we bring more intelligence to the edge?”
Trusting the Machine
All of it, Kandasamy suggested, is in aid of making sure that when the chips are down, the compute will be there.
“When the cloud goes dark, the mission needs to continue,” he said, in closing.
In terms of defense applications, Kandasamy’s company has pioneered the Ruggedized AI Toolkit (RTK) and the Latent AI Efficient Inference Platform (LEIP). What are these? I asked ChatGPT.
“The Ruggedized AI Toolkit (RTK) is Latent AI’s field-deployable edge-AI system,” ChatGPT responded after “thinking” for 8 seconds. “Mil-spec hardware plus software that lets non-expert warfighters retrain, update, and redeploy AI models on-site, offline, in contested or bandwidth-denied environments, using a simplified, no-code workflow tightly integrated with Latent AI’s LEIP platform. … RTK couples a rugged Jetson box to a phone-like UI so non-experts can run, adapt, and redeploy edge AI models on drones/vehicles entirely in the field, without the cloud or a data-science team.”
As for the question, what is a Jetson box, the model had this to say:
“A “Jetson box” is just shorthand for a small NVIDIA Jetson computer packaged in a rugged case: NVIDIA Jetson = a family of tiny, power-efficient GPUs made to run AI models at the edge (on robots, drones, cameras, vehicles, etc.). The “box” = the Jetson module plus memory, storage, ports (USB, Ethernet, camera inputs), power management, and a protective enclosure—often mil-spec or industrial.”
I liked this little bit:
“So when people say “Jetson box,” think: a tough little AI PC that can sit on a robot/drone/vehicle and run neural nets locally, without the cloud.”
All of this serves to explain some of the thinking behind powerful edge systems, and why, after years of prodding companies to get with the program and use the cloud, we’re often coming back to an on-prem build. To be sure, there’s some nuance here, and business leaders have some big questions to answer. Stay tuned.
