Not many huge things come out of Boise and that is the very reason I live here. But something that should disrupt the semiconductor industry (the world in fact) is Micron’s Automata. It’s been occupying much of my thoughts since it was announced a few weeks ago. The Idaho Statesman’s front-page headline today is Big Data, Big Move for Micron with New Processor.
In a nutshell this is hybrid memory that is capable of having a large number of custom operations programmed directly on it that can be run in massively parallel fashion. From a software developer’s point of view, the parallelism is the key value. But from an IT guy’s point of view, not needing to depend on a trip to the CPU and back for every operation drastically cuts out a ton of processing overhead.
When I’ve told colleagues about this, many see this as just faster throughput which would be beneficial for a class of software applications – niche. This is like the dawn of the jet airplane; it’s not just a faster way to get from New York to Los Angeles. Automata isn’t ultimately a niche product only useful for research organizations or the NSA. We all live in a complex world where outcomes are dependent on things happening in massively parallel fashion.
Really, the way we use computers today is what is niche. They are useful for a subset of relatively simple (still within the realm of “complicated” as opposed to “complex”) problems that are deterministic, well-defined, limited in volume, and pretty linear. We’ve taken these well-defined applications, propagated and socialized them to severe extents, and forgot that these are really reduced, simplified versions of the real world. Then we consider the applications attempting to embrace real-world complexity as niche.
It will be incredibly interesting to see what players jump onto Automata first. Probably Hadoop with a retrofitted Automata version. But it could be Microsoft’s CEP (complex event processing) offering, StreamInsight, something from Oracle, maybe IBM’s ILog, or a yet unknown startup.
Regarding that yet unknown startup, it probably won’t be SCL. Ever since I wrote the SCL language, interpreter, base test-harness UI (Map Rock is in part a rich SCL UI), I’ve taken several approaches to writing a bad-ass “SCL Server”. They actually worked fairly well, but I knew I needed superior parallelism to what was offered by the current methods, lots of servers with lots of CPUs, in order to achieve what I really envisioned. Then Hadoop and all the associated Big Data things came along. Anyway, weighing everything, I decided to aim way downstream (Map Rock).
However, keep in mind that the kind of massively parallel processing that Hadoop can do is different from the sort I’m thinking of. I’m thinking of Hadoop as a homogeneous MPP where a long list of like tasks is divided amongst like processors. The sort I’m thinking of is a much more compelling heterogeneous sort of MPP where thousands of points of view work on streams of data in order to promote robust recognition.
The key question is, Can Automata-based Big Data servers resolve problems that currently are just too expensive (in terms of cost and time) to resolve today? Apparently it can, according to some of the sparse literature on Automata (see slide 12, titled Breakthrough Performance, of this file). If so, that’s disruptive.
Micron readily admits that the concepts of Automata-like technology has been around for a while. Micron showed a lot of courage sticking to the development of something so different instead of the sort of incremental improvements we’ve been seeing lately. It would be unfair to call technologies like multicore processors and NUMA merely incremental, but Automata is truly fresh and timely.
On a light note, related to not much coming out of Boise, on my flight back to Boise from my client site the other day, I overheard something very funny in a conversation behind me. This woman was talking to a guy about how people react when we say we’re from Idaho. Of course, it’s always something about potatoes. She said she tells people she’s the Bubba (from Forest Gump) of potatoes; French fried, au gratin, baked, mashed, … Then others started throwing in their potato styles too; twice-baked, potato skins, potato pancakes …