SAN JOSE, Calif. – IBM is applying technology developed for large server clusters to cloud computing systems. The move is another example of how the company is focusing its efforts on the web datacenters that are the fastest growing slice of today's server market
IBM sees a wide array of applications for its distributed job scheduler, now the basis for a U.S. Patent 8,645,745. It described possible uses including trading, retail, online gaming, and medical research services.
"There are a wide variety of job scheduler techniques to optimize resource usage in a cluster, for example by scheduling jobs in parallel," said Eric Barsness, one of the inventors for the IBM patent. "We flipped that model to have multiple job schedulers running in parallel, each managing a subset of jobs," he wrote in an email exchange.
The technique aims to "help large clusters handle a large volume of concurrent requests for resources... eliminating a potential bottleneck," Barsness said. "You might not use this in a small cluster or one that isn't very busy, however, because it could add unnecessary complexity.
"A downside is that there isn't one scheduler that sees the big picture, which means some jobs with very large resource needs could take a bit longer to be scheduled," he added. "That is because multiple schedulers might have to cooperate together to find enough resources to run the job."
Because the technique was conceived to run on a single system, it also faces challenges prioritizing jobs when running across computers from multiple companies. "For example, when all the resources are used you would have to decide which company's jobs get priority to run," he wrote.
The original target for the technology was a big academic or government supercomputer such as IBM's Blue Gene. "We're still in the preliminary stages of deciding how and where to integrate it with our cloud offerings," Barsness wrote.
The effort parallels IBM's recent announcement it will make open-source its Power 8 processor through the Open Power Foundation. The move similarly targets a rising tide of massive datacenters for web services.
— Rick Merritt, Silicon Valley Bureau Chief, EE Times