Aug 29, 2004 (04:08 PM EDT)
Primer: BI Modeling And Optimization
Read the Original Article at InformationWeek
At the apex of Business Intelligence is BI-based Modeling and Optimization (BIMO), a set of technologies for delivering better decision-making options to executives and managers. BI-based Modeling and Optimization combines the rich processed information provided from BI tools and data marts and feeds them into modeling and optimization tools to arrive at the best possible options or courses of action in a problem situation. These decision situations can be operational -- determining the best way to run a refinery given record breaking oil prices, for example. Or the problem can be strategic -- determining the best portfolio of capital projects given BI-based market projections and costing estimates.
The key attractions of BIMO are that it can deliver outstanding return on investment both in day-to-day operations as well as one-time decisions. In addition, BIMO methods really help to refine and discipline the thinking and processes around decision making. In fact, BIMO tends to produce better returns in high-flux, high-change periods both operationally and strategically. Unfortunately, many BIMO systems are quietly hidden away, as corporate executives do not want to reveal their competitive advantage-producing systems. Think of the optimizers in supply chain and production scheduling packages: In some cases the optimizing tools are so elegantly embedded that staff do not know the power of the tool harnessed for their benefit. Rightfully, users view the tool as a total BI-based optimizing system.
Traditionally, there have been three major risks associated with delivering consistent, high-quality BIMO results. First, the underlying disciplines of linear programming, network modeling and non-linear optimization are not only difficult and complex but also different in methodology among themselves. Second, getting expertise can be expensive because, as often as not, linear programming experts cannot help with non-linear math optimization problems. Sometimes the data and algorithms can be shared; but the underlying models and solution methods are distinct, even disjointed. (Methods used for linear programming are very different among themselves as well as from those used for non-linear optimization.)
But the most challenging problem for BIMO is marshaling together all the data sources necessary to support the models. What can make BIMO data-handling even more demanding is the fact that BIMO inputs frequently have to go through several stages of processing as they filter up the BI stack: ETL, consolidation, analysis and forecasting. Fortunately, the quiet revolution in BI -- over the past 10 years BI tools have become easier to use, better integrated, and broadly available -- has provided improved data warehousing, ETL, and analytics at lower cost and with greater ease of use. In addition, new technologies such as XML, business process management and Web services further ease the data-marshaling burden. The result is that BIMO weathered the dot-com-induced IT slump with a solid 3-9% annual growth rate and diversified new offerings. So let's take a look at the BIMO vendors.
The Optimization Players
BIMO vendors have traditionally divided themselves along two lines -- server-based versus desktop-based tools. They further divide among solution methods: linear programming, quadratic methods, discrete or network optimization, and non-linear programming. But in the past decade the market has evolved such that there are now three categories: solvers, modelers and component-makers.
The solver products include tools such as Dash's Xpress-MP, Sunset System's XA, ILOG's CPLEX, Mosek's Solver and over a dozen others. These vendors concentrate on providing solver-engines that implement the advanced mathematics required for optimization solutions. These algorithms are quite refined and still evolving (see the sidebar below on optimizers) -- making larger problems solvable in shorter periods of time. Many of these companies sell directly to end users, but they also market their products to the two other BIMO vendor-types.
The modelers like Lindo Inc, Maximal Software, Paragon Decision Technology, ILOG and others create front-end modeling systems that make BIMO solving much more approachable. Modeling tools considerably simplify the process of setting up a problem, refining it, and then reporting the results. For example, Paragon's AIMMS product adds data connection wizards for XML, SQL, and XLS data sources, while ILOG's OPLScript simplifies linking the stages as more solutions become multi-step.
As noted in the sidebar on optimizing, systems with hundreds of thousands of variables and equations are not uncommon. Marshalling the data for these equations is a data mart problem of no small proportion. Producing the results is a report-writing problem of equally taxing proportions. Modeling tools simplify the set-up and refining of a particular optimizing solution. The problems often require many iterations as variables, equations, and constraints are added, changed and/or dropped. Modeling systems take feeds from data marts and warehouses, OLAP systems, and various accounting and scorecard systems. Some modeling tools have built-in mini-ETL systems able to read CSV, XML, XLS, and other analytic and data feeds.
The idea behind modeling systems is not only to set up problems quickly, but also to respond to the inevitable "what-if" or "changed condition" situations. The key to the new BIMO proliferation is that the modeling systems allow organizations to respond to change with a new set of best-policy recommendations. Many times, the new optimum does not generate enough return to make the internal changes required. Executives then know "steady-as-you-go" is the best policy.
Other times, however, the solution points to changes, and then the reporting capabilities of the modeling system produce template reports that detail the exact changes required. These reports can be delivered in a variety of formats including PDF, XML, SQL scripts, and a variety of other formats. The whole process of results-delivery has become so important that it has spawned the third BIMO category – embedded or component BIMO systems.
BIMO vendors like ILOG, Frontline, Tomlab and others supply optimizing components that are embedded by diverse ISVs ranging from OptiChrome and Arbed/Sidmar to PeopleSoft, Manugistics, i2 and other software vendors. These components help simplify or customize any one of the BIMO input, processing or output stages. Often users do not know that under the hood they have a powerful optimizing system that is determining the best possible project portfolio or crew schedule. In effect, more BIMO technology is becoming transparently embedded in more enterprise systems.
Talking to BIMO vendors about their customer successes is like getting a glimpse of the Sultan's harem. Customers are very reluctant to discuss their trade secrets. Nonetheless, one can see by the steady expansion of the BIMO market and the flourishing of embedded systems that BIMO is doing very well. And this writer, having worked in the petroleum-refining and transportation/distribution sectors, can testify that BIMO applications are critical success factors in both sectors. As markets become more competitive, watch for BIMO solutions to lead the way in a wide range of industries and applications.
Sidebar: Optimization Methods
Optimization is all about finding the best possible solution given a stated goal (maximizing profits, for example) but subject to many constraints (say, plant capacities, resource prices, pollution limits, etc,). Optimization methods are taught in most MBA programs for three reasons. First, in many business sectors, optimizing methods have become a vital part of both operational and strategic policies. Second, the discipline of optimizing methods with goals, constraints, feasible solutions, and modeling behavior is vital to thinking about business problems in general. Finally, the tools have become so flexible and the hardware and software so inexpensive, that BIMO methods now are cropping up in an ever-wider range of business situations.
Optimization's basic methods are drawn from linear algebra, networks, and calculus. The number and size of problems (literally hundreds of thousands of variables and equations) that can be solved with linear programming are remarkable. Linear programming is the first and basic optimizing method. The difference between linear equations (2x + 3y – 4z) and non-linear equations (2x * 3log(y)/min(4z)) is that linear systems have simple boundaries. In contrast, non-linear solution spaces are just that -- so complex that identifying the global optima is quite tricky. The result is that linear-programming solvers are so fast and reliable compared to non-linear ones that BIMO developers have taken to using them to solve non-linear problems. The trade-off is the expanded size of the problem versus the speed and reliability of the linear methods.
Anyone who has specialized in math knows there are a whole range of problems when solutions are constrained to discrete or integer values. In linear programming programs, many solutions have fractional values: 1.3967 etc. But fractions like 1.3967 can lead to ambiguous results (do you round up or round down, etc.?). So when integer or discrete values are required, an entirely new mathematical space is opened, with correspondingly unique solution methods. Finally, complex constraints, many of them contingent on previous results, further add to the complexity of BIMO solvers. So it's no wonder there's a thriving community, mostly drawn from academia, exploring the latest in optimization methods. Again, it should be no surprise that developing solver engines is a craft on its own. So the market splits between creating solver engines, then embedding those engines in either general modeling tools or into customized, often industry-specific solution packages.
Jacques Surveyer is a consultant and trainer; see some of his tips and tutorials at theOpenSourcery.com