Showing posts with label Business Intelligence. Show all posts
Showing posts with label Business Intelligence. Show all posts

Sunday, March 9, 2008

BI applications

BI applications include the activities of decision support systems, query and reporting, online analytical processing (OLAP), statistical analysis, forecasting, and data mining.

Business intelligence applications can be:

  • Mission-critical and integral to an enterprise's operations or occasional to meet a special requirement
  • Enterprise-wide or local to one division, department, or project
  • Centrally initiated or driven by user demand

BI with Microsoft products

Introduction

Microsoft® SQL Server™ 2005 is a complete business intelligence (BI) platform that provides the features, tools, and functionality to build both classic and innovative kinds of analytical applications. This paper provides an introduction to the tools that you will use to build an analytical application, and highlights new functionality that makes it easier than ever to build and manage complex BI systems.

The SQL Server 2005 Business Intelligence toolset delivers end-to-end BI application integration,

Design:
Business Intelligence Development Studio is the first integrated development environment designed for the business intelligence system developer. Built on Visual Studio® 2005, the Business Intelligence Development Studio delivers a rich, integrated, professional development platform for BI system developers. Debugging, source control, and script and code development are available for all components of the BI application.



Integrate:
Integration Services has been rewritten to perform complex data integration, transformation, and synthesis at high speed for very large data volumes. The Business Intelligence Development Studio makes building and debugging packages positively fun. Integration Services, Analysis Services, and Reporting Services work together to present a seamless view of data from heterogeneous sources.



Store:
SQL Server 2005 blurs the lines between relational and multidimensional databases. You can store data in the relational database, in the multidimensional database, or use the new Proactive Cache feature to get the best of both worlds.



Analyze:
Data Mining in SQL Server has always been easy to use. Now it's even better with the addition of important new algorithms, including Association Rules, Time Series, Regression Trees, Sequence Clustering, Neural Nets, and Naïve Bayes. Important new analytical capabilities have been added to Analysis Services cubes as well: Key Performance Indicator framework, MDX scripts, and other built-in advanced business analytics. The Reporting Services report delivery and management framework enables easy distribution of complex analytics to the widest possible audience.



Report:
Reporting Services extends the Microsoft Business Intelligence platform to reach the business user who needs to consume the analysis. Reporting Services is an enterprise managed reporting environment, embedded and managed using web services. Reports can be personalized and delivered in a variety of formats and with a range of interactivity and printing options. Complex analyses can reach a broad audience through the distribution of reports as a data source for downstream business intelligence. New in SQL Server 2005 is Report Builder. Report Builder provides for self service ad hoc reporting by the end user. Ad hoc query and analysis tools from Microsoft and their partners will continue to be a popular choice for accessing data in Analysis Services and relational databases.



Manage:
The SQL Server Management Studio integrates the management of all SQL Server 2005 components. Through Management Studio, BI platform components gain enhanced scalability, reliability, availability, and programmability. These enhancements provide significant benefits to the business intelligence practitioner.

Business Intelligence and Data Warehousing in SQL Server 2005

Microsoft® SQL Server™ 2005 is a complete business intelligence (BI) platform that provides the features, tools, and functionality to build both classic and innovative kinds of analytical applications. This paper provides an introduction to the tools that you will use to build an analytical application, and highlights new functionality that makes it easier than ever to build and manage complex BI systems.

The following table presents an overview of the components of a business intelligence system, and the corresponding Microsoft SQL Server 2000 and SQL Server 2005 components.

Component

SQL Server 2000

SQL Server 2005

Extract, transformation, and load

Data Transformation Services (DTS)

SQL Server 2005 Integration Services

Relational data warehouse

SQL Server 2000 relational database

SQL Server 2005 relational database

Multidimensional database

SQL Server 2000 Analysis Services

SQL Server 2005 Analysis Services

Data mining

SQL Server 2000 Analysis Services

SQL Server 2005 Analysis Services

Managed reporting

SQL Server 2000 Reporting Services

SQL Server 2005 Reporting Services

Ad hoc reporting


SQL Server 2005 Reporting Services

Ad hoc query and analysis

Microsoft Office products (Excel, Office Web Components, Data Analyzer, SharePoint Portal)

Microsoft Office products (Excel, Office Web Components, Data Analyzer, SharePoint Portal)

Database development tools

SQL Server 2000 Enterprise Manager, Analysis Manager, Query Analyzer, various other

SQL Server 2005 Business Intelligence Development Studio (New!)

Database management tools

Enterprise Manager, Analysis Manager

SQL Server 2005 SQL Server Management Studio (New!)


Two components are new for SQL Server 2005: SQL Server Management Studio and SQL Server Business Intelligence Development Studio. The other primary BI components – Integration Services, Analysis Services OLAP, Analysis Services Data Mining, and Reporting Services – are substantially different and improved in SQL Server 2005. The SQL Server 2005 relational database contains several significant new features. Although the Microsoft Office query and portal tools are not part of SQL Server, the current releases will continue to work with SQL Server 2005. The BI functionality in Microsoft Office will evolve with the Office product release cycle.

Business Intelligence Applications

BI systems beneficiaries include a wider and wider group of users starting from specialists in controlling, financial reporting and finance, through salespeople, up to members of the board. Sectors that use BI systems most frequently include trading companies, insurance companies, banks and a financial sector, telecommunications and manufacturing companies,

Retail industry

  • Forecasting. Using scanning data to forecast demand and based on the forecast, to define inventory requirements more accurately
  • Ordering and replenishment. Using information to make faster decisions about items to order and to determine optimum quantities
  • Marketing. Providing analyzes of customer transactions (what is selling, who is buying)
  • Merchandising. Defining the right merchandise for the market at any point in time, planning store level, refine inventory
  • Distribution and logistics. Helping distribution centers manage increased volumes. Can use advance shipment information to schedule and consolidate inbound and outbound freight
  • Transportation management. Developing optimal load consolidation plans and routing schedules
  • Inventory planning. Helping identify the inventory needed level, ensure a given grade of service

Insurance

  • Claims and premium analysis. The ability to analyze detailed claims and premium history by product, policy, claim type, and other specifics
  • Customer analysis. Analyze client needs and product usage patterns, develop marketing programs on client characteristics, conduct risk analysis, improving client service
  • Risk analysis. Identify high-risk market segments and opportunities in specific segments, relate market segments, reduce frequency of claims


Banking, finance and securities

  • Customer profitability analysis. Determinate the overall profitability of individual customer, current and long term, provide the basis for high-profit sales and relationship banking, maximize sales to high-value customers, reduce costs to low-value customers, provide the means to maximize profitability of new products and services
  • Credit management. Establish patterns of credit problem progression by customers class and type, warn customers to avoid credit problems, to manage credit limits, evaluate of the bank’s credit portfolio, reduce credit losses
  • Branch sales. Improve customer service and account selling, facilitate cross selling, improve customer support, strengthen customer loyalty

Telecommunications

  • Customer profiling and segmentation. Determine high-profit product profiles and customer segments, provide detailed, integrated customer profiles, develop of individualized frequent-caller programs, determine future customer needs
  • Customer demand forecasting. Forecast future product needs or service activity, provide basis for churn analysis and control for improving customer retention

Manufacturing industry

  • Sales. Provide analyzes of customer-specific transaction data
  • Forecasting. Forecast demand, define inventory requirements
  • Ordering and replenishment. Order optimum quantities of items
  • Purchasing. Helping distribution centers manage increased volumes.
  • Distribution and logistics. Can use advance shipment information to schedule and consolidate inbound and outbound freight
  • Transportation management. Developing optimal load consolidation plans and routing schedules
  • Inventory planning. Identify the inventory level needed, ensure a given grade of service

Thursday, March 6, 2008

Introduction to BI

We live in the information economy. Knowledge is the power. Information is the lifeblood of companies. It’s clear that this idea contains a lot of truth. Smart use of information and information technology can be one of the most effective ways to gain a competitive advantage and thrive in times of change with dynamic business environment. Also most of the executives find that they and others in their organizations fall far short of where they ought to be with regard to having optimal capabilities for processing the information.



People in most organizations are overwhelmed with ever-increasing mountains of data, yet most have difficulty transforming data into meaningful information and actionable insights.

Nearly everything people do generates data these days, and organizations are capturing that data by the gigabyte. Data is collected because people think information is an asset, so it seems good to have a lot of it. But data is only potential information. Few organizations get high grades for extracting value from the data they collect, so instead of rich insights, most organizations get stuck with ‘data vaults.’ Turning raw data into
useful information is not a trivial task, but without that capability, decision-makers fail to gain the potential insights and benefit that could help them be more successful.

To overcome those issues with regard, the managing data and to refine the raw data into reports and analytical summaries different kinds of systems come to the market. Mainly those systems are kinds of ERPs, CRMs and SCMs etc. These management information systems are doing great job for business community by planning, managing data and providing reports. But these system unable to provide complex reports such as cross functional analysis, also multi dimension view of information. But due sophistication of business process and dynamic environment business community needs such complex analysis of data.



















The propose solution to overcome those issue Business Intelligence system.BI enable quickly, effectively, and economically get access to, analyze, report on, and share the information need to achieve corporate objectives. Business intelligence usually refers to the information that is available for the enterprise to make decisions on. A data warehousing (or data mart)system is the backend, or the infrastructural, component for achieving business intelligence. Business intelligence also includes the insight gained from doing data mining analysis, as well as unstructured data for content management systems.

BI systems use complex queries and stored data in multi dimension cubes rather than tables. BI systems can provide valuable intelligences, rather than information to the business community. Hence they can capitalize on opportunities before others do and get competitive advantages. By a BI system business organization can fulfill the information gap.

Business Intelligence

  1. Business intelligence (BI) is a business management term that dates to 1958. It refers to applications and technologies that are used to gather, provide access to, and analyze data and information about company operations. Business intelligence systems can help companies have a more comprehensive knowledge of the factors affecting their business, such as metrics on sales, production, and internal operations, and they can help companies to make better business decisions. Business Intelligence should not be confused with competitive intelligence, which is a separate management concept.

  2. Most companies collect a large amount of data from their business operations. To keep track of that information, a business and would need to use a wide range of software programs , such as Excel, Access and different database applications for various departments throughout their organization. Using multiple software programs makes it difficult to retrieve information in a timely manner and to perform analysis of the data.

    The term Business Intelligence (BI) represents the tools and systems that play a key role in the strategic planning process of the corporation. These systems allow a company to gather, store, access and analyze corporate data to aid in decision-making. Generally these systems will illustrate business intelligence in the areas of customer profiling, customer support, market research, market segmentation, product profitability, statistical analysis, and inventory and distribution analysis to name a few.

Business Intelligence Systems

Efforts undertaken to develop BI systems have resulted in many business solutions that allow for effective support of manager’s work. Practice shows that the most significant business effects are obtained while using the following analyzes offered by the BI systems,

  • analysis that supports cross selling and up selling;
  • customer segmentation and profiling;
  • analysis of parameters importance;
  • survival time analysis;
  • analysis of customer loyalty and customer switching to competition;
  • credit scoring;
  • fraud detection;
  • logistics optimizations;
  • forecasting of strategic business processes development;
  • web mining (analysis and assessment of the Internet services performance); and
  • web-farming (analysis of the Internet content)


Analysis that Supports Cross Selling and up Selling

Marketing techniques of cross selling or up selling involve selling products to specific customers taking their previous purchases into consideration. Cross/up selling increases customer’s trust in the company they deal with, and reduces the risk of customer’s switching to competition. It leads to a remarkable increase in company’s incomes and customer loyalty level. Data mining model helps to select marketing campaign objectives optimally and, what is more, show the best
cross/up selling offers for customers in such a way that they correspond with customers’ present needs. There are many advanced methods that are used to find interdependencies between purchased products. One of them - Market Basket Analysis – provides knowledge on what kind of services and products should be sold together in sets or which set should be recommended to a particular customer. Using classification models to select customers who are the most susceptible to a particular offer is another practical application of the discussed solution. It allows to direct marketing activities correctly and – as a result – to reduce costs of the campaign while simultaneously increasing its effectiveness.


Customer Segmentation and Profiling

Customer segmentation and profiling is based on grouping customers in some homogeneous segments. BI systems enable both descriptive and predictive segmentation. Within descriptive segmentation the following segmentations are carried out:

• demographic segmentation (on the basis of the data including customer’s income, age,
sex, education, marital status, ethnic group, religion, etc.);
• behavioral segmentation (on the basis of the data including frequency of shopping,
amount and sort of purchased products, etc.); and
• motivational segmentation (on the basis of variables that describe reasons of customers’
purchases – this kind of data usually come from questionnaires and surveys carried out).

Subsequently, predictive segmentation is useful when it is necessary to distinguish ‘good’ customers from the ‘bad’ ones. At the very beginning, a variable that describes ‘good’ customers is determined (e.g. on the basis of total shopping they have done so far), and then, other variables that greatly influence the initial variable are determined. Such analyzes allow to create a specific approach to a particular segment of customers, and this approach is supported by dynamic updating of segmentation and analyzes of customers’ migration between segments. Segmentation and profiling of customers together with identification of potential cross/up selling offers and testing of different hypotheses enable to create a customized offer that enjoys huge potential of meeting future, new and loyal customers’ needs. Segmentation and profiling of customers provide some knowledge that is useful while designing new products and addressing marketing campaigns appropriately, as well. They allow for much more individualized customer service and optimization of marketing activities and sales, thus deriving profits from data concerning customers.

Analysis of Parameters Importance

Analysis of parameters importance allows for determination of the most important (from the perspective of company’s benefits) variables that describes products, processes and customers in the situation when there are different variables that describe analyzed objects. Knowledge obtained this way is used to identify directions to be taken while perfecting products and customer service, and planning marketing actions, etc. The Bivariate statistical analysis, stepwise regression algorithm or artificial neuronal networks are mainly used in this case.

Survival Time Analysis

Survival time analysis evaluates customer’s survival time length and a possibility that they leave during that time (leaving is understood as customer’s switching to other supplier of a particular product). The analysis describes a distribution of survival time for individuals of a given population, monitors strength of other parameters impact on the expected survival time, and additionally, it enables to compare distributions of survival time between different sub-populations. Taking advantage of this method, a company may be given an invaluable insight into customer behavior and find some ways to prolong customer’s survival time.


Analysis of Customer Loyalty and Customer Switching to Competition

Analysis of customer loyalty usually concerns four categories: time of co-operation, amount (volume) of co-operation, closeness of co-operation and quality of co-operation. It is strictly related to analyzes of customer’s switching to competition. That results in identifying customers who are inclined to leave a company and join competition. Discovery of factors that result in switching to competition enables a company to direct – appropriately - its activities that aim at retaining customers. Moreover, distinguishing groups of customers characterized by different risk levels of leaving allows for construction of effective loyalty programs and more attention paid to loyal customers.


Credit Scoring

Credit scoring models enable to determine financial risk that is related to particular customers. Such a process may be performed at the very moment a contract with a customer is concluded, and it is based on the data that come from application forms provided by a customer subject to analysis. Appropriate dealing with customers who are characterised by high risk of stopping payments makes it possible to reduce losses effectively. Credit scoring finds its application in, inter alias, banking (cash loans, assessment and tolerance of late payments) and in many other sectors related, for instance, to renting or leasing property and machinery. A good example of a credit scoring application may be also provided by contracts concluded to render telecommunications services connected with selling cellular phones. Credit scoring may be performed according to different models. Correct selection of the models depends on the analysis objective and specifics of the analyzed data:

•application scoring – used in case of new customers; information on them is available
only on the basis of the completed application forms;
• behavioral scoring – paying attention to additional information on customers’ track records;
it predicts customers’ future behavior; and
• profit scoring – expanding of the basic scoring model; it pays attention not only to probability of paying credits back by customers, but also helps to assess what sort of profit may be expected as a result of co-operation with a particular customer; it is a more sophisticated model because it considers several additional economic factors.


Fraud Detection

Fraud detection is a well-tried and incredibly efficient method due to which a company may save vast amounts of money, and keep good relations with customers. Fraud detection means identification of suspicious transfers, orders and other illegal activities that target a company in question. Fraud detection models may be divided into application assessment and behavioral assessment. The former is used to detect suspicious customers at the early stage of signing a contract with a company in question, and is based on data derived from submitted applications. However, the latter is formulated on the basis of all data gathered during ‘lifetime’ of customer’s activity including, inter alias, transactional data, use of services or performance track record. Fraud detection is frequently applied in order to prevent credit card frauds (e.g. Internet transaction frauds, telemarketing frauds or identity thefts), breaching of computer systems security, ‘money laundering’, telecommunications frauds, etc.


Logistics Optimizations

Logistics optimization problem involves offering the best possible plan of logistics activities (including transportation or distribution), simultaneously taking already known limitations and available potential into consideration. Wrongly prepared plan of logistics optimizations may result in huge delays of e.g. production or distribution that would consequently bring about a necessity for bearing higher costs - thus decreasing potential profits to be obtained. Employing advanced data mining techniques, it is possible to show the best available solution for actual and complex optimization problems. Quality of such solutions is usually much higher than the quality offered by traditional solutions of optimization methods.


Forecasting of Strategic Business Processes Development

Abilities to understand and forecast development of strategic business processes make up a foundation of the correct planning of any business activity. That is why, modeling of multidimensional forecasts based on historical, present and anticipated data is so important. Analyzes of time series make it possible to identify and analyze hidden trends and fluctuations (e.g. in marketing data or sales data). Taking seasonal nature and other marketing factors into consideration, it is possible to foresee potential behavior of market or customers, developments of customer expectations and customers’ purchases.


Web Mining

Analysis and assessment of the performance of Internet services (web mining) helps to obtain knowledge who uses services, when, why and how. Application of advance classification, grouping, matching and regression models with reference to data on service performance and its users (‘log’ files), business data (databases), and data on users’ experience with the service (questionnaires and interviews) allow to perform identification of customers and their preferences. Analysis and assessment of the performance of Internet services is limited to discovering and analyzing information that is stored in the service (web content mining), discovering and analyzing patterns of using the service by its users (web usage mining), and analyzing service structure (web structure mining). This way, valuable, dialectic knowledge is acquired – the knowledge on company’s offer attractiveness and its formulation in such a way that it corresponds to individual needs of particular customers. As a result, it is possible to customize service, automate navigation, shape pricing and promotional strategy and develop ‘intelligent’ e-business.


Web-Farming

Web-farming involves systematic analyses of the Internet content in order to provide a company with themes and issues of fundamental nature for company performance. Internet is more and more frequently treated as a powerful resource of important economic information on potential customers, suppliers and competitors, information on the latest market bargains, technological trends and development of the world economy. Therefore, each company that wishes to remain its competitiveness must explore the web that is understood as a valuable source of knowledge. Web-farming offers a possibility of constant analyzing of the Internet; finding important business information there; acquiring such information; saving it in data warehouse of a company; and delivering processed information to adequate persons or departments in a company. Major benefits obtained while carrying out web-farming include permanent monitoring of strategic business information sources, extracting of essential facts and their fluent matching with the internal system of company data storing. All these operations may be performed by means of advanced data mining tools.

Industry Risk – Poor Business Intelligence Costing Fortune 500 Companies Millions

A study conducted by market research firm Dynamic Markets has yesterday revealed that business intelligence systems are failing to improve operational performance or impact decision making within US and UK businesses.

A study conducted by market research firm Dynamic Markets has yesterday revealed that business intelligence systems are failing to improve operational performance or impact decision making within US and UK businesses. The research, based on interviews with 218 operational executives and front line management and conducted between November 2006 and August 2007, provides a snapshot of current use and satisfaction with Business intelligence systems and their impact on operational performance.
Although many organizations have adopted BI tools in quantity, buying thousands of user licenses, results of this research show that the majority fall short in meeting the expectations of their promise. For example, in many cases, IT executives find themselves forced into a position where they have to make decisions before all the information they need is available (76 percent), believe that BI reports end up being simply reference documents to justify decisions that have already been made (63 percent), and do not receive reports that provide predictions about potential problems or provide potential opportunities (more than 70 percent).

The impact of inadequate intelligence on the business is huge. The average cost - discovered by the research – in lost revenue to an organization is $478,686. In fact, one operations manager admitted to a direct $5,000,000 cost to his area of the business. By extrapolating this annual average as representative across the market, then the survey indicates that the Fortune 500 companies are losing approximately $250 million per year in missed business opportunities as a result of inadequate business intelligence.

Fortune 500 companies lose c. $250 million per year from poor business intelligence

A study conducted by market research firm Dynamic Markets has today revealed that business intelligence systems are failing to improve operational performance, or impact decision making within US and UK businesses. The research, based on interviews with 218 operational executives and front line management and conducted between November 2006 and August 2007, provides a snapshot of current use and satisfaction with Business Intelligence systems and their impact on operational performance. Although many organizations have deployed BI tools in quantity, buying thousands of user licenses, results of this research show that the majority fall short in meeting the expectations of their promise. The results of this research are shocking:
  • 76% find themselves forced into a position where they have to make decisions before the information they need is available.
  • 66% state that the data is either out of date or arrives too late to be of any use.
  • 63% believe that BI reports end up being simply reference documents that are only consulted to justify decisions already made;
  • 74% do not receive any kind of report that provides predictions about potential problems and 78% do not receive any reports about potential business opportunities;
  • 58% admit that business opportunities have been missed or problems have not been spotted as a result of not having access to relevant information at the fight time;
  • The impact of inadequate intelligence on the business is huge. The average cost--discovered by the research--in lost revenue to an organization is US$478,686 (260,155 [pounds sterling]). In fact, one operations manager admitted to a direct US $5,000,000 cost to his area of the business. By extrapolating this annual average as representative across the market, then the survey indicates that the Fortune 500 companies are losing approximately $250 million per year in missed business opportunities as a result of inadequate business intelligence.
Business intelligence was developed to help make businesses more efficient, and to give managers the information they needed to make smarter decisions. However, the research, commissioned bySeeWhy Software, has unearthed interesting insights into the apparent failure of BI systems. Charles Nicholls, CEO and Founder of SeeWhy Software, commented: "What is clear from this research is that all is not well in the world of BI. BI tools are perceived as hard to use, reports are out of date and largely irrelevant to daily operational decision making; and BI is seen as inherently retrospective. Yet it is clear that managers strive for more, seeking information that can make a difference; that is relevant to operations now; that can give early warning of problems; or can present opportunities for the business." Although managers were very aware of the problems that exist with their current BI solutions, they were also aware of the potential benefits of moving to event driven approaches that provide real time operational 'event intelligence' rather than retrospective 'business' intelligence. Almost all managers surveyed (90%) believed that there would be significant benefits to the business if they could embed greater intelligence into daily operational processes, Benefits mentioned included:
  • 72% believed that their company would employ more efficient processes.
  • 71% felt that there would be an improvement in customer service.
  • 65% thought that there would be an increase in revenues and 65% anticipated an increase in profitability.
  • 59% believe there will be lower levels of risk to the business.
  • 57% felt it would make them more competitive.
  • 52% said that it would enable better compliance with regulations.
Charles Nicholls explained: "We've seen order of magnitude increases in performance when business intelligence is built into operational processes* In some cases the improvement exceeds 10x (1000%). It's clear that operations managers in the US in particular are becoming increasingly aware of the potential to transform key operational processes in this way". Alerting technologies disappoint The survey also examined use of alerting technologies in daily business operations, and these too are failing to deliver what is required in the majority of cases: 55% of all managers stated that their current operational alerts arrive after the event, are not specific enough, or are not delivered m an operational context. This means that the alerts cannot be acted upon to improve business performance. Nicholls commented: "This research has highlighted the need for a new, better and more business focussed way to aid decision making. SeeWhy gives customers an easy way to inject intelligence into their business processes and to gain a competitive edge from doing so. Building in-memory analytics into business processes produces dramatic results." A surprising result from the research was the difference between the US and the UK. The survey found that more UK managers compared to US managers receive reports to help them run their respective parts of the business. UK managers also receive a greater variety of reports that can be used in an operational context for daily tactical decision making:
  • 68% of UK operations managers receive snapshot reports with only 44% of their US colleagues doing so.
  • 62% of UK managers receive reports alerting them to problems that have already occurred compared to only 44% in the US.
  • US managers find themselves more pushed for time to check all the facts and figures, compared to the UK sample group.
A full research report is available at www.seewhysoftware.com/bisurvey The survey was conducted between November 2006 and August 2007 in the US and the UK. The total sample size was 218 interviews, using both quantitative and qualitative surveys. Where any differences exist that are significant at a 95% confidence level, they are described accordingly in this report.


Implementing Business Intelligence Systems: An Organizational Learning Approach

Can organization theory inform the designers and implementers of business intelligence (BI) systems to help increase the likelihood of their success? The link between BI and the organization that uses it is that an organization collectively is a cognitive system: it senses the environment, makes a representation of it, acts on the basis of the representation and learns from the results of its actions, storing its experience as institutional memory. A BI system supports this process at various levels of organizational learning.

This article, a follow-up to a previous article published in the June 2003 issue of DM Review, "The Limits of Business Intelligence: An Organizational Learning Approach," looks at how designers can reference organizational theory to more effectively implement BI systems. There are two key areas where designers can exercise leverage over BI implementation to help assure their success:

  1. Correctly assessing the level of organizational learning and intervention that the BI project requires to be successful.
  2. Integrating change at the correct level of the organization's institutional memory or knowledge structure.

Levels of Organizational Learning: Single and Double Loops

There are many ways that learning occurs in an organization. Chris Argyris and Donald Schon, scholars at Harvard and MIT respectively, have articulated a useful learning framework.1 They describe organizational learning as comprising single-loop and double-loop modes of change. Single-loop learning occurs when an organization encounters a situation that it can resolve using its current systems, ideas and values. Double-loop learning requires a revision of these artifacts and assumptions. Argyris and Schon also define a third construct, the organization's action theory that describes the collective task knowledge of the organization that it and individuals within the organization access to respond to events. The action theory consists of both explicit knowledge that everyone sees and knows about and tacit knowledge that describes why people actually do what they do in the organization (i.e., it is the organization's collective "moxie").

In the single-loop learning mode, an organization inquires into a situation by referencing its current action theory or repository of institutional knowledge and memory. This can include the collective systems and information resources of the organization, its espoused (public) values, and its basic underlying values and assumptions ­– the components of institutional memory. In single-loop learning, members of the organization know these things and they are not questioned or changed as the situation is resolved. Figure 1 illustrates how single-loop learning works.


Figure 1: Single-Loop Learning

Here, a single feedback loop, mediated by inquiry into the current organizational knowledge base, connects an outcome of action mismatched by expectations (i.e., negative feedback) that is, therefore, surprising to the organization.2 This is similar to how a room thermostat works. The temperature deviates from some preset level, and it triggers the furnace to kick in until the room temperature rises to the preset level again. In a business example, perhaps sales drop for a quarter and management responds with the usual tactics: cut prices, increase advertising, etc. When sales respond, the intervention is deemed successful. No major changes have been incorporated into the business as a result. The situation has responded to tactics dictated by the current action theory. Single-loop learning thus accommodates and responds to negative feedback from the business environment.

My experience has been that many BI efforts are intended to address a single-loop learning/change situation. Users and technicians design a data warehouse (DW) that feeds data into functional data marts and/or "cubes" of data for query and analysis by BI users. The functional data marts represent business domains such as marketing, finance, production and planning. These are used by functional analysts and managers to inquire into current operating issues and to facilitate incremental improvements and plans. This is seen in Figure 2, a diagram similar to those used by BI vendors to illustrate how BI informs the overall organizational learning process. Figure 2 is a single-loop model of learning.


Figure 2: BI Single-Loop Learning Model

In contrast, a double-loop learning event results in a change to the organization's action theory and its knowledge base. This is shown in Figure 3.


Figure 3: Double-Loop Learning

An additional learning loop connects the observed events with the strategies needed to formulate action (i.e., positive feedback). The action theory and even the knowledge base can undergo revision, as shown in Figure 3. In BI terms, this means that the business intends to integrate the BI technology in such a way as to change its operating model, not just to seek incremental improvement.

In the BI world, examples of double-loop implementation are more difficult to find and are risky when attempted. I can think of one case that I observed where a company planned to use its BI system (a CRM analytic DW) to enable it to migrate to performance-based pricing with its customers. In this case, the BI suite was a technical success but the company did not sell its customers on the pricing idea, nor did its management follow through with the organizational changes needed to accomplish the goal. The end result was that the BI suite was an expensive tool that did not give the business the strategic competitive edge it sought from the technology. It is a lot easier to talk about BI technology changing the business operating model than it is to attempt it when the time comes!

Impact of Learning Modes for the BI Designer

Single-loop methods are workable when inquiry and corrective action take place within the framework of values and norms dictated by the current action theory. However, when events require the organization to revise its action theory so that a new theory is available to members, double-loop learning is needed. The process of inquiry between the organization and its environment, including double-loop type events that result in changes to the action theory, puts a lot of stress on managers and technical staff who must support this mode with BI. It is much more risky. Double-loop organizational learning is becoming a necessity for organizations when environmental change is rapid and disruptive.

For the BI designer, this means that he/she has to assess the level of change and learning that the BI suite is intended to support and then to factor that into the BI project risk. If the BI suite will support a change in the organization's action theory (double-loop change), then there are a lot of factors that will be outside of the BI designer's scope of control but will be necessary to assure the success of the project. These might include changes in organization structure, metrics, compensation formulas and so forth. For example, an enterprise CRM system that includes an analytic DW component integrated with channel contact applications is a project that is likely to be double-loop in scope. According to the industry statistics I have seen, such projects do not have a good track record of success. The rubric about people, processes and technology needing to be in alignment has never been more true when it comes to double-loop change situations!

A Framework for Implementing BI Change

An organizational learning framework for successfully implementing BI change will have to consider both: the type of change that the BI system is intended to support (single- or double-loop) and, how the components of organizational knowledge (institutional memory) are affected. Before presenting this framework, I want to review how organizations store and process knowledge.

Earlier, I described the construct of the action theory that consists of both explicit knowledge that everyone sees and knows about and tacit knowledge that describes why people actually do what they do in an organization. In my prior article, I discussed the structure of organizational knowledge as seen in Figure 4.


Figure 4: Action Theory Encoded in Organizational Knowledge

Figure 4 illustrates that the action theory is "encoded" in a schema of three levels of knowledge that comprise the organization's institutional memory.3Artifacts are the visible and tangible products of culture in an organization and include items such as the physical layout, reward systems, important business processes, information systems (e.g., the BI suite), symbols of status, logos and dress codes. Espoused values are what people in the organization say that they believe; they are the organization's official view of itself and usually come into being as a result of management action (e.g., a mission statement or promotion of people who exemplify some desired behavior). Finally, basic assumptions are the real operative principles that underlay a culture. Basic assumptions tell the members of the organization what to pay attention to, how to react emotionally, and what to do in various situations; they are embedded in the organization's unconscious knowledge and are less visible. All of these are the repository for explicit and implicit rules that comprise the organization's action theory and, depending on the level of change that a BI project entails, need to be considered.

Putting It All Together

Figure 5 is a framework for BI designers to assess the level of change and risk that a project requires and to plan accordingly for implementation. The framework shows the scope of the learning or change effort for a BI implementation (single-loop and double-loop) matched to the levels of organizational knowledge that the effort entails.


Figure 5: Framework for BI Change

For example, a data mart that is used for financial analysis by a few users would likely involve only single-loop change; it is not changing the organization's business model or impacting many other users. Thus, its impact is mainly at the artifact level of technology and business processes. My guess is that most departmental BI projects fall into this category and are represented in the lower left-hand corner of the diagram in Figure 5.

In contrast, BI projects that initiate double-loop learning impact the more nebulous levels of organizational knowledge and are represented by the right-hand side of Figure 5. An enterprise CRM system that connects an analytic DW to e-business contact channels such as a user Web site, a contact center and a sales force automation application is likely to involve a lot more organizational change. This type of system could provide both the information and functionality that changes the organization's view of itself and its environment. It will impact the levels of artifacts, espoused values (there might be quality standards involved, especially with customer service functions) and possibly even basic assumptions (e.g., moving to a relationship pricing model with customers). Thus, the project that implements this level of change will need to consider the impact on the higher and more nebulous levels of institutional knowledge. This is represented in the upper right-hand corner of Figure 5. Working at this level of organizational change is sometimes referred to as organization development or OD. The main issue for BI implementers working at this level is to help senior managers to understand what their basic assumptions actually are and how the BI project is likely to influence them (and vice versa).

In short, the BI implementer is faced with the task of assessing the degree of change that a project will require in the target organization and factoring this into his/her plans. The more that a BI project impacts knowledge levels above that of artifacts, the more change effort is required of the organization and on the part of BI project leadership to assure success. Planning and budgeting for the organizational impact of BI, especially when it involves double-loop learning, can greatly increase the likelihood of success of the BI project as a business initiative.

In this article, I have attempted to explain how designers can plan for successful BI implementation using the constructs of organization theory. There are two key areas where designers can exercise leverage over BI implementation to help assure success:

  1. Correctly assessing the level of organizational learning and intervention that the BI project requires to be successful.
  2. Integrating change into the right components of the organization's institutional memory or knowledge structure.

This is accomplished by assessing whether the change that is needed for BI can be accommodated within the organization's current schema of knowledge (single-loop learning) or requires a revision of the schema (double-loop learning). Subsequently, BI designers have to focus change efforts at the right levels of knowledge within the organizations' collective or institutional memory.