All businesses work with data – information generated by many internal and external sources of the company. These data feeds serve as management’s senses, providing them with information about what is happening with the business and the market. Consequently, any misconception, inaccuracy or lack of information can lead to a distorted perception of the market situation and an incorrect understanding of internal operations, which in turn leads to erroneous decisions.

Making data-driven decisions requires clear visibility into every aspect of your business, even the ones you don’t think about. But how do you turn unstructured pieces of data into something useful? Business intelligence will help you with this.

We have already talked about the strategy for organizing machine learning. In this article we will talk about how to integrate business intelligence into your existing corporate infrastructure. You will learn how a business intelligence strategy is prepared and tools are integrated into the company’s work processes.

What is business intelligence?

Let’s start with a definition: business intelligence (BI) is a set of practices for collecting, structuring, analyzing and turning raw data into a business picture that allows you to make decisions. BI uses techniques and tools that transform unstructured data sets, compiling them into understandable reports or information dashboards. The main purpose of BI is to create a picture of the business and justify decision-making using data.

Example of an interactive dashboard for the sales department

The entire business intelligence process can be divided into four stages:

• Data collection
• Data cleaning/standardization
• Analysis
• Reporting

The biggest part of implementing BI is using the tools that do the data processing. The BI infrastructure is made up of various tools and technologies. Most often, this infrastructure contains the following technologies for storing and processing data, as well as generating reports:

• Data sources
• ETL (Extract, Transform, Load) or data integration tools
• Data store
• Online analytical processing data cubes
• Data kiosks
• Reporting tools (BI)

Business intelligence is a technological process that is highly dependent on input data. Technologies used in BI for transforming unstructured or semi-structured data can also be used for data mining, as well as in the front end for working with big data.

Business intelligence and predictive analytics

The definition of business intelligence is often confusing because it overlaps with other areas of knowledge, in particular predictive analytics. It would be a serious mistake to assume that business intelligence and predictive analytics are the same thing.

Essentially, business intelligence is a data analysis technique that answers the questions: what happened? and what’s going on? This type of data processing is also called descriptive analytics. Using descriptive analytics, companies can examine the state of the market in their industry, as well as their own internal processes. Analysis of historical data helps to identify business weaknesses and potential.

Predictive analytics deals with making predictions based on data processing of past events. Predictive analytics does not analyze historical events, but makes predictions about future business trends. Both of these types of forecasts are based on analysis of past events. Therefore, BI and predictive analytics can use the same techniques to process data. To some extent, predictive analytics can be considered the next stage of business intelligence. You can read more about this in our article on analytics maturity models.

Both analysis techniques address three main types of data management:

1. Descriptive Analytics (BI)
2. Predictive analytics
3. Prescriptive analytics

Prescriptive analytics is the third type, aimed at finding solutions to business problems; he suggests actions to solve them. Currently, prescriptive analytics is implemented using rich BI tools, but this area of knowledge as a whole has not developed to a sufficiently reliable level.

The entire process of integrating BI tools into your organization can be divided into familiarizing company employees with business intelligence as a concept and the integration of tools and applications itself. Below we will talk about the main points of BI integration and reveal some of the difficulties.

Stage 1: familiarization of employees and management with business intelligence

Let’s start with the basics. To start using business intelligence in your organization, the first step is to explain the meaning of BI to the entire leadership team. Mutual understanding is important here, since employees from various departments will be involved in data processing. Therefore, consistency needs to be ensured so that no one confuses business intelligence with predictive analytics.

Another purpose of this stage is to explain the BI concept to key managers involved in data management. You need to define the task you will work on, set KPIs and organize specialists to launch your own business intelligence project.

It is important to note that at this stage you will, strictly speaking, make assumptions about the data sources and standards that will be specified to manage the flow of data. In subsequent steps, you can test the validity of your assumptions and shape your data processing process. That is why you need to be prepared to change data acquisition channels and team structure.

We set goals, KPIs and requirements

The first important step after securing a common vision of the situation is to identify the problem or group of problems that you will solve with the help of business intelligence. Setting goals will allow you to define high-level BI parameters, such as:

• What data sources will be used? (CRM, ERP, website analytics, external sources and so on.)
• What type of data do we need to accept? (Sales figures, reports, website traffic, etc.)
• Who needs access to this data? (Senior management, market analysts, other specialists.)
• What types of reports do we need and how should they be presented? (Spreadsheets, charts, operational reports, or interactive dashboards.)
• How will progress be measured?

At this stage, along with goals, you need to think about possible KPIs and evaluation metrics to verify the success of the task. These may be physical limitations (development budget) or performance metrics such as query speed or the frequency of bug reports.

By the end of this stage, you will already be able to configure the initial requirements for the future product. This could be a list of features in the product backlog consisting of a user story, or a simplified version of this requirements document. The main thing here is that based on the requirements you should understand what type of architecture, features and capabilities are required in your BI software/hardware.

Step 2: Select Tools or Decide to Develop Your Own Solution

Drawing up a document with requirements for a business intelligence system is a key point in understanding which tool you need. Large companies should consider developing their own BI ecosystem for the following reasons:

1. Sometimes enterprise-level organizations cannot trust third-party companies to handle their valuable data.
2. BI tools are mainly differentiated by serving the needs of a specific industry. It may be that there is no supplier in the market that provides services in your industry.
3. Processing large amounts of information or working with big data may justify developing your own BI instead of searching for a supplier, since your own system increases flexibility in choosing a cloud infrastructure supplier.

For smaller companies, the BI market offers a huge number of tools capable of running both embedded and cloud (Software-as-a-Service) technologies. You can find offerings that cover all or most of your industry’s data analytics needs and are flexible.

Based on the requirements, industry type, size and business needs, you can decide whether to invest in a custom BI tool. Otherwise, you can choose a vendor who will take on the burden of implementation and integration.

Stage 3: gathering a business intelligence team

Next, you will need to gather a group of people from different departments of the company to work on the business intelligence strategy. Why create such a group at all? The answer is simple: the BI team helps bring together representatives from different departments to facilitate communication and get departmental input on the required data and its sources. That is, the structure of the BI team should include two main categories of people:

Subject matter representatives from different departments

These people will be responsible for providing the team with access to data sources. They also invest their domain knowledge in selecting and interpreting different types of data. For example, a marketer might determine whether the valuable data types are website traffic, bounce rates, or the number of newsletter sign-ups. A customer service specialist can provide valuable advice about interacting with customers. Plus, you’ll have access to marketing or sales information from one person.

BI related positions

The second category of people are the BI people who will lead the development process and make architectural, technical and strategic decisions. That is, you need to appoint people to the following positions:

Head of BI. This person must have theoretical, practical and technical knowledge to support the implementation of the strategy and tools. This could be a manager with knowledge of business intelligence and access to data sources. The BI Lead is the decision maker that drives the implementation.

A BI engineer is a technical team member who specializes in creating, implementing, and configuring BI systems. Typically, BI engineers have experience in software development and database configuration. They should also be proficient in data integration methods and techniques. A BI engineer can lead the IT department in implementing BI tools. You can learn more about data scientists and their responsibilities in our article.

Also part of the BI team should be a data analyst who can apply his knowledge in data validation, processing and visualization.

Step 4: Document your BI strategy

Once you’ve assembled your team and selected the data sources needed to solve a specific problem, you can begin to develop a BI strategy. You can document your strategy using traditional strategy documents like a product road map. A business intelligence strategy can include various components depending on the industry, company size, competition and business model. However, the following required components are recommended:

Data sources

This is documentation of the selected data source feeds. It should include all types of channels, be it the manager, industry analytics as a whole, or information from employees and departments. Examples of such channels could be Google Analytics, CRM, ERP, and so on.

Industry/own KPIs

Documenting industry standard and unique KPIs can demonstrate a complete picture of your business’s growth and loss. Ultimately, BI tools are designed to track these KPIs, supporting them with additional data.

Reporting Standards

At this stage, you need to determine what type of reporting you need to easily extract valuable information. If you have a custom BI system, you can choose between visual or textual representation. If you have already selected a vendor, your choice of reporting standards may be limited because the vendor sets its own. This section can also include the types of data you want to work with.

Reporting stream type and end users

The end user is the person who will be viewing the data through the reporting tool interface. Depending on the end users, you can choose different types of reporting flow:

Traditional BI. Traditionally, BI was designed solely for management. Since the number of users and data types is limited, there is no need for full automation. Therefore, in the traditional type of BI flow, technical personnel are required as an intermediary between the reporting tool and the end user. If the end user wants to extract some data, then he can make a request and the technical staff will generate a report from the required data. In this case, the IT department acts as a power user – a user who has access to the data and influences its transformation.

The traditional approach offers a more secure and controlled data flow. However, having to rely on the IT department can lead to delays that reduce flexibility and speed when processing large volumes of data (especially in the case of big data). If you want more control over reporting and accuracy of reports, then assemble a separate IT team that will handle queries and report generation.

Self-service BI. Modern companies and solution providers use self-service BI. This approach allows business users and management to receive reports automatically generated by the system. Automatic reporting does not require power users (administrators) from the IT department processing each request to the data warehouse; however, technical personnel are still needed to configure the system.

Automation can reduce the quality of the resulting reports and their flexibility, and may also be limited by how the reporting is designed. However, this approach has an advantage: the constant participation of technical personnel is not required to work with the system. Non-technical users will be able to create a reference themselves or access a dedicated section of the data storage.

Stage 5: Prepare Data Integration Tools

The tool integration phase will require a lot of time and work from the IT department. If you need to build your own solution, you will have to develop many different building blocks of the BI architecture. In other cases, you can choose a supplier on the market that provides the implementation and data structuring that suits you.

One of the basic elements of any BI architecture is the data warehouse. A warehouse is a database that stores information in a specified format, usually structured, categorized, and error-free. If the data is not pre-processed, the BI tool or IT department will not be able to query it. Therefore, you cannot directly connect the data warehouse to information sources. Instead, you should use ETL (Extract, Transform, Load) or data integration tools. They will pre-process the raw data from the original sources and transfer it to the warehouse in three sequential steps:

1. Data extraction. An ETL tool obtains data from data sources such as ERP, CRM, analytics, and spreadsheets.
2. Data conversion. Once extracted, the ETL tool begins processing the data. All extracted data is analyzed, cleared of duplicates, and then standardized, sorted, filtered and verified.
3. Loading data. At this stage, the converted data is loaded into the warehouse.

Typically, ETL tools are provided off-the-shelf along with vendor-developed BI tools. (We will look at the most popular ones below). To learn what you need to clean and prepare your data, read our article.

Stage 6: Configuring the Data Warehouse and Selecting an Architectural Solution

Data store

Having configured the transfer of data from the selected sources, you need to configure the storage. Data warehouses in business intelligence are special types of databases that typically store historical information in SQL formats. On the one hand, the warehouses are connected to data sources and ETL systems, on the other, to reporting tools or dashboard interfaces. This allows you to display data from different systems in a single interface.

However, the storage usually contains huge amounts of information (from 100 GB), which is why responses to requests are quite slow. In some cases, data may be stored in an unstructured or semi-structured form, which leads to a high error rate when parsing data to generate a report. Analytics may require a specific type of data, which is grouped into one storage space for ease of use. That is why companies use additional technologies to provide accelerated access to small thematic blocks of information.

There are different types of solutions used to provide analytics with small pieces of data from a warehouse. The most popular of these is Online Analytical Processing and Data Mart. These technologies provide faster reporting and easier access to the necessary data.

Recommendations: if you do not have large amounts of data, then it will be enough to use a simple SQL storage. Additional building blocks such as a data mart will require large additional costs without providing any value. This option is suitable for small businesses or industries that work with relatively small amounts of data.

Data warehouse + data cubes Online Analytical Processing

The data stored in the warehouse has two dimensions, reminiscent of a typical spreadsheet format (tables and rows). The method of storing data in such a store is also called a relational database. A single database can contain thousands of types of data, so processing requests to the data warehouse takes a significant amount of time. To satisfy the needs of the analyst and provide quick access to data, analyze it in different dimensions, and group it as needed, OLAP data cubes are used.

OLAP or online analytical processing is a technology that processes data and provides access to it simultaneously in several dimensions. Structuring data into cubes overcomes the limitations of a data warehouse.

An OLAP data cube is a data structure optimized for quickly analyzing data from SQL (warehousing) databases. The original cube data from the data warehouse is a smaller version of its description. However, the data structure assumes that there are more than two dimensions (spreadsheet row and column format). Dimensions are critical elements for generating a report. For example, for the sales department they could be like this:

• Sales Manager
• Volume of sales
• Product
• Region
• A period of time

Cubes form a multidimensional database of information that can be adapted to group information in different ways to speed up reporting. OLAP data cubes dedicated to different data topics form OLAP databases. Warehousing and OLAP are used together because cubes store a relatively small amount of data and are used for ease of processing.

Recommendation: The “data warehouse + OLAP data cubes” architecture can be considered typical. It can be used by companies of any size that require data storage and complex multidimensional information analysis. If you don’t want to overload your storage with queries, consider using an OLAP-based architecture.

Data warehouse + data mart technologies

Storage is the first and largest element of business intelligence architecture. A smaller description of data storage arrays is a data mart. A data mart is a specialized part of a warehouse that collects thematically similar information related to a specific department. With data marts, departments can access the data they need because the kiosks provide information specific to one area of the business. This means that developers won’t have to set up a permission-based request system for end users.

Recommendation: Data warehouse + data mart is the second most popular architectural style based on the use of data marts to distribute required information across departments. This approach can be used to set up ongoing reporting or simplify access to information without granting permissions to end users.

Hybrid architecture

Enterprise businesses may require different data management options. Data marts and data cubes are different technologies, but both are used to describe smaller volumes of information from a warehouse. Data marts describe a subset of a data warehouse relevant to a specific task, but they can be implemented in different ways. Implementation options include relational databases (warehouse or any other SQL database) and multidimensional structures, which are essentially OLAP data cubes. That is, both technologies can be used to manage data and distribute it across departments of the organization.

Recommendation: Both technologies can be used because they support the same concept but serve different purposes. Data marts can be implemented as part of a data warehouse to provide security, data aggregation, or availability. Or you can use kiosks to describe multiple dimensions of an OLAP data cube. However, it is worth keeping in mind that both kiosks and OLAP data cubes will require separate database setup

Stage 7: Implementation of the end-user interface – tools and reporting dashboards

Data organized into convenient thematically related blocks of information in Online Analytical Processing cubes or data marts is presented using the BI tools user interface. This is where descriptive analysis benefits the end user.

Modern BI tools allow you to present the required data in many different ways. In the past, business intelligence could only create static reports based on events in the future and past. Today, BI is capable of creating interactive dashboards with customizable pieces of information. However, reporting templates remain the most popular way to present data.

The most valuable way to present information is considered to be an ad hoc report. Live reporting allows users to drill down into the details of a standard report using any type of data for one-time use. This type of reporting is used instead of daily or monthly reports as a more complete version, since the user will retrieve data from the warehouse (cube or data mart) directly at the time of viewing the report. This ensures the freshness of the information provided by querying the databases for each piece of information. So, essentially, a live report is a customizable, real-time report used to find answers to a specific business question.

Step 8: Conduct end user training

To ensure a smooth employee onboarding process, we highly recommend conducting training sessions. These sessions can take a variety of forms: If you’re using an analytics tool built into your CRM or ERP, you can implement onboarding practices like video tutorials or interactive onboarding tools that walk users through each step step by step.

If you don’t have a budget for automation training, your manager or BI team members should still provide it.

Basic business intelligence tools on the market

It is important to mention that BI tool providers provide users with data integration, ETL, reporting (dashboards), and storage services. This means that, more often than not, you will have a complete BI architecture integrated into your system. Below we will talk about some examples of business intelligence tool providers.

Sisense

Sisense is one of the biggest names in the business intelligence market. The company’s product provides access to data analysis systems in the backend and frontend for users with different technical levels. Sisense also offers storage services, which means it provides a full-featured solution. The pricing model is an annual subscription, but the cost is greatly influenced by the number of users, data volumes and type of project.

Zoho Analytics

Another big name in the business intelligence industry is Zoho Analytics. Zoho offers a complete infrastructure with a scalable interface for both small and large businesses. Among other useful features, it offers open RESTFUL APIs for connecting all the necessary CRS and ERP systems, a collaboration platform for employees or management.

Tableau

Tableau is a cloud-based BI solution that pioneered the use of drag-and-drop interfaces in reporting tools. Tableau software also has collaboration features: analysts can create a single login page to access the dashboard and share information. You can query data so that it is sent to the mobile device. In the Tableau app, you can edit reports and save changes directly from your phone.

SAP

SAP is an international company that offers a variety of technical solutions, including the Business Objects Business Intelligence and Cloud Analytics products. The first product is a basic solution for businesses of any size. The platform provides smart query and operational reporting services. In addition, dashboard reporting uses a position-based format, meaning any user can customize an analytical dashboard depending on their position. An additional benefit is the ease of integration of SAP products with Microsoft Office products.

BusinessQ

BusinessQ’s BI solutions are designed specifically for small and medium-sized businesses. BusinessQ offers both a separate web application and an embedded version that is built into the client’s application.

Domo

The Domo BI platform is a cloud-first solution designed for businesses of all sizes. The service is scalable, which allows it to work with both big data and small corporate databases. Domo provides access to real-time dashboards and uses data marts implemented in OLAP cubes to provide multidimensional analysis and division of data by department.

Qlik

Qlik is a business intelligence service provider that provides a variety of products for data visualization, interactive dashboarding, and self-service reporting. The infrastructure can be implemented at the client’s facilities or in the cloud. In addition, Qlik offers access to a list of public datasets as information sources.

Finally

Business intelligence tools have been around for over twenty years. However, the appearance and basic functionality of a “standard” BI tool has changed significantly. Instead of simple static reporting, every vendor today offers operational reporting or interactive dashboards for analysts to collaborate. In addition, self-service BI is becoming the standard for routine business tasks, allowing entrepreneurs to perform analytics with less wastage of resources. Following general technical trends, innovations such as cloud platforms and mobile BI reporting have appeared in BI.

Thus, knowing the main trends and technologies used in this industry, you can create your own BI system or choose a ready-made one; this will allow you to create easy-to-understand reports to support your decisions. Business intelligence is no longer the privilege of senior management, it is a collaboration tool for the entire organization. Find the right vendor and use all the necessary features to ensure your employees benefit from BI results.