Big Data and the Goldilocks Principle

I was inspired to write this post (I can hear all of you sighing ‘Yet Another on Big Data’) due to another ‘Big’ reason. I listened to a TED (www.ted.com) talk by David Christian titled ‘The History of the World in 18 minutes’ in which he narrates a complete history of the universe, from the Big Bang to the Internet, in a riveting 18 minutes. This is “Big History”: an enlightening, wide-angle look at complexity, life and humanity, set against our slim share of the cosmic timeline. Check out his website – www.bighistoryproject.com , and I promise you that this ‘Big’ has nothing to do with Big Data, as we know it. But what got me interested in his talk is his reference to the ‘Goldilocks moment’ – a moment so precisely right for certain thresholds to be reached to enable higher forms of complexity (life) in the universe.

That got me thinking – Is Big Data the ‘Goldilocks moment’ for organizations with respect to analytics helping them towards achieving better business outcomes?

I think the answer is ‘Yes’ and this stems from the following hypothesis – An organization can utilize analytics for better business outcomes if:

a)      they have more data points to be analyzed (volume)

b)      have the ability to perform sophisticated analysis on large and diverse datasets (variety)

c)       and can do it at a much faster rate than before (velocity)

In that context, I really liked the picture (given below) from one of the IBM articles, which illustrates how Big Data when synthesized  properly along with standard transactional data can help in better business decision making (in this case, it was Fraud Detection)

Source: IBM – Understanding Big Data by Paul Zikopoulos

On the other hand, the exponential increase in processing power of CPUs, the steep fall in memory prices and high bandwidth availability, have enabled the practical use of Big Data techniques. From the human angle, people are creating digital data, viz. social media chatter, video sharing, blogs, mobility etc. at a rapid pace that organizations (with help of Big Data techniques, of course) can potentially solve the ‘Innovators Dilemma’ by providing new products and services that the consumers did not ask for simply because they couldn’t figure out what they actually want.

All in all, I think we are at a precise moment in history (the Goldilocks moment) where organizations can greatly increase their ability to provide better products & services for their consumers using Big Data techniques.

Source: http://blogs.hexaware.com/business-intelligence/big-data-and-the-goldilocks-principle/

Continue Reading

Business Focused Analytics – The Starting Point

Having been a Business Intelligence practitioner for the last 13 years, there has never been a more exciting time to practice this art, as organizations increasingly realize that a well implemented BI & Analytics system can provide great competitive advantage for them. This leads us to the question of – ‘What is a well implemented BI system?’ Let us follow the Q&A below.

Q: What is a well implemented BI system?

A: A well implemented BI system is one that is completely business focused.

Q: Well, that doesn’t make it any easier. How can we have BI that is completely business focused?

A: BI & Analytics becomes completely business focused when they have ‘business decisions’ as the cornerstone of their implementation. The starting point to build / re-engineer a BI system is to identify the business decisions taken by business stakeholders in their sphere of operations. Business decisions can be operational in nature (taken on a daily basis) and/or strategic (taken more infrequently but they tend to have a longer term impact). To reiterate, the starting point for BI is to catalog the business decisions taken by business stakeholders and collect the artifacts that are currently used to take those decisions.

Q: The starting point is fine – What are the other pieces?

A: The next step is to identify the metrics and key performance indicators that support decision making. In other words, any metric identified should be unambiguously correlated to the decision taken with the help of that metric and by whom. Next we need to identify the core datasets in the organization. Please refer to my earlier blog post titled ‘Thinking by Datasets’  on this subject.

Q: What about the operational systems in the landscape? Aren’t they important?

A: Once we have documented the relationship between Business Decisions to Metrics to Datasets, we need to focus on the transactional applications. The key focus items are:

• Inventory of all Transactional Applications
• Identify the business process catered by these applications
• Identify the datasets generated as part of each of business process
• Next step is to drill-down into individual entities that make up each of the datasets
• Once the Facts & Dimensions are identified from the entities, sketch out the classic ‘Bus Matrix’ which would form the basis for dimensional data modeling

 

Q: All this is good if we are building a BI system from scratch – How about existing BI systems?

A: For existing BI applications, the above mentioned process could be carried out as a health-check on the BI landscape. The bottomline is that every single report / dashboard / any other analytical component should have traceability into the metrics shown which should then link to the decisions taken by business users. BI & Analytics exist to help organizations take better business decisions and that defines its purpose & role in an enterprise IT landscape.

The answers mentioned above provide the high-level view of Hexaware’s approach to Business Intelligence projects. We have worked with many organizations across industries and a business focused analytical approach has provided good value for our customers.

Thanks for reading. Please do share your thoughts.

Continue Reading

Business Focused Analytics – The Starting Point – Part 2

Business decisions are the cornerstone of a successful BI implementation. Cataloging the decisions taken by the key stakeholders in an organization is the first step in understanding the information requirements for a data warehouse. Capturing business decisions – strategic and operational, is not a simple task as most business decisions tend to be complex requiring diverse data points. Further, when all decisions are collected, how will we know the decisions that have the most impact on the business?

Here’s a simple framework based on the six primitive interrogatives that Hexaware has effectively used while assessing information requirements. This framework helps systematically uncover important dimensions of information and organize them in a format that is easy to comprehend.

Question	Description	Comments	Example
Who?	The decision-maker	Stakeholder	Service Delivery Manager (SDM)
What?	The decision	A decision requiring supporting data / information	Resourcing for a project
Why?	The motivation for the decision	The significance of the decision to the business	Getting the right project team is critical to the success of a services project.
When?	When is the decision made	frequency (or) a point in time	Made during the planning stage and reviewed at periodic intervals
How?	The basis for making the decision – KPI / metrics / a logic	the metrics and datasets required for making the decision	By comparing the skill set requirement and project schedule in the Project plan to the  availability of resources (HR and PMO databases) with the right skills (skills database) and good track record (appraisal database)
Where?	The place where the decision is made	Specifies mobility /  additional access requirement	This information needs to be accessed through extranet

Depending on the stakeholder (mostly), the decisions could be strategic or operational. A manager responsible for carrying out a business process will have an operational view of information and will be making operational decisions relating to his/her sphere of operations.  Decisions taken by top management personnel with longer-term business responsibilities tend to be more strategic.  The above framework helps capture both strategic and operational decisions along with the datasets required to make the decisions.

Successfully capturing the decisions and the relevant metrics and the datasets is only half the story in the assessment for a data warehouse. Let’s reserve the other half for a subsequent blog.

Hope this information was useful. Please do share your comments/suggestions.

Continue Reading