Business Intelligence Team Roles: Data Modeling in San Diego
Posted by Daniel Upton at 11/7/2009 11:31 AM 
Categories: Big Picture on Business Intelligence,Beginners Mind,Dimensional Modeling,Skills - Interpersonal
Tags: Dimensional Data Model Gemini Business Intelligence Project Team Roles QlikView San Diego OLAP Requirements
Categories: Big Picture on Business Intelligence,Beginners Mind,Dimensional Modeling,Skills - Interpersonal
Tags: Dimensional Data Model Gemini Business Intelligence Project Team Roles QlikView San Diego OLAP Requirements
Business Intelligence and Data Modeling in San Diego
The Role of The Data Modeler on the BI Project Team — Although Business Intelligence projects usually require significant participation by non-IT business stakeholders for such roles as requirements analysis, user acceptance and, lest we forget, sponsorship and budgets, the bulk of daily BI work, for better or worse, remains largely technical, thus requiring database and software technology pros as core participants. Duh!
Obvious, right? Yes, and this rule applies in San Diego as elsewhere, but let's consider some real-world business intelligence situations from both the technical and business perspectives and see what challenges the hefty influence of technologists presents. Technical: Many technical people in data warehousing and business intelligence directly observe how projects suffer from a lack of sustained, helpful participation by the business side. Business: Supervisors of business-side BI project team members will, justifiably, find other tasking for their staff who return from such meetings talking more more about technology — (whether it be cool (Web 3.0-style collaberation) or un-cool (firewall port exception #'s)) — more than they talk about how the team is preparing to deliver hard data support for specific business decisions. Technology: Murphy's Law - At the next project team meeting, which was supposed to include a discussion on business requirements gathering processes between the database dev lead and the requirements analyst, few are surprised that the requirements analyst is double-booked and misses out.
Business/Technology Executives: After the next executive briefing, IT management issues a terse, "hurry up and expedite a prototype" order, and away we go. To we're now driving big nails (think 'ETL' here) without a blueprint.
Business Intelligence Project Risk: Development without Architecture — By now, the team must gloss over one of the most important milestones prior to development, a well-conceived dimensional data model. Instead, the tech lead, who is rarely a lover of formal processes or documentation, will tend to conceive of a model that requires minimal transformation from source data, and from which only he and other database engineers understand how it will need to be manipulated into satisfying actual business requirements. In fact, many OLTP database pros view their OLTP database as "the business", such that any transformations to it equate to a distortion. Business Intelligence pros like myself, however, tend to agree that OLTP databases transact the business, while transformed dimensional- and OLAP-databases provide the most insight into it. Even if a hastily transformed, or non-transformed, data model can be tortured into satisfying the documented requirement (which we've already identified as being insufficient), it is unlikely that it is flexible enough to extend itself to satisfy complimentary analytic needs. In truth, no amount of text documentation and end-user mock-ups can express a multi-dimensional architecture in a way that a variety of stakeholders can appreciate. So, what is the solution?
The Solution is a Smart Dimensional Data Model — one that has makes it visually obvious how most expected queries will be resolved. The dimensional data model serves as the 'stake in the ground' not only for much of the business logic that will be delivered to the business, but also serves as the guiding document for the big work of ETL. In business intelligence, ETL that begins without a destination schema is, in my not-so-humble opinion, a sin! A solid dimensional data model informs ETL engineers of the essentials of required data granularity, hierarchic relationships and field naming. It also makes clear what data quality tasks are priorities. Why did I refer above to 'most expected queries' and not all? If OLAP cubes are included, then the cube itself will likely contain MDX-based calculated metrics or key performance indicators (KPIs), downstream of the dimensional model. In that light, although the data model will not display those metrics directly, it still needs to surface the exact fields from which they will be calculated.
Data Modelers to the Rescue — An excellent data modeler — who combines expertise in dimensional design, interpersonal communications relevent business domain knowledge and, as applicable, expertise with OLAP cubes — is, in fact, the BI Project Team's key bridge-player to the business. If a business specification leaves dimensional modelling questions un-answered, the data modeler must insist on the spec's revision. Subsequently, if the data modeler cannot convincingly demonstrate to the business requirements analyst exactly how the dimensional model satisfies business query requirements, then the model needs revision. When both of the above conditions are satisfied, the project is ready for the big work of ETL — work which is now informed by a strong destination data model which will provide inherent answers to the many ETL issues that will arise.
The Data Modeler's Role in Relation to System Architects, ETL Architects, Data Warehouse Architects, Business Intelligence Architects, OLAP Developers, Dashboard Developers and Managers — Far from a being a handful of best-case talking points, dimensional data modeling is critical to BI success and thus requiries an owner. If a skilled data modeler also demonstrates business domain data expertise and, as applicable, principals of OLAP, data visualization/dashboards and reporting, he can fill the role of Business Intelligence Architect alongside an experienced senior engineer who will oversee hardware, system architecture and ETL. If, from the technology end, a senior database engineer understands everything up to, but necessarily including, dimensional data models, he can serve as data warehouse architect alongside less experienced modelers and developers of OLAP cubes, dashboards and reports. Lastly, on all BI Project teams, and especially smaller teams, any role with the word "Architect" in it implies a degree of leadership not only in design, but also technical implementation.
If, as often happens, most of this responsibility falls on the multi-tasked database manager, then their bandwidth to truly own the data modeling process is very limited, especially in the context of their competing concerns for expediency and the myriad fire-extinguishing tasks.
Real World Business Intelligence Roles — Do most DW/BI projects include a dedicated system architect, ETL architect, data warehouse architect, and business intelligence architect? Unless it's a NASA mission, I doubt it — there would be no room left at the meeting room table for business analysts, who specifiy the requirement or developers who build the solution.
Bonus Topic...
Dimensional Data Modeling in the Age of QlickView and Gemini / PowerPivot — These new 'in-memory' quasi-cube-generation platforms, which empower non-technical business analysts to rapidly build their own ad-hoc BI solutions, are indeed a paradigm shift in business intelligence. How will they effect the important of data modeling? My take is that, sooner or later, they will only increase the importance of standardized, conformed data modeling, serving as the organization's last chance to offer up standardized data to business units and in return, receive the covetted "single version of the truth". Since the source data will be directly accessible to the newly-empowered analyst, its warts will be harder to conceal. So, whether the tasking is to design good data models from the get-go or to make re-do bad models later, I think data modelers will continue to stay busy. But, OLAP people who don't yet know modelling beter hit the books.
Daniel Upton
Business Intelligence in San Diego
The Role of The Data Modeler on the BI Project Team — Although Business Intelligence projects usually require significant participation by non-IT business stakeholders for such roles as requirements analysis, user acceptance and, lest we forget, sponsorship and budgets, the bulk of daily BI work, for better or worse, remains largely technical, thus requiring database and software technology pros as core participants. Duh!
Obvious, right? Yes, and this rule applies in San Diego as elsewhere, but let's consider some real-world business intelligence situations from both the technical and business perspectives and see what challenges the hefty influence of technologists presents. Technical: Many technical people in data warehousing and business intelligence directly observe how projects suffer from a lack of sustained, helpful participation by the business side. Business: Supervisors of business-side BI project team members will, justifiably, find other tasking for their staff who return from such meetings talking more more about technology — (whether it be cool (Web 3.0-style collaberation) or un-cool (firewall port exception #'s)) — more than they talk about how the team is preparing to deliver hard data support for specific business decisions. Technology: Murphy's Law - At the next project team meeting, which was supposed to include a discussion on business requirements gathering processes between the database dev lead and the requirements analyst, few are surprised that the requirements analyst is double-booked and misses out.
Business/Technology Executives: After the next executive briefing, IT management issues a terse, "hurry up and expedite a prototype" order, and away we go. To we're now driving big nails (think 'ETL' here) without a blueprint.
Business Intelligence Project Risk: Development without Architecture — By now, the team must gloss over one of the most important milestones prior to development, a well-conceived dimensional data model. Instead, the tech lead, who is rarely a lover of formal processes or documentation, will tend to conceive of a model that requires minimal transformation from source data, and from which only he and other database engineers understand how it will need to be manipulated into satisfying actual business requirements. In fact, many OLTP database pros view their OLTP database as "the business", such that any transformations to it equate to a distortion. Business Intelligence pros like myself, however, tend to agree that OLTP databases transact the business, while transformed dimensional- and OLAP-databases provide the most insight into it. Even if a hastily transformed, or non-transformed, data model can be tortured into satisfying the documented requirement (which we've already identified as being insufficient), it is unlikely that it is flexible enough to extend itself to satisfy complimentary analytic needs. In truth, no amount of text documentation and end-user mock-ups can express a multi-dimensional architecture in a way that a variety of stakeholders can appreciate. So, what is the solution?
The Solution is a Smart Dimensional Data Model — one that has makes it visually obvious how most expected queries will be resolved. The dimensional data model serves as the 'stake in the ground' not only for much of the business logic that will be delivered to the business, but also serves as the guiding document for the big work of ETL. In business intelligence, ETL that begins without a destination schema is, in my not-so-humble opinion, a sin! A solid dimensional data model informs ETL engineers of the essentials of required data granularity, hierarchic relationships and field naming. It also makes clear what data quality tasks are priorities. Why did I refer above to 'most expected queries' and not all? If OLAP cubes are included, then the cube itself will likely contain MDX-based calculated metrics or key performance indicators (KPIs), downstream of the dimensional model. In that light, although the data model will not display those metrics directly, it still needs to surface the exact fields from which they will be calculated.
Data Modelers to the Rescue — An excellent data modeler — who combines expertise in dimensional design, interpersonal communications relevent business domain knowledge and, as applicable, expertise with OLAP cubes — is, in fact, the BI Project Team's key bridge-player to the business. If a business specification leaves dimensional modelling questions un-answered, the data modeler must insist on the spec's revision. Subsequently, if the data modeler cannot convincingly demonstrate to the business requirements analyst exactly how the dimensional model satisfies business query requirements, then the model needs revision. When both of the above conditions are satisfied, the project is ready for the big work of ETL — work which is now informed by a strong destination data model which will provide inherent answers to the many ETL issues that will arise.
The Data Modeler's Role in Relation to System Architects, ETL Architects, Data Warehouse Architects, Business Intelligence Architects, OLAP Developers, Dashboard Developers and Managers — Far from a being a handful of best-case talking points, dimensional data modeling is critical to BI success and thus requiries an owner. If a skilled data modeler also demonstrates business domain data expertise and, as applicable, principals of OLAP, data visualization/dashboards and reporting, he can fill the role of Business Intelligence Architect alongside an experienced senior engineer who will oversee hardware, system architecture and ETL. If, from the technology end, a senior database engineer understands everything up to, but necessarily including, dimensional data models, he can serve as data warehouse architect alongside less experienced modelers and developers of OLAP cubes, dashboards and reports. Lastly, on all BI Project teams, and especially smaller teams, any role with the word "Architect" in it implies a degree of leadership not only in design, but also technical implementation.
If, as often happens, most of this responsibility falls on the multi-tasked database manager, then their bandwidth to truly own the data modeling process is very limited, especially in the context of their competing concerns for expediency and the myriad fire-extinguishing tasks.
Real World Business Intelligence Roles — Do most DW/BI projects include a dedicated system architect, ETL architect, data warehouse architect, and business intelligence architect? Unless it's a NASA mission, I doubt it — there would be no room left at the meeting room table for business analysts, who specifiy the requirement or developers who build the solution.
Bonus Topic...
Dimensional Data Modeling in the Age of QlickView and Gemini / PowerPivot — These new 'in-memory' quasi-cube-generation platforms, which empower non-technical business analysts to rapidly build their own ad-hoc BI solutions, are indeed a paradigm shift in business intelligence. How will they effect the important of data modeling? My take is that, sooner or later, they will only increase the importance of standardized, conformed data modeling, serving as the organization's last chance to offer up standardized data to business units and in return, receive the covetted "single version of the truth". Since the source data will be directly accessible to the newly-empowered analyst, its warts will be harder to conceal. So, whether the tasking is to design good data models from the get-go or to make re-do bad models later, I think data modelers will continue to stay busy. But, OLAP people who don't yet know modelling beter hit the books.
Daniel Upton
Business Intelligence in San Diego
Comments