While the topic of Information Life Cycle Management (ILM) is huge, the BW ILM roadmap is quite simple:
BW ILM: Create -> Curate -> Collect -> Combine -> Consume -> Conserve -> Combust.
7Cs of BW Information Lifecycle Management
Create and Curate are tightly coupled to business processes that involve all aspects of the business. Including enterprise personnel, vendors and customers. It all revolves around the data that reflects the status of the business processes. This data lives in Online Transaction Processing (OLTP) systems.
Collect, Combine and Consume is what happens to the Created and Curated data when you need more than just the raw data. This is where ‘Business Intelligence (BI)’ comes into the picture. The BI solution mostly benefits the personnel within the enterprise while assisting a limited number of related enterprises like vendors, customers and statutory authorities (ASIC, ATO, ASX, etc). This data lives in Online Analytical Processing (OLAP) systems.
Conserve is a waiting game. While the data is consumed it is slowly becoming less useful. The very detailed operational data has a shorter life span than it’s summarised representation used in trending and KPI reports. An enterprise without an ILM strategy will stop at this point; their Data Warehouse will grow year-on-year while the stale data slows down performance, increases operational costs and wastes valuable personnel time.
Combust is what should happen to the stale data. You need to burn it out of the system as it is weighing down the business processes. As a general rule:
“the older and more granular data
is the least useful to business personnel”
However, you should always independently evaluate the value of each DataSet to the enterprise; looking at time, effort and cost of the IT department versus the wasted man-hours of the rest of the enterprise personnel (while they wait for systems to respond). Are you being sued? Then you could consider data migration and isolation instead of Combustion.
A solid Combustion process will first move the data to near-line storage, then archive (offline storage) and then delete it later.
Information Lifecycle Management and Integrated BW Systems
As a BI Specialist you will spend most of your work day in the Collect, Combine and Consume sections of the ILM roadmap. The path to becoming a BI Solution Architect is understanding the impact of Create, Curate, Conserve and Combust on the BI system.
Situation: When the ECC source system gets a performance boost from archiving data, the BI system can no longer do a full historical extract of that data from the original ECC tables.
Solution: A BI Solution Architect will ensure the BI DataModel has a Staging Layer with the raw/untransformed historical data. It should be a wider record that contains additional fields not used in reporting but has real business data (for future reporting needs). The Transformation Layer can be completely reloaded from the Staging Layer without needing to go back to the source system (which no longer has the full history due to archiving).
With the above in place, the BI demands on the technical implementation of the ECC archiving solution are greatly simplified. The BI system will not insist that the ECC archived data be able to be reloaded back in, loaded up in another client/system or be available through an integrated near-line storage solution. This means the ECC ILM process should require less effort and not need an overly complicated archiving solution; it will only need to consider its own system requirements and not the connected systems.
Next Step: Talk to your colleagues and find out where the ILM process is up to in each of your enterprise systems.