![]() These are the Data Warehouse Jedi Masters, and you owe it to yourself and your clients to read as much of their material as you can and analyze how you can apply their recommendations to your applications.Īdditionally, I recommend my prior CODE Magazine article from March/April 2013, where I talk about implementing the Kimball Dimensional modeling patterns: #1: Populate the Data Model with Test Data as Early as Possible Margy Ross and Bob Becker lead Decision Works Consulting ( ). Fear not: This only strengthens the resolve of those who partnered with the Kimball family. Ralph and Julie Kimball retired at the end of 2015. I almost envy anyone who is about to dive into the Kimball world for the first time. You can also find a good overview that accompanies the ETL book here. ![]() If you're new to the Kimball methodologies and want to start exploring, here's a good place to start. I've found countless tips on data warehouse modeling and ETL subsystems from these books and website articles. Although aggregate counts of activity don't automatically translate into meaningful experience, I'll put my professional reputation on the line by making a recommendation that served me and others well: if you work in this space, you should read the Ralph and Julie Kimball series of books on data warehousing and dimensional modeling. I've held leadership roles in data warehousing for a decade. I taught SQL Server and Data Warehousing for five years. I've been building database applications (transactional, data warehouse, and reporting databases) for thirty years. Baker's Dozen Potpourri: Miscellaneous ETL thoughts.Baker's Dozen Spotlight: Collecting all activity in ETL logs.Use of the MERGE and composable DML and extract engines.Define primary keys in source data, and what happens when source systems delete data.Identify and account for any specific data type challenges.Establishing necessary source data, profile source data, and source primary keys.Defining a high-level roadmap of physical data sources and processes.Populating the end-result data model as early as possible.Better Extract/Transform/Load (ETL) Practices in Data Warehousing (Part 2 of 2).Many ETL processes have a series of common patterns/activities, whether the end-game is a full-blown data warehouse or some type of operational transaction system or data store. If your job includes retrieving (extracting) data from one location, manipulating/validating (transforming) that data in the middle, and then saving (loading) that data somewhere else, you might find value from some of these tips. “But what if I don't work in Data Warehousing?”Įven if you don't build full-blown data warehouses, you might still benefit from this article. The second part will apply these tips to a specific data warehouse project using the newest features in SQL Server, and will focus on applying those methods using Microsoft technologies. This first part presents 13 general tips that focus on practice and methodology, although I'll go into a certain level of detail when necessary. Because this general topic is so deep, I'm writing it as a two-part article. Because of this, CODE Magazine asked me to write an article on the subject, so, in my traditional Baker's Dozen format, I'll present thirteen tips that you can incorporate into your data warehouse ETL projects. ![]() I've worked on many data warehouse initiatives, led several data warehouse projects, and also taught and mentored on the subject. I've studied the large volumes of material on data warehouse ETL practices and have collected many useful thoughts along the way. When you combine that statistic with the palpable and sobering objective of a data warehouse as the “single version of trust,” good processes are essential. In the world of data warehousing, many industry journals report that Extract/Transform/Load (ETL) development activities account for a large majority (as much as 75%) of total data warehouse work. Those with experience will recommend these as practices, with the idea that following these practices leads to better, cleaner, and more productive outcomes. Nearly every professional endeavor contains processes and methods associated with carrying out tasks in that discipline.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |