DATA ENGINEERING
ESTABLISHING THE DATA FOUNDATION
Great strides have been made in data technologies over the past decade. Cloud infrastructure, along with new storage paradigms, have provided a new dimension to the arsenal of tools available to tackle the deluge of data we face.
A modern data warehouse and data lake architecture that combines SQL as well as No-SQL technologies is a preferred way to store and enrich enterprise data where high-quality “small data” is seen as the goal, placing the focus on quality rather than quantity.
Our view is that a well-structured data warehouse - with SQL at its core - still provides a level of technological maturity that is lacking in these new-world tech solutions that remain relevant to financial service providers. The data lake and streaming patterns are valuable in instances where there is a high level of data volume, variety and velocity.
The following high-level stages are generally prescribed when implementing a data warehouse that will provide a return on investment for the business: