Data engineering is the process of organizing the data generated in a company’s business processes in such a way that business analysts and data scientists can easily analyze that data and help the company to make better decisions. 

The basic input to architect a data engineering platform is a list of the metrics or KPIs the organization wants to be analyzed. With this list, a data engineer will start a process of identifying the underlying data to be displayed for those metrics and, more importantly, the sources in the company that data is coming from. In conjunction with the business analyst or data scientist, the data engineer will design the data structures (data warehouses or data lakes) that the data analysis dashboards will connect to, either to display the information or to enable the data scientist to query the unified data. 

Once the data sources have been determined and the data warehouses or data lakes have been designed, the data engineer will identify the available interfaces to communicate with those data sources to extract the information and will design a workflow to initiate the data extraction, transform that data to adjust it to the data warehouse or data lake structure, and save that data into them. These components are called ETLs in data engineering (Extract, Transform, and Load components). 

The following diagram depicts the data engineering process explained above: 

A data engineering platform will be as simple or complex as the metrics and KPIs to be analyzed, the amount of data to move from the data sources to the data warehouses, and how well structured the data is organized in those data sources. Another very important factor to determine the complexity of the data engineering platform is the availability of interfaces to communicate with the data sources (known as APIs in the software engineering field) and how mature and well documented those interfaces are. 

Nevertheless, the most common and basic architecture of a data engineering platform is shown in the following diagram: 

There are different PAAS environments where you can host such a data engineering platform. In Softcial, we know very well how to do that by using Microsoft Azure, so let me explain how you can leverage that cloud environment to implement your data engineering platform.  

Let’s recreate the basic architecture diagram of the Data Engineering Platform by using Softcial’s recommended Microsoft Azure components: 

The best option to implement your database in Microsoft Azure will depend on how complex the metrics and KPIs to analyze are and how much data will be moved from the data sources to the data warehouse. For small and medium-sized companies, using an Azure SQL Server will be good enough. However, for larger companies or companies that generate large amounts of data, upgrading to a more robust and scalable option such as data lakes implemented using Azure Data Lake could be a better option. 

When talking about the ETLs, let’s remember that these components are workflows or pipelines that extract and transform the information from the data sources and load (or save) that information into the data warehouses. Microsoft Azure offers a very good solution for implementing those workflows in the form of Azure Data Factories.  With a data factory, you can create pipelines that use connectors to interface with the data sources and the data repositories. There are multiple built-in connectors to many different environments (like the most popular SAAS solutions like CRMs or ERPs), but you can also create your own custom connectors. To implement those custom connectors in Softcial, we use Azure Functions. 

The Azure Functions communicate with the data sources by using their available REST APIs, and they are made available to the pipelines implemented by using Azure Data Factories to save the data to the data warehouses. It is very important to notice that you can create triggers for the data factories to refresh the data in the data warehouse on practically every possible time basis (hourly, daily, weekly, monthly, and yearly).