Azure Data Engineer
Job Description
Our client, a leading global IT service provider,is recruiting for aAzure Data Engineer to join their business in Germany.
Required Skills and Roles & Responsibilities :
Bachelor's or Master's degree in Computer Science, Engineering, or related field
8+ years of experience in big data engineering, data warehousing, and data integration
3+ years of experience in cloud-based big data platform development using Azure cloud services
Experience in developing CI / CD pipelines for Azure SQL, Azure Databricks, and Azure Data Factory
Experience in developing and managing big data processing pipelines using Apache Spark, Hadoop, and other big data technologies
Experience in developing and managing machine learning pipelines for building and deploying machine learning models on big data
Experience in developing and managing data visualization and reporting mechanisms
Design and develop big data platform architecture based on Azure cloud services such as Azure HDInsight, Azure Databricks, Azure Data Lake Store, and others
Design and implement data ingestion pipelines for ingesting data from various sources such as databases, APIs, and other sources
Develop and manage big data processing pipelines for performing data transformations, aggregations, and analytics using Apache Spark and other big data technologies
Design and implement data storage mechanisms based on Azure cloud services such as Azure Data Lake Store, Azure Blob Storage, and others
Work closely with other members of the team such as data scientists, data analysts, and software engineers to understand their requirements and integrate their solutions into the big data platform
Develop and implement test strategies, plans, and cases for Azure SQL, Azure Databricks, and Azure Data Factory
English language fluent, German language desirable.
Technical Skills :
Expertise in Azure cloud services such as Azure HDInsight, Azure Databricks, Azure Data Lake Store, and others.
Expertise in big data processing technologies such as Apache Spark, Hadoop, and others
Proficiency in programming languages such as Python, Java, Scala, and others
Experience in developing and managing CI / CD pipelines using tools such as Azure DevOps, Jenkins, and others
Experience in developing and managing data integration pipelines using tools such as Azure Data Factory, Apache Nifi, and others
Experience in developing and managing data storage mechanisms based on Azure cloud services such as Azure Data Lake Store, Azure Blob Storage, and others.