Data Engineer ( Databricks )
Unison Consulting Pte Ltd, India
Job Description
Job Description:
As an Azure Data Engineer, you will play a pivotal role in the design, development, and maintenance of data solutions on the Azure platform. Your primary responsibilities will revolve around building and optimizing data pipelines, ensuring data quality, and implementing data processing and transformation logic. To excel in this role, your expertise in Azure Databricks, Python, SQL, Azure Data Factory (ADF), and PySpark/Scala will be essential. Your contributions will enable data-driven decision-making and insights for our organization.
Key Responsibilities:
1. Designing and Developing Data Pipelines:
- Design and implement scalable and efficient data pipelines using Azure Databricks, PySpark, and Scala.
- Manage data ingestion, data transformation, and data loading processes.
2. Data Modeling and Database Design:
- Create and implement data models to support efficient data storage, retrieval, and analysis.
- Collaborate with relational databases, data lakes, and other storage solutions on the Azure platform.
3. Data Integration and Orchestration:
- Leverage Azure Data Factory (ADF) to orchestrate data integration workflows.
- Manage data movement across various data sources and targets, including scheduling and monitoring data pipelines.
4. Data Quality and Governance:
- Implement data quality checks, validation rules, and data governance processes.
- Ensure data accuracy, consistency, and compliance with relevant regulations and standards.
5. Performance Optimization:
- Optimize data pipelines and queries to enhance system performance and reduce processing time.
- Tune SQL queries, optimize data transformation logic, and employ caching techniques as needed.
6. Monitoring and Troubleshooting:
- Continuously monitor data pipelines and identify performance bottlenecks.
- Proactively troubleshoot issues related to data ingestion, processing, and transformation.
- Collaborate with cross-functional teams to resolve data-related problems.
7. Documentation and Collaboration:
- Document data pipelines, data flows, and data transformation processes.
- Collaborate closely with data scientists, analysts, and other stakeholders to understand their data requirements and provide data engineering support.
Requirements
5+ years of experience in a Data Engineer role.
- Proficiency in Python or Scala programming.
- Strong problem-solving skills in a big data environment.
- Experience in building 'big data' data pipelines, architectures, and data sets from various business applications.
- Familiarity with message queuing, stream processing, and scalable 'big data' data stores.
- Entrepreneurial spirit, dedicated to delivering high-quality results.
- Degree in Computer Science or a related field, such as Applied Mathematics.
- Experience in a technical customer-facing role (preferably solution engineering) or a willingness to undertake such a role.
- Strong expertise in Azure Databricks, Python, SQL, ADF, PySpark, and Scala.
- Proficiency in designing and developing data pipelines and ETL processes.
- Sound understanding of data modeling concepts and database design principles.
- Knowledge of data integration and orchestration using Azure Data Factory.