About the company
Sandisk understands how people and businesses consume data and we relentlessly innovate to deliver solutions that enable today’s needs and tomorrow’s next big ideas. With a rich history of groundbreaking innovations in Flash and advanced memory technologies, our solutions have become the beating heart of the digital world we’re living in and that we have the power to shape.
Responsibilities
- Design, develop, and maintain high-performance ETL/ELT pipelines using PySpark and Spark SQL using cloud-native components in Databricks
- Build and orchestrate data workflows in Azure
- Implement hybrid data integration between on-premise databases and Azure Databricks using tools such as ADF, HVR/Fivetran, and secure network configurations
- Enhance/optimize Spark jobs for performance, scalability, and cost efficiency
- Implement and enforce best practices for data quality, governance, and documentation
- Collaborate with data analysts, data scientists, and business users to define and refine data requirements
- Support CI/CD processes and automation tools and version control systems like Git
- Perform root cause analysis, troubleshoot issues, and ensure the reliability of data pipelines
Requirements
- Bachelor's degree in Computer Science, Engineering, or related field
- 6+ years of hands-on experience in data engineering
- Proficiency in PySpark, Spark SQL, and distributed processing
- Strong knowledge of Azure cloud services including ADF, Databricks, and ADLS
- Experience with SQL, data modeling, and performance tuning
- Familiarity with Git, CI/CD pipelines, and agile practices
- Preferred: Experience with orchestration tools such as Airflow or ADF pipelines
- Preferred: Expertise in Databricks, Delta Lake, Unity Catalog and Azure Data Services
- Preferred: Knowledge of real-time streaming tools (Kafka, Event Hub, HVR)
- Preferred: Exposure to APIs, data integrations, and cloud-native architectures
- Preferred: Familiarity with enterprise data ecosystems (S/4 HANA, BDC)