Complex enterprise environment focused on scalable, reliable data platforms.
Responsibilities
Design scalable and reliable data architectures, leading complex technical decisions on data pipelines, ETL, migration, performance, and reliability.
Review technical implementations and provide guidance on engineering standards and best practices.
Analyze existing systems and identify improvements in data processing, architecture, and performance.
Create high-level technical designs covering ETL processes, data dictionaries, metadata, file layouts, data flows, validation, transformation, and loading.
Gather and analyze business and application requirements and translate them into technical solutions.
Plan and execute Agile sprints across the software development lifecycle.
Develop technical documentation for parallel jobs, batch processing, sequencers, and data migration.
Participate in architecture, code, and design reviews.
Lead and mentor junior engineers.
Requirements
Bachelor’s degree or foreign equivalent in Computer Science, Electronic Engineering, or related field, plus at least 7 years of progressive experience as a Senior Data Engineer, IT Consultant, or related role.
At least 7 years building data pipelines, data warehouse solutions, and reporting frameworks.
Strong experience gathering requirements and developing solutions within an Agile SDLC.
Proven ability to build data pipelines from flat files, databases, APIs, and streaming applications.
Hands-on experience with IBM DataStage, Talend, and ODI.
Experience with Spark, Python, Talend, DataStage, ODI, Unix, PL/SQL, Teradata, AWS, and Java.
Experience designing and automating ETL processes for data validation and reconciliation.
Proficiency with Python, Unix, Java, JavaScript, C, C++, REST APIs, and security keys.
Extensive experience with AWS services: S3, Glue, Athena, Lake Formation, Lambda, EC2, RDS, EMR, Redshift, IAM.
Experience with AWS, Azure, and GCP.
Experience developing and integrating REST APIs.
Familiarity with CI/CD tools such as Bitbucket, GitHub, and Jenkins.
Strong technical leadership, communication, problem-solving, and mentoring skills.
Ability to work remotely from any location within the United States.