KDA Consulting Inc. is seeking a highly skilled Data Engineer to design, build, and maintain scalable data pipelines and architectures supporting mission-critical programs within the Intelligence Community (IC).
Responsibilities
Design, develop, and maintain scalable ETL/ELT pipelines to ingest, transform, and load data from multiple sources
Build and optimize data workflows supporting both batch and real-time processing
Ensure high availability and reliability of data pipelines across enterprise systems
Leverage AWS services such as S3, Glue, Lambda, Redshift, EMR, and Kinesis
Design and manage cloud-based data architectures that are secure, scalable, and cost-efficient
Implement best practices for data storage, partitioning, and lifecycle management
Optimize data storage and retrieval for performance, scalability, and cost efficiency
Implement indexing, partitioning, and query optimization strategies
Monitor and troubleshoot data pipeline performance issues
Ensure data accuracy, consistency, and integrity across all data pipelines and systems
Implement validation, monitoring, and error-handling mechanisms
Support data governance practices and compliance with security and IC standards
Work closely with data scientists, software engineers, and mission stakeholders
Integrate data solutions with enterprise applications, APIs, and downstream analytics platforms
Participate in Agile development processes
Requirements
Active TS/SCI W/ Polygraph Required
Bachelor's degree in Computer Science, Engineering, or a related technical field (or equivalent experience)
Demonstrated experience in data engineering and pipeline development
Strong experience with AWS data services (e.g., S3, Glue, Redshift, Lambda, EMR)
Proficiency in SQL and Python for data processing and transformation
Experience working with large-scale data sets and big data technologies