About the company
Bailey Abbott are a progressive and dynamic IT Consultancy business working across both Public and Private sectors. People are at the core of everything we do. We’re all about nurturing and inspiring people – ours and our clients. We bring a positive attitude and mindset to everything we do. We confidently use our knowledge and skills to solve problems, finding a better and easier way forward.
We deliver transformational outcomes to take businesses further, fearlessly. Confidence and trust is fundamental to delivering superior outcomes and we take ownership and responsibility for our commitments.
Responsibilities
- Production ML Pipelines: Build, test, and optimize production-grade data processing and model training pipelines in Apache Spark and Databricks.
- MLOps & Lifecycle Automation: Standardise model deployment, versioning, monitoring, and automated retraining workflows (CI/CD, MLflow).
- Code Quality & Architecture: Apply software engineering best practices—modular code architecture, automated testing, containerisation, and version control—to analytical and statistical codebases.
- R & Statistical Workloads: Optimise and scale R-based modeling workflows to run efficiently over distributed data architectures using Sparklyr/SparkR.
- Collaboration & Integration: Partner with data architects, data engineers, and domain analysts to integrate predictive outputs cleanly into downstream systems and analytical platforms.
- Performance & Reliability: Monitor pipeline latency, manage compute cluster sizing, and ensure data integrity and governance across production workspaces.
Requirements
- Software Engineering Fundamentals: Solid foundation in core software engineering principles (design patterns, unit/integration testing, Git version control, CI/CD pipelines).
- Databricks Ecosystem: Proven hands-on experience orchestrating workloads, managing workspaces, and using MLflow / Unity Catalog within Databricks.
- Distributed Computing (Apache Spark): Deep practical understanding of Spark internals, data partitioning, query optimization, and memory management.
- R for Production / Advanced Analytics: Strong proficiency in R (including Spark integrations like sparklyr or SparkR) for statistical modeling, forecast automation, or data manipulation.
- Data Layer Experience: Comfort querying complex relational data, working with Delta Lake formats, and handling large-scale time-series or tabular datasets.
Desirable Skills
- Familiarity with Python/PySpark alongside R for cross-language workflows.
- Experience with cloud infrastructure (Azure, AWS, or GCP) and Infrastructure as Code (Terraform).
- Understanding of model drift detection, data lineage, and enterprise data governance frameworks.