AI Engineering Team Lead

Guiding team members on using platform features for training, deployment, and monitoring. You'll troubleshoot common pipeline issues and ensure efficient use of managed services.

Writing Dockerfiles for ML services, understanding basic K8s concepts (Pods, Deployments), and debugging container issues. You'll ensure our models are packaged and orchestrated correctly.

Enforcing team standards for logging experiments, using the UI to compare runs, and helping the team find the best models. You'll ensure reproducibility and traceability of our ML work.

Reading and understanding existing Infrastructure as Code plans. You might make minor edits to existing modules for new services, but you won't be writing complex modules from scratch yet.

Managing backlogs, sprints, and roadmaps for your team. You'll create detailed technical documentation and decision logs, ensuring everyone knows what's happening and why.

Understanding how your team's data pipelines and model training jobs interact with these platforms. You'll help debug access or performance issues and work with Data Engineering on optimising data flows.

• Senior AI Engineering Lead (your direct manager)
• Product Managers (who define what needs building)
• Data Scientists (who build the initial models)
• Data Engineering Team (who provide the data)
• Platform Engineering Team (who manage the infrastructure)
• Other AI Engineering Teams (for collaboration and shared services)

• Cloud Service Providers (e.g., AWS, GCP, Azure)
• MLOps Tool Vendors (e.g., MLflow, Weights & Biases)

Cloud costs are a constant battle, and simply using managed services isn't enough. You'll need to become an expert in optimising every penny spent, from GPU instance types to serverless inference patterns. This directly impacts our bottom line.

Cost-aware Architecture Design · Performance Tuning for ML Workloads · Multi-Cloud & Hybrid Cloud Strategies

• This week: Review your team's current cloud spend reports and identify the top 3 cost drivers.
• This month: Propose and implement one specific cost-optimisation technique for a training pipeline or inference service.
• Month 2: Complete an advanced course or certification in cloud cost management for your primary cloud provider.
• Month 3: Lead a workshop for your team on best practices for cloud resource management and cost awareness.

Quick win: Implement automated shutdown policies for development and staging ML environments outside of working hours.

As our datasets grow and models become more complex (think foundation models), training and serving them will require increasingly sophisticated distributed systems. You'll need to move beyond single-machine training to orchestrating massive compute clusters.

Distributed Training Frameworks (e.g., Horovod, Ray) · Data Parallelism vs. Model Parallelism · High-Performance Computing (HPC) for ML

• This week: Read up on the fundamentals of distributed computing for machine learning.
• This month: Experiment with a distributed training framework on a small dataset using a cloud-managed service.
• Month 2: Evaluate a new distributed ML framework (e.g., Ray) for a potential future project within your team.
• Month 3: Present a summary of distributed ML best practices to your team and identify potential use cases.

Quick win: Refactor one of your team's existing training scripts to use a basic distributed data loader, even if it's on a single machine, to understand the patterns.