Staff Healthcare Data Engineer

Designing and deploying production ETL pipelines using Glue/EMR, architecting Redshift/Spectrum schemas for optimal performance, implementing Lake Formation security for PHI, and managing IAM roles for secure data access.

Designing and implementing complex data models (star schema, data vault), managing RBAC for sensitive data, optimising clustering keys and query performance for large healthcare datasets, and leveraging advanced features like Snowpipe and Streams.

Designing and maintaining complex, idempotent DAGs for data orchestration. Building and maintaining scalable dbt models, implementing CI/CD for data pipelines, and establishing best practices for workflow management.

Developing robust data processing libraries, implementing advanced Spark optimisations (partitioning, broadcasting), writing comprehensive unit/integration tests, and building custom data APIs (e.g., with FastAPI) for FHIR servers.

Developing complex channels to transform and route HL7/FHIR data between systems. Building custom APIs against a FHIR server, and troubleshooting interoperability issues with external healthcare providers.

Publishing and managing certified data sources, implementing row-level security for PHI, and optimising extract performance for critical healthcare dashboards. While not a primary focus, you'll need to understand how your data models support BI.

• Director, Healthcare Data Platform
• Other Lead Engineers (e.g., Machine Learning, Platform)
• Product Owners (especially for data products)
• Clinical Informatics Team
• Security and Compliance Teams
• Analytics and Business Intelligence Teams

• Healthcare Providers (hospitals, clinics)
• Regulatory Bodies (e.g., NHS Digital, CQC)
• Third-party Data Vendors
• Interoperability Standard Organisations (e.g., HL7)

Healthcare is moving towards real-time decision-making, from patient monitoring to immediate clinical alerts. Our data platform needs to support this with robust, low-latency streaming capabilities, which means moving beyond traditional batch processing.

Event-Driven Architectures · Stream Processing Frameworks · Complex Event Processing (CEP) · Guaranteed Delivery & Fault Tolerance

• This month: Take an online course on Apache Kafka or AWS Kinesis, focusing on architectural patterns.
• Month 2: Prototype a small real-time data pipeline for a non-critical dataset (e.g., website logs) using a streaming framework.
• Month 3: Research how other healthcare organisations are using real-time data for clinical decision support or operational efficiency.
• Month 4: Propose a roadmap for integrating real-time streaming into one of our existing batch pipelines, outlining the benefits and challenges.

Quick win: Set up a local Kafka instance and experiment with producing and consuming simple messages. It's a great way to get a feel for stream processing.

As data platforms become more complex, simply monitoring uptime isn't enough. We need to understand data quality, lineage, and performance proactively. AIOps for data uses AI to automate monitoring, anomaly detection, and root cause analysis in data pipelines.

Data Quality Monitoring (Automated) · End-to-End Data Lineage (Automated) · Predictive Anomaly Detection · Automated Root Cause Analysis

• This month: Research leading data observability platforms (e.g., Monte Carlo, Soda, Databand).
• Month 2: Implement automated data quality checks (e.g., Great Expectations) on one of our critical datasets.
• Month 3: Explore how to integrate data quality metrics and pipeline health into a unified dashboard, potentially using an AIOps tool.
• Month 4: Lead a discussion on how we can improve our current data monitoring strategy using observability principles.

Quick win: Add a simple data quality check to your next dbt model or Airflow DAG that flags unexpected null values or out-of-range numbers, and ensure it sends an alert.