Senior Databricks Data Platform Engineer

Project description

We are seeking a highly skilled and motivated Databricks Platform Engineer to join its core Lakehouse Platform team. This role focuses on engineering and evolving the Databricks-based Lakehouse platform itself, building advanced platform capabilities, reusable services, automation frameworks, and enterprise integrations that enable ADIA’s Data & AI ecosystem at scale. While the platform supports data pipelines and analytics workloads, this is not a traditional data engineering role focused on building ETL jobs or delivering datasets. Instead, the role is centred on: • Designing and developing shared platform capabilities • Building internal libraries and APIs • Implementing governance, security, and integration patterns • Ensuring scalability, reliability, and operational excellence The ideal candidate combines strong Python software engineering expertise with solid infrastructure understanding and experience integrating enterprise systems. Clear, precise, and articulate communication is essential. The primary focus of this role is building and enhancing platform capabilities, not developing business ETL pipelines or reporting datasets.

Responsibilities

• Platform Capability Engineering
• Collaborate with stakeholders during requirements clarification and sprint planning to ensure alignment with platform strategy and enterprise standards.
• Design and implement technical solutions on ADIA’s Lakehouse platform (Databricks), including: o Prototyping and productising new platform capabilities. o Developing reusable Python libraries, SDKs, and internal frameworks. o Exposing platform capabilities to support ADIA’s Data Products strategy and broader Data & AI ecosystem.
• Build automation for provisioning, governance controls, configuration management, and operational standards. Enterprise Integration & APIs
• Integrate the Lakehouse platform with enterprise systems, including: o Incident and monitoring systems (e.g., ServiceNow). o Identity and access management solutions. o Observability platforms (e.g., Dynatrace, Prometheus, Grafana).
• Design and implement secure API-based integrations using enterprise authentication and authorisation mechanisms. Reliability, Quality & Operations
• Develop and maintain unit and integration tests to ensure quality and resilience of platform components.
• Support QA teams during acceptance testing of platform features.
• Act as a third-line engineer for complex production incidents affecting platform stability.
• Collaborate with cloud and infrastructure teams to deliver secure, scalable, and reliable platform architecture.

SKILLS

Must have

• Minimum 10 Years of experience in IT/Data.
• Minimum 3 years of expertise in Databricks from a platform enablement perspective.
• 3+ years of experience in designing, writing, and maintaining APIs used for Platform automation, Serverless workflows, Deployment pipelines, release management and repository management
• Advanced Python skills for production-grade library and service development.
• Solid understanding of distributed processing concepts (PySpark knowledge beneficial, but not limited to ETL use cases).
• Experience designing and integrating REST APIs.
• Advanced SQL skills for troubleshooting and performance optimisation (e.g., Oracle, MS SQL).
• Experience with Git and CI/CD workflows.
• Familiarity with monitoring and observability tools (e.g., ServiceNow, Prometheus, Grafana).
• Knowledge of scheduling and orchestration tools (e.g., Stonebranch, Control-M, Airflow).
• Understanding of data quality and governance frameworks (e.g., Great Expectations).
• Strong grasp of cloud infrastructure fundamentals: o DNS o Certificates o Identity & access management o Networking concepts o Load balancing

Nice to have

• Containerization & Orchestration: Docker, Kubernetes. • Infrastructure as Code (IaC): Terraform. • Understanding of Investment Data domain (desired).