Data Engineer

Job Title: Databricks EngineerLocation: Melbourne, VIC (Hybrid)Experience: 5–8+ YearsAbout the RoleViable Solutions is seeking an experienced Databricks Engineer to join our team and deliver scalable, high-performance data engineering and analytics solutions for enterprise clients. You will be responsible for designing, developing, and optimising data pipelines and lakehouse architectures on the Databricks platform. This is a great opportunity for someone with deep Databricks expertise who enjoys working on large-scale data processing, machine learning pipelines, and cloud-native data solutions.Key ResponsibilitiesDatabricks Platform DevelopmentDesign, develop, and maintain data pipelines and workflows on the Databricks Lakehouse PlatformBuild and optimise Apache Spark jobs for large-scale data processing and transformationDevelop and maintain Delta Lake tables — schema management, optimisation, and time travelImplement Databricks Workflows and Delta Live Tables (DLT) for pipeline orchestrationManage and optimise Databricks clusters — configuration, autoscaling, and cost managementDevelop notebooks and reusable libraries using Python, Scala, or SQLData Engineering & PipelinesDesign and implement ELT/ETL pipelines for ingesting, transforming, and loading data at scaleWork with structured, semi-structured, and unstructured data sourcesImplement lakehouse architecture patterns — Bronze, Silver, and Gold layersIntegrate Databricks with upstream and downstream systems — databases, APIs, and data warehousesImplement data quality checks, validation, and observability across pipelinesManage Unity Catalog for data governance, lineage, and access controlMachine Learning & AnalyticsBuild and manage MLflow experiments, model tracking, and model registrySupport data scientists in operationalising ML models on DatabricksDevelop feature engineering pipelines for ML workloadsImplement Databricks AutoML and experiment management best practicesCloud & DevOpsDeploy and manage Databricks workspaces on AWS, Azure, or GCPImplement infrastructure as code for Databricks — Terraform or Databricks Asset BundlesBuild and maintain CI/CD pipelines for Databricks workloads — GitHub Actions, Azure DevOps, or JenkinsImplement GitOps practices for notebook and pipeline version controlMonitor and optimise Databricks workloads for performance and cost efficiencyGovernance & SecurityImplement Unity Catalog for data governance, metadata management, and access controlEnsure data lineage, traceability, and compliance across all data assetsApply row-level and column-level security across Delta Lake tablesDocument data models, pipeline architectures, and operational runbooksRequired Skills & Experience5–8+ years of experience in data engineering or a related roleStrong hands-on experience with Databricks Lakehouse Platform (mandatory)Strong proficiency in Apache Spark — PySpark, Spark SQL, and Spark Structured Streaming (mandatory)Strong proficiency in Python — data engineering and pipeline development (mandatory)Experience with Delta Lake — table management, optimisation, ACID transactions, and time travelExperience with Delta Live Tables (DLT) and Databricks WorkflowsStrong SQL skills — complex querying and data transformationExperience with Unity Catalog — data governance and access controlExperience with MLflow — experiment tracking and model registryHands-on experience with cloud platforms — AWS, Azure, or GCPExperience with CI/CD tools — GitHub Actions, Azure DevOps, or JenkinsExperience with Terraform for infrastructure as codeExperience working in Agile / Scrum delivery environments