Description
Remote
Our client is seeking a highly experienced Staff AI & Data Engineer to design and build enterprise-grade data and AI solutions on Databricks. This hands-on role focuses on strengthening the Databricks platform, onboarding new data sources, building reusable ingestion and transformation frameworks, creating trusted Gold-layer data products, enabling cross-domain analytics, and developing semantic and agent-ready capabilities. The ideal candidate brings strong data engineering expertise with modern AI and GenAI engineering skills and works across pipelines, modeling, data quality, governance, semantic layers, natural-language analytics, and AI agents.
Due to client requirements, applicants must be willing and able to work on a w2 basis. For our w2 consultants, we offer a great benefits package that includes Medical, Dental, and Vision benefits, 401k with company matching, and life insurance.
Responsibilities
- Design and build production-grade data pipelines on Databricks.
- Ingest data from enterprise applications, databases, APIs, SaaS platforms, files, and other source systems.
- Build and maintain Bronze, Silver, and Gold medallion architectures.
- Implement incremental processing and change data capture patterns where appropriate.
- Standardize types, codes, units, keys, and business definitions across source systems.
- Implement deduplication, historical tracking, schema evolution, and data lineage.
- Optimize pipelines for reliability, performance, scalability, and cost.
- Build a configuration-driven source onboarding framework and Lakeflow pipeline templates.
- Automate source registration, deployment, monitoring, lineage, and operational controls.
- Implement data quality expectations, anomaly detection, monitoring, and quarantine processes.
- Implement job and pipeline monitoring, alerting, audit logging, and operational dashboards.
- Support automated deployment and promotion using Databricks Asset Bundles and CI/CD.
- Develop conformed dimensions and shared-key strategies across domains.
- Build integrated Gold-layer products and reusable data contracts.
- Build AI-ready Gold data products with ownership, lineage, definitions, and quality information.
- Develop semantic context, business glossary, ontology, and certified KPIs in Unity Catalog.
- Build governed Unity Catalog functions and agent-ready tools and endpoints.
- Implement tracing, evaluation, testing, monitoring, and governance for AI and agent workflows.
- Support development of AI agents that use trusted enterprise data.
Experience Requirements
- Advanced hands-on experience with Databricks, Apache Spark, Delta Lake, and SQL.
- Hands-on experience with AWS services related to data analytics and AI, including EMR and Bedrock.
- Strong Python and SQL development skills.
- Experience building scalable batch and/or streaming data pipelines.
- Deep understanding of Bronze/Silver/Gold medallion architectures.
- Experience with Databricks Lakeflow or comparable orchestration technologies.
- Experience implementing enterprise data quality, lineage, observability, and monitoring.
- Experience with Unity Catalog and governed enterprise data environments.
- Strong understanding of dimensional modeling, data products, semantic models, and cross-domain integration.
- Experience with Git-based development, CI/CD, automated testing, and production deployment.
- Ability to troubleshoot complex distributed data and pipeline issues.
- Experience building AI and GenAI solutions against enterprise data.
- Understanding of semantic layers and natural-language-to-data architectures.
- Experience with Databricks Genie or comparable natural-language analytics technology.
- Experience building tools, functions, APIs, or services consumed by AI agents.
- Understanding of MCP, agent tool calling, tracing, evaluation, and AI governance.
- Experience developing production AI solutions with security, observability, and governance controls.
- Preferred: Databricks Data Engineer Professional or equivalent experience.
- Preferred: Experience with Databricks AI/BI, Genie, MLflow, and MCP-based integrations.
- Preferred: Experience in Real Estate, Finance, People/Workplace, Operations, or similar domains.
- Preferred: Experience building metadata-driven or configuration-driven ingestion frameworks.