Job Title: Senior Databricks Engineer
Location: Onsite – Sugarland, Tx.
Employment Type: Full-Time

About the Role:

We are seeking an experienced Senior Databricks Engineer to design, develop, optimize, and manage large-scale data pipelines and analytics solutions using Databricks, Spark, and modern cloud platforms (Azure/AWS/GCP). The ideal candidate has deep expertise in distributed data processing, Delta Lake architecture, and end-to-end data engineering best practices.You will work with cross-functional teams—including product, analytics, and delivery—to deliver scalable data solutions for enterprise clients.

Key Responsibilities

Databricks & Spark Engineering
• Design, develop, and optimize Spark/Databricks ETL/ELT pipelines at scale.
• Implement efficient Delta Lake architectures (bronze/silver/gold layers).
• Write high-performance PySpark/SQL code following best practices.
• Tune clusters, jobs, and workflows for performance and cost efficiency.

Data Architecture & Modeling:
• Develop data models and schemas to support analytics, reporting, and ML workloads.
• Implement medallion architecture following industry and Databricks standards.
• Design scalable and secure storage patterns using cloud-native services.

Cloud Engineering & Integration:
• Work with cloud platforms (Azure preferred, AWS or GCP acceptable).
• Implement CI/CD pipelines for Databricks using Git, Azure DevOps, or GitHub Actions.
• Integrate Databricks with third-party systems, ADF, Event Hub, Kafka, APIs, etc.
• Configure workflows in Databricks (Jobs, Workflows, Unity Catalog).

Data Quality, Governance & Security
• Implement data validation, observability, and monitoring frameworks.
• Apply best practices in data governance, lineage, and cataloging.
• Enforce security guidelines using RBAC, Unity Catalog, and encryption policies.

Collaboration & Leadership
• Lead technical solutioning for client projects and POCs.
• Provide mentorship to junior engineers.
• Collaborate closely with business analysts, product managers, and stakeholders.
• Prepare architectural documentation, standards, and reusable accelerators.

Troubleshooting & Optimization
• Investigate data issues, performance bottlenecks, and operational failures.
• Drive continuous improvement across data engineering processes.
• Ensure reliable and timely data delivery for downstream applications.