Our team members are at the heart of everything we do. At Cencora, we are united in our responsibility to create healthier futures, and every person here is essential to us being able to deliver on that purpose. If you want to make a difference at the center of health, come join our innovative company and help us improve the lives of people and animals everywhere. Apply today!
Job Details POSITION SUMMARY
The Internal Audit Data Analytics team is seeking an experienced Databricks Data Engineer Lead to own the technical direction and delivery of Internal Audit’s Databricks-based analytics environment. This role leads the design and implementation of scalable, governed data pipelines and lakehouse solutions that enable stand-alone audits, continuous auditing, and enterprise risk monitoring across the organization.
Reporting to the Data Analytics Sr. Manager – Internal Audit, the Data Engineer Lead is accountable for ensuring reliable ingestion, data quality, audit-ready lineage, and operational excellence of Internal Audit datasets and pipelines. The role partners with auditors, analysts, IT teams, and business stakeholders to deliver governed, reusable datasets that reduce manual effort and improve audit coverage and timeliness.
This is a hands-on engineering leadership role requiring deep expertise in Databricks, Spark, Delta Lake, and cloud data engineering, plus the ability to lead delivery planning, architecture decisions, design reviews, and mentoring across the team.
PRIMARY DUTIES AND RESPONSIBILITIES
Technical Leadership, Architecture & Roadmap Ownership
Own and evolve the data engineering roadmap for Internal Audit’s Databricks lakehouse, including standards for ingestion patterns, data modeling, governance, and operational reliability.
Define and enforce reference patterns for ETL/ELT, CDC/incremental processing, streaming, schema evolution, reconciliation, and publishing across bronze/silver/gold layers.
Lead architecture and design reviews for new pipelines and enhancements; ensure alignment to engineering standards, maintainability, cost efficiency, and audit requirements.
Delivery Leadership & Project / Program Management
Lead end-to-end delivery for priority initiatives by translating business/audit needs into an executable delivery plan (scope, milestones, dependencies, and resourcing).
Coordinate execution across offshore resources and onshore teams to manage work intake, backlog and prioritization to ensure delivery of reusable, certified datasets and reliable pipelines supporting audits and risk monitoring.
Proactively identify delivery risks (technical, data availability, access constraints, performance, upstream system changes) and drive mitigation plans.
Data Engineering & Platform Build
Design, build, and maintain large-scale, fault-tolerant pipelines using Python/PySpark, Spark SQL, Databricks, Delta Lake, and orchestration tools (Databricks Jobs, ADF, Airflow).
Lead implementation of ingestion patterns across files, databases, APIs, including streaming pipelines (Structured Streaming).
Own implementation of CDC, incremental loads, and full refresh patterns; manage schema evolution and reconciliation for high-confidence audit datasets.
Own curated lakehouse modeling (bronze/silver/gold) and publishing datasets for BI/analytics consumption.
Drive performance and cost optimization for production workloads (partitioning, Z-ORDER, file sizing, caching strategies, cluster policies, job tuning).
Data Quality, Monitoring, Governance & Auditability
Establish and enforce data quality standards and implement automated checks, anomaly detection routines, monitoring and alerting aligned to SLAs.
Establish repeatable processes for lineage documentation, validation, reconciliation, and test coverage; ensure pipeline versioning and evidence is maintained for auditability.
Implement scalable frameworks for metadata management and schema validation to support reliable and governed operations.
Audit Collaboration & Stakeholder Leadership
Serve as the lead SME for Internal Audit on data engineering capabilities, data availability Serve as the primary SME for data engineering within Internal Audit—guiding stakeholders on data availability, structures, pipeline behavior, and limitations.
Partner with auditors/analysts to define requirements and deliver reusable datasets that accelerate audit execution and continuous monitoring.
Collaborate with IT and business data owners to resolve access, upstream data issues, and changes that affect pipeline stability and audit outcomes.
Mentorship, Standards & Team Enablement
Establish consistency in engineering practices: code review standards, documentation requirements, development patterns, testing discipline, and CI/CD expectations.
Mentor and coach engineers and analysts on Databricks engineering practices, performance tuning, and production readiness expectations.
Contribute to hiring/interview loops and onboarding by defining technical screens, building onboarding guides, and promoting best practices across the team
.
EDUCATION AND EDUCATIONAL REQUIREMENTS
Bachelor’s or master’s degree in computer science, Data Engineering, Information Systems, Analytics, or related discipline (or equivalent work experience).
REQUIRED EXPERIENCE
8+ years of relevant experience (data engineering, analytics engineering, platform engineering), with 5+ years hands-on engineering in Databricks / Spark production environments & a team handling experience of 3+ years.
Demonstrated ownership of end-to-end data pipelines and lakehouse implementations (ingestion → transformation/modeling → publishing → monitoring/operations).
Proven experience acting as a technical lead, including architecture decisions, design reviews, mentoring, and delivery leadership across multiple initiatives.
REQUIRED TECHNICAL SKILLS
Deep proficiency in Python, PySpark/Spark, and SQL, including performance tuning practices and understanding of Spark execution concepts.
Strong experience implementing Delta Lake patterns: table design, upserts/merge logic, schema enforcement/evolution, incremental processing, and optimization strategies.
Experience with orchestration tools and production scheduling (e.g., Azure Data Factory, Airflow, Databricks Jobs).
Experience with data quality, monitoring, and alerting patterns that support reliable, SLA-driven operations.
Strong engineering discipline: version control, CI/CD practices, and documentation/runbook creation.
PREFERRED EXPERIENCE / NICE TO HAVE
Azure data ecosystem experience (e.g., ADLS, ADF, key enterprise patterns) and experience operating governed platforms in regulated environments.
Experience building semantic/consumption layers and enabling BI/analytics usage patterns.
Experience working effectively with US-based onshore stakeholders and offshore delivery teams.
PREFERRED CERTIFICATIONS
CORE BEHAVIORAL COMPETENCIES
Ownership & accountability; ability to make decisions and drive outcomes.
Strong communication (technical and non-technical), stakeholder influence, prioritization, and execution under ambiguity.
Coaching and mentorship mindset; raises the engineering bar through standards and examples.
What Cencora offersBenefit offerings outside the US may vary by country and will be aligned to local market practice. The eligibility and effective date may differ for some benefits and for team members covered under collective bargaining agreements.
Full time
Affiliated CompaniesAffiliated Companies: CENCORA BUSINESS SERVICES INDIA PRIVATE LIMITED
Equal Employment OpportunityCencora is committed to providing equal employment opportunity without regard to race, color, religion, sex, sexual orientation, gender identity, genetic information, national origin, age, disability, veteran status or membership in any other class protected by federal, state or local law.
The company’s continued success depends on the full and effective utilization of qualified individuals. Therefore, harassment is prohibited and all matters related to recruiting, training, compensation, benefits, promotions and transfers comply with equal opportunity principles and are non-discriminatory.
Cencora is committed to providing reasonable accommodations to individuals with disabilities during the employment process which are consistent with legal requirements. If you wish to request an accommodation while seeking employment, please call 888.692.2272 or email hrsc@cencora.com. We will make accommodation determinations on a request-by-request basis. Messages and emails regarding anything other than accommodations requests will not be returned
