12+ years of experience in data engineering with Databricks and Spark
Strong knowledge of Databricks architecture, tools and capabilities
Experience in Design, develop, and optimize ETL/ELT data pipelines using Databricks on AWS.
Good understanding of AWS native services – S3, DataSync, DMS, Lambda, SNS, SQS.
Deep understanding of distributed computing and how to use spark for data processing
SQL and PySpark : strong command over querying databases and proficiency in PySpark.
Experience with Delta Lake and building data lakes.
Familiarity with CI/CD pipelines using tools such as Gitlab, or CodePipeline.
Experience with infrastructure as code (IaC) tools like Terraform or AWS CloudFormation.
Strong analytical and problem-solving skills, with the ability to optimize complex data workflows.
Excellent communication skills and the ability to work in a fast-paced, collaborative environment
Experienced in complex customer interactions
Ability to drive technical and business conversations with C-level executives of financial institutions to develop relationships, orchestrate teams and influence outcomes to achieve long-term customer success.
Skilled in identifying and prioritizing key use cases, resource allocations, and investment focus.
Prior experience with coding in a core programming language (i.e., Python, Java, Scala) and willingness to learn a base level of Spark.
Hands-on expertise with complex proofs-of-concept and public cloud platform(s).
Know how to provide technical solutions for specialized customer needs and navigate a competitive landscape.

Roles & Responsibilities:

Accountable for the design, development and architecture of new and current systems
Participates in enterprise architecture reviews and provides recommendations and guidance to
align with corporate systems
Act as technical lead for medium to large complex project/feature enhancements
Work with the Product Management team, Project Management, Business and User Experience teams
to identify and prioritize initiatives and features
Participate in all grooming and priority setting sessions and accountable for project sizing and estimation
Evaluates new tools, technologies and processes to improve speed, efficiency and scalability of Datalake Platform
Work closely with other development teams across organization to troubleshoot issues, performance
tuning and identify opportunities for seamless integrations across systems
Works closely with the Production Support team to train, guide and troubleshoot production issues
Provides coaching and technical guidance for Software Engineers at different levels of experience
Thrives on working in a fast-paced environment.

#LI-RJ2

Salary Range-$100,000-$150,000 a year

Databricks Cloud Architect

This job is now closed

Job Description

About the company

Other job opportunities