Data Engineer w/ Spark/Scala Exp
Posted Today by 1764911289
Negotiable
Outside
Remote
USA
Summary: The Data Engineer role focuses on designing, developing, and maintaining scalable big data solutions using Apache Spark and Scala. The position requires implementing complex SQL queries and developing Power BI dashboards while ensuring data quality and compliance. Collaboration with cross-functional teams is essential for integrating data pipelines and reporting solutions. Candidates should possess strong analytical skills and relevant experience in the financial domain or ERP systems.
Key Responsibilities:
- Design, develop, and maintain scalable big data solutions using Apache, Spark and Scala.
- Implement complex SQL queries for data transformation and analytics.
- Develop and optimize Power BI dashboards for business reporting and visualization.
- Collaborate with cross-functional teams to integrate data pipelines and reporting solutions.
- Ensure data quality, security, and compliance across all systems.
Key Skills:
- Strong proficiency in Apache, Spark with hands-on experience in Scala.
- Solid understanding of SQL for data manipulation and analysis.
- Experience in Power BI for creating interactive reports and dashboards.
- Familiarity with distributed computing concepts and big data ecosystems (Hadoop, Hive, etc.).
- Ability to work with large datasets and optimize data workflows.
- Spark Performance Tuning Expertise: Proven ability to optimize Spark jobs for efficiency and scalability.
- Knowledge of cluster resource management and troubleshooting performance bottlenecks.
- Experience with Azure cloud services for big data solutions (e.g., Azure Data Lake, Azure Databricks, Synapse Analytics).
- Exposure to other cloud platforms (AWS or Google Cloud Platform) is a plus.
- Experience working in financial domain or with ERP systems (e.g., SAP, Oracle ERP).
- Understanding of compliance and regulatory requirements in financial data processing.
- Strong problem-solving and analytical skills.
- Excellent communication and collaboration abilities.
- Bachelor's or Master's degree in Computer Science, Engineering, or related field.
Salary (Rate): undetermined
City: undetermined
Country: USA
Working Arrangements: remote
IR35 Status: outside IR35
Seniority Level: undetermined
Industry: IT
Key Responsibilities:
Design, develop, and maintain scalable big data solutions using Apache, Spark and Scala.
Implement complex SQL queries for data transformation and analytics.
Develop and optimize Power BI dashboards for business reporting and visualization.
Collaborate with cross-functional teams to integrate data pipelines and reporting solutions.
Ensure data quality, security, and compliance across all systems.
Required Skills & Experience
Strong proficiency in Apache, Spark with hands-on experience in Scala.
Solid understanding of SQL for data manipulation and analysis.
Experience in Power BI for creating interactive reports and dashboards.
Familiarity with distributed computing concepts and big data ecosystems (Hadoop, Hive, etc.).
Ability to work with large datasets and optimize data workflows.
Highly Desirable
Spark Performance Tuning Expertise: Proven ability to optimize Spark jobs for efficiency and scalability.
Knowledge of cluster resource management and troubleshooting performance bottlenecks.
Experience with Azure cloud services for big data solutions (e.g., Azure Data Lake, Azure Databricks, Synapse Analytics).
Exposure to other cloud platforms (AWS or Google Cloud Platform) is a plus.
Experience working in financial domain or with ERP systems (e.g., SAP, Oracle ERP).
Understanding of compliance and regulatory requirements in financial data processing.
Additional Qualifications
Strong problem-solving and analytical skills.
Excellent communication and collaboration abilities.
Bachelor s or Master s degree in Computer Science, Engineering, or related field.