Data Engineer
Posted Today by 1756881063
Negotiable
Outside
Remote
USA
Summary: The Data Engineer role involves designing, developing, and maintaining scalable ETL/ELT pipelines using tools such as Informatica and Azure Data Factory. The position requires processing and transforming data from various sources into a centralized data lake or warehouse while ensuring integration with Azure services. Candidates should have extensive experience in data engineering and strong programming skills, particularly in Python, Scala, or SQL. This is a contract-to-hire full-time position based in Texas, USA, with remote work options available.
Key Responsibilities:
- Design, develop, and maintain scalable ETL/ELT pipelines using Informatica, Azure Data Factory and Databricks.
- Process and transform structured and unstructured data from various sources into a centralized data lake or warehouse.
- Leverage Azure Data Lake Storage and Databricks Lakehouse to build efficient data storage and analytics solutions.
- Ensure seamless integration with Azure services such as Azure Event Hubs, Azure Stream Analytics, and Azure Functions.
- Optimize data pipelines for performance, scalability, and cost-effectiveness.
- Implement partitioning, indexing, and caching strategies to handle large-scale datasets.
Key Skills:
- 5 years of experience as a data engineer or in a similar role.
- Hands-on expertise with Informatica, Azure Data Factory, Azure Synapse, and Azure Data Lake.
- Proficiency in Databricks for data transformation, analytics, and pipeline orchestration.
- Strong programming skills in Python, Scala, or SQL.
- Experience with distributed data processing frameworks (e.g., Apache Spark).
- Familiarity with CI/CD pipelines and version control systems (e.g., Azure DevOps, Git).
- Experience with Delta Lake and Databricks.
- Knowledge of real-time data ingestion and processing.
- Exposure to BI tools such as Power BI or Tableau.
- Strong problem-solving and debugging capabilities.
- Excellent communication and collaboration skills.
- Knowledge of data modeling techniques and schema designs for analytics.
- Preferred Azure / Databricks certifications.
Salary (Rate): undetermined
City: San Antonio
Country: USA
Working Arrangements: remote
IR35 Status: outside IR35
Seniority Level: undetermined
Industry: IT
No Third party Vendors, No Sponsorship
Contract to hire full-time
Location: Anywhere in Texas, USA
Key Responsibilities
- Design, develop, and maintain scalable ETL/ELT pipelines using Informatica, Azure Data Factory and Databricks.
- Process and transform structured and unstructured data from various sources into a centralized data lake or warehouse.
- Leverage Azure Data Lake Storage and Databricks Lakehouse to build efficient data storage and analytics solutions.
- Ensure seamless integration with Azure services such as Azure Event Hubs, Azure Stream Analytics, and Azure Functions.
- Optimize data pipelines for performance, scalability, and cost-effectiveness.
- Implement partitioning, indexing, and caching strategies to handle large-scale datasets.
Key Skills :
- 5 years of experience as a data engineer or in a similar role.
- Hands-on expertise with Informatica, Azure Data Factory, Azure Synapse, and Azure Data Lake.
- Proficiency in Databricks for data transformation, analytics, and pipeline orchestration.
- Strong programming skills in Python, Scala, or SQL.
- Experience with distributed data processing frameworks (e.g., Apache Spark).
- Familiarity with CI/CD pipelines and version control systems (e.g., Azure DevOps, Git).
- Experience with Delta Lake and Databricks.
- Knowledge of real-time data ingestion and processing.
- Exposure to BI tools such as Power BI or Tableau.
- Strong problem-solving and debugging capabilities.
- Excellent communication and collaboration skills.
- Knowledge of data modeling techniques and schema designs for analytics.
- Preferred Azure / Databricks certifications