Senior/Lead Data Engineer-Python,Spark & SQL (Only W2)
Posted 1 day ago by 1766053896
Negotiable
Outside
Remote
USA
Summary: We are seeking a Senior Data Engineer with expertise in SQL, Python, and Spark, who has experience in AI-related tasks and API integrations for chatbot development. The role involves collaborating with stakeholders to define data requirements, designing scalable data models, and developing AI-driven features. The ideal candidate will also have a solid background in database engineering and software development, particularly in cloud services. Flexibility and a continuous improvement mindset are essential for success in this position.
Key Responsibilities:
- Partner closely with stakeholders to define requirements for data, data processing, reporting, and analytics.
- Design and maintain scalable data models and extraction processes.
- Develop and integrate chatbot functionality using AWS Bedrock foundation models, Knowledge Bases, and Agents.
- Implement AWS Bedrock Converse API for multi-turn conversations and tool use patterns.
- Build AI-driven features including conversational interfaces, intelligent automation workflows, and contextual response systems.
- Approach tasks with flexibility, adapting to ad-hoc requirements as needed.
- Foster a continuous improvement mindset for data practices and visualizations.
Key Skills:
- A solid background in database engineering and software development.
- Experience with AI Agents Design and Integration.
- Demonstrated ability with Python and Spark within data-intensive environments.
- Experience with cloud services, particularly integrating foundation models through Bedrock runtime APIs.
- Understanding of prompt engineering, system messages, and model parameter tuning for optimal AI responses.
- Eager to learn and apply new technologies in data visualization, automation, AI.
Salary (Rate): undetermined
City: undetermined
Country: USA
Working Arrangements: remote
IR35 Status: outside IR35
Seniority Level: Senior
Industry: IT
Description:
We are in search of a Senior Data Engineer who is adept in SQL, Python, and Spark, and ideally with an experience in AI-related tasks and API integrations for chatbot development/integration.
- SQL Expertise: Essential for complex query handling and database manipulation.
- Python Proficiency: Required for automation, data processing, AWS Bedrock API integration, and chatbot development workflows using Boto3 SDK
- Spark Familiarity: Needed for large-scale data processing.
Responsibilities:
- Partner closely with stakeholders to define requirements for data, data processing, reporting, and analytics.
- Design and maintain scalable data models and extraction processes.
- Develop and integrate chatbot functionality using AWS Bedrock foundation models, Knowledge Bases, and Agents
- Implement AWS Bedrock Converse API for multi-turn conversations and tool use patterns
- Build AI-driven features including conversational interfaces, intelligent automation workflows, and contextual response systems
- Approach tasks with flexibility, adapting to ad-hoc requirements as needed.
- Foster a continuous improvement mindset for data practices and visualizations.
Requirements
- A solid background in database engineering and software development.
- Experience with AI Agents Design and Integration
- Demonstrated ability with Python and Spark within data-intensive environments
- Experience with cloud services, particularly integrating foundation models through Bedrock runtime APIs
- Understanding of prompt engineering, system messages, and model parameter tuning for optimal AI responses
- Eager to learn and apply new technologies in data visualization, automation, AI.
Nice to have:
- Experience working on Amazon Web Services (in particular using AWS Glue, AWS Athena)
- Familiarity with AWS Bedrock Agents, Knowledge Bases, and RAG patterns
- Prior experience building chatbots using various foundation models (Claude, Llama, Titan) available on AWS Bedrock
- Knowledge of multi-modal AI applications and streaming response implementations
- Understanding of AWS security best practices, IAM policies, and model access management for Bedrock.