Senior AWS Data Engineer (AI & Migration Tooling)

Project DescriptionThis is a strategic data engineering engagement focused on planning and enabling the migration of the client’s enterprise-scale data processing and ETL estate from Matillion to AWS Glue.During the mobilisation phase, the team will work closely with the client’s engineering teams to assess and reverse-engineer the existing landscape, define the target-state architecture, and create a detailed delivery blueprint for the full-scale migration.The engagement covers hundreds of ETL workflows and includes the development of reusable migration patterns, automated conversion tooling, validation and reconciliation frameworks, and a proof of concept to verify the proposed approach. The team will also support the client’s internal architecture and governance processes, including Architectural Significance Assessments, Architecture Review Boards, and the formal Gate 1 decision.The resulting framework and delivery plan will serve as the foundation for a multi-phase migration programme executed by a larger delivery organisation. This role combines hands-on data engineering, AI-powered automation, technical architecture, and close collaboration with stakeholders within a Tier 1 financial services environment.ResponsibilitiesDesign and develop an agentic AI framework to automate and accelerate ETL migration activities.Build and maintain a library of reusable migration patterns and templates for common Matillion-to-AWS Glue conversion scenarios.Develop automated pipelines for converting Matillion jobs, configurations, and transformation logic into target-state AWS components.Parse, interpret, and transform Matillion job definitions stored in JSON format.Integrate GitHub Copilot and GenAI capabilities into development and migration workflows to accelerate code generation and conversion.Develop automated testing, validation, and reconciliation mechanisms to verify migration accuracy.Contribute to the proof of concept and help assess the scalability and reliability of the proposed migration approach.Provide technical guidance, tooling support, and troubleshooting assistance to migration factory pods.Collaborate with data engineers, architects, and client stakeholders to continuously improve migration patterns and automation tools.Produce clear technical documentation for migration tooling, reusable components, and implementation standards.Mandatory SkillsStrong hands-on Python development experience, particularly in building engineering tools, automation frameworks, and data-processing utilities.Practical experience integrating GenAI or LLM capabilities into applications, development workflows, or internal engineering tools.Hands-on prompt engineering experience, including designing, testing, evaluating, and optimising prompts for reliable LLM-based automation.Experience developing GenAI-powered solutions using AWS Bedrock.Practical experience with GitHub Copilot and its use within software development and code-generation workflows.Experience designing and building automation frameworks and technical tooling from the ground up.Strong knowledge of automated testing, validation, and reconciliation approaches for data and code conversion.Experience parsing, interpreting, and transforming complex JSON structures; experience with Matillion job definitions is required.Hands-on experience with AWS Database Migration Service (AWS DMS).Strong understanding of AWS data engineering services and cloud-based ETL architecture.Ability to work directly with technical stakeholders and communicate complex engineering decisions clearly.Nice-to-Have SkillsExperience with AWS Glue and Matillion.Experience delivering large-scale ETL or data platform migrations.Experience working in banking, financial services, or another highly regulated environment.Familiarity with enterprise architecture governance and technical review processes.LanguagesEnglish: C2 Proficient