AWS Data Pipeline AWS Glue & AWS Step Functions
Serverless data integration (AWS Glue) and workflow orchestration (AWS Step Functions) services that replace legacy AWS Data Pipeline workflows.
Naming timeline
- AWS Data Pipeline
Launched at AWS re:Invent 2012 (API version 2012-10-29) to schedule and orchestrate data movement and transformation tasks across Amazon S3, RDS, DynamoDB, and EMR.
- AWS Glue & AWS Step FunctionsCurrent name
AWS removed Data Pipeline console procedures in May 2023, closed new customer onboarding, and redirected the product page to AWS Glue alongside official migration paths to AWS Glue and AWS Step Functions.
What replaced AWS Data Pipeline?
AWS Data Pipeline has been superseded by AWS Glue & AWS Step Functions, with the latest change on 26 May 2023 (Console Removal & Migration Guidance). Serverless data integration (AWS Glue) and workflow orchestration (AWS Step Functions) services that replace legacy AWS Data Pipeline workflows. The underlying legacy API or CLI surface is deprecated, so a technical migration is required.
Also known as
- Data Pipeline
Technical identifiers
Legacy identifiers deprecated. The underlying API or CLI surface for the legacy name is officially deprecated, so this change requires a technical migration rather than a label update.
| AWS Service APIs / SDKs | datapipeline glue states |
|---|---|
| AWS CLI Commands | aws datapipeline aws glue aws stepfunctions |
| IAM Actions & Policies | datapipeline:* glue:* states:* |
| Terraform Resources (aws_*) | aws_datapipeline_pipeline aws_glue_job aws_sfn_state_machine |
The datapipeline API, CLI, and aws_datapipeline_pipeline Terraform resource are closed to new customers and superseded by AWS Glue (glue) and AWS Step Functions (states).
Notes & disambiguation
- AWS placed AWS Data Pipeline in maintenance mode, removed AWS Management Console creation procedures in May 2023, and closed the service to new customers while redirecting https://aws.amazon.com/datapipeline/ to https://aws.amazon.com/glue/.
- Official AWS migration guidance directs customers to migrate Apache Spark and data movement ETL jobs to AWS Glue (glue) and multi-service workflow orchestration to AWS Step Functions (states) or Amazon MWAA.
- The legacy datapipeline API, aws datapipeline CLI, datapipeline:* IAM actions, and aws_datapipeline_pipeline Terraform resource are in maintenance mode for existing customers only.
- AWS placed AWS Data Pipeline in maintenance mode, removed AWS Management Console creation procedures on May 26, 2023, and closed the service to new customers effective July 25, 2024 while redirecting https://aws.amazon.com/datapipeline/ to https://aws.amazon.com/glue/.
Sources & official documentation
- Official documentation ↗
- Product page ↗
- Migrating workloads from AWS Data Pipeline to AWS Glue and AWS Step Functions
- Document History - AWS Data Pipeline Developer Guide
- What is AWS Data Pipeline? - AWS Data Pipeline Developer Guide
- Migrate workloads from AWS Data Pipeline (Closed to New Customers Effective July 25, 2024)
Catalog last updated . Spotted an error? Suggest an edit