How to orchestrate and automate workflows with Prefect running on ECS Fargate with a private Docker registry
This article is a step by step guide of how I deployed a Prefect Server with a Fargate ECS Cluster using Docker Storage on a private registry. A few articles covered deployment with Prefect Cloud and different combinations of runners and storage. As I ran through a few issues when deploying this specific architecture, I decided to detail the steps in this article.
By the end of this tutorial, we will have :
- Prefect Server running on EC2
- a ECSAgent running on EC2
- packaged our flows in docker images in a private registry
- run a flow in an ECS cluster through the UI and GraphQL API
You can choose to use different Agents and Storage options, this article is only a base guideline to the many options of Prefect. Although I will not detail Prefect and AWS concepts, you will find references to relevant documentation to help you.
- AWS Account
- A Docker registry
- Basic Python skills
If it is your first experience with Prefect, I recommend trying to deploy it locally first to grasp Prefect concepts easier.
The first step is to deploy Prefect Server on an EC2 instance. Create and launch an EC2 instance. I used a g4dn.2xlarge instance as I deployed multiple services on it, but a small one can be enough if you plan to use only Prefect on it. You will need to update the security rules to allow TCP connections on ports 8080 to access the UI, and 4200 for the backend.
You may need to install docker-compose if it’s not already on your instance.
Next, SSH on your instance and install prefect and run :
pip install “prefect[aws]”
See other options of prefect installation here.
prefect backend server
Create the file ~/.prefect/config.toml containing the following information :
apollo_url = “http://X.X.X.X:4200/graphql"
Replace X.X.X.X with your instance public adress as it allows to connect the UI to the backend.
You can now start Prefect with :
prefect server start
From now, you can connect to the UI from any machine (authorized by your security rules) by going on http://X.X.X.X:8080
The agents are responsible for starting and monitoring your runs. Here, we deploy it on the same EC2 instance as Prefect Server. You can choose to deploy it on a second instance by providing the Prefect Server’s instance IP address in the config.toml file or throught the command line.
In both cases, you need to add :