ECS Networking 정리

ECS Task networking

• The networking behavior of Amazon ECS tasks hosted on Amazon EC2 instances is dependent on the network mode defined in the task definition
• Amazon ECS recommends using the awsvpc network mode unless you have a specific need to use a different network mode
• awsvpc
  • The task is allocated its own elastic network interface (ENI) and a primary private IPv4 address
  • ENI
    • VPC에서 가상 네트워크 카드를 나타내는 논리적 네트워킹 구성 요소
    • A primary private IPv4 address from the IPv4 address range of your VPC
    • One or more secondary private IPv4 addresses from the IPv4 address range of your VPC
    • One Elastic IP address (IPv4) per private IPv4 address
    • One public IPv4 address
    • One or more IPv6 addresses
    • One or more security groups
    • A MAC address
    • A source/destination check flag
    • A description
  • gives the task the same networking properties as Amazon EC2 instances
  • This mode allocates an elastic networking interface to each running task, providing a dynamic private IP address and internal DNS name
  • simplifies container networking management and operations, allowing tasks to run with full networking features on AWS
  • Advantages
    • Addressable by IP addresses and the DNS name of the elastic network interface
    • Attachable as ‘IP’ targets to Application Load Balancers and Network Load Balancers
    • Observable from VPC flow logs
    • Integration into CloudWatch logging and Container Insights
    • Access controlled by security groups
    • Enables running multiple copies of the same task definition on the same instance, without needing to worry about port conflicts
    • Higher performance because there is no need to perform any port translations or contend for bandwidth on the shared docker0 bridge, as you do with the bridge networking mode
  • EC2 instance type일 때 ENI 수에 limit 있음. 아래 링크로 해결.
    https://ecsworkshop.com/networking_sd/ecs_networking/awsvpc/#:~:text=For%20details%20see-,here,-For%20comprehensive%20considerations
• bridge
  • utilizes Docker’s built-in virtual network which runs inside each Amazon EC2 instance hosting the task
  • The task will get an IP address out of the bridge’s network IP range
  • Containers use this docker0 virtual bridge interface to communicate with endpoints outside of the instance, using the primary elastic network interface of the instance on which they are running
  • Dynamic port mapping을 하면 static port mapping 과 달리 같은 host에서 랜덤의 host port를 사용하여 port가 겹치는 containert를 서비스할 수 있음.
    • Task definition의 hostPort를 0으로 입력
  • Considerations
    • Additional network hop and decreased performance by using docker0 virtual bridge interface
    • Containers are not addressable with the IP address allocated by Docker
    • Host port mapping requires additional operational efforts and considerations
    • Single EC2 ENI is shared by multiple containers
    • No network policy enforcement for a single container with fine grained security groups possible
    • No integration into AWS network observability
• awsvpc vs bridge
  • https://aws.amazon.com/blogs/compute/introducing-cloud-native-networking-for-ecs-containers/
•  host
  • bypasses Docker’s built-in virtual network and maps container ports directly to the ENI of the Amazon EC2 instance hosting the task
  • can’t run multiple instantiations of the same task on a single Amazon EC2 instance when port mappings are used
  • useful to optimize performance, and in situations where a container needs to handle a large range of ports, as it does not require network address translation (NAT), and no “userland-proxy” is created for each port
  • Considerations
    • Not possible to run more than a single instantiation of a particular task per host, as only the first task will be able to bind to its required port on the EC2 instance.
    • No way to remap a container port when using host networking mode, any port collisions or conflicts must be managed by changing the configuration of the application inside the container.
• none
  • no external network connectivity
  • want to completely disable the networking stack on a container, you can use none as network mode when starting the container or in ECS Task Definition
  • Within the container, only the loopback device is created
  • cannot leverage the new “ECS exec” feature to access the container in “none” networking mode
• NAT
  • Docker for Windows uses a different network mode (known as NAT) than Docker for Linux
  • When you register a task definition with Windows containers, you must not specify a network mode. If you use the AWS Management Console to register a task definition with Windows containers, you must choose the <default> network mode
• every Amazon ECS task on Fargate is provided an elastic network interface (ENI) with a primary private IP addres

ECS 컨테이너 사이의 통신

1. Service Discovery using AWS Cloud Map

• awsvpc 방식을 사용하면 continter 간 통신할 때 service discovery 측면에서 각 task 의 A record(private IP)만으로 파악이 가능 vs Bridge mode는 같은 IP를 공유하고 port가 다르기 때문에 SRV Record(public IP:Port)를 사용해야 함. 
• awsvpc 방식을 사용하면 각 서비스에 unique security group을 설정할 수 있음.
• Service discovery를 이용한 service-to-service communication 의 단점은 connection failure 를 대비한 extra logic을 구현해야 한다는 것. application은 retry와 bad backend에 대한 로직을 다룰 필요성이 생김

2. Internal Load Balancer 사용

• 모든 네트워크 방식 (awsvpc, bridge, ...)에 효과적.
• 비용적인 측면에서 disadvantage. but 하나의 load balancer를 사용하고 path-based mapping 으로 구성하면 하나의 ALB 만 사용하여 비용 낮출 수 있음.

3. App Mesh 사용

• envoy sidecar proxy가 container inbound & outbound traffic 모두 관장.
• App Mesh Control plane은 Cloud Map으로 모든 가용 서비스들의 IP를 가져와 proxy sidecar에 전달.
• Applications don't implement as much load balancing logic within their code because the Envoy proxy sidecar handles that load balancing
• can also be configured to use mTLS to encrypt traffic in transit, and ensure that your applications are communicating to a verified destination
• with Envoy proxy, you're responsible for deploying and managing your own Envoy proxy sidecar. This adds some overhead to the task resource consumption, and additional operational workload to maintain and update the proxy when needed
• App Mesh and an Envoy proxy allows for extremely low latency between tasks. This is because Envoy proxy runs collocated to each task
• only one instance to instance network jump, between one Envoy proxy and another Envoy proxy. This means there's also less network overhead compared to when using load balancers(two jumps).

Network Optimizing and troubleshooting

https://docs.aws.amazon.com/AmazonECS/latest/bestpracticesguide/networking-troubleshooting.html

awsvpc

bridge

host