ecs_composex.common package

Subpackages

• ecs_composex.common.stacks package
  • Module contents
    • ComposeXStack
      • ComposeXStack.add_dependencies()
      • ComposeXStack.add_parameter()
      • ComposeXStack.attributes
      • ComposeXStack.get_top_root_stack()
      • ComposeXStack.is_void
      • ComposeXStack.mappings
      • ComposeXStack.mark_nested_stacks()
      • ComposeXStack.render()
      • ComposeXStack.render_parameters_list_cfn()
      • ComposeXStack.set_vpc_parameters_from_vpc_stack()
      • ComposeXStack.set_vpc_params_from_vpc_lookup()
      • ComposeXStack.write_config_file()
    • process_stacks()
    • render_codepipeline_config_file()

Submodules

ecs_composex.common.aws module

Common functions and variables fetched from AWS.

ecs_composex.common.aws. assert_can_create_stack ( client , name )

Checks whether a stack already exists or not

ecs_composex.common.aws. assert_can_update_stack ( client , name )

Checks whether a stack already exists or not

ecs_composex.common.aws. define_lookup_role_from_info ( info , session )

Function to override ComposeXSettings session to specific session for Lookup

Parameters :

• info –

• session –

Returns :

boto3 session from lookup settings

Return type :

boto3.session.Session

ecs_composex.common.aws. define_tagsgroups_filter_tags ( tags )

Function to create the filters out of tags list

Parameters :

tags ( list ) – list of Key/Value dict

Returns :

filters

Return type :

list

ecs_composex.common.aws. deploy ( settings , root_stack )

Function to deploy (create or update) the stack to CFN. :param ComposeXSettings settings: :param ComposeXStack root_stack: :return:

ecs_composex.common.aws. find_aws_resource_arn_from_tags_api ( info , session , aws_resource_search , types = None , allow_multi = False )

Function to find the RDS DB based on info

Parameters :

• info ( dict ) –

• session ( boto3.session.Session ) – Boto3 session for clients

• aws_resource_search ( str ) – Resource type we are after within the AWS Service, ie. cluster, instance

• types ( dict ) – Additional types to match.

Returns :

ecs_composex.common.aws. get_change_set_status ( client , change_set_name , settings )

ecs_composex.common.aws. get_cross_role_session ( session , arn , region_name = None , session_name = None )

Function to override ComposeXSettings session to specific session for Lookup

Parameters :

• session ( boto3.session.Session ) – The original session fetching the credentials for X-Role

• arn ( str ) –

• region_name ( str ) – Name of region for session

• session_name ( str ) – Override name of the session

Returns :

boto3 session from lookup settings

Return type :

boto3.session.Session

ecs_composex.common.aws. get_resources_from_tags ( session , aws_resource_search , search_tags )

Parameters :

• session ( boto3.session.Session ) – The boto3 session for API calls

• aws_resource_search ( str ) – AWS Service short code, ie. rds, ec2

• search_tags ( list ) – The tags to search the resource with.

Returns :

ecs_composex.common.aws. handle_multi_results ( arns , name , res_type , regexp , allow_multi = False )

Function to evaluate more than one result to see if we can match an unique name.

Parameters :

• arns ( list ) –

• name ( str ) –

• res_type ( str ) –

• regexp ( str ) –

Raises :

LookupError –

Returns :

The ARN of the resource matching the name.

ecs_composex.common.aws. handle_search_results ( arns , name , res_types , aws_resource_search , allow_multi = False )

Function to parse tag resource search results

Parameters :

• arns ( list ) –

• name ( str ) –

• res_types ( dict ) –

• aws_resource_search ( str ) –

Returns :

ecs_composex.common.aws. plan ( settings , root_stack )

Function to create a recursive change-set and return diffs :param ComposeXSettings settings: :param ComposeXStack root_stack: :return:

ecs_composex.common.aws. set_filters_from_tags_list ( tags )

Simple function to define the tags filters to use

Return type :

list

ecs_composex.common.aws. validate_search_input ( res_types , res_type )

Function to validate the search query

Parameters :

• res_types ( dict ) –

• res_type ( str ) –

Returns :

ecs_composex.common.aws. validate_stack_availability ( settings , root_stack )

Function to check that the stack can be updated :type settings: :param settings: :type root_stack: :param root_stack: :return:

ecs_composex.common.cfn_conditions module

Common Conditions across the templates

ecs_composex.common.cfn_conditions. define_stack_name ( template = None )

Function to return Stack name construct. Adds the conditions and parameters if template is given.

Parameters :

template ( troposphere.Template ) – the template to add it to.

Returns :

ecs_composex.common.cfn_conditions. pass_root_stack_name ( )

Function to add root_stack to a stack parameters

Returns :

rootstack name value based on condition

ecs_composex.common.cfn_params module

” Common parameters for CFN This is a crucial part as all the titles, marked _T are string which are then used the same way across all imports, which gives consistency for CFN to use the same names, which it heavily relies onto.

You can change the names values so you like so long as you keep it Alphanumerical [a-zA-Z0-9]

class ecs_composex.common.cfn_params. Parameter ( title , return_value = None , group_label = None , label = None , ** kwargs )

Bases: Parameter

Class to extend the default Parameter behaviour

title : str

ecs_composex.common.ecs_composex module

Global settings and variables re-used across the project

ecs_composex.common.envsubst module

Module to do a better env variables handling.

ecs_composex.common.envsubst. expandvars ( path , default = None , skip_escaped = True )

Expand environment variables of form $var and ${var}.

If parameter ‘skip_escaped’ is True, all escaped variable references (i.e. preceded by backslashes) are skipped. Unknown variables are set to ‘default’. If ‘default’ is None, they are left unchanged.

ecs_composex.common.files module

Functions to manage a template and wheter it should be stored in S3

class ecs_composex.common.files. FileArtifact ( file_name , settings , file_format = None , template = None , content = None )

Bases: object

Class to handle files artifacts, such as configuration files or templates. It will allow to upload the content to S3 or write to local filesystem. It also handles CloudFormation templates validation.

Variables :

• url ( str ) – The URL in S3 where the file will be uploaded to or available from.

• body ( str ) – The content of the FileArtifact

• template ( troposphere.Template ) – the CFN template

• file_name ( str ) – the base name of the file

• mime ( str ) – MIME-type of the file

• session ( boto3.session.Session ) – session for clients to make API calls to AWS

• can_upload ( bool ) – Indicate whether or not config allows for upload to S3.

• no_upload ( bool ) – Turns off upload if True

• validate ( bool ) – Indicates whether the template is validated.

• output_dir ( str ) – Path to the local director to output the file to.

• file_path ( str ) – Output file path for the FileArtifact

define_body ( )

Method to define the body of the file artifact. Sets the mime type that will be used for upload into S3.

define_file_specs ( file_name , file_format , settings )

Method to set the file body from template if self.template is Template

Parameters :

• file_name – name of the file

• file_format – format to use for the file.

• settings – The settings for execution

Returns :

file_path = None

mime = 'text/plain'

upload ( settings )

Method to handle uploading the files to S3.

validate ( settings )

Method to validate the CloudFormation template, either via URL once uploaded to S3 or via TemplateBody

write ( settings )

Method to write the files to local filesystem based on parameters (directory name etc.)

exception ecs_composex.common.files. RetryThis

Bases: Exception

ecs_composex.common.files. upload_file ( body , bucket_name , file_name , settings , prefix = None , mime = None )

Upload template_body to a file in s3 with given prefix and bucket_name

Parameters :

• body ( str ) – Template body, would come from troposphere template to_json() or to_yaml()

• bucket_name ( str ) – name of the bucket to upload the file to

• file_name ( str ) – Name of the file

• prefix ( str , optional ) – override default prefix for the file in S3

Returns :

url_path, the https://s3.amazonaws.com/ URL to the file

Return type :

str

ecs_composex.common.files. validate_wrapper ( session , body = None , url = None )

Parameters :

• session –

• body ( typing.Optional [ str ]) –

• url ( typing.Optional [ str ]) –

Returns :

ecs_composex.common.logging module

class ecs_composex.common.logging. ErrorFilter ( name = '' )

Bases: Filter

Inspired from https://stackoverflow.com/a/16066513

filter ( rec )

Determine if the specified record is to be logged.

Returns True if the record should be logged, or False otherwise. If deemed appropriate, the record may be modified in-place.

class ecs_composex.common.logging. InfoFilter ( name = '' )

Bases: Filter

Inspired from https://stackoverflow.com/a/16066513

filter ( rec )

Determine if the specified record is to be logged.

Returns True if the record should be logged, or False otherwise. If deemed appropriate, the record may be modified in-place.

class ecs_composex.common.logging. MyFormatter ( fmt = None , datefmt = None , style = '%' , validate = True )

Bases: Formatter

date_format = '%Y-%m-%d %H:%M:%S'

debug_format = '%(asctime)s [%(levelname)8s] (%(filename)s.%(lineno)d , %(funcName)s,) %(message)s'

default_format = '%(asctime)s [%(levelname)8s] %(message)s'

format ( record )

Format the specified record as text.

The record’s attribute dictionary is used as the operand to a string formatting operation which yields the returned string. Before formatting the dictionary, a couple of preparatory steps are carried out. The message attribute of the record is computed using LogRecord.getMessage(). If the formatting string uses the time (as determined by a call to usesTime(), formatTime() is called to format the event time. If there is exception information, it is formatted using formatException() and appended to the message.

Return type :

str

ecs_composex.common.logging. setup_logging ( )

ecs_composex.common.settings module

Module for the ComposeXSettings class

class ecs_composex.common.settings. ComposeXSettings ( content = None , profile_name = None , session = None , for_macro = False , ** kwargs )

Bases: object

Class to handle the settings to use for ECS ComposeX.

Variables :

• {str ( dict of ) – ComposeFamily} families: Map of families and services

• root_stack ( ComposeXStack ) –

• mod_manager ( ModManager ) –

active_commands = [{'name': 'up', 'help': 'Generates & Validates the CFN templates, Creates/Updates stack in CFN'}, {'name': 'render', 'help': 'Generates & Validates the CFN templates locally. No upload to S3'}, {'name': 'create', 'help': 'Generates & Validates the CFN templates locally. Uploads files to S3'}, {'name': 'plan', 'help': 'Creates a recursive change-set to show the diff prior to an update'}]

add_new_family ( family_name , service , assigned_services )

Return type :

None

add_service_to_family ( family_name , service , assigned_services )

Return type :

None

all_commands = [{'name': 'up', 'help': 'Generates & Validates the CFN templates, Creates/Updates stack in CFN'}, {'name': 'render', 'help': 'Generates & Validates the CFN templates locally. No upload to S3'}, {'name': 'create', 'help': 'Generates & Validates the CFN templates locally. Uploads files to S3'}, {'name': 'plan', 'help': 'Creates a recursive change-set to show the diff prior to an update'}, {'name': 'config', 'help': 'Merges docker-compose files to provide with the final compose content version'}, {'name': 'init', 'help': 'Initializes your AWS Account with prerequisites settings for ECS'}, {'name': 'version', 'help': 'ECS ComposeX Version'}]

allowed_formats = ['json', 'yaml', 'text']

arn_arg = 'RoleArn'

bucket_arg = 'BucketName'

cluster_name_arg = 'ClusterName'

command_arg = 'command'

config_render_arg = 'config'

create_arg = 'create'

create_ec2_arg = 'AddComputeResources'

create_vpc_arg = 'CreateVpc'

default_format = 'json'

default_output_dir = '/tmp/1709638667'

default_vpc_cidr = '100.127.254.0/24'

deploy_arg = 'up'

property disable_rollback : bool

ecr_arg = 'SkipScanEcrImages'

property ecs_cluster : EcsCluster

evaluate_private_namespace ( )

Method to go over all services and figure out if any of them requires cloudmap. If so it will also expect x-dns.PrivateNamespace to be set.

property family_names : list [ str ]

find_resource ( compose_resource_arn )

Return type :

ecs_composex.compose.x_resources.XResource

format_arg = 'TemplateFormat'

get_resource_attribute ( compose_resource_arn )

Return type :

tuple

get_x_resources ( include_new = True , include_mappings = True )

Returns the x_resources with filters

Parameters :

• include_new – Whether to add the new resources in the list

• include_mappings – Whether to add the new resources in the listF

Return type :

list

import_regional_mapping ( )

init_s3 ( )

Method to initialize S3 settings

Returns :

input_file_arg = 'DockerComposeXFile'

name_arg = 'Name'

neutral_commands = [{'name': 'init', 'help': 'Initializes your AWS Account with prerequisites settings for ECS'}, {'name': 'version', 'help': 'ECS ComposeX Version'}]

output_dir_arg = 'OutputDirectory'

override_session ( session , profile_name , kwargs )

Method to set the session based on input params

Parameters :

• session ( boto3.session.Session ) – The session to override the API calls with

• profile_name ( str ) – Name of a profile configured in .aws/config

• kwargs ( dict ) – CLI kwargs

parse_command ( kwargs , content = None )

Method to analyze the command and set execution settings accordingly.

Parameters :

• kwargs ( dict ) –

• content ( dict ) –

Returns :

plan_arg = 'plan'

region_arg = 'RegionName'

render_arg = 'render'

requires_vpc ( )

Determines whether the execution will require a VPC.

property service_names : list [ str ]

set_bucket_name_from_account_id ( )

Defines the default bucket name to use from the AWS Account ID

set_content ( kwargs , content = None , fully_load = True )

Method to initialize the compose content

Parameters :

• kwargs ( dict ) –

• content ( dict ) –

• fully_load ( bool ) –

set_efs ( )

Method to add a x-efs definition to the compose-x definition when a volume is flagged as using NFS/EFS

Return type :

None

set_families ( )

Method to define the list of families. Creating services_to_assign which won’t get duplicate services added, as they already are in other functions.

Return type :

None

set_networks ( vpc_stack )

Maps top level docker-compose networks with x-vpc subnets when applicable.

Return type :

None

set_output_settings ( kwargs )

Method to set the output settings based on kwargs

set_secrets ( )

Function to parse the settings compose content and define the secrets.

Return type :

None

set_services ( )

Method to define the ComposeXResource for each service. :rtype: None :return:

set_volumes ( )

Method configuring the volumes at root level :rtype: None :return:

single_nat_arg = 'SingleNat'

property stacks

validation_commands = [{'name': 'config', 'help': 'Merges docker-compose files to provide with the final compose content version'}]

vpc_cidr_arg = 'VpcCidr'

property x_resource_repr

property x_resources

Iterates over all resources defined and returns the list of them Only resources that are created from XResource(and children classes) are considered. Avoids having to go through stacks down to resources and work backwards

Returns: the list of XResource in the execution.

zones_arg = 'Zones'

ecs_composex.common.tagging module

From the docker ComposeX definition file, allows to add generic tags to all objects supporting AWS Tags from CFN

Obviously as per AWS CFN API, when you create a stack with Tags, these tags propagate to all resources that support tagging via CFN. Which is great, but very little people use that. And given that the AWS Stack itself has no cost, the FinOps people usually only are able to track the resources that generate actual billing.

By adding the tags to the resources wherever supported and have these as parameters, this enforces the presence of some tags, even though their values might differ. Using parameters to set the values also allows to copy-paste templates within the same organization and simply change the values at the time of creating the CFN Stack.

You might have IAM policies in place to try to force tagging. I find this doesn’t use a lot of parameters unless you had an insane number of tags.

When defining the tags in ComposeX as a list, the names of your tags can contain some special characters which otherwise you could not, i.e., vpc::usage::ecsapps

ecs_composex.common.tagging. add_all_tags ( root_template , settings , params = None , xtags = None )

Function to go through all stacks of a given template and update the template It will recursively render sub stacks defined. If there are no substacks, it will go over the resources of the template add the tags.

Parameters :

• root_template ( troposphere.Template ) – the root template to iterate over the resources.

• settings ( ecs_composex.common.settings.ComposeXSettings ) – Execution settings

• params ( list ) – Parameters to add to template if any

• xtags ( troposphere.Tags ) – List of Tags to add to the resources.

ecs_composex.common.tagging. add_object_tags ( obj , tags )

Function to add tags to the object if the object supports it

Parameters :

• obj – Troposphere object to add the tags to

• tags ( troposphere.Tags ) – list of tags as defined in Docker composeX file

ecs_composex.common.tagging. apply_tags_to_resources ( settings , resource , params , xtags )

Parameters :

• settings ( ecs_composex.common.settings.ComposeXSettings ) – Execution settings

• resource – The resource to add the tags to

• params ( list ) – Parameters to add to template if any

• xtags ( troposphere.Tags ) – List of Tags to add to the resources.

Returns :

ecs_composex.common.tagging. default_tags ( )

Function to return default tags to set on resource :return: default compose-x tags :rtype: troposphere.Tags

ecs_composex.common.tagging. define_extended_tags ( tags )

Function to generate the tags to be added to objects from x-tags

Parameters :

tags ( list or dict ) – tags as defined in composex file

Returns :

Tags() or None

Return type :

troposphere.Tags or None

ecs_composex.common.tagging. define_tag_parameter_title ( tag_name )

Returns the formatted name title for a given tag

Parameters :

tag_name ( str ) – name of the tag as defined in the ComposeX file

Returns :

reformatted tag name to work on CFN

Return type :

str

ecs_composex.common.tagging. expand_launch_template_tags_specs ( lt , tags )

Function to expand the LaunchTemplate TagSpecifications with defined x-tags.

Parameters :

• lt – the LaunchTemplate object

• tags ( troposphere.Tags ) – the Tags as built from x-tags

Type :

troposphere.ec2.LaunchTemplate

ecs_composex.common.tagging. generate_tags_parameters ( tags )

Function to generate a list of parameters used for the tags values

Returns :

list of parameters and tags to add to objects

Return type :

tuple

ecs_composex.common.tagging. merge_tags_lists ( x_data , y_data )

ecs_composex.common.troposphere_tools module

ecs_composex.common.troposphere_tools. add_defaults ( template )

Function to CFN parameters and conditions to the template which are used across ECS ComposeX

Parameters :

template ( Template ) – source template to add the params and conditions to

ecs_composex.common.troposphere_tools. add_outputs ( template , outputs )

Function to add parameters to the template

Parameters :

• template ( troposphere.Template ) – the template to add the parameters to

• outputs ( list<troposphere.Output> ) – list of parameters to add to the template

ecs_composex.common.troposphere_tools. add_parameter_to_group_label ( interface_metadata , parameter )

Simply goes over the ParameterGroups of the metadata.AWS::CloudFormation::Interface and if already exists, adds to group, else, create group and adds first element

Parameters :

• interface_metadata ( dict ) –

• parameter ( ecs_composex.common.cfn_params.Parameter ) –

Return type :

None

ecs_composex.common.troposphere_tools. add_parameters ( template , parameters )

Function to add parameters to the template

Parameters :

• template ( troposphere.Template ) – the template to add the parameters to

• parameters ( list<ecs_composex.common.cfn_params.Parameter> ) – list of parameters to add to the template

Return type :

None

ecs_composex.common.troposphere_tools. add_parameters_metadata ( template , parameter )

Simple function that will auto-add AWS::CloudFormation::Interface to the template if the parameter has a group and labels defined

Parameters :

• template –

• parameter –

Returns :

ecs_composex.common.troposphere_tools. add_resource ( template , resource , replace = False )

Function to add resource to template if the resource does not already exist

Parameters :

• template ( troposphere.Template ) –

• resource ( troposphere.AWSObject ) –

• replace ( bool ) –

Return type :

troposphere.AWSObject

ecs_composex.common.troposphere_tools. add_update_mapping ( template , mapping_key , mapping_value , mapping_subkey = None )

Parameters :

• template ( troposphere.Template ) –

• mapping_key ( str ) –

• mapping_value ( dict ) –

• mapping_subkey ( str ) – If set, applies the value to a sub-key of the mapping on update

Returns :

ecs_composex.common.troposphere_tools. add_update_parameter_recursively ( ext_stack , compose_settings , attribute_settings )

Recursively adds parameters to an external stack and updates the parameters as it goes.

if current external stack has no parent or the parent is the root stack,

use

attribute_settings[“ImportValue”] is the GetAtt stack.Outputs.<output_name> for that value.

Otherwise, we consider that we have started from a lower node, we add the direct reference to the value, and recursively go up the parent stacks until the first condition is met. We set the value to Ref given the value is given via parameter coming from the parent stack. If however the parameter is a List<> as defined in the AWS CFN Docs, we flatten the list with Join for nested Stack parameters.

ecs_composex.common.troposphere_tools. build_template ( description = None , * parameters )

Entry point function to creating the template for ECS ComposeX resources

Parameters :

• description ( str , optional ) – Optional custom description for the CFN template

• parameters ( List<troposphere.Parameters> , optional ) – List of optional parameters to add to the template.

Returns template :

the troposphere template

Return type :

Template

ecs_composex.common.troposphere_tools. init_template ( description = None )

Function to initialize the troposphere base template

Parameters :

description ( str ) – Description used for the CFN

Returns :

template

Return type :

Template

ecs_composex.common.troposphere_tools. no_value_if_not_set ( props , key , is_bool = False )

Function to simplify setting value if the key is in the dict and else Ref(AWS_NO_VALUE) for resource properties

Parameters :

• props ( dict ) –

• key ( str ) –

• is_bool ( bool ) –

Returns :

ecs_composex.common.troposphere_tools. set_get_resource ( template , resource )

Function to add resource to template if the resource does not already exist Returns the resource if it already does.

Module contents

Most commonly used functions shared across all modules.

ecs_composex.common. clpow2 ( x )

Function to return the closest power of two from given x

Parameters :

x – Number to look the closest power of two for

Returns :

int() closest power of two

ecs_composex.common. get_nested_property ( top_object , property_path , separator = None , to_update = None )

ecs_composex.common. nxtpow2 ( x )

Function to find the next power of two from given x number

Parameters :

x – number to look for the next power of two

Returns :

next power of two number