x-kinesis

x-kinesis:
  stream:
    Properties: {} # AWS Kinesis CFN definition
    Settings: {}
    MacroParameters: {}
    Services: {}

Define Kinesis Data Streams, new or existing ones, that you wish your services to consume/produce from/to.

Services

As per the generic Services definition, we have a list of object, name and access, which define how the service can access the stream.

For AWS Kinesis streams, we have the following permissions.

ReturnValues

To access the Ref value, use StreamId

See AWS CFN Kinesis Return Values for available values.

IAM permissions

The following predefined permissions are available (see JSON definition of the IAM policy statement below).

• Producer

• Consumer

• PowerUser

IAM permissions pre-defined for your services.

{
  "Consumer": {
    "Effect": "Allow",
    "Action": [
      "kinesis:DescribeStream",
      "kinesis:GetRecords",
      "kinesis:ListShards"
    ],
    "Resource": [
      "${ARN}"
    ]
  },
  "Producer": {
    "Effect": "Allow",
    "Action": [
      "kinesis:DescribeStream",
      "kinesis:DescribeStreamSummary",
      "kinesis:DescribeStreamConsumer",
      "kinesis:SubscribeToShard",
      "kinesis:RegisterStreamConsumer",
      "kinesis:PutRecord",
      "kinesis:PutRecords",
      "kinesis:ListShards"
    ],
    "Resource": [
      "${ARN}"
    ]
  },
  "PowerUser": {
    "Effect": "Allow",
    "NotAction": [
      "kinesis:CreateStream",
      "kinesis:DeleteStream"
    ],
    "Resource": [
      "${ARN}"
    ]
  },
  "kinesis_firehose": {
    "kinesisSource": {
      "Effect": "Allow",
      "Action": [
        "kinesis:DescribeStream",
        "kinesis:GetShardIterator",
        "kinesis:GetRecords",
        "kinesis:ListShards"
      ],
      "Resource": [
        "${ARN}"
      ]
    }
  }
}

Properties

The Properties are the AWS CFN definition for AWS Kinesis streams .

Hint

If you leave the Properties empty, default values are used and ShardCount is set to 1.

MacroParameters

No specific MacroParameters for Kinesis data streams. Given the AWS definition is very straightforward, just define the properties. The only truly required property is the ShardCount

Examples

Services definition example

services: [serviceA, serviceB]

x-kinesis:
  streamA:
    Properties:
      ShardCount: 2
    Services:
      serviceA:
        Access: Producer
      serviceB:
        Access: Consumer

JSON Schema

Model

x-kinesis

x-kinesis.spec.json
x-kinesis specification
type	object
properties
Lookup	x-resources.common.spec.json#/definitions/Lookup
Properties	https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-kinesis-stream.html
	type	object
Settings	x-resources.common.spec.json#/definitions/Settings
Services	x-resources.common.spec.json#/definitions/Services
MacroParameters	type	object
	properties
	SetupIAMForKCL	type	object
		patternProperties
		^[a-zA-Z0-9-_\.]+$	#/definitions/SetupIAMForKCLDef
additionalProperties	False
definitions
SetupIAMForKCLDef	oneOf	Enables IAM permissions as described https://docs.aws.amazon.com/streams/latest/dev/tutorial-stock-data-kplkcl-iam.html
		type	boolean
		type	object
		properties
		CloudWatchPutMetricData	If true, allows PutMetricData to Cloudwatch Metrics
			type	boolean
		DynamoDB	oneOf	Grant all recommend IAM permissions for DynamoDB. Very permissive, use at own risks
				type	boolean
				type	object
				properties
				EnableCreateDeleteTables	If true, adds IAM permissions to create/delete any tables.
					type	boolean
				TableNames	List of DynamoDB tables to restrict the create/delete to instead of *
					type	array
					items	type	string
				additionalProperties	False
		additionalProperties	False

Definition

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "id": "x-kinesis.spec.json",
  "$id": "x-kinesis.spec.json",
  "title": "x-kinesis",
  "description": "x-kinesis specification",
  "type": "object",
  "additionalProperties": false,
  "properties": {
    "Lookup": {
      "$ref": "x-resources.common.spec.json#/definitions/Lookup"
    },
    "Properties": {
      "type": "object",
      "description": "https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-kinesis-stream.html"
    },
    "Settings": {
      "$ref": "x-resources.common.spec.json#/definitions/Settings"
    },
    "Services": {
      "$ref": "x-resources.common.spec.json#/definitions/Services"
    },
    "MacroParameters": {
      "type": "object",
      "properties": {
        "SetupIAMForKCL": {
          "type": "object",
          "patternProperties": {
            "^[a-zA-Z0-9-_\\.]+$": {
              "$ref": "#/definitions/SetupIAMForKCLDef"
            }
          }
        }
      }
    }
  },
  "oneOf": [
    {
      "required": [
        "Lookup"
      ]
    },
    {
      "required": [
        "Properties"
      ]
    }
  ],
  "definitions": {
    "SetupIAMForKCLDef": {
      "oneOf": [
        {
          "type": "boolean",
          "description": "Enables IAM permissions as described https://docs.aws.amazon.com/streams/latest/dev/tutorial-stock-data-kplkcl-iam.html"
        },
        {
          "type": "object",
          "additionalProperties": false,
          "properties": {
            "CloudWatchPutMetricData": {
              "type": "boolean",
              "description": "If true, allows PutMetricData to Cloudwatch Metrics"
            },
            "DynamoDB": {
              "oneOf": [
                {
                  "type": "boolean",
                  "description": "Grant all recommend IAM permissions for DynamoDB. Very permissive, use at own risks"
                },
                {
                  "type": "object",
                  "additionalProperties": false,
                  "properties": {
                    "EnableCreateDeleteTables": {
                      "type": "boolean",
                      "description": "If true, adds IAM permissions to create/delete any tables."
                    },
                    "TableNames": {
                      "type": "array",
                      "items": {
                        "type": "string"
                      },
                      "description": "List of DynamoDB tables to restrict the create/delete to instead of *"
                    }
                  }
                }
              ]
            }
          }
        }
      ]
    }
  }
}

Test files

You can find the test files here to use as reference for your use-case.