x-kinesis

x-kinesis:
  stream:
    Properties: {} # AWS Kinesis CFN definition
    Settings: {}
    MacroParameters: {}
    Services: {}

Define Kinesis Data Streams, new or existing ones, that you wish your services to consume/produce from/to.

Services

As per the generic Services definition, we have a list of object, name and access, which define how the service can access the stream.

For AWS Kinesis streams, we have the following permissions.

ReturnValues

To access the Ref value, use StreamId

See AWS CFN Kinesis Return Values for available values.

IAM permissions

The following predefined permissions are available (see JSON definition of the IAM policy statement below).

  • Producer

  • Consumer

  • PowerUser

IAM permissions pre-defined for your services.
{
  "Consumer": {
    "Effect": "Allow",
    "Action": [
      "kinesis:DescribeStream",
      "kinesis:GetRecords",
      "kinesis:ListShards"
    ],
    "Resource": [
      "${ARN}"
    ]
  },
  "Producer": {
    "Effect": "Allow",
    "Action": [
      "kinesis:DescribeStream",
      "kinesis:DescribeStreamSummary",
      "kinesis:DescribeStreamConsumer",
      "kinesis:SubscribeToShard",
      "kinesis:RegisterStreamConsumer",
      "kinesis:PutRecord",
      "kinesis:PutRecords",
      "kinesis:ListShards"
    ],
    "Resource": [
      "${ARN}"
    ]
  },
  "PowerUser": {
    "Effect": "Allow",
    "NotAction": [
      "kinesis:CreateStream",
      "kinesis:DeleteStream"
    ],
    "Resource": [
      "${ARN}"
    ]
  },
  "kinesis_firehose": {
    "kinesisSource": {
      "Effect": "Allow",
      "Action": [
        "kinesis:DescribeStream",
        "kinesis:GetShardIterator",
        "kinesis:GetRecords",
        "kinesis:ListShards"
      ],
      "Resource": [
        "${ARN}"
      ]
    }
  }
}

Properties

The Properties are the AWS CFN definition for AWS Kinesis streams .

Hint

If you leave the Properties empty, default values are used and ShardCount is set to 1.

MacroParameters

No specific MacroParameters for Kinesis data streams. Given the AWS definition is very straightforward, just define the properties. The only truly required property is the ShardCount

Examples

Services definition example
services: [serviceA, serviceB]

x-kinesis:
  streamA:
    Properties:
      ShardCount: 2
    Services:
      serviceA:
        Access: Producer
      serviceB:
        Access: Consumer

JSON Schema

Model

x-kinesis

x-kinesis.spec.json

x-kinesis specification

type

object

properties

  • Lookup

x-resources.common.spec.json#/definitions/Lookup

  • Properties

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-kinesis-stream.html

type

object

  • Settings

x-resources.common.spec.json#/definitions/Settings

  • Services

x-resources.common.spec.json#/definitions/Services

  • MacroParameters

type

object

properties

  • SetupIAMForKCL

type

object

patternProperties

  • ^[a-zA-Z0-9-_\.]+$

#/definitions/SetupIAMForKCLDef

additionalProperties

False

definitions

  • SetupIAMForKCLDef

oneOf

Enables IAM permissions as described https://docs.aws.amazon.com/streams/latest/dev/tutorial-stock-data-kplkcl-iam.html

type

boolean

type

object

properties

  • CloudWatchPutMetricData

If true, allows PutMetricData to Cloudwatch Metrics

type

boolean

  • DynamoDB

oneOf

Grant all recommend IAM permissions for DynamoDB. Very permissive, use at own risks

type

boolean

type

object

properties

  • EnableCreateDeleteTables

If true, adds IAM permissions to create/delete any tables.

type

boolean

  • TableNames

List of DynamoDB tables to restrict the create/delete to instead of *

type

array

items

type

string

additionalProperties

False

additionalProperties

False

Definition

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "id": "x-kinesis.spec.json",
  "$id": "x-kinesis.spec.json",
  "title": "x-kinesis",
  "description": "x-kinesis specification",
  "type": "object",
  "additionalProperties": false,
  "properties": {
    "Lookup": {
      "$ref": "x-resources.common.spec.json#/definitions/Lookup"
    },
    "Properties": {
      "type": "object",
      "description": "https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-kinesis-stream.html"
    },
    "Settings": {
      "$ref": "x-resources.common.spec.json#/definitions/Settings"
    },
    "Services": {
      "$ref": "x-resources.common.spec.json#/definitions/Services"
    },
    "MacroParameters": {
      "type": "object",
      "properties": {
        "SetupIAMForKCL": {
          "type": "object",
          "patternProperties": {
            "^[a-zA-Z0-9-_\\.]+$": {
              "$ref": "#/definitions/SetupIAMForKCLDef"
            }
          }
        }
      }
    }
  },
  "oneOf": [
    {
      "required": [
        "Lookup"
      ]
    },
    {
      "required": [
        "Properties"
      ]
    }
  ],
  "definitions": {
    "SetupIAMForKCLDef": {
      "oneOf": [
        {
          "type": "boolean",
          "description": "Enables IAM permissions as described https://docs.aws.amazon.com/streams/latest/dev/tutorial-stock-data-kplkcl-iam.html"
        },
        {
          "type": "object",
          "additionalProperties": false,
          "properties": {
            "CloudWatchPutMetricData": {
              "type": "boolean",
              "description": "If true, allows PutMetricData to Cloudwatch Metrics"
            },
            "DynamoDB": {
              "oneOf": [
                {
                  "type": "boolean",
                  "description": "Grant all recommend IAM permissions for DynamoDB. Very permissive, use at own risks"
                },
                {
                  "type": "object",
                  "additionalProperties": false,
                  "properties": {
                    "EnableCreateDeleteTables": {
                      "type": "boolean",
                      "description": "If true, adds IAM permissions to create/delete any tables."
                    },
                    "TableNames": {
                      "type": "array",
                      "items": {
                        "type": "string"
                      },
                      "description": "List of DynamoDB tables to restrict the create/delete to instead of *"
                    }
                  }
                }
              ]
            }
          }
        }
      ]
    }
  }
}

Test files

You can find the test files here to use as reference for your use-case.