How to consolidate list of inner elements from map into a map with a value of set type

Question

Hello I'm trying to convert a set of bucket notifications defined as follows to a map: sequence structure so I can iterate over the sequence with a dynamic block in my bucket notifications.

buckets_with_events = [
  {
    "bucket" = "bucket1"
    "events" = [
      "s3:ObjectCreated:*",
    ]
    "filter_prefix" = tostring(null)
    "filter_suffix" = ".xml"
    "function" = "lambda1"
  },
  {
    "bucket" = "bucket1"
    "events" = [
      "s3:ObjectCreated:*",
    ]
    "filter_prefix" = tostring(null)
    "filter_suffix" = ".csv"
    "function" = "lambda1"
  },
  {
    "bucket" = "bucket2"
    "events" = [
      "s3:ObjectCreated:*",
    ]
    "filter_prefix" = tostring(null)
    "filter_suffix" = ".csv"
    "function" = "lambda1"
  }]

I need to have the buckets as a unique key and have all events consolidated as a single sequence under that key. I have been trying the following, but I can't seem to correctly assign the tuple/set/sequence to the map key.

  bucket_events = flatten([
    for parent_obj in local.buckets_with_events:
        parent_obj.bucket => [
          for obj in local.buckets_with_events:
            {events = obj.events
             filter_suffix = obj.filter_suffix} if parent_obj.bucket == obj.bucket
    ]
  ])

The output I want to achieve is the following:

bucket_events = {
  "bucket1" = [{
    
    "events" = [
      "s3:ObjectCreated:*",
    ]
    "filter_prefix" = tostring(null)
    "filter_suffix" = ".xml"
    "function" = "lambda1"
  },
  {
    "events" = [
      "s3:ObjectCreated:*",
    ]
    "filter_prefix" = tostring(null)
    "filter_suffix" = ".csv"
    "function" = "lambda1"
  }]
  "bucket2" = [{
    "events" = [
      "s3:ObjectCreated:*",
    ]
    "filter_prefix" = tostring(null)
    "filter_suffix" = ".csv"
    "function" = "lambda1"
  }]

In this way, I can easily generate all my bucket notifications that are required using for_each and dynamic as below --

resource "aws_s3_bucket_notification" "dynamic_s3_triggers" {
  for_each = var.bucket_events
  bucket = each.key
  dynamic "lambda_function" {

      content {
      lambda_function_arn = aws_lambda_function.functions[lambda_function.function].function].arn
      ...
       }
    }
}

Could you please edit your question and describe the expected output for variable buckets_with_events? — Rui Jarimba
– Rui Jarimba, Commented Nov 13, 2024 at 17:42
@RuiJarimba I updated the question with more detail about the expected output and context around how I want to use it. — Robin Tanner
– Robin Tanner, Commented Nov 13, 2024 at 17:48

Lorenzo Felletti · Accepted Answer · 2024-11-13 23:20:41Z

(Possible duplicate of this one)

The following code achieves what you're asking for:

  map_bucket_all_events = {
    for bucket in local.buckets_with_events : bucket.bucket => {
      events        = bucket.events
      filter_prefix = bucket.filter_prefix
      filter_suffix = bucket.filter_suffix
      function      = bucket.function
    }...
  }

The "magic" is done using the ellipsis operator, which is used to perform a sort of "group by" operation (docs here) based on the key.

I've tested it locally using terraform console and that's the output I get:

{
  "bucket1" = [
    {
      "events" = [
        "s3:ObjectCreated:*",
      ]
      "filter_prefix" = tostring(null)
      "filter_suffix" = ".xml"
      "function" = "lambda1"
    },
    {
      "events" = [
        "s3:ObjectCreated:*",
      ]
      "filter_prefix" = tostring(null)
      "filter_suffix" = ".csv"
      "function" = "lambda1"
    },
  ]
  "bucket2" = [
    {
      "events" = [
        "s3:ObjectCreated:*",
      ]
      "filter_prefix" = tostring(null)
      "filter_suffix" = ".csv"
      "function" = "lambda1"
    },
  ]
}

Rui Jarimba · Accepted Answer · 2024-11-13 23:41:39Z

0

Consider the following code:

locals {
  buckets_with_events = [
    {
      "bucket" = "bucket1"
      "events" = [
        "s3:ObjectCreated:*",
      ]
      "filter_prefix" = tostring(null)
      "filter_suffix" = ".xml"
      "function"      = "lambda1"
    },
    {
      "bucket" = "bucket1"
      "events" = [
        "s3:ObjectCreated:*",
      ]
      "filter_prefix" = tostring(null)
      "filter_suffix" = ".csv"
      "function"      = "lambda1"
    },
    {
      "bucket" = "bucket2"
      "events" = [
        "s3:ObjectCreated:*",
      ]
      "filter_prefix" = tostring(null)
      "filter_suffix" = ".csv"
      "function"      = "lambda1"
  }]

  distinct_buckets = distinct([
    for be in local.buckets_with_events : be.bucket
  ])

  buckets_map = {
    for bucket_event in local.distinct_buckets :
    bucket_event => [
      for be in local.buckets_with_events :
      {
        events        = be.events
        filter_prefix = be.filter_prefix
        filter_suffix = be.filter_suffix
        function      = be.function
      }
      if be.bucket == bucket_event
    ]
  }
}

output "buckets_map" {
  value = local.buckets_map
}

Running terraform plan:

Changes to Outputs:
  + buckets_map = {
      + bucket1 = [
          + {
              + events        = [
                  + "s3:ObjectCreated:*",
                ]
              + filter_prefix = null
              + filter_suffix = ".xml"
              + function      = "lambda1"
            },
          + {
              + events        = [
                  + "s3:ObjectCreated:*",
                ]
              + filter_prefix = null
              + filter_suffix = ".csv"
              + function      = "lambda1"
            },
        ]
      + bucket2 = [
          + {
              + events        = [
                  + "s3:ObjectCreated:*",
                ]
              + filter_prefix = null
              + filter_suffix = ".csv"
              + function      = "lambda1"
            },
        ]
    }

edited Nov 13, 2024 at 23:41

answered Nov 13, 2024 at 18:07

Rui Jarimba

18.5k11 gold badges64 silver badges101 bronze badges

3 Comments

Robin Tanner Over a year ago

Okay this is great but you can see in the output the results are actually duplicated. How do I remove the duplicates? Also it is a list of lists, how do I unnest the lists?

Rui Jarimba Over a year ago

@RobinTanner my apologies, I was completely distracted. I'll fix it soon.

Rui Jarimba Over a year ago

@RobinTanner answer was updated.

Collectives™ on Stack Overflow

How to consolidate list of inner elements from map into a map with a value of set type

2 Answers 2

Comments

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related