AWS EventBridge Integration with Snowpipe

Read Time:2 Minute, 39 Second

During this post we are going to discuss the latest release from Snowflake i.e. AWS EventBridge integration with Snowpipe. Recently in April month, Snowflake has announced the Amazon EventBridge support for Snowpipe auto-ingest. As we know that in current scenario, we can configure the Snowpipe with below approaches:

Amazon SQS (Simple Queue Service)notifications for an S3 bucket.
Configuring Amazon SNS to Automate Snowpipe Using SQS Notifications

Therefore, in extension to this we can set up Amazon Event Bridge to automate Snowpipe in your snowflake account.

Amazon EventBridge, Serverless event bus that integrated SaaS applications (Snowflake for our use case), and AWS services. It helps in streaming real-time data from sources like S3 (in our use case) to destinations like Snowflake tables.

Though at this moment I don’t have a very strong use case for opting the EventBridge except one SNS limitation compared with EventBridge. The number of destinations or target systems in SNS are limited as compared to EventBridge, For example monitoring the status of GLUE job and sent an notification based on the Job Status or calling Lambda functions via SNS is not supported. On the other hand, EventBridge supports more than 15 targets and in addition to content-based event filtering etc.

Steps to be included:

Firstly, Enable EventBridge Notification on S3 Bucket.

SNS Topic Configure

Select Everyone for this use case in Access Policy-optional section.
Copy the JSON policy and save in TEXT file, We will modify it at later stage.
Create Topic and noted down the generated ARN.

2.1 Now Configure the SNS with Snowflake:

Copy the Topic ARN from AWS
Go to the snowflake and paste the Topic ARN in below Query

Select system$get_aws_sns_iam_policy('{}');

select system$get_aws_sns_iam_policy(arn:aws:sns:us-east-1:913267004595:eventbridge’);

Copy the text inside the Statement Array.
Go to the JSON Policy we saved in previous step and modify it.
Add Comma after Resource } and paste the value we copied from Statement Array.
Now Copy the modified JSON value and go to the SNS Topic you created earlier.
Modify the Access Policy : Delete the existing one and paste the Copied JSON value.

3. Snowflake Stage and PIPE setup

Go to Snowflake and create the STAGE and PIPE.

create or replace stage demo_db.public.event_csv_stage URL = 's3://eventbridgetesting/landing' STORAGE_INTEGRATION = s3_int file_format = demo_db.public.csv_format;

create or replace pipe eventbridge_pipe auto_ingest = true aws_sns_topic = 'arn:aws:sns:us-east-1:913267004595:eventbridge' as copy into prod_tbl from @event_csv_stage;

show pipes;

A subscription record will be created under TOPIC.