1
0
A Kinesis data stream is basically a publish/subscribe system used to collect and process large streams of data records in real time. Amazon Kinesis Data Streams (KDS) is a massively scalable and durable real-time data streaming service. KDS can continuously capture gigabytes of data per second from hundreds of thousands of sources such as website clickstreams, database event streams, financial transactions, social media feeds, location-tracking events etc . A Data Pipeline created using Stream, Kinesis Firehose and Snowflake would ingest the data inside Snowflake Data warehouse.
Web applications generates the streaming data (e.g. Log) and this data can be gathered by tools like Amazon Kinesis Streams. Kinesis Firehose is Amazon’s data-ingestion product offering for Kinesis. It is used to capture and load streaming data into other Amazon services such as S3 and Redshift.
It will consist of producers, delivery streams, and destination. Producers will produce data records of your preference and send it to delivery streams. Amazon Firehose is used to capture and load data into other Amazon services such as S3 and Redshift.
Kinesis Data Streams and Firehose are designed to work well together to build data pipeline. Firehose performs two tasks here:
It converts the data format to JSON
It packs the data in batches which it writes to an Amazon S3 bucket.
Snowflake consumes this data via SNOWPIPE in VARIANT column . Later on SQL is being used to segregate the JSON data into tabular form.
For Technical Implementation, Please Click here.
