1
0
Monitoring and optimizing cloud costs is a key challenge for businesses operating in cloud environments. Snowflake provides detailed usage insights, but integrating this data with AWS CloudWatch using External Functions allows organizations to track cost in real-time, set up alerts, and optimize warehouse utilization.
What if we could integrate Snowflake warehouse cost tracking with AWS CloudWatch?
This would allow businesses to visualize Snowflake warehouse costs in CloudWatch dashboards, detect anomalies, and trigger alerts when costs exceed thresholds
In this blog, we’ll explore how to push Snowflake warehouse usage data to AWS CloudWatch in real time using Snowflake External Functions and AWS Lambda. A unique and innovative solution that bridges data warehousing and cloud monitoring seamlessly.
Use Case: Real-Time Warehouse Cost Monitoring
Imagine a data-driven company using Snowflake for analytics and AWS CloudWatch for monitoring infrastructure. The finance and DevOps teams want to track Snowflake warehouse usage costs in real time to:
- Detect cost anomalies (e.g., unexpected increases in warehouse usage)
- Trigger alerts when warehouse spending crosses thresholds
- Optimize warehouse configurations based on historical usage trends
By integrating Snowflake and AWS CloudWatch, we can push Snowflake cost metrics to CloudWatch and visualize trends or trigger alarms when costs exceed limits.
Solution Overview
This solution consists of five main components:
Components Involved
1️Snowflake Warehouse Cost Data – Extracted from ACCOUNT_USAGE.WAREHOUSE_METERING_HISTORY.
2️Snowflake External Function – Calls an AWS Lambda function via API Gateway.
3️AWS API Gateway – Exposes an HTTP endpoint for Snowflake to send data.
4️AWS Lambda Function – Processes data and pushes metrics to AWS CloudWatch.
5️AWS CloudWatch – Stores and visualizes Snowflake cost data
Implementation Steps
Implementation Steps
1️Create AWS Lambda Function for CloudWatch Integration
The following AWS Lambda function receives Snowflake data, extracts warehouse costs, and pushes them to AWS CloudWatch.
Configure the Proxy Service i.e. Amazon API Gateway.
- Create an API Gateway linked to your Lambda function.
- Configure the API Gateway to accept requests and invoke the Lambda function.
- Deploy the API Gateway and note the endpoint URL (e.g., https://<your-api-endpoint>).
Create the API Integration.
create or replace api integration my_SEND_CLOUDWATCH_METRIC
api_provider = aws_api_gateway
api_aws_role_arn = 'arn:aws:iam::XXXXXXXX:role/custom_cloudwatch'
api_allowed_prefixes = ('https://XXXXX.execute-api.xx-east-1.amazonaws.com/cloudwatch_stage/cloudwatch_resource')
enabled = true;
Create the External Function.
CREATE OR REPLACE EXTERNAL FUNCTION SEND_CLOUDWATCH_METRIC(
METRICNAME STRING,
VALUE NUMBER,
UNIT STRING,
NAMESPACE STRING,
DIMENSION_JSON ARRAY
)
RETURNS VARIANT
API_INTEGRATION = my_SEND_CLOUDWATCH_METRIC
AS 'https://tjnaqtshgh.execute-api.us-east-1.amazonaws.com/cloudwatch_stage/cloudwatch_resource';
Call the External function.
This function sends requests to the API Gateway, which invokes the Lambda function. Query Snowflake Warehouse Costs & Send to CloudWatch.
Monitoring in AWS CloudWatch
Once the query executes, data appears in AWS CloudWatch under the namespace Snowflake/WarehouseMetrics.
- Go to AWS CloudWatch → Metrics
- Select Namespace: Snowflake/WarehouseMetrics
- View warehouse cost trends and set up alerts for anomalies.
Conclusion
By integrating Snowflake warehouse cost tracking with AWS CloudWatch, businesses can gain real-time visibility into cloud spending, set up proactive alerts, and optimize resource allocation.
