2
0
Dataframe Error: During the last post ,we saw LIMIT Clause has been used in SOQL to fetch data from big object. Now the question comes what happens, if remove the LIMIT from Query. We moved ahead and removed the Clause which allows to extract the complete data from Big object. Currently the objects holds 62K records in Salesforce. To our surprise during the snow_df.to_sql we get the following error and table remained blank:
There is SQL error implies that we are trying to insert the exceeded number of expressions in one go. While we have seen in earlier post, Ingestion works fine if we insert 10K records in one go. Hence there should be some mechanism or logic needs to be derive where dataframe should be divided into batches. In other words instead of ingesting the bulk dataframe in one step, we should segregate this frame into batches of 10K each.
Please find the below logic to divide the dataframe into batches.
We have replaced the above code instead of snow_df.to_sql. This code inserts the data into Snowflake in the batches of 10K records and resolved Dataframe Error. We have executed the cell and verified the records in Snowflake .
Sample statement to insert records in Big object:
/*List<snowdata__b> acc = new List<snowdata__b>();
for(integer i=0;i<10000;i++)
{
snowdata__b a = new snowdata__b(name__c=’Sachin’+i,snow__c=’test’+i,Age__C=10);
acc.add(a);
}
database.insertImmediate(acc);*/
