SalesForce SSIS Source not using Bulk API

Discussion of issues, suggestions and bugs of Devart SSIS Data Flow Components, our product line for building SSIS-based ETL solutions, performing data access to popular cloud applications and databases.
Post Reply
sharon
Posts: 1
Joined: Tue 29 May 2018 14:55

SalesForce SSIS Source not using Bulk API

Post by sharon » Tue 29 May 2018 15:05

I'm currently evaluating the SalesForce SSIS Data Flow Components to see if it would be a suitable replacement in our company for our existing solution.

We source millions of rows from SalesForce to a SQL Server database on a daily basis.

I have set up a sample SSIS package to pull from the SalesForce Account table. I have set the property of the Devart SalesForce Source to have BulkQuery = True. However, it doesn't appear like it is actually using the bulk API. I checked the "Monitor Bulk Data Load Jobs" page in SalesForce and its not showing my username there. Is there something else I have to do to get it to use the bulk API?

I also have an additional question. The maximum batch size it will allow me to select is 2000. We will very quickly burn through our 10,000 batches with this setting. Is there a way to increase the batch size to avoid exceeding the bulk API batch limit?

Thanks!

Shalex
Site Admin
Posts: 8327
Joined: Thu 14 Aug 2008 12:44

Re: SalesForce SSIS Source not using Bulk API

Post by Shalex » Fri 01 Jun 2018 19:16

sharon wrote:
Tue 29 May 2018 15:05
I have set up a sample SSIS package to pull from the SalesForce Account table. I have set the property of the Devart SalesForce Source to have BulkQuery = True. However, it doesn't appear like it is actually using the bulk API. I checked the "Monitor Bulk Data Load Jobs" page in SalesForce and its not showing my username there. Is there something else I have to do to get it to use the bulk API?
Bulk API is designed to quickly load data INTO Salesforce. So, this feature is used by Devart Salesforce Destination.

The BulkQuery property of Devart Salesforce Source determines whether to use optimizations for retrieving large amount of data. If True, provider tries to apply ORDER BY and LIMIT to get parts of result set step by step to avoid possible timeouts. Default value is False.
sharon wrote:
Tue 29 May 2018 15:05
I also have an additional question. The maximum batch size it will allow me to select is 2000. We will very quickly burn through our 10,000 batches with this setting. Is there a way to increase the batch size to avoid exceeding the bulk API batch limit?
The BatchSize property of Devart Salesforce Source specifies the number of rows to be retrieved as a batch. The maximum is 2000. Default is 500.

Refer to https://www.devart.com/dotconnect/sales ... Calls.html (doc is for our different product but it will help you to understand the way how API calls are countered, the difference is that the "SSIS Data Flow Components" product doesn't use local cache files but caches data in memory).

Post Reply