UniDirectional and FetchRows
Posted: Tue 31 Jul 2018 13:27
Hi,
I am helping a customer with some software development.
They use UniDAC. At some point in time I have set a UniDAC Query to UniDirectional and FetchRows to 50000 records to avoid memory problems, they have millions of tuples in the database with hundreds of fields on each tuple.
Basically they run through the recordset and for each database-record the create a delphi-record and keep this in a sortable list.
It works perfectly. Memory consumption works now better, as the unidac records are released every time we get to retreiving more tuples.
But it only works perfectly when the exe-file is run locally. So let's say we have a client machine and a database server.
If, however the exe-file is installed on an application server, with a shared drive, and I run the application from a remote path like \\myserver\myApp\Databaseapp.exe something weird happens with performance.
I have timed the calls to query.open, and what we have and seemingly actually retreiving the data goes fine. But calls to retreiving data out of a dataset record and/or calling query.next is really messing something up.
So
myrec.Info1 := query.Fields[1].asString;
myrec.info2 := query.Fields[2].asString;
followed by query.next;
Is really really really slow if the executable is on a remote drive, but only with UniDirectional := true and FetchAll := False;
Running through a million records (and handling whatever handling needs done) takes :
Unidirection := true; FetchAll := False; RemoteDrive ==> 1 hour (give or take).
UniDirection := true; FetachAll := False; LocalDrive ==> 5 minutes
UniDirection := False; FetchAll := True; RemoteDrive ==> 6 minutes
UniDirection := False; FetchAll := True; LocalDrive ==> 5 minutes.
So, the combination of a remote drive, Unidirectional and Fetchall is false has a serious impact.
Can anyone give a fair explanation of why? I mean I understand there should be an impact on running on a remote drive, but why is this impact 12-fold in this exact combination??
Thanks in advance
Jens Fudge
I am helping a customer with some software development.
They use UniDAC. At some point in time I have set a UniDAC Query to UniDirectional and FetchRows to 50000 records to avoid memory problems, they have millions of tuples in the database with hundreds of fields on each tuple.
Basically they run through the recordset and for each database-record the create a delphi-record and keep this in a sortable list.
It works perfectly. Memory consumption works now better, as the unidac records are released every time we get to retreiving more tuples.
But it only works perfectly when the exe-file is run locally. So let's say we have a client machine and a database server.
If, however the exe-file is installed on an application server, with a shared drive, and I run the application from a remote path like \\myserver\myApp\Databaseapp.exe something weird happens with performance.
I have timed the calls to query.open, and what we have and seemingly actually retreiving the data goes fine. But calls to retreiving data out of a dataset record and/or calling query.next is really messing something up.
So
myrec.Info1 := query.Fields[1].asString;
myrec.info2 := query.Fields[2].asString;
followed by query.next;
Is really really really slow if the executable is on a remote drive, but only with UniDirectional := true and FetchAll := False;
Running through a million records (and handling whatever handling needs done) takes :
Unidirection := true; FetchAll := False; RemoteDrive ==> 1 hour (give or take).
UniDirection := true; FetachAll := False; LocalDrive ==> 5 minutes
UniDirection := False; FetchAll := True; RemoteDrive ==> 6 minutes
UniDirection := False; FetchAll := True; LocalDrive ==> 5 minutes.
So, the combination of a remote drive, Unidirectional and Fetchall is false has a serious impact.
Can anyone give a fair explanation of why? I mean I understand there should be an impact on running on a remote drive, but why is this impact 12-fold in this exact combination??
Thanks in advance
Jens Fudge