Copying an entire contents of TCustomDADataSet to TVirtualTable row-by-row

[aoven]

Hi,

I'm looking for a faster way to do this:

Code: Select all

while not SrcDataSet.Eof do begin
  DstDataSet.Append;
  try
    for I := 0 to SrcDataSet.FieldCount - 1 do
      DstDataSet.Fields[I].Value := SrcDataSet.Fields[I].Value;
  finally
    DstDataSet.Post;
  end;

  SrcDataSet.Next;
end;

Please bear in mind that this is a contrived example. In real code I'm sending rows in batches from a background thread to the UI, so it's a requirement that I do it row-by-row. Since SrcDataSet can contain a large number of records (hundreds of thousands), this loop with Variant-based copying of values makes for a considerable overall slowdown (I confirmed it with a profiler).

Field structures of both datasets are identical, so I believe it should be possible to somehow directly assign row buffers from SrcDataSet to DstDataSet. I've tried experimenting with this code:

Code: Select all

DstDataSet.Append;
try
  Move(SrcDataSet.ActiveBuffer^, DstDataSet.ActiveBuffer^, SrcDataSet.RecordSize);
finally
  DstDataSet.Post;
end;

It runs fine, but DstDataSet doesn't seem to see the changes - all fields in the DstDataSet have Null values afterwards.

Is there any other way? I don't mind changing the source, if necessary. I want to avoid the unnecessary looping, variant-based assignments and intermediate buffering of any kind. I also wish to avoid triggering as many events and internal checks in both datasets as possible in order to gain speed.

Access is strictly sequential and datasets are identical copies, as seen from my example above, so error checking and nulling field values before copying them (which Append does internally) is all completely superfluous.

I suspect the solution, if it exists, will be close to what I did above with ActiveBuffer assignment, but probably with additional manipulation of the internal Data of the DstDataSet (which is TVirtualTable), so that changes become visible to DstDataSet clients.

Any help will be greatly appreciated. Thanks in advance!

[Antaeus]

There is no such possibility in TVirtualTable. The recommend way of filling TVirtualTable from other datasets is the TCRBatchMove component or the TVirtualTable.Assign method.

[aoven]

Well, I found a way to make it work, at least with TMSQuery -> TVirtualTable (haven't tested TOraQuery -> TVirtualTable yet).

This is what I did:

Code: Select all

type
  TFastVirtualTable = class(TVirtualTable)
  public
    procedure AddRecordFast(aBuffer: PChar);
  end;

  THackData = class(TData);

procedure TFastVirtualTable.AddRecordFast(aBuffer: PChar);
var
  RS, I: Integer;
  RecBuf, Indicator: PAnsiChar;
begin
  Append();

  RS := RecordSize;
  I := THackData(Data).IndicatorSize;

  GetActiveRecBuf(RecBuf);
  Move(aBuffer^, RecBuf^, RS - I);

  Indicator := PAnsiChar(Cardinal(RecBuf) + Cardinal(RS - I));
  FillChar(Indicator^, I, 0);

  Post();
end;

...

var
  SrcDataSet: TMSQuery;
  DstDataSet: TFastVirtualTable;
begin
  ...
  SrcDataSet.Options.FlatBuffers := True; // This is required to even out RecordSize in both datasets.
  SrcDataSet.Open;
  while not SrcDataSet.Eof do begin
    DstDataSet.AddRecordFast(SrcDataSet.ActiveBuffer);
    SrcDataSet.Next;
  end;
end;

It seems to be working and it's light years faster on large datasets than the loop-based assignment of field values.

The only thing I'm not quite satisfied with is the handling of the IndicatorSize block at the end of the record buffer. As you can see, my solution was to simply zero it out. It works, but it feels ugly. Maybe someone from CoreLab team can improve it further.

Anyway, I'd be interested to hear everyone's opinions.

Aleksander

[aoven]

Just letting everyone know: it works with TOraQuery, too!

Aleksander