Fast Loading of into SqLite DB
Posted: Tue 14 Jul 2015 03:18
Hello,
I have a procedure I'm trying to optimize (currently taking several hours to load data).
Is there any way to speed this process up? The FileList.Count is around 20K per file and I have about 30 files to process. Using Unidac 6.1.3 for XE8, Win 7
Thanks for any suggestions.
Bryan
I have a procedure I'm trying to optimize (currently taking several hours to load data).
Is there any way to speed this process up? The FileList.Count is around 20K per file and I have about 30 files to process. Using Unidac 6.1.3 for XE8, Win 7
Thanks for any suggestions.
Bryan
Code: Select all
procedure ReadModelFileToDB(inFile, inStormID : String);
var
FileList : TStringList;
x : integer;
aRec : TModelRecord;
aModel : TModelObject;
aList : TList<TModelRecord>;
begin
DM.sqliteQuery.SQL.Clear;
//Create the table if it does not exist...
DM.sqliteQuery.SQL.Add('CREATE TABLE IF NOT EXISTS '+QuotedStr('file_'+inStormID)
+' (PosDateTime varchar(24), Model varchar(5), Tau Int, Lat double, Lon double, Wind Int, Pres Int)');
DM.sqliteQuery.Execute;
//add the data
aList := TList<TModelRecord>.Create;
aModel := TModelObject.Create;
FileList := TStringList.Create;
FileList.LoadFromFile(InFile);
Try
for x := 0 to FileList.Count -1 do
// reading a csv file (I make some formatting changes with the ExtractModelDataToModelRecord)
begin
aRec := aModel.ExtractModelDataToModelRecord(FileList.Strings[x]);
DM.sqliteQuery.SQL.Clear;
DM.sqliteQuery.SQL.Add('INSERT OR REPLACE INTO '+QuotedStr('file_'+inStormID)
+' VALUES ('+QuotedStr(aRec.PosDateTimeStr)+', '+ QuotedStr(aRec.MType)+', '
+ QuotedStr(IntToStr(aRec.TAU))+', '+ QuotedStr(FloatToStr(aRec.Lat))+', '
+ QuotedStr(FloatToStr(aRec.Lon))+', '+ QuotedStr(IntToStr(aRec.Wind))+', '
+ QuotedStr(IntToStr(aRec.Pressure))
+')');
DM.sqliteQuery.Execute;
end;
Finally
FileList.Free;
aModel.Free;
aList.Free;
End;
end;