Connection pooling setup

Discussion of open issues, suggestions and bugs regarding UniDAC (Universal Data Access Components) for Delphi, C++Builder, Lazarus (and FPC)
andrefm
Posts: 37
Joined: Wed 23 Oct 2013 10:02

Re: Connection pooling setup

Post by andrefm » Thu 25 Jun 2020 12:13

Sorry for another question in regards the pooling setup, but I am making some tests and discovering other behaviours which I am not sure if it is normal.
It seems the UniConnection does actually already create a pool in background, is this correct? Because If I run the code below, I do see UniDAC creating multiple connections to database even when Pooling = False.

Code: Select all

var
  LPooling: string;
  LConnection1: TUniConnection;
begin
  Button1.Enabled := False;
  Button1.Caption := 'Running..';

  LConnection1 := TUniConnection.Create(nil);
  LConnection1.ProviderName := 'MySQL';
  LConnection1.Server := '192.168.0.171';
  LConnection1.Database := 'sakila';
  LConnection1.Username := 'tutorial';
  LConnection1.Password := 'Tutorial.123!';
  LConnection1.LoginPrompt := False;
//  LConnection1.PoolingOptions.MaxPoolSize := 5;
//  LConnection1.PoolingOptions.MinPoolSize := 5;
//  LConnection1.PoolingOptions.ConnectionLifetime := 5000; // 60 seconds
//  LConnection.PoolingOptions.Validate := True;
  LConnection1.Pooling := False;
  LConnection1.Macros.Clear;
  LConnection1.Connect;

    TTask.Run(
    procedure
    var
      LIndex: Integer;
      LStopWatch: TStopWatch;
      LTasks: TArray<ITask>;

    begin
      LStopWatch := TStopWatch.StartNew;
      LTasks := [];
      for LIndex := 1 to 1000 do
      begin
        LTasks := LTasks + [TTask.Run(
          procedure
          var
            LQuery: TUniQuery;
            i: Integer;
          begin
            try
              LQuery := TUniQuery.Create(nil);
              try
                LQuery.Connection := LConnection1;
                LQuery.UniDirectional := true;
                LQuery.FetchRows := 30000;
                LQuery.Options.StrictUpdate := false;
                LQuery.SQL.Text := ' select * from film_list ';
                LQuery.Open;
                LQuery.Last;
                LQuery.Close;
              finally
                FreeAndNil(LQuery);
              end;
            finally

            end;
          end)];

      end;
      TTask.WaitForAll(LTasks);

      LStopWatch.Stop;
      TThread.Synchronize(nil,
        procedure
        begin
          Memo1.Lines.Add('MYSQL - '+LPooling + LStopWatch.Elapsed.ToString());
          Button1.Enabled := True;
          Button1.Caption := 'Button1'
        end);
    end);
But If I add a second connection and do the same (TTask ...), I only get one connection (at least this is what I noticed) for the second UniConnection which made me sad again as I need more than one pool...

oleg0k
Devart Team
Posts: 190
Joined: Wed 11 Mar 2020 08:28

Re: Connection pooling setup

Post by oleg0k » Tue 30 Jun 2020 15:27

Hello,
You can leave your suggestions at our UserVoice page (https://devart.uservoice.com/forums/104 ... y_id=18939), and if there are many votes for your suggestion, we will consider the possibility to implement it.
Please send us an example demonstrating the incorrect behavior, along with the DDL scripts for creating and populating the database objects, through the contact form on our website:
https://devart.com/company/contactform.html

wbr, Oleg
Devart Team

oleg0k
Devart Team
Posts: 190
Joined: Wed 11 Mar 2020 08:28

Re: Connection pooling setup

Post by oleg0k » Tue 30 Jun 2020 15:35

Hello,
You can leave your suggestions at our UserVoice page ( https://devart.uservoice.com/forums/104 ... y_id=18939 ),
and if there are many votes for your suggestion, we will consider the possibility to implement it.
Please send us an example demonstrating the incorrect behavior, along with the DDL scripts for creating and populating the database objects, through the contact form on our website:
https://devart.com/company/contactform.html

wbr, Oleg
Devart Team

andrefm
Posts: 37
Joined: Wed 23 Oct 2013 10:02

Re: Connection pooling setup

Post by andrefm » Thu 16 Jul 2020 00:36

I added the request for multiple connection pools in one application to UserVoice, but I would appreciate any feedback and suggestions on how I could try to implement this as I believe this change won't happen any time soon from DevArt side.
Thank you

oleg0k
Devart Team
Posts: 190
Joined: Wed 11 Mar 2020 08:28

Re: Connection pooling setup

Post by oleg0k » Fri 24 Jul 2020 11:09

Hello,
We appreciate you taking the time to post your suggestion on UserVoice, but we cannot guarantee that it will be implemented anytime soon, since we have many features and improvements on the roadmap. We'll consider adding the requested feature once your suggestion receives more upvotes.

wbr, Oleg
Devart Team

darnocian
Posts: 5
Joined: Sun 23 Aug 2020 11:07

Re: Connection pooling setup

Post by darnocian » Wed 26 Aug 2020 13:44

I just have a related issue... trying to figure out how connection + connection pool + transactions would work in a multithreaded environment.... rather than just on a gui with relatively static components...

I would have assumed a component decomposition:
- connnection pool
- connection connected to connection pool (active = true it is not in 'free' pool and in use, active=false it is back in 'free' pool)
- transaction is associated with a connection. with pooling enabled on a connection, I don't know what 'pooling' actually means... is there some thread tracking taking place via tls variables?

anyone done multitheeading with transactions?

oleg0k
Devart Team
Posts: 190
Joined: Wed 11 Mar 2020 08:28

Re: Connection pooling setup

Post by oleg0k » Wed 02 Sep 2020 11:20

Hello,

Connection pooling reduces the amount of time a user must wait to establish a connection to the database and increases performance. It has nothing to do with transactions. Read more in our tutorial: https://www.devart.com/unidac/docs/work_pooling.htm
It seems like you're getting an exception because a transaction is started for the first thread and then it becomes active for all other threads. If you need a separate transaction for each thread, you should create an instance of TUniTransaction for each thread and control it programatically. Some servers do not allow creating several transactions per connection, in which case you need to create a separate TUniConnection for each thread.

wbr, Oleg
Devart Team

andrefm
Posts: 37
Joined: Wed 23 Oct 2013 10:02

Re: Connection pooling setup

Post by andrefm » Sun 26 Sep 2021 22:58

oleg0k wrote: Tue 30 Jun 2020 15:35 Hello,
You can leave your suggestions at our UserVoice page ( https://devart.uservoice.com/forums/104 ... y_id=18939 ),
and if there are many votes for your suggestion, we will consider the possibility to implement it.
Please send us an example demonstrating the incorrect behavior, along with the DDL scripts for creating and populating the database objects, through the contact form on our website:
https://devart.com/company/contactform.html

wbr, Oleg
Devart Team
Hi Oleg,
It is me again about this. Since I really need this, I started checking the source code to find how to change and implement this functionality. The first thing I noticed is that UNIDAC was meant to support multiple connection pools, however, it is not working properly (bug or implementation issue). I didn't debug the whole code to identify what exactly is wrong, but we can clearly see that you have a PoolManager and you do create multiple pools (TCRConnectionPoolManager.GetConnectionPool) when the connection string differs. However, only the first connection pool works properly and the second only establish one connection.
In your help you also have:
Connections belong to the same pool if they have identical values for the following parameters: MinPoolSize, MaxPoolSize, Validate, ConnectionLifeTime, Server, Username, Password.
I sent an example using the contact form and hope your team can fix this.
Thank you

andrefm
Posts: 37
Joined: Wed 23 Oct 2013 10:02

Re: Connection pooling setup

Post by andrefm » Wed 29 Sep 2021 20:54

Any feedback about the issue? According to the website when I sent the enquiry via the contact form, it mentioned that I should have received a feedback within 2 business day. Cheers

ViktorV
Devart Team
Posts: 3168
Joined: Wed 30 Jul 2014 07:16

Re: Connection pooling setup

Post by ViktorV » Thu 30 Sep 2021 09:21

Hi André!

Thank you for contacting Devart and for your inquiry!

We do apologize for the slight delay in our response! We carefully examined your sample and came to the following results:
The sample you sent does not demonstrate the bug you specified in the work of our pulling. The reason for this filling of ListBoxes is in the order of thread starts.
If you display in the ListBox not only information that the connection is open, but also information about the start of the thread, you will see that the threads from the second cycle are launched 1-2 seconds later than the first, respectively, and connections are first opened in the first cycle and with some delay in the second.
You can use your example code slightly modified by us.

Code: Select all

procedure TForm1.Button1Click(Sender: TObject);
var
  LPooling: string;
begin
  Button1.Enabled := False;
  Button1.Caption := 'Running..';

  TTask.Run(
    procedure
    var
      LIndex: Integer;
      LStopWatch: TStopWatch;
      LTasks: TArray<ITask>;
      LConnection: TUniConnection;
    begin
      LConnection := TUniConnection.Create(nil);
      LConnection.ConnectString := 'Provider Name=mySQL;Database=sakila;User ID=admin;Password=Teste_1234;Connection Timeout=15;Data Source=10.211.55.13;Port=3306';
      LConnection.LoginPrompt := False;
      LConnection.PoolingOptions.MaxPoolSize := 4;
      LConnection.PoolingOptions.MinPoolSize := 4;

      LConnection.PoolingOptions.ConnectionLifetime := 5000;

      LConnection.PoolingOptions.Validate := True;
      LConnection.Pooling := True;
      //LConnection.Open;
      //LConnection.Close; // Some caching should happen !!!
      //LConnection.Open; // Open before timing starts
      LStopWatch := TStopWatch.StartNew;
      LTasks := [];
      for LIndex := 1 to 100 do
      begin
         LTasks := LTasks + [TTask.Run(
          procedure
          var
            LLConnection: TUniConnection;
            LQuery: TUniQuery;
            j: Integer;
          begin
            j := TTask.CurrentTask.Id;

            TThread.Synchronize(nil, procedure
                                        begin
                                          listbox1.items.add('Started: ' + IntToStR(j));
                                          listbox1.update;
                                          ListBox1.ItemIndex := ListBox1.Items.Count - 1;
                                        end);

            LLConnection := TUniConnection.Create(nil, LConnection.ConnectString);
            LLConnection.Open;
            try
              LQuery := TUniQuery.Create(nil);
              try
                LQuery.Connection := LLConnection;
                LQuery.SQL.Text := 'select  * from customer';

                  TThread.Synchronize(nil, procedure
                                              begin
                                                listbox1.items.add('Connected: ' + IntToStR(j));
                                                listbox1.update;
                                                ListBox1.ItemIndex := ListBox1.Items.Count - 1;
                                              end);
                  LQuery.Open;
                  LQuery.Last;
                  sleep(300);
                  LQuery.Close;
                  TThread.Synchronize(nil, procedure
                                              begin
                                                listbox1.items.delete(listbox1.items.IndexOf('Started: ' + IntToStR(j)));
                                                listbox1.items.delete(listbox1.items.IndexOf('Connected: ' + IntToStR(j)));
                                                listbox1.update;
                                                listbox2.ItemIndex := listbox2.Items.Count - 1;
                                              end);
              finally
                FreeAndNil(LQuery);
              end;
            finally
              LLConnection.Close;
              FreeAndNil(LLConnection);
            end;
          end)];

      end;
      TTask.WaitForAll(LTasks);
      LConnection.Close;
      FreeAndNil(LConnection);
      LStopWatch.Stop;
      TThread.Synchronize(nil,
        procedure
        begin
          Memo1.Lines.Add('MYSQL 1 - '+LPooling + LStopWatch.Elapsed.ToString());
        end);
    end);

  TTask.Run(
    procedure
    var
      LIndex: Integer;
      LStopWatch: TStopWatch;
      LTasks: TArray<ITask>;
      LConnection: TUniConnection;
    begin
      LConnection := TUniConnection.Create(nil);
      LConnection.ConnectString := 'Provider Name=mySQL;Database=sakila;User ID=admin;Password=Teste_1234;Connection Timeout=15;Data Source=10.211.55.13;Port=3306';
      LConnection.LoginPrompt := False;
      LConnection.PoolingOptions.MaxPoolSize := 4;
      LConnection.PoolingOptions.MinPoolSize := 4;

      //DIFFERENT VALUE, NEW CONNECTION POOL USED
      LConnection.PoolingOptions.ConnectionLifetime := 6000;

      LConnection.PoolingOptions.Validate := True;
      LConnection.Pooling := True;
      //LConnection.Open;
      //LConnection.Close; // Some caching should happen !!!
      //LConnection.Open; // Open before timing starts
      LStopWatch := TStopWatch.StartNew;
      LTasks := [];
      for LIndex := 1 to 100 do
      begin
        LTasks := LTasks + [TTask.Run(
          procedure
          var
            LLConnection: TUniConnection;
            LQuery: TUniQuery;
            j: Integer;
          begin
            j := TTask.CurrentTask.Id;

            TThread.Synchronize(nil, procedure
                                        begin
                                          listbox2.items.add('Started: ' + IntToStR(j));
                                          listbox2.update;
                                          listbox2.ItemIndex := listbox2.Items.Count - 1;
                                        end);

            LLConnection := TUniConnection.Create(nil, LConnection.ConnectString);
            LLConnection.Open;
            try
              LQuery := TUniQuery.Create(nil);
              try
                LQuery.Connection := LLConnection;
                LQuery.SQL.Text := 'select  * from customer';

                  TThread.Synchronize(nil, procedure
                                              begin
                                                 listbox2.items.add('Connected: ' + IntToStR(j));
                                                 listbox2.update;
                                                 listbox2.ItemIndex := listbox2.Items.Count - 1;
                                              end);
                  LQuery.Open;
                  LQuery.Last;
                  sleep(300);
                  LQuery.Close;
                  TThread.Synchronize(nil, procedure
                                              begin
                                                listbox2.items.delete(listbox2.items.IndexOf('Started: ' + IntToStR(j)));
                                                listbox2.items.delete(listbox2.items.IndexOf('Connected: ' + IntToStR(j)));
                                                listbox2.update;
                                                listbox2.ItemIndex := listbox2.Items.Count - 1;
                                              end);

              finally
                FreeAndNil(LQuery);
              end;
            finally
              LLConnection.Close;
              FreeAndNil(LLConnection);
            end;
          end)];

      end;
      TTask.WaitForAll(LTasks);
      LConnection.Close;
      FreeAndNil(LConnection);
      LStopWatch.Stop;
      TThread.Synchronize(nil,
        procedure
        begin
          Memo1.Lines.Add('MYSQL 2 - '+LPooling + LStopWatch.Elapsed.ToString());
        end);
    end);

end;
Then you will see that there are times when 4 connections are open in both the first and second threads at the same time (see screenshot).
Image

Also, you can use the TUniSQLMonitor component to get information about the work of our connection pools. To do this, just place this component on the form and set tfPool in the TraceFlag enumeration (see screenshot).
Image
Then you should launch TUniSQLMonitor, for this you just need to click on it twice, and then launch your application and click Button1. You will see identical and correct behavior for both pools.

We will consider the possibility of adding Id to PoolingOptions so that to create two independent pools you do not have to look for wordaround of job type of different ConnectionLifetime.

Please, let us know if you have any questions!

Thanks in advance,
Viktor

andrefm
Posts: 37
Joined: Wed 23 Oct 2013 10:02

Re: Connection pooling setup

Post by andrefm » Thu 30 Sep 2021 14:09

Hi Viktor,
Still only one connection at second pool. Sent videos.

My environment:
- Delphi 11
- Windows 10 64bit in a VM running with Parallels (macbook pro 16)
- UNIDAC 9.0.1
- DB: MariaDB. (same with SQL Server 2019) ; Both using Direct Connection
-VCL or FMX, same behaviour

The behaviour is same if I run the app on macOS. However in Linux (Ubuntu 20.04.3 LTS) besides the second pool having one connection, the first pool I see always one connection lower than the MaxPoolSize. Can you also check if there is something different in Linux?

Best regards,
Andre

andrefm
Posts: 37
Joined: Wed 23 Oct 2013 10:02

Re: Connection pooling setup

Post by andrefm » Fri 01 Oct 2021 11:41

Hi Viktor,
I made some changes to my application to use 2 pools and I can confirm that I can use multiple pools in one application without issues and that it is working as expected (at least in Windows). I will try to test in Linux later and report in case I identify anything different from the expected behaviour.

I assume the sample with the queries used were not appropriate for a proper test.

It would be great if you could add an "Id" to PoolingOptions so that we can create two independent pools without having to use a workaround by using for example different ConnectionLifetime.

Thank you for the help

ViktorV
Devart Team
Posts: 3168
Joined: Wed 30 Jul 2014 07:16

Re: Connection pooling setup

Post by ViktorV » Fri 01 Oct 2021 15:55

Hi André!

Thank you for your question and interest in our product!

Note, the PoolingOptions.MinPoolSize option is not the minimum number of used connections in the pool (any number of connections that do not exceed the PoolingOptions.MaxPoolSize value can be used in the pool). We will add this clarification to the UniDAC help.
PoolingOptions.MinPoolSize is the maximum number of unused connections that will remain during the pool cleanup procedure. That is, when the pool cleanup procedure is performed and when the number of unused connections exceeds MinPoolSize, the extra ones are closed until the number of unused connections grows early or less than MinPoolSize.

Should you have any questions, do not hesitate to ask!

Thanks in advance,
Viktor

FredS
Posts: 272
Joined: Mon 10 Nov 2014 17:52

Re: Connection pooling setup

Post by FredS » Fri 01 Oct 2021 16:32

ViktorV wrote: Fri 01 Oct 2021 15:55 That is, when the pool cleanup procedure is performed and when the number of unused connections exceeds MinPoolSize, the extra ones are closed until the number of unused connections grows early or less than MinPoolSize.
Hello ViktorV,

I've been watching this thread and have a couple of questions:
  • In DB Monitor I see one Pool Manager and two pools, since the log file shows one pool start immediately after the monitor starts I assume that is for the monitor, correct?
  • Leaving my app idle i see the Pool being destroyed but MinPoolSize is four

    Code: Select all

    Connection destroyed in pool. Pool has 2 connection(s) Complet
    Connection destroyed in pool. Pool has 1 connection(s) Complet
    Connection destroyed in pool. Pool has 0 connection(s) Complet
    Connection pool destroyed Complete
    Maybe you could clarify the quoted text in BOLD.. just making sure because what I see is pretty much what I expected..
thanks
Fred

FredS
Posts: 272
Joined: Mon 10 Nov 2014 17:52

Re: Connection pooling setup

Post by FredS » Fri 01 Oct 2021 17:28

Not that I'm a technical writer but what I see is summed up by this:
MinPoolSize: Used to specify the minimum unused connections maintained in the Pool until the cleanup procedure is performed, in approximately 4 minutes of idle time.
..but doesn't match this: https://www.devart.com/unidac/docs/work_pooling.htm
The pool frees connections that are held in the pool during a long time. If no new connections are placed to the pool it becomes empty after approximately 4 minutes. This pool behaviour is intended to save resources when the count of connections in the pool exceeds the count that is needed by application. If you set the PoolingOptions.MinPoolSize property to a non-zero value, this prevents the pool from freeing all pooled connections. When connection count in the pool decreases to MinPoolSize value, remaining connection will not be freed except if they are broken.
FYI: my connection was to a local Server..

Post Reply