sync db when the source ha highly usage

Discussion of open issues, suggestions and bugs regarding database management and development tools for PostgreSQL
Post Reply
royb
Posts: 2
Joined: Thu 22 Apr 2021 09:17

sync db when the source ha highly usage

Post by royb » Thu 22 Apr 2021 09:35

I am wondering if this tool is the right tool for my needs and i hope this is the place to ask.
I plan to run a compare every few minutes while one of the tables at my source db is a table with over 30 millions records.
Every minute this table get bigger by at least 300 records.
The target db is a lazy db it is not going to be updated beside the sync process.
So my questions are:
1 - Is that the right tool for the job?
2 - How does the sync process works, I have seen that the first compare taking me around one hour, what happened at the source? does the db get locked? (i think that i have seen that the insert jobs are getting idle).
3 - What will happened in the other compares are they going to be shorter?

Hope to find that this tool is the right one for my usage.

Thanks.

dzhanhira
Devart Team
Posts: 67
Joined: Mon 26 Oct 2020 13:49

Re: sync db when the source ha highly usage

Post by dzhanhira » Fri 23 Apr 2021 11:25

Let me, please, clarify that. If the table has about 30 million records and every minute the table grows by 300 records and the synchronization occurs every few minutes, then our tool will hardly work.
To compare the data, their source and target are selected on the computer where it is installed and the records are shipped there. The data is fetched by a request:
SELECT <column_list> FROM <table_name> ORDER BY <column_key_comparison>;
I don't think this is a suitable query for regularly fetching data from a large table.

royb
Posts: 2
Joined: Thu 22 Apr 2021 09:17

Re: sync db when the source ha highly usage

Post by royb » Sun 25 Apr 2021 07:18

Thanks for the replay.
I will add a question:
I have seen that you can customize queries so my question if there is a way to do it that every few minutes will compare only from last run timestamp to the current run timestamp.
Thanks

Post Reply