Does PgSqlLoader time out?
Posted: Wed 25 Jul 2012 13:03
We've recently made some changes to our code that are intended to speed up our loading of data. It's a complicated process involving transmission of data over the wire using XMPP from our server software to the machine where Postgres is installed. This process was taking a minimum of 4.5 hours + to download 2.5 million rows of data, often taking longer. This is much too long. Also, the disk ended up being so busy that the machine was totally unusable while the download was happening.
We started using PgSqlLoader to make use of the COPY command and this has reduced the download time and disk activity on the machine. While disk activity is still somewhat high, the machine is now usable.
We've been seeing some timeout error occurring in our code during the download process. We've set the timeout value to 0 everywhere there was one, however, there is no time out setting in the PgSqlLoader class. I suspect that the timeout errors are coming from this class.
Is there any way that we can set a time out on the COPY operation?
Tony
We started using PgSqlLoader to make use of the COPY command and this has reduced the download time and disk activity on the machine. While disk activity is still somewhat high, the machine is now usable.
We've been seeing some timeout error occurring in our code during the download process. We've set the timeout value to 0 everywhere there was one, however, there is no time out setting in the PgSqlLoader class. I suspect that the timeout errors are coming from this class.
Is there any way that we can set a time out on the COPY operation?
Tony