Greetings
I am processing 70 million records in iIDEA, the PC becomes very slow when trying to join the table that has 70 million records with the header table to get the accounting movements of a period, the question is, would I have to change the PC for one with higher processing power and more memory? Well, this union lasts approximately more than 8 hours, where I had to reset the PC and it does not make the union. The union is made between the transaction account and the header account.
what dou you recommend ?
ortizmario6025@...
Offline
Last seen: 3 weeks 2 days ago
Joined: 09/03/2021 - 11:54
Depending on the stats of your machine doing a join with 70 million records will take some time. Some things that might help speed things up is eliminate any fields that aren't necessary. I don't know how many fields your primary or secondary file contain but if they have a large number of fields try and limit the join to the fields that are necessary for the analysis. This will potentially save quite a bit of time writing the file.
Since you seem to be resetting your computer try doing it in several steps. First create the join index in the primary and secondary file and then do the join. This way the join can skip performing the index as it is already present. This will speed it up.
Also how complex is your join? Is it over multiple fields or single field. A join on a single field I believe should be faster then multiple fields but I have never actually tested that.
Thanks, I'll check these comments