IMD files are excessively huge

Joel Edwards

Forums

Hi all
Firstly, my apologies if I've posted to the wrong forum section...
 
At work we use Caseware IDEA (v11.1 if it matters) to analyse large datasets, but something I've noticed is that the output files (saved as *.IMD) end up being colossal in size. I've opened some smaller ones in a text editor and I see a massive amount of references to "NULL" in the values, which I'm assuming is an empty value in the databases that were imported.
In some datasets we end up with over 300GB of "NULL" in a single file and this is making our space management a nightmare. I've tried compressing these files with apps like WinRAR and it can get sizes down from 300GB to 1.5GB so I'm confident that's exactly whats taking the space.
My question is, can IDEA be configured to store a literal empty space for empty values, instead of writing "NULL" on every single empty cell? This will free up more than 99% of my space issues with IDEA, and presumably make it work much faster since there's less data to read/write...

scotchy33 Fri, 10/15/2021 - 14:00

Hi Joel,
It would help if you could attach a sample of a file containing the "NULL"s, to replicate this issue.
Scott

osaajah Sun, 10/17/2021 - 14:35

hi joel,
was your IDEA file source from Excel file?
if it was, I guessed that you have pressed delete key on the keyboard to delete some area / range in your Excel file rather than used right-click and selected "delete" from pop-up menu.
 

Brian Element Mon, 10/18/2021 - 06:37

Hi Joel, it is hard to say without knowing more about the file but it sounds like you might have lots of empty fields in your file or empty rows.  I would suggest looking through your file and deleting or not importing fields that are empty or not to be used.  CaseWare has a utility that you can download from Passport, it is under SmartAnalyzer - Utilities and one of the apps in there will look through your file and show you columns that are either blank or contain the same value.  These columns can generally be dropped.