Skip to main content

How to delete rows or remove duplicates with python

Hi Brian
I wanted to know if it is possible for me to apply python scripts to remove duplicates from a table. I would like to consider all columns (more than 100 columns) as duplicates.
I know how to write a function; the problem is I should add a column and apply the function in that column. I would like to apply Python scripts to whole table.
For instance, apply this Python script in a table; data.drop_duplicates(keep=False)
Or delete rows with python

Brian Element Mon, 12/12/2022 - 13:45

Hi talebi,

If you want to use Python then you would need to take the IDEA database and move it into Python, do your removal of columns and such and then send it back to IDEA.  You can use the module found in IDEA to easily move the data to a Panda dataframe, then do your updating on the dataframe then use the module to send the dataframe back to IDEA.  

Also this might be something you can do directly in IDEA with IDEAScript.

talebi Mon, 12/12/2022 - 16:38

In reply to by Brian Element

Thank you BrianDo you have any link or example for that? That would be great if I see an example to make some scripts faster.