How to delete rows or remove duplicates with python

5 posts / 0 new
Last post
talebi
Offline
Joined: 11/23/2021 - 14:15
How to delete rows or remove duplicates with python

Hi Brian
I wanted to know if it is possible for me to apply python scripts to remove duplicates from a table. I would like to consider all columns (more than 100 columns) as duplicates.
I know how to write a function; the problem is I should add a column and apply the function in that column. I would like to apply Python scripts to whole table.
For instance, apply this Python script in a table; data.drop_duplicates(keep=False)
Or delete rows with python
 
Thanks,
 

Brian Element's picture
Brian Element
Offline
Joined: 07/11/2012 - 19:57

Hi talebi,

If you want to use Python then you would need to take the IDEA database and move it into Python, do your removal of columns and such and then send it back to IDEA.  You can use the IDEALib.py module found in IDEA to easily move the data to a Panda dataframe, then do your updating on the dataframe then use the IDEALib.py module to send the dataframe back to IDEA.  

Also this might be something you can do directly in IDEA with IDEAScript.

talebi
Offline
Joined: 11/23/2021 - 14:15

Thank you BrianDo you have any link or example for that? That would be great if I see an example to make some scripts faster.
Thanks,

Brian Element's picture
Brian Element
Offline
Joined: 07/11/2012 - 19:57

If you start watching around the 44 minute mark I talk about how to use the IDEALib.py in your python programs - https://www.youtube.com/watch?v=SxLShiRuRdY&t=2681s

talebi
Offline
Joined: 11/23/2021 - 14:15

Thank you