Removing Duplicates from a List

In this chapter, we will learn how to remove duplicate elements from a list.

Key Points:

• Python Set: A set is an unordered collection of unique elements.

• Python List: A list is a collection of items arranged in a specific linear order. Basic operations on lists include searching, inserting, and deleting elements.

Example 1: Using Sets

list_1 = [1, 2, 1, 4, 6]

# Convert list to set to remove duplicates, then convert back to list
print(list(set(list_1)))

Output:

[1, 2, 4, 6]

In this example, converting the list to a set removes duplicates, and converting it back to a list gives us the result without duplicate elements.

Maintaining Original Order

To preserve the order of elements in the original list, use an auxiliary set to track seen elements and build a new list.

Example 2: Using an Auxiliary Set

# Function to remove duplicates while maintaining order
def remove_duplicates(lst):
    seen = set()
    unique_list = []
    for item in lst:
        if item not in seen:
            seen.add(item)
            unique_list.append(item)
    return unique_list

# Example
original_list = [1, 2, 2, 3, 4, 4, 5]
unique_list = remove_duplicates(original_list)
print(unique_list)  # Output: [1, 2, 3, 4, 5]

Using dict.fromkeys()

The dict.fromkeys() method can also be used to remove duplicates while maintaining order, as dictionaries in Python 3.7 and later versions preserve insertion order.

Example 3: Using dict.fromkeys()

# Function to remove duplicates while maintaining order
def remove_duplicates(lst):
    return list(dict.fromkeys(lst))

# Example
original_list = [1, 2, 2, 3, 4, 4, 5]
unique_list = remove_duplicates(original_list)
print(unique_list)  # Output: [1, 2, 3, 4, 5]

Using List Comprehensions

List comprehensions with conditional checks can also achieve deduplication.

Example 4: Using List Comprehensions

# Function to remove duplicates using list comprehension
def remove_duplicates(lst):
    unique_list = []
    [unique_list.append(item) for item in lst if item not in unique_list]
    return unique_list

# Example
original_list = [1, 2, 2, 3, 4, 4, 5]
unique_list = remove_duplicates(original_list)
print(unique_list)  # Output: [1, 2, 3, 4, 5]

Removing Duplicates Between Two Lists

In the following example, elements that appear in both lists will be removed.

Example 5: Removing Duplicates Between Two Lists

list_1 = [1, 2, 1, 4, 6]
list_2 = [7, 8, 2, 1]

# Find elements that are in either list but not in both
print(list(set(list_1) ^ set(list_2)))

Output:

[4, 6, 7, 8]

In this example, converting the lists to sets removes duplicates, and using the ^ operator finds the symmetric difference between the two sets, which excludes overlapping elements.