23Oct/20

Python remove duplicates similar to bash uniq + sort

2020-10-23Linux, Programmingkossboss

# citing: https://www.reddit.com/r/Python/comments/jfx7wy/how_to_quickly_remove_duplicates_from_a_list/

# Let's make some duplicates (one million numbers between 1 and 100)

from random import randrange
DUPLICATES = [ randrange(100) for _ in range(1_000_000) ]

### METHOD 1 ### 

# Not very efficient - Similar to "sort | uniq" but the original order is kept

unique = []
for element in DUPLICATES:
    if element not in unique:
        unique.append(element)
print(unique)

# Output:
[95, 62, 73, 27, 89, 88, 31, 50, 26, 97, 13, 18, 54, 64, 17, 94, 72, 65, 83, 14, 40, 91, 1, 21, 22, 52, 61, 15, 7, 80, 24, 42, 37, 96, 74, 9, 59, 44, 66, 20, 6, 84, 39, 81, 2, 67, 32, 77, 36, 5, 8, 47, 55, 82, 60, 35, 33, 75, 57, 68, 29, 11, 23, 86, 69, 99, 25, 78, 76, 19, 12, 92, 10, 90, 16, 3, 51, 4, 28, 0, 71, 46, 79, 85, 58, 48, 93, 56, 53, 98, 87, 30, 45, 63, 70, 38, 41, 43, 34, 49]

### METHOD 2 ###

# Very efficient - Similar to "sort | uniq"

list(set(DUPLICATES))

# This works because sets contain unique items by definition

# Output:
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99]

### METHOD 3 ###

# Very efficient but order preserved
# Not sure which bash function this is similar to
# Resembles "sort | uniq" but with the original order preserved
# But sets are unordered! What if we need to preserve the order?
# Use this dict.fromkeys() trick!
list(dict.fromkeys(DUPLICATES))

# Output:
[95, 62, 73, 27, 89, 88, 31, 50, 26, 97, 13, 18, 54, 64, 17, 94, 72, 65, 83, 14, 40, 91, 1, 21, 22, 52, 61, 15, 7, 80, 24, 42, 37, 96, 74, 9, 59, 44, 66, 20, 6, 84, 39, 81, 2, 67, 32, 77, 36, 5, 8, 47, 55, 82, 60, 35, 33, 75, 57, 68, 29, 11, 23, 86, 69, 99, 25, 78, 76, 19, 12, 92, 10, 90, 16, 3, 51, 4, 28, 0, 71, 46, 79, 85, 58, 48, 93, 56, 53, 98, 87, 30, 45, 63, 70, 38, 41, 43, 34, 49]

### METHOD 4 ###

# Older Python - Similar to "uniq | sort"
# But it only works for Python 3.6 and above
# For Python 2.7 and 3.0-3.5, use OrderedDict:from collections import OrderedDict
list(OrderedDict.fromkeys(DUPLICATES))

#### METHOD 5 ###

# Crunch duplicates as they appear and show count
# Similar to bashes "uniq -c"
# I will show you how to do "uniq -c" on a string
# Just change the input to a list and it will work on a list

# METHOD 5.1 #

from itertools import groupby
input = "Raaanndommmm Leetters"
for i,v in groupby(input):
   print(len(list(v)),i)

# Output:
1 R
3 a
2 n
1 d
1 o
4 m
1
1 L
2 e
2 t
1 e
1 r
1 s

# METHOD 5.2 #

# Or you can return a list for an output:

l=[]
for i,v in groupby(input):
   l.append(str(len(list(v)))+" "+str(i))
print(l)

# Output:
['1 R', '3 a', '2 n', '1 d', '1 o', '4 m', '1  ', '1 L', '2 e', '2 t', '1 e', '1 r', '1 s']

#  METHOD 5.3 #

# Same solution as a one liner:

print([ str(len(list(v)))+" "+str(i) for i,v in groupby(input)])

# Output:
['1 R', '3 a', '2 n', '1 d', '1 o', '4 m', '1  ', '1 L', '2 e', '2 t', '1 e', '1 r', '1 s']

### METHOD 6 ###

# Crunch out duplicates but dont show count

# METHOD 6.1 #

from itertools import groupby
input = "Raaanndommmm Leetters"
for i,v in groupby(input):
   print(str(i))

# Output:
R
a
n
d
o
m

L
e
t
e
r
s

# METHOD 6.2 #

l=[]
for i,v in groupby(input):
   l.append(str(i))
print(l)

# Output:
['R', 'a', 'n', 'd', 'o', 'm', ' ', 'L', 'e', 't', 'e', 'r', 's']

# METHOD 6.3 #

print([ str(i) for i,v in groupby(input)])
# Output:

['R', 'a', 'n', 'd', 'o', 'm', ' ', 'L', 'e', 't', 'e', 'r', 's']

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

# citing: https://www.reddit.com/r/Python/comments/jfx7wy/how_to_quickly_remove_duplicates_from_a_list/

# Let's make some duplicates (one million numbers between 1 and 100)

from random import randrange

DUPLICATES = [ randrange(100) for _ in range(1_000_000) ]

### METHOD 1 ###

# Not very efficient - Similar to "sort | uniq" but the original order is kept

unique = []

for element in DUPLICATES:

if element not in unique:

unique.append(element)

print(unique)

# Output:

[95, 62, 73, 27, 89, 88, 31, 50, 26, 97, 13, 18, 54, 64, 17, 94, 72, 65, 83, 14, 40, 91, 1, 21, 22, 52, 61, 15, 7, 80, 24, 42, 37, 96, 74, 9, 59, 44, 66, 20, 6, 84, 39, 81, 2, 67, 32, 77, 36, 5, 8, 47, 55, 82, 60, 35, 33, 75, 57, 68, 29, 11, 23, 86, 69, 99, 25, 78, 76, 19, 12, 92, 10, 90, 16, 3, 51, 4, 28, 0, 71, 46, 79, 85, 58, 48, 93, 56, 53, 98, 87, 30, 45, 63, 70, 38, 41, 43, 34, 49]

### METHOD 2 ###

# Very efficient - Similar to "sort | uniq"

list(set(DUPLICATES))

# This works because sets contain unique items by definition

# Output:

[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99]

### METHOD 3 ###

# Very efficient but order preserved

# Not sure which bash function this is similar to

# Resembles "sort | uniq" but with the original order preserved

# But sets are unordered! What if we need to preserve the order?

# Use this dict.fromkeys() trick!

list(dict.fromkeys(DUPLICATES))

# Output:

### METHOD 4 ###

# Older Python - Similar to "uniq | sort"

# But it only works for Python 3.6 and above

# For Python 2.7 and 3.0-3.5, use OrderedDict:from collections import OrderedDict

list(OrderedDict.fromkeys(DUPLICATES))

#### METHOD 5 ###

# Crunch duplicates as they appear and show count

# Similar to bashes "uniq -c"

# I will show you how to do "uniq -c" on a string

# Just change the input to a list and it will work on a list

# METHOD 5.1 #

from itertools import groupby

input = "Raaanndommmm Leetters"

for i,v in groupby(input):

print(len(list(v)),i)

# Output:

1 R

3 a

2 n

1 d

1 o

4 m

1 L

2 e

2 t

1 e

1 r

1 s

# METHOD 5.2 #

# Or you can return a list for an output:

l=[]

for i,v in groupby(input):

l.append(str(len(list(v)))+" "+str(i))

print(l)

# Output:

['1 R', '3 a', '2 n', '1 d', '1 o', '4 m', '1 ', '1 L', '2 e', '2 t', '1 e', '1 r', '1 s']

# METHOD 5.3 #

# Same solution as a one liner:

print([ str(len(list(v)))+" "+str(i) for i,v in groupby(input)])

# Output:

['1 R', '3 a', '2 n', '1 d', '1 o', '4 m', '1 ', '1 L', '2 e', '2 t', '1 e', '1 r', '1 s']

### METHOD 6 ###

# Crunch out duplicates but dont show count

# METHOD 6.1 #

from itertools import groupby

input = "Raaanndommmm Leetters"

for i,v in groupby(input):

print(str(i))

# Output:

# METHOD 6.2 #

l=[]

for i,v in groupby(input):

l.append(str(i))

print(l)

# Output:

['R', 'a', 'n', 'd', 'o', 'm', ' ', 'L', 'e', 't', 'e', 'r', 's']

# METHOD 6.3 #

print([ str(i) for i,v in groupby(input)])

# Output:

['R', 'a', 'n', 'd', 'o', 'm', ' ', 'L', 'e', 't', 'e', 'r', 's']

18Jul/19

Bash convert seconds to human readable

2019-07-18BSD, Linuxkossboss

function displaytime { local T=$1 local D=$((T/60/60/24)) local H=$((T/60/60%24)) local M=$((T/60%60)) local S=$((T%60)) (( $D > 0 )) && printf ‘%d days ‘ $D (( $H > 0 )) && printf ‘%dRead More…

28Apr/19

Wrong Port Connected Check With ifconfig

2019-04-28BSD, Linuxkossboss

We need to run ifconfig and look for the status line. Remember this about ifconfig output: * if “status: active“, the port is linked up and sending traffic (now you just needRead More…

18Apr/19

Grepping Thru All Rotated Logs

2019-04-18BSD, Linuxkossboss

If your system gzips and rotates your logs. You can use this trick to search thru all of them. # zgrep -i — “SEARCHTERM” $(ls -1Str /var/log/messages*) Another way # zgrep -iRead More…

27Apr/18

Find UPNP devices on your network – ex: ReadyDLNA or UPnP routers

2018-04-27Linux, Otherkossboss

Got info from here: https://stackoverflow.com/questions/18363833/how-can-i-list-upnp-server-renderer-in-command-line-console-mode-on-linux Get the Upnp test tools

# apt-get install gupnp-tools

1	# apt-get install gupnp-tools

Example 1: Find all Upnp devices (note their target names start with urn:…. It will hint at what service it is, ex: Layer3Forwarder,Read More…

24Jan/18

bash simple variable substitution

2018-01-24Linuxkossboss

You can use bash to do simple substitutions of variables, just like you can with sed. The bash trick turns out to take up less characters (bytes). Lets set some variable

VAR1="this is a sentence"

1	VAR1="this is a sentence"

Bash scripting dealing with any characters in strings or filenames

2017-11-15BSD, Linuxkossboss

Imagine a file list, list.txt, like this one

image1.jpg
image2.jpg
image3.jpg

image1.jpg

image2.jpg

image3.jpg

Naturally you could deal with it like so

cat list.txt | while read i; do
   FILE="$i"
   dosomething "$FILE"
done

cat list.txt | while read i; do

FILE="$i"

dosomething "$FILE"

done

Or just use the $i variable as its already the file and do aRead More…

26May/17

Repeat Command Until Success – recursive method – rrsync, repeat, repeatrecurse

2017-05-26BSD, Linuxkossboss

Here I show you how to repeat a command until it finishes successfully. This should only be done with programs that support resume and also have legit exit codes (or else theyRead More…

21Oct/16

FreeBSD – Show info about running process – SIGINFO

2016-10-21BSDkossboss

When you are running a command hit Control-T anytime, its very useful when a command looks stuck. It causes the kernel to spit out a line similar to the following: load: 0.69Read More…

27Sep/16

Enable Mouse Scrolling with Screen (tested in Putty)

2016-09-27BSD, Linuxkossboss

Screen the app that can split screens like so, is not too intuitive when it comes to scrolling thru your bash shell. Notice where the cursor is in the diagram below, itsRead More…

infotinks

My Notes, Articles & Guides for Linux, Windows and Networking.

Category Archives: Linux

Python remove duplicates similar to bash uniq + sort

Bash convert seconds to human readable

Wrong Port Connected Check With ifconfig

Grepping Thru All Rotated Logs

Find UPNP devices on your network – ex: ReadyDLNA or UPnP routers

bash simple variable substitution

Bash scripting dealing with any characters in strings or filenames

Repeat Command Until Success – recursive method – rrsync, repeat, repeatrecurse

FreeBSD – Show info about running process – SIGINFO

Enable Mouse Scrolling with Screen (tested in Putty)