12Nov/20

Backup file[s] to Dropbox (without syncing)

2020-11-12Linuxkossboss

This github page, https://github.com/andreafabrizi/Dropbox-Uploader, has the dropbox-uploader tool which I use to backup content from servers (Linux, Mac, etc) without having to sync my Dropbox content to the local disk on yourRead More…

09Nov/20

xargs parallel note to self

2020-11-09Linuxkossboss

Xargs is useful to run in parallel. Its parallel processing is very efficient. Read this post about its efficiency and this one about basic commands. Below, is my favorite way to runRead More…

23Oct/20

Python remove duplicates similar to bash uniq + sort

2020-10-23Linux, Programmingkossboss

# citing: https://www.reddit.com/r/Python/comments/jfx7wy/how_to_quickly_remove_duplicates_from_a_list/

# Let's make some duplicates (one million numbers between 1 and 100)

from random import randrange
DUPLICATES = [ randrange(100) for _ in range(1_000_000) ]

### METHOD 1 ### 

# Not very efficient - Similar to "sort | uniq" but the original order is kept

unique = []
for element in DUPLICATES:
    if element not in unique:
        unique.append(element)
print(unique)

# Output:
[95, 62, 73, 27, 89, 88, 31, 50, 26, 97, 13, 18, 54, 64, 17, 94, 72, 65, 83, 14, 40, 91, 1, 21, 22, 52, 61, 15, 7, 80, 24, 42, 37, 96, 74, 9, 59, 44, 66, 20, 6, 84, 39, 81, 2, 67, 32, 77, 36, 5, 8, 47, 55, 82, 60, 35, 33, 75, 57, 68, 29, 11, 23, 86, 69, 99, 25, 78, 76, 19, 12, 92, 10, 90, 16, 3, 51, 4, 28, 0, 71, 46, 79, 85, 58, 48, 93, 56, 53, 98, 87, 30, 45, 63, 70, 38, 41, 43, 34, 49]

### METHOD 2 ###

# Very efficient - Similar to "sort | uniq"

list(set(DUPLICATES))

# This works because sets contain unique items by definition

# Output:
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99]

### METHOD 3 ###

# Very efficient but order preserved
# Not sure which bash function this is similar to
# Resembles "sort | uniq" but with the original order preserved
# But sets are unordered! What if we need to preserve the order?
# Use this dict.fromkeys() trick!
list(dict.fromkeys(DUPLICATES))

# Output:
[95, 62, 73, 27, 89, 88, 31, 50, 26, 97, 13, 18, 54, 64, 17, 94, 72, 65, 83, 14, 40, 91, 1, 21, 22, 52, 61, 15, 7, 80, 24, 42, 37, 96, 74, 9, 59, 44, 66, 20, 6, 84, 39, 81, 2, 67, 32, 77, 36, 5, 8, 47, 55, 82, 60, 35, 33, 75, 57, 68, 29, 11, 23, 86, 69, 99, 25, 78, 76, 19, 12, 92, 10, 90, 16, 3, 51, 4, 28, 0, 71, 46, 79, 85, 58, 48, 93, 56, 53, 98, 87, 30, 45, 63, 70, 38, 41, 43, 34, 49]

### METHOD 4 ###

# Older Python - Similar to "uniq | sort"
# But it only works for Python 3.6 and above
# For Python 2.7 and 3.0-3.5, use OrderedDict:from collections import OrderedDict
list(OrderedDict.fromkeys(DUPLICATES))

#### METHOD 5 ###

# Crunch duplicates as they appear and show count
# Similar to bashes "uniq -c"
# I will show you how to do "uniq -c" on a string
# Just change the input to a list and it will work on a list

# METHOD 5.1 #

from itertools import groupby
input = "Raaanndommmm Leetters"
for i,v in groupby(input):
   print(len(list(v)),i)

# Output:
1 R
3 a
2 n
1 d
1 o
4 m
1
1 L
2 e
2 t
1 e
1 r
1 s

# METHOD 5.2 #

# Or you can return a list for an output:

l=[]
for i,v in groupby(input):
   l.append(str(len(list(v)))+" "+str(i))
print(l)

# Output:
['1 R', '3 a', '2 n', '1 d', '1 o', '4 m', '1  ', '1 L', '2 e', '2 t', '1 e', '1 r', '1 s']

#  METHOD 5.3 #

# Same solution as a one liner:

print([ str(len(list(v)))+" "+str(i) for i,v in groupby(input)])

# Output:
['1 R', '3 a', '2 n', '1 d', '1 o', '4 m', '1  ', '1 L', '2 e', '2 t', '1 e', '1 r', '1 s']

### METHOD 6 ###

# Crunch out duplicates but dont show count

# METHOD 6.1 #

from itertools import groupby
input = "Raaanndommmm Leetters"
for i,v in groupby(input):
   print(str(i))

# Output:
R
a
n
d
o
m

L
e
t
e
r
s

# METHOD 6.2 #

l=[]
for i,v in groupby(input):
   l.append(str(i))
print(l)

# Output:
['R', 'a', 'n', 'd', 'o', 'm', ' ', 'L', 'e', 't', 'e', 'r', 's']

# METHOD 6.3 #

print([ str(i) for i,v in groupby(input)])
# Output:

['R', 'a', 'n', 'd', 'o', 'm', ' ', 'L', 'e', 't', 'e', 'r', 's']

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

# citing: https://www.reddit.com/r/Python/comments/jfx7wy/how_to_quickly_remove_duplicates_from_a_list/

# Let's make some duplicates (one million numbers between 1 and 100)

from random import randrange

DUPLICATES = [ randrange(100) for _ in range(1_000_000) ]

### METHOD 1 ###

# Not very efficient - Similar to "sort | uniq" but the original order is kept

unique = []

for element in DUPLICATES:

if element not in unique:

unique.append(element)

print(unique)

# Output:

[95, 62, 73, 27, 89, 88, 31, 50, 26, 97, 13, 18, 54, 64, 17, 94, 72, 65, 83, 14, 40, 91, 1, 21, 22, 52, 61, 15, 7, 80, 24, 42, 37, 96, 74, 9, 59, 44, 66, 20, 6, 84, 39, 81, 2, 67, 32, 77, 36, 5, 8, 47, 55, 82, 60, 35, 33, 75, 57, 68, 29, 11, 23, 86, 69, 99, 25, 78, 76, 19, 12, 92, 10, 90, 16, 3, 51, 4, 28, 0, 71, 46, 79, 85, 58, 48, 93, 56, 53, 98, 87, 30, 45, 63, 70, 38, 41, 43, 34, 49]

### METHOD 2 ###

# Very efficient - Similar to "sort | uniq"

list(set(DUPLICATES))

# This works because sets contain unique items by definition

# Output:

[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99]

### METHOD 3 ###

# Very efficient but order preserved

# Not sure which bash function this is similar to

# Resembles "sort | uniq" but with the original order preserved

# But sets are unordered! What if we need to preserve the order?

# Use this dict.fromkeys() trick!

list(dict.fromkeys(DUPLICATES))

# Output:

### METHOD 4 ###

# Older Python - Similar to "uniq | sort"

# But it only works for Python 3.6 and above

# For Python 2.7 and 3.0-3.5, use OrderedDict:from collections import OrderedDict

list(OrderedDict.fromkeys(DUPLICATES))

#### METHOD 5 ###

# Crunch duplicates as they appear and show count

# Similar to bashes "uniq -c"

# I will show you how to do "uniq -c" on a string

# Just change the input to a list and it will work on a list

# METHOD 5.1 #

from itertools import groupby

input = "Raaanndommmm Leetters"

for i,v in groupby(input):

print(len(list(v)),i)

# Output:

1 R

3 a

2 n

1 d

1 o

4 m

1 L

2 e

2 t

1 e

1 r

1 s

# METHOD 5.2 #

# Or you can return a list for an output:

l=[]

for i,v in groupby(input):

l.append(str(len(list(v)))+" "+str(i))

print(l)

# Output:

['1 R', '3 a', '2 n', '1 d', '1 o', '4 m', '1 ', '1 L', '2 e', '2 t', '1 e', '1 r', '1 s']

# METHOD 5.3 #

# Same solution as a one liner:

print([ str(len(list(v)))+" "+str(i) for i,v in groupby(input)])

# Output:

['1 R', '3 a', '2 n', '1 d', '1 o', '4 m', '1 ', '1 L', '2 e', '2 t', '1 e', '1 r', '1 s']

### METHOD 6 ###

# Crunch out duplicates but dont show count

# METHOD 6.1 #

from itertools import groupby

input = "Raaanndommmm Leetters"

for i,v in groupby(input):

print(str(i))

# Output:

# METHOD 6.2 #

l=[]

for i,v in groupby(input):

l.append(str(i))

print(l)

# Output:

['R', 'a', 'n', 'd', 'o', 'm', ' ', 'L', 'e', 't', 'e', 'r', 's']

# METHOD 6.3 #

print([ str(i) for i,v in groupby(input)])

# Output:

['R', 'a', 'n', 'd', 'o', 'm', ' ', 'L', 'e', 't', 'e', 'r', 's']

18Jul/19

Bash convert seconds to human readable

2019-07-18BSD, Linuxkossboss

function displaytime { local T=$1 local D=$((T/60/60/24)) local H=$((T/60/60%24)) local M=$((T/60%60)) local S=$((T%60)) (( $D > 0 )) && printf ‘%d days ‘ $D (( $H > 0 )) && printf ‘%dRead More…

28Apr/19

Wrong Port Connected Check With ifconfig

2019-04-28BSD, Linuxkossboss

We need to run ifconfig and look for the status line. Remember this about ifconfig output: * if “status: active“, the port is linked up and sending traffic (now you just needRead More…

18Apr/19

Grepping Thru All Rotated Logs

2019-04-18BSD, Linuxkossboss

If your system gzips and rotates your logs. You can use this trick to search thru all of them. # zgrep -i — “SEARCHTERM” $(ls -1Str /var/log/messages*) Another way # zgrep -iRead More…

27Apr/18

Find UPNP devices on your network – ex: ReadyDLNA or UPnP routers

2018-04-27Linux, Otherkossboss

Got info from here: https://stackoverflow.com/questions/18363833/how-can-i-list-upnp-server-renderer-in-command-line-console-mode-on-linux Get the Upnp test tools

# apt-get install gupnp-tools

1	# apt-get install gupnp-tools

Example 1: Find all Upnp devices (note their target names start with urn:…. It will hint at what service it is, ex: Layer3Forwarder,Read More…

24Jan/18

bash simple variable substitution

2018-01-24Linuxkossboss

You can use bash to do simple substitutions of variables, just like you can with sed. The bash trick turns out to take up less characters (bytes). Lets set some variable

VAR1="this is a sentence"

1	VAR1="this is a sentence"

Bash scripting dealing with any characters in strings or filenames

2017-11-15BSD, Linuxkossboss

Imagine a file list, list.txt, like this one

image1.jpg
image2.jpg
image3.jpg

image1.jpg

image2.jpg

image3.jpg

Naturally you could deal with it like so

cat list.txt | while read i; do
   FILE="$i"
   dosomething "$FILE"
done

cat list.txt | while read i; do

FILE="$i"

dosomething "$FILE"

done

Or just use the $i variable as its already the file and do aRead More…

26May/17

Repeat Command Until Success – recursive method – rrsync, repeat, repeatrecurse

2017-05-26BSD, Linuxkossboss

Here I show you how to repeat a command until it finishes successfully. This should only be done with programs that support resume and also have legit exit codes (or else theyRead More…

infotinks

My Notes, Articles & Guides for Linux, Windows and Networking.

Category Archives: Linux

Backup file[s] to Dropbox (without syncing)

xargs parallel note to self

Python remove duplicates similar to bash uniq + sort

Bash convert seconds to human readable

Wrong Port Connected Check With ifconfig

Grepping Thru All Rotated Logs

Find UPNP devices on your network – ex: ReadyDLNA or UPnP routers

bash simple variable substitution

Bash scripting dealing with any characters in strings or filenames

Repeat Command Until Success – recursive method – rrsync, repeat, repeatrecurse