28May/21

Python creating your own range() function with yield

Generator functions allow for yielding which is an important skill to have when you are becoming advanced with python. It basically allows creating lists but not caring about the whole list, only about the next number. This allows for time savings in many situations.

Anyhow, these are just my own practice notes on the yield call. I don’t have any clever explanations for it. Besides that when the code executes yield, it leaves the function back to where it was called (in the example its the list function), then when it goes back in (in the example: the list function will attempt to go thru the entirety of the function being yielded), it remembers at which line it was and continues (it keeps on looping). For your own exercise, you can add print() methods to see how it proceeds.

Also, the list() method is not the best way to show yields, as its simply prints everything the generator is trying to express. This could have been done with none yielding code that appends a new number to a list and returns a list.

Instead, we can really show the power of these generator functions using the next() method.

Note that we define the new_range variable which becomes a generator object. Then if we were to call list() on this object we would get [6,7,8] – not shown here as similar concept is shown in the first code output. However, if we call next(new_range), we get the next value of this generator – the key concept is that if it hasn’t been called yet it will provide the first value of the generator. So the first next() will output the first number, which is 6. Then 7. Then 8. If we call it again, it will error as it’s the end of the generator object.

Benefits: Imagine the range was r1(1,100000000). Then imagine we needed the 1000th item. We could generate the list() of it and then get the 1000th item, but that will take a while to generate the list. So instead we can just yield thru until the 1000th item without caring about the rest. Speed 🙂

Thats all folks.

16Mar/21

WebEx – Simple Python Script To Find Room By Name and Send Message To It

This is a simple python script which uses the WebEx API. WebEx is a chat system created by Cisco. However its alot more then just that, its full on collaboration tool that can be used like Slack and Zoom at the same time. So it is more then just chatting, you can share screens as well.

What we do with this example is we simply use the API to write a message to the room (or specifically a small space). First we have the API get the room list, find the room by its name, then we get that rooms ID, then we send a message to that room ID.

Sidenote: If you have the room ID already you can shortcut a lot of the code

Sidenote: A room is any space (many people) or one on one conversation that you are having.

Getting API Key For Yourself for 12 hours:

More info on the API & developing with it is found here:

https://developer.webex.com/docs/api/getting-started

In fact, you need to go to that site to get the API Key. First login at the top left corner. Then scroll thru the guide and look up any guide. A good starting point is. API Reference -> Rooms . You should find the ability to copy your API Key into the clipboard directly from the page (make sure your browser is fully expanded on the screen).

This will get a 12 hour API Key.

Getting A More Permanent API Key with a Bot:

To get a more permanent key (100 years) you have to create a WebEX API Bot / app (create bot here|more info on bots here). You give the bot a name, and a unique username, an icon (just like a user), and a description. The bot then gets a domain name tagged to it automatically with domain webex.bot – like an email username; so for bot username bot1, the full email username will be bot1@webex.bot. You can use this username to add your bot to your room. After submitting, you will get a Bot ID and Bot Token (You use the Bot Token as your API Key; its good for 100 years).

Sidenote: you can create bots, integrations, and guest issuers, and also login as yourself for 12 hours (explained above).

The Steps

  1. First open WebEx
  2. Create a room or space called “MyRoom-Test” (or whatever you want to call it)
  3. Make sure the user which API key is used is joined to the room.
    • If you created an APP or Bot, make sure to join it to the room.
  4. Next, make sure you fulfill your python requirements, create the script and call the script with the correct environment variables in place.

The python script uses the requests python package which needs to be installed. Requests is an HTTP[s] package for python. Allows for easy manipulation of said protocol and therefore using REST API (which WebEx relies on)

I will not go in depth here on the code or REST API as there are plenty articles online, and this is mostly for my notes.

Sidenote: One thing, I didn’t do in this code is error handling.

Other Requirements:

  • Python 3.9 as I use the f string format {var=} that was introduced then
  • requests python package which can be installed with: pip install requests

Script (Find Room and Send Message):

Note: that you can use the api.ciscospark.com and webexapis.com endpoint when calling the API, both will achieve same results.

How To Run It:

First set your WBX_KEY API Key environment variable. Replace PasteLongApiKeyHere with your actual key:

Example Output:

Personal information has been edited for security reasons.

Also, you will see your “test message” in the actual WebEx application. If you used your API key it will come from you. If you used your Bots API key it will come from your bot.

The Script Assuming You Know The RoomID:

If you know the roomID you can make the code alot smaller obviously as we don’t have to find the roomID by title.

Then you would call this code like this:

First set your WBX_KEY api key environment variable by replacing PasteLongApiKeyHere with your actual key. Do the same but with the long Room ID for PasteLongRoomIDHere.

The output will be similar to the previous section but shorter as we don’t have to find the room and you will get a text message in your WebEx as expected.

The end,

11Mar/21

Trick to recall “ln -s” symlink argument order

I can never remember the order of arguments for the ln command (link command).

The easiest way to think about it is to rethink of the “cp” (copy command) instead of “cp source destination” think of it as “cp existing new“. With symlinks, it is hard to comprehend what is the source and what is destination. Switching your thinking to understanding what exists and what will be new helps. So the argument order is just like it is with cp, existing then new. “ln -s existing new” does the trick.

Rule of thumb:

Also apparently, I am not the only one that has troubles with this as there is this link – https://news.ycombinator.com/item?id=1984456 . I stole the trick of the top comment and made an article, now I am bound to not forget (I hope).

The end

04Mar/21

Sublime Text Editor – Show All Whitespace

The ability to see all whitespace (spaces, tabs, etc) is not very clear in sublime and not easily accessible.

You can change it by doing this:

Open your user settings (Preferences -> Settings) and add this item in to your settings in between the { and }.

Make sure to properly end it with a comma if its not the last item in the dict, and if it is the last item in the dict, you can remove the final comma character.

However, you easily set yourself some keyboard shortcuts by editing the keymap (Preferences -> Key Bindings). Note put this in between your [ and ] and characters (They keybindings is a list of dicts with keys of “keys”, “command” and “args”).

Likewise, make sure to remove the comma at the end if its the last item and keep it if you have further items in there.

Shift+Alt+D to show all whitespace

Shift+Alt+A to revert to normal

On a MAC replace Alt with Option key

Sidenote: My key bindings so far are simple so its just:

11Dec/20

rhood – robinhood portfolio analysis tool (better net profits per symbol)

Github: https://github.com/bhbmaster/rhood <- download location. install & run instructions.

Using the robin-stocks python module, I created my own robinhood portfolio analyzer called rhood. It parses all of your Robinhood account information provided by the API and outputs a single text output containing all of your portfolio information, order + open positions + dividend information. Mainly, it parses all of your orders and outputs sorted orders, open positions, informative profits, and dividend information. It provides a good figure to your total net gain, and net gain per any position currently owned and previously owned (currently the default Robinhood app doesn’t show this information nicely).

It requires python, the robin-stocks pip package (robinhood api client), and pyopt pip package (2factor authentication module). It can be run on Windows, MAC, or Linux.

This prints a lot of information about your stocks, crypto, and options (see note 1):

  • all of the orders (creates csvs from them as well)
  • all open positions
  • net profit calculations

It provides useful information that I couldn’t find on robinhood app itself; i.e. your profit per stock. Robinhood has a section to show total return, however that seems to clear out if you sell the whole stock. My application doesn’t do that, and it shows you total profit (or loss) for each symbol: stock, crypto, option (see note 1).

* Note 1 – Work in progress: options are not implemented yet. So if you are only using stocks and or crypto you are set, otherwise options are skipped/ignored.

* Note 2: I used the robin-stocks module. However I see there are some other modules that talk with the robinhood API as well. I didn’t use these, as they seem to be older. https://github.com/robinhood-unofficial/pyrh and here https://github.com/mstrum/robinhood-python

17Nov/20

Create Github Repo On The Go From The Shell – Github API

Sometimes you start coding something and you don’t yet realize if it will be a project worth sharing on github. We don’t always think about this as we begin coding. This is a write up on how to deal with the time you code something up, and then – after awhile – you realize you want to share it. This method, also works for creating fresh new repos that you haven’t started coding yet.

Of course the normal practice is to create a github repo from the browser and then follow the code to git init it.

Thats all fine and nice, but its time consuming to open up the browser and create the github repo. Luckily, we can do it using the github api.

First you need to get a github API key. Then using curl you can send a command to github to create your repo for you. There are many settings you can tweak. Specifically for us we want the basic; public repo without any commits or files (no README.md files).

Step 1 – Create API Token

First you have to create an authentication token – personal token:

  • Login to github.com on your browser
  • Go to Settings -> Developer settings -> Create Personal Access Token
  • Hit Generate button
  • In the note textbox, write its purpose. ex: “creating repos from command line
  • Then give it proper access in the scopes:
    • Check on everything in the repo section
    • Check on gist
  • That is it. When you submit this info, it will give you an access token (long alphanumeric string)
  • Save that access token string. We will be using it in our curl commands. This token is the equivalent of providing a username and password (so don’t lose it and dont share it)

Step 2 – Command Line

After you get your key you can now use it in the shell. For example’s sake we use abc123 as the key (your key will have more characters).

Here is how it will look like in your workflow:

  • First, create directory and code some stuff
  • Realize you are making a repo
  • git init the repo. That only saves it locally
  • Make a commit
  • Now create the github repo using curl command. Note we set auto_init to false (by default its true) so that it doesn’t create a first commit with a template README.md file. Also, we make it a public repo, so we set private to true.
    • Change abc123 to your authorization token alphanumeric value
    • Change REPONAME to your repo name. Only use these chars alphabet, number, dot, underscore and minux: A-Za-z0-9_. -.
  • Then set the git origin, which is the remote repository server and repo. Make sure to use the https://github.com/USERNAME/REPONAME.git link; If you use wrong link remove with git remote remove origin. This link is seen in the curl output (look for “clone_url“)
    • Change USERNAME to your github username and REPONAME to your reponame
  • set the branch (master or main) to upstream and push

More Info:

  • More information on github api. Such as more options to pass in the json string with the -d argument to affect the type of repo that gets created: https://developer.github.com/v3/repos/#create-a-repository-for-the-authenticated-user
  • The simplest form of this call curl -H "Authorization: token abc123" https://api.github.com/user/repos -d '{"name":"REPONAME"}' would create a repo that has an initial commit. However, for a fluid work process we don’t want that, so we add the option auto_init: false (if not provided; this option is set to true). Also we set private to false, so that we get a public repo.
  • Previously, you could use the api without a token using your username and password; that has been deprecated out as its unsafe. the commands looked like this: curl -u user:pass https://api.github.com/user/repos -d '{"name":"REPONAME"}'

Making an alias for easier use

It might be annoying to always type those long commands. So you can write an alias and stick it in your .bashrc or .bashprofile. However, that alias is really long. I prefer to create an environment function – it’s a bash function that can be called from the shell; it’s just a regular bash function that was created in the shell instead of in a script.

Here is what I have in my bashrc/bashprofile:

Don’t forget to comment out whichever alias function you don’t want to use (ALIAS 1 or ALIAS 2). I personally use ALIAS 1 as I don’t like having extra files).

For ALIAS 1, don’t forget to put the Auth Token in the variable, thus replacing abc123.

For ALIAS 2, don’t forget to create file ~/.github-api-key with your key: echo "abc123" > ~/.github-api-key

If you just created the script don’t forget to source your .bashrc or .bashprofile (or whichever file you put the script in): source ~/.bashrc

Using the Alias / Function on the Go

Now finally you can use like this – just an example (you can use it other ways – like you can start with the CreateGitRepo command):

  • Create a directory Project1:
  • Change into the dir:
  • Code some stuff up:
  • Initialize the local git repo in the current dir:
  • Stage the current files (wow.js):
  • Commit the staged to the local repo with a descriptive comment:
  • Now create the empty public remote repo (in otherwords create the repo on github):
  • This will show you a lot of lines output if all worked correct.
  • So far the repo has been created but the code has not been pushed to it yet. the final 2 lines of CreateGitRepo command help you with the final 2 commands to set origin and push the code to github. for this example those 2 commands would look like this:

Create Private Git Repo

After using this function a bit, I realized having a private repo creator is just as useful. Here is the same alias and function made for private. The alias and function have an extra suffix to differentiate them. Copy and paste into your .bashrc or .zshrc.

Final Thoughts

All of these commands create a public repo. You can modify that by setting private to true (change "private": false to "private": true). If you want you can even create your own function for that.

The end

12Nov/20

Backup file[s] to Dropbox (without syncing)

This github page, https://github.com/andreafabrizi/Dropbox-Uploader, has the dropbox-uploader tool which I use to backup content from servers (Linux, Mac, etc) without having to sync my Dropbox content to the local disk on your server.

It uses the Dropbox API (docs here) . You can use the API pretty easily with curl commands, but for files over 150 MiB it gets complicated (chunking and such).

Before using the uploader, get an API key dropbox, Here is how you do that.

Note that there are 2 levels of access each API gets (which you configure):

  • Full access – allows the API to access your entire Dropbox (so if it fell into wrong hands it could access/write/delete into anything – even go as far as delete everything). The benefit is that a user can specify which directory in their Dropbox file system to backup files to.
  • Application access – This access limits the API to only the /Apps/<appname> directory (if the /Apps/<appname> directory is missing it will be created on the root of your Dropbox file system). So you can only backup content to /Apps/<appname>/. Within that directory the API has full control (well the access can be fine tuned)

For backup purposes just use the application level access, there is no reason for it to potentially have access to all of your Dropbox data.

More info on these access levels and Oauth authentication is here, https://www.dropbox.com/lp/developers/reference/oauth-guide

Steps by example:

First install dropbox_uploader.sh to any location of your choosing in your filesystem.

Now create a db.conf file. If you run dropbox_uploader.sh without the -f option, it will prompt you for your OAUTH KEY and create the conf file in its default location. I prefer putting it in a custom location.

echo "OAUTH_ACCESS_TOKEN=gvi4325ffFAKEKEYwadfasd-i234asdfa" > ~/backups/db.conf

Then if I want to copy a file called yourfile.txt, first I delete it from the destination. I do this to avoid possible time consuming hash-checks, because if the file already exists at the destination, this script does a hash check.


./dropbox_uploader.sh -f ~/backups/db.conf delete /yourfile.txt

Then I upload the file, by specifyin the source and destination location. Note the destination location must start with a forward slash. If your key have full access it will go to Dropbox:yourlocation. If your key has app access it will go to Dropbox:/Apps/appname/yourlocation. If you are unsure if the directories, exist, don’t worry the API creates the needed directories (even if they are a few nested ones).

./dropbox_uploader.sh -f ~/backups/db.conf upload ~/yourfile.txt /yourfile.txt

Cron

I use this tool to backup servers (data compressed with tar+xz => good high compression) to dropbox. In my backup script which is called by cron daily or however often, I do all of the above steps. I create the db.conf everytime, that way I can see all of the settings and commands from the backup script and don’t have to rummage thru my filesystem for my config file.

The end

09Nov/20

xargs parallel note to self

Xargs is useful to run in parallel. Its parallel processing is very efficient. Read this post about its efficiency and this one about basic commands.

Below, is my favorite way to run with a single-command method repeated (or parallelized):

Replace inputlist with anything. Each line of input list gets run once by command. In the command you can use {} if you need to call the line.

Below is a multi-command method:

Note: recommend to use single quotes on the outside as seen here

Note: you can redirect the output at the very left outside of the single quotes, this way each run’s output is saved. you can save each seperate run if you redirect inside the quotes.

Ex redirection examples:

The end

29Oct/20

Python + Guitar + All Notes

I came across this medium article about python and guitar strings and plotting scales. It has an interesting Jupyter notebook to work with, allowing to plot scales for all of the chords. It was great except it only covered 20 frets. My guitar has 24 frets. So I modified the script to allow for 24 frets & saving the plots. (I have note tested with more then 24)

So you can get something like this:

C Major scale (every whole note):

E Minor Blues scale:

Etc, the rest can be viewed from my Github (go into the Scales directory)

I then came across articles like this that also mapped each guitar string note to a midi value from 16 (low E) to N. N is the highest note. So in my case of 24 frets, N is 64 as that would be the 24th fret on the high E string. Each note increment is +1. I found this fascinating. So I modified the scripts to print all notes + their midi values. Immediately you see the pattern that every 5th fret is the same as the string above it (besides the change from G to B string).

Here is every whole note with midi values. This is also the C major scale:

If the images are too small then just zoom in.

23Oct/20

Python remove duplicates similar to bash uniq + sort