urlopener re-written in C

8/3/2019

For the longest time I was using a urlopener script I had written in Python. I pipe links from my terminal into dmenu, then into my urlopener. The urlopener launches certain programs depending on the domain and file extensions. I recently thought it'd be fun to rewrite it in C. It was fun.

The source code can be found in my git repo.

---

Terrible, hacky database syncing

30/1/2019

Someone in my family wanted a custom inventory system made that was accessible from any device and had some features they couldn't find in any pre-made solution so I made my own. It works great (code not distributed, if you want it ask, GPL3), however prior to creating it they never mentioned wanting to view it away from their LAN. So I had to come up with a solution.

The server that the website (it's PHP/mysql) is located on has no access to external servers, so I had to route everything through my own computer.

My solution was cronjobs, sqldumps and sftp. Perfectly hacky.

Every hour (*/59 * * * *, each computer has a different localtime, so there isn't any concerns there) the server hosting the website creates an sql dump of the database and backups the uploaded images, my local computer then pulls that dump and backup and uploads the sql dump to my external server, which then imports it.

The images are not uploaded to the external server, but this is fine, the user indicated they only need access to the textual data off-site. It would be just too much data otherwise.

I also created a cut-down viewer for the data removing the ability to view images, add, edit and delete items.

sql dump and images backup:

name=$(date +"%d-%b-%Y-%H-%M")
mysqldump -u inventory -ppasswordhere inventory > /home/user/inventory.sql

tar -zcvf /webuser/backups/inventory_directory.tar.gz /var/www/html/inventory

cp /webuser/backups/inventory_directory.tar.gz /home/user/inventory_directory.tar.gz

downloading and uploading the backups and sql dump:

sftp user@tewi:/home/user <<EOF
get /home/user/inventory.sql /home/daniel_j/programming/inventory.sql
get /home/user/inventory_directory.tar.gz /home/daniel_j/programming/inventory_directory.tar.gz
exit
EOF

sftp -oPort=port user@danieljon.es:/home/user	 <<EOF
put /home/daniel_j/programming/inventory.sql
exit
EOF

Importing the database:

mysql -u inventory -ppasswordhere inventory < /home/user/inventory.sql

---

listen.moe song history

10/1/2019

I had someone who listens to listen.moe complain that they had heard a song they liked but couldn't get the name of because they were driving. The website only gives you the last 3 songs played so I used their API to parse their song list and display every one that was played on a particular day in order. They found the song, and it sure is great.

getsongs.py

#!/usr/bin/python2.7
import json
import urllib2
import codecs

def request(url, header, jsonpayload):
	baseurl = "https://listen.moe/api/{}";
	req = urllib2.Request(baseurl.format(url));
	req.add_header("Content-Type", "application/json");
	req.add_header("Accept", "application/vnd.listen.v4+json");
	if header is not None:
		req.add_header(header[0], header[1]);
	response = urllib2.urlopen(req, json.dumps(jsonpayload) if jsonpayload is not None else None);
	return response;

loginpayload = {
	"username": "name",
	"password": "password"
}

def main():
	# login and get response with our token
	response = request("login", None, loginpayload);
	# extract token
	token = json.loads(response.read());
	token = token["token"];
	# request songs list
	response = request("songs", ["Authorization", "Bearer " + token], None);
	response = json.loads(response.read());
	# dump song list to file
	with codecs.open("songs.json", 'w', encoding="utf-8") as f:
		json.dump(response, f, indent=2, sort_keys=True, ensure_ascii=False);
	exit();

if __name__ == "__main__":
	main();
	

parse.py

#!/usr/bin/python2.7
import json

date = "2019-01-09";
songs = [];

def main():
	infile = "songs.json";
	with open(infile) as f:
		data = json.load(f)
	for song in data["songs"]:
		if date in song["lastPlayed"]:
			songs.append(song);
	# sort songs by timestamp
	songs.sort(key=lambda x:x["lastPlayed"]);
	for song in songs:
		print("{} last played {}".format(song["title"].encode("utf-8"), song["lastPlayed"])); 

if __name__ == "__main__":
	main();
	

---

A simple binary counter and LED matrix

20/12/2018

Using this same technique I emulated an 8x8 LED matrix and created a bouncing ball demo. The code can be found in my git repo.

#include <stdio.h>
#include <unistd.h>

// website generator breaks with unicode, so fill in the unicode characters yourself
#define CIRCLE "CIRCLE"
#define BLACKCIRCLE "BLACKCIRCLE"

void
printleds(unsigned char value)
{
	printf("\0337");
	printf("%s", "\033[1;0F");
	printf("%s", "\033[0J");
	printf("%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n",
			(value & 1 << 7) ? BLACKCIRCLE : CIRCLE,
			(value & 1 << 6) ? BLACKCIRCLE : CIRCLE,
			(value & 1 << 5) ? BLACKCIRCLE : CIRCLE,
			(value & 1 << 4) ? BLACKCIRCLE : CIRCLE,
			(value & 1 << 3) ? BLACKCIRCLE : CIRCLE,
			(value & 1 << 2) ? BLACKCIRCLE : CIRCLE,
			(value & 1 << 1) ? BLACKCIRCLE : CIRCLE,
			(value & 1 << 0) ? BLACKCIRCLE : CIRCLE
	      );
	printf("\0338");
}

int
main(void)
{
	unsigned char value = 0;
	while (1)
	{
		printleds(value);
		value++;
		if (value >= 255)
			value = 0;
		sleep(1);
	}
	return 0;
}

---

Switched to dash as my shell

3/12/2018

For a number of weeks now I have been using dash as my shell, replacing bash.

I have been taking tab-completion and the command history for granted and felt it appropriate to lose those luxuries.

I have also left PS1 as default, meaning 'pwd' has become a close friend.

I also re-linked /bin/sh to point to dash, meaning I had to remove all the bash-isms from my various scripts which was a fun process.

Glob patterns have replaced tab completion and carefully thinking about what I am typing removes the need for the command history and arrow-key navigation (which isn't a thing in dash, meaning you can't go back to add or change a character).

I have no intentions on switching shells back to bash, I like this kind of punishment.

---

Raspberry Pi 4 digit 7 segment display effects

27/11/2018

After the last post I began playing with a real 7 segment display with 4 digits using the Pi. My goal was to create text effects without using libraries.

displayword(unsigned char str[4]) // simply display a word, use in a loop
blinkword(unsigned char str[4], int repeat) // blink word once plus any integer > 0. < 0 will blink forever
runclock(void) // display the time on the display, forever
scrollword(char *string) // scroll the string from right to left with a small buffer of extra characters

Scrolling text:


Blinking text:

The source code can be found here.

---

7 segment display

11/11/2018

The gif recreated using this program:



The code for this version can be found here.

---

#include <stdio.h>

/*
     A
     _
   F| |B
   G -
   E| |C
     -
     D

   A 7 segment display (unsurprisingly) has 7 individual segments that can conveniently make 0-9 and A-F
   perfect for hexadecimal.
   The state of a single 7 segment character can also conveniently be stored in 8 bits (a byte, in C a char),
   almost as if these aren't coincidences (they're not).
 */

enum segments
{
	/* refer to the diagram to know which value represents which segment */
	SEG_A 	= 1 << 0,
	SEG_B 	= 1 << 1,
	SEG_C	= 1 << 2,
	SEG_D	= 1 << 3,
	SEG_E	= 1 << 4,
	SEG_F	= 1 << 5,
	SEG_G	= 1 << 6,
};

enum characters
{
	CHAR_ZERO	= SEG_A | SEG_B | SEG_C | SEG_D | SEG_E | SEG_F,
	CHAR_ONE	= SEG_B | SEG_C,
	CHAR_TWO	= SEG_A | SEG_B | SEG_D | SEG_E | SEG_G,
	CHAR_THREE	= SEG_A | SEG_B | SEG_C | SEG_D | SEG_G,
	CHAR_FOUR	= SEG_B | SEG_C | SEG_F | SEG_G,
	CHAR_FIVE	= SEG_A | SEG_C | SEG_D | SEG_F | SEG_G,
	CHAR_SIX	= SEG_A | SEG_C | SEG_D | SEG_E | SEG_F | SEG_G,
	CHAR_SEVEN	= SEG_A | SEG_B | SEG_C,
	CHAR_EIGHT	= SEG_A | SEG_B | SEG_C | SEG_D | SEG_E | SEG_F | SEG_G,
	CHAR_NINE	= SEG_A | SEG_B | SEG_C | SEG_D | SEG_F | SEG_G,
	CHAR_A		= SEG_A | SEG_B | SEG_C | SEG_E | SEG_F | SEG_G,
	CHAR_B		= SEG_C | SEG_D | SEG_E | SEG_F | SEG_G,
	CHAR_C		= SEG_A | SEG_D | SEG_E | SEG_F,
	CHAR_D		= SEG_B | SEG_C | SEG_D | SEG_E | SEG_G,
	CHAR_E		= SEG_A | SEG_D | SEG_E | SEG_F | SEG_G,
	CHAR_F		= SEG_A | SEG_E | SEG_F | SEG_G,
};

void
printdisplay(char segs)
{
	/* line 1 */
	printf("\t %c\n", (segs & SEG_A) ? '_' : ' ');
	/* line 2 */
	printf("\t%c %c\n", (segs & SEG_F) ? '|' : ' ',
			    (segs & SEG_B) ? '|' : ' ');
	/* line 3 */
	printf("\t %c\n", (segs & SEG_G) ? '-' : ' ');
	/* line 4 */
	printf("\t%c %c\n", (segs & SEG_E) ? '|' : ' ',
			    (segs & SEG_C) ? '|' : ' ');
	/* line 5 */
	printf("\t %c\n", (segs & SEG_D) ? '-' : ' ');
}


int
main(void)
{
	printdisplay(CHAR_FIVE);
}

---

This program outputs:

	 _
	|  
	 -
	  |
	 -

---

Bit manipulation

3/11/2018

Bit manipulation is something I find interesting, to strengthen my understanding I thought it'd be best to put some of the basics in text.

First, an understanding of binary is required:

128	64	32	16	8	4	2	1
0	0	1	0	1	1	0	0

= 44

Got it? I knew you would. If not see my post on binary.

Next you need an understanding of bitwise operations:

AND (&)	copies bit if it is set in both operands 11 AND 5 (1011 AND 0101) = 1 1011 0101 ---- 0001
OR (|)	inclusive OR, copies bit if it is set in either operand 13 OR 7 (1101 OR 0111) = 15 1101 0111 ---- 1111
XOR (^)	exclusive OR, copies bit if it is set in either operand but not if it is set in both 13 XOR 7 (1101 XOR 0111) = 10 1101 0111 ---- 1010
NOT (~)	flips each bit obtaining the numbers one's complement NOT 5 (101) = 2 101 --- 010
Left Shift (<<)	Shifts the bits of the left operand by the number in the right operand to the left 5 << 2 (101 << 010) = 20 000101 <<< (the 101 is moved to the left by 2 bits) ------ 010100
Right Shift (>>)	Shifts the bits of the left operand by the number in the right operand to the right, underflow is discarded 22 >> 3 (10110 >> 00011) = 2 10110 >>>>>(the 10110 is move to the right by 3 bits) ----- 00010

Now that those are understood, let's take a look at some bit manipulation, I'm going to use C bitwise operators from now on.

First note, counting bits starts from the least significant bit (right-most), which is the 0th bit.

Setting a bit	To set a bit to 1 OR the number with 1 left shifted with the desired bit to set number |= (1 << bit) number = 5, bit = 3 5 | (1 << 3) = 13 1 << 3 = 8 0001 <<< ----- 1000 5 | 8 = 13 0101 | 1000 ------ 1101 ^ our 3rd bit (0,1,2,3) is now set
Clearing a bit	To clear a bit AND the number with the one's complement of 1 left shifted with the desired bit to clear number &= ~(1 << bit) number = 9, bit = 3 9 & ~(1 << 3) = 1 1 << 3 = 8 0001 <<< ---- 1000 ~8 = 7 1000 ~ ------ 0111 9 & 7 = 1 1001 & 0111 ------ 0001 ^ our 3rd bit (0,1,2,3) is now 0
Getting a bit	To get a bit right shift the number by the desired bit to retrieve, then AND it by 1 result = (number >> bit) & 1 number = 14, bit = 3 result = (14 >> 3) & 1 result = 1 14 >> 3 = 1 1110 >>> ----- 0001 1 & 1 = 1 01 01 -- 01
Toggling a bit	To toggle a bit XOR the number by 1 left shifted by the desired bit number ^= 1 << bit number = 12, bit = 4 12 ^= 1 << 4 = 28 1 << 4 = 16 00001 <<<< ----- 10000 12 ^ 16 = 28 01100 ^ 10000 ------- 11100 ^ our 4th bit (0,1,2,3,4) is now set

Let's do something with this knowledge and use the bits of a char as individual flags that we can use. Using an enum is a much better way to do this.

#include <stdio.h>

int
dectobin(int dec)
{
	/* return the binary representation of dec */
	if (dec == 0) return 0;
	if (dec == 1) return 1;
	return (dec % 2) + 10 * dectobin(dec / 2);
}

int
getbit(int num, int bit)
{
	return (num >> bit) & 1;
}

int
togglebit(int number, int bit)
{
	return number ^= 1 << bit;
}

int
setbit(int number, int bit)
{
	return number |= 1 << bit;
}

int
clearbit(int number, int bit)
{
	return number &= ~(1 << bit);
}

int
main(void)
{
	unsigned char flags = 0; /* char because it is 1 byte, unsigned because we don't need negatives */
	printf("flag has value %d, in binary: %08d\n", flags, dectobin(flags));
	puts("let's make the number 145");
	puts("we need to toggle bits: 0, 4, 7");
	flags = setbit(flags, 0);
	puts("set bit 0");
	printf("flag has value %d, in binary: %08d\n", flags, dectobin(flags));
	flags = setbit(flags, 4);
	puts("set bit 4");
	printf("flag has value %d, in binary: %08d\n", flags, dectobin(flags));
	flags = setbit(flags, 7);
	puts("set bit 7");
	printf("flag has value %d, in binary: %08d\n", flags, dectobin(flags));

	puts("let's check the value of bit 7");
	printf("bit 7 is %d\n", getbit(flags, 7));
	puts("we don't want bit 7 set anymore");
	flags = clearbit(flags, 7);
	puts("cleared bit 7");
	printf("flag has value %d, in binary: %08d\n", flags, dectobin(flags));
	return 0;
}

output:

flag has value 0, in binary: 00000000
let's make the number 145
we need to toggle bits: 0, 4, 7
set bit 0
flag has value 1, in binary: 00000001
set bit 4
flag has value 17, in binary: 00010001
set bit 7
flag has value 145, in binary: 10010001
let's check the value of bit 7
bit 7 is 1
we don't want bit 7 set anymore
cleared bit 7
flag has value 17, in binary: 00010001

---

string replacement in files

14/10/2018

I'm slowly rewriting my website generator in C and for it I required an easy way to write an input file to an output file replacing every instance of something with another string. So I made this convenience function that might be useful elsewhere one day.

int replaceinfile(char *originalfile, char *destinationfile, char *placeholder,
								char *replacement);
// 0 = failure, 1 = success

replaceinfile("input.txt", "output.txt", "%PLACEHOLDER%", "replacement string");

input.txt

Normal text file

%PLACEHOLDER%
%PLACEHOLDER%
%PLACEHOLDER%

With lines and whatever, %PLACEHOLDER%

123%PLACEHOLDER%123	
ok %PLACEHOLDER% ok

output.txt

Normal text file

replacement string
replacement string
replacement string

With lines and whatever, replacement string

123replacement string123	
ok replacement string ok

The source is available in my git repository.

---

startup scripts

6/10/2018

I setup and open a few things when I log into my computer through .bash_profile, these include two instances of irssi, mutt, a slack-to-irc gateway and cmus.

This is achieved through three shell scripts:

main.sh - this script launches my other two scripts. It checks for the file /tmp/started, if it exists nothing is done and the script exits. If the file does exist the file /tmp/started is created (empty) and my two other scripts start. /tmp is wiped on boot.

#!/bin/bash

cd "$HOME/programming/bash/autostart"

if [ ! -f /tmp/started ]; then
	TMPFILE="/tmp/started"
	touch $TMPFILE
	./startup.sh
	./slack.sh
fi
cd $HOME

startup.sh - This script is responsible for launching irssi, cmus and mutt. I open these in a tmux session named main. It has 3 windows named irssi. cmus and mutt respectively.

#!/bin/bash

tmux new-session -d -s main -n irssi
tmux new-window -t main:1 -n cmus
tmux new-window -t main:2 -n mutt

tmux send-keys -t main:0 "firejail irssi" C-m
tmux send-keys -t main:1 "cd $HOME/music/japanese;cmus" C-m
tmux send-keys -t main:2 "firejail mutt" C-m
cd $HOME

slack.sh - This script is responsible for launching my slack-to-irc gateway and an irssi instance that automatically connects to it. Once again a tmux session is created with two windows named lsirc and irssi.

#!/bin/bash
BASE="$HOME/programming/python/localslackirc"
cd $BASE
tmux new-session -d -s slack -n lsirc
tmux new-window -t slack:1 -n irssi

tmux send-keys -t slack:0 "cd $BASE;python3 ./irc.py -j" C-m
sleep 1
tmux send-keys -t slack:1 "firejail irssi --home $HOME/.slackirssi -c 127.0.0.1 -p 9007" C-m

While simple, its really effective and saves a lot of effort every boot. Is there a better way to run a script once when you initially log in? Probably but the effort to search was more than it was worth.

---