danieljon.es

index posts opinions likes dislikes interesting cgit

Posts

My posts about programming and things.
Date format is day/month/year because I'm sane.

prev 1 2 3 4 5 6 7 8 9 10 next
direct link

Terrible, hacky database syncing

30/1/2019

Someone in my family wanted a custom inventory system made that was accessible from any device and had some features they couldn't find in any pre-made solution so I made my own. It works great (code not distributed, if you want it ask, GPL3), however prior to creating it they never mentioned wanting to view it away from their LAN. So I had to come up with a solution.

The server that the website (it's PHP/mysql) is located on has no access to external servers, so I had to route everything through my own computer.

My solution was cronjobs, sqldumps and sftp. Perfectly hacky.

Every hour (*/59 * * * *, each computer has a different localtime, so there isn't any concerns there) the server hosting the website creates an sql dump of the database and backups the uploaded images, my local computer then pulls that dump and backup and uploads the sql dump to my external server, which then imports it.

The images are not uploaded to the external server, but this is fine, the user indicated they only need access to the textual data off-site. It would be just too much data otherwise.

I also created a cut-down viewer for the data removing the ability to view images, add, edit and delete items.

sql dump and images backup:

name=$(date +"%d-%b-%Y-%H-%M")
mysqldump -u inventory -ppasswordhere inventory > /home/user/inventory.sql

tar -zcvf /webuser/backups/inventory_directory.tar.gz /var/www/html/inventory

cp /webuser/backups/inventory_directory.tar.gz /home/user/inventory_directory.tar.gz
downloading and uploading the backups and sql dump:
sftp user@tewi:/home/user <<EOF
get /home/user/inventory.sql /home/daniel_j/programming/inventory.sql
get /home/user/inventory_directory.tar.gz /home/daniel_j/programming/inventory_directory.tar.gz
exit
EOF

sftp -oPort=port user@danieljon.es:/home/user	 <<EOF
put /home/daniel_j/programming/inventory.sql
exit
EOF
Importing the database:
mysql -u inventory -ppasswordhere inventory < /home/user/inventory.sql

direct link

listen.moe song history

10/1/2019

I had someone who listens to listen.moe complain that they had heard a song they liked but couldn't get the name of because they were driving. The website only gives you the last 3 songs played so I used their API to parse their song list and display every one that was played on a particular day in order. They found the song, and it sure is great.

getsongs.py

#!/usr/bin/python2.7
import json
import urllib2
import codecs

def request(url, header, jsonpayload):
	baseurl = "https://listen.moe/api/{}";
	req = urllib2.Request(baseurl.format(url));
	req.add_header("Content-Type", "application/json");
	req.add_header("Accept", "application/vnd.listen.v4+json");
	if header is not None:
		req.add_header(header[0], header[1]);
	response = urllib2.urlopen(req, json.dumps(jsonpayload) if jsonpayload is not None else None);
	return response;

loginpayload = {
	"username": "name",
	"password": "password"
}

def main():
	# login and get response with our token
	response = request("login", None, loginpayload);
	# extract token
	token = json.loads(response.read());
	token = token["token"];
	# request songs list
	response = request("songs", ["Authorization", "Bearer " + token], None);
	response = json.loads(response.read());
	# dump song list to file
	with codecs.open("songs.json", 'w', encoding="utf-8") as f:
		json.dump(response, f, indent=2, sort_keys=True, ensure_ascii=False);
	exit();

if __name__ == "__main__":
	main();
	
parse.py
#!/usr/bin/python2.7
import json

date = "2019-01-09";
songs = [];

def main():
	infile = "songs.json";
	with open(infile) as f:
		data = json.load(f)
	for song in data["songs"]:
		if date in song["lastPlayed"]:
			songs.append(song);
	# sort songs by timestamp
	songs.sort(key=lambda x:x["lastPlayed"]);
	for song in songs:
		print("{} last played {}".format(song["title"].encode("utf-8"), song["lastPlayed"])); 

if __name__ == "__main__":
	main();
	



direct link

A simple binary counter and LED matrix

20/12/2018



Using this same technique I emulated an 8x8 LED matrix and created a bouncing ball demo. The code can be found in my git repo.



#include <stdio.h>
#include <unistd.h>

// website generator breaks with unicode, so fill in the unicode characters yourself
#define CIRCLE "CIRCLE"
#define BLACKCIRCLE "BLACKCIRCLE"

void
printleds(unsigned char value)
{
	printf("\0337");
	printf("%s", "\033[1;0F");
	printf("%s", "\033[0J");
	printf("%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n",
			(value & 1 << 7) ? BLACKCIRCLE : CIRCLE,
			(value & 1 << 6) ? BLACKCIRCLE : CIRCLE,
			(value & 1 << 5) ? BLACKCIRCLE : CIRCLE,
			(value & 1 << 4) ? BLACKCIRCLE : CIRCLE,
			(value & 1 << 3) ? BLACKCIRCLE : CIRCLE,
			(value & 1 << 2) ? BLACKCIRCLE : CIRCLE,
			(value & 1 << 1) ? BLACKCIRCLE : CIRCLE,
			(value & 1 << 0) ? BLACKCIRCLE : CIRCLE
	      );
	printf("\0338");
}

int
main(void)
{
	unsigned char value = 0;
	while (1)
	{
		printleds(value);
		value++;
		if (value >= 255)
			value = 0;
		sleep(1);
	}
	return 0;
}


direct link

Switched to dash as my shell

3/12/2018

For a number of weeks now I have been using dash as my shell, replacing bash.

I have been taking tab-completion and the command history for granted and felt it appropriate to lose those luxuries.

I have also left PS1 as default, meaning 'pwd' has become a close friend.

I also re-linked /bin/sh to point to dash, meaning I had to remove all the bash-isms from my various scripts which was a fun process.

Glob patterns have replaced tab completion and carefully thinking about what I am typing removes the need for the command history and arrow-key navigation (which isn't a thing in dash, meaning you can't go back to add or change a character).

I have no intentions on switching shells back to bash, I like this kind of punishment.


direct link

Raspberry Pi 4 digit 7 segment display effects

27/11/2018

After the last post I began playing with a real 7 segment display with 4 digits using the Pi. My goal was to create text effects without using libraries.

displayword(unsigned char str[4]) // simply display a word, use in a loop
blinkword(unsigned char str[4], int repeat) // blink word once plus any integer > 0. < 0 will blink forever
runclock(void) // display the time on the display, forever
scrollword(char *string) // scroll the string from right to left with a small buffer of extra characters
Scrolling text:


Blinking text:

The source code can be found here.

direct link

7 segment display

11/11/2018

post87_1

The gif recreated using this program:

post87_2

The code for this version can be found here.

---

#include <stdio.h>

/*
     A
     _
   F| |B
   G -
   E| |C
     -
     D

   A 7 segment display (unsurprisingly) has 7 individual segments that can conveniently make 0-9 and A-F
   perfect for hexadecimal.
   The state of a single 7 segment character can also conveniently be stored in 8 bits (a byte, in C a char),
   almost as if these aren't coincidences (they're not).
 */

enum segments
{
	/* refer to the diagram to know which value represents which segment */
	SEG_A 	= 1 << 0,
	SEG_B 	= 1 << 1,
	SEG_C	= 1 << 2,
	SEG_D	= 1 << 3,
	SEG_E	= 1 << 4,
	SEG_F	= 1 << 5,
	SEG_G	= 1 << 6,
};

enum characters
{
	CHAR_ZERO	= SEG_A | SEG_B | SEG_C | SEG_D | SEG_E | SEG_F,
	CHAR_ONE	= SEG_B | SEG_C,
	CHAR_TWO	= SEG_A | SEG_B | SEG_D | SEG_E | SEG_G,
	CHAR_THREE	= SEG_A | SEG_B | SEG_C | SEG_D | SEG_G,
	CHAR_FOUR	= SEG_B | SEG_C | SEG_F | SEG_G,
	CHAR_FIVE	= SEG_A | SEG_C | SEG_D | SEG_F | SEG_G,
	CHAR_SIX	= SEG_A | SEG_C | SEG_D | SEG_E | SEG_F | SEG_G,
	CHAR_SEVEN	= SEG_A | SEG_B | SEG_C,
	CHAR_EIGHT	= SEG_A | SEG_B | SEG_C | SEG_D | SEG_E | SEG_F | SEG_G,
	CHAR_NINE	= SEG_A | SEG_B | SEG_C | SEG_D | SEG_F | SEG_G,
	CHAR_A		= SEG_A | SEG_B | SEG_C | SEG_E | SEG_F | SEG_G,
	CHAR_B		= SEG_C | SEG_D | SEG_E | SEG_F | SEG_G,
	CHAR_C		= SEG_A | SEG_D | SEG_E | SEG_F,
	CHAR_D		= SEG_B | SEG_C | SEG_D | SEG_E | SEG_G,
	CHAR_E		= SEG_A | SEG_D | SEG_E | SEG_F | SEG_G,
	CHAR_F		= SEG_A | SEG_E | SEG_F | SEG_G,
};

void
printdisplay(char segs)
{
	/* line 1 */
	printf("\t %c\n", (segs & SEG_A) ? '_' : ' ');
	/* line 2 */
	printf("\t%c %c\n", (segs & SEG_F) ? '|' : ' ',
			    (segs & SEG_B) ? '|' : ' ');
	/* line 3 */
	printf("\t %c\n", (segs & SEG_G) ? '-' : ' ');
	/* line 4 */
	printf("\t%c %c\n", (segs & SEG_E) ? '|' : ' ',
			    (segs & SEG_C) ? '|' : ' ');
	/* line 5 */
	printf("\t %c\n", (segs & SEG_D) ? '-' : ' ');
}


int
main(void)
{
	printdisplay(CHAR_FIVE);
}
---

This program outputs:
	 _
	|  
	 -
	  |
	 -

direct link

Bit manipulation

3/11/2018

Bit manipulation is something I find interesting, to strengthen my understanding I thought it'd be best to put some of the basics in text.

First, an understanding of binary is required:

128
64 32 16 8 4 2 1
0 0 1 0 1 1 0 0
= 44

Got it? I knew you would. If not see my post on binary.

Next you need an understanding of bitwise operations:
AND (&)
copies bit if it is set in both operands
11 AND 5 (1011 AND 0101) = 1
1011
0101
----
0001
OR (|)
inclusive OR, copies bit if it is set in either operand
13 OR 7 (1101 OR 0111) = 15
1101
0111
----
1111
XOR (^)
exclusive OR, copies bit if it is set in either operand but not if it is set in both
13 XOR 7 (1101 XOR 0111) = 10
1101
0111
----
1010
NOT (~)
flips each bit obtaining the numbers one's complement
NOT 5 (101) = 2
101
---
010
Left Shift (<<)
Shifts the bits of the left operand by the number in the right operand to the left
5 << 2 (101 << 010) = 20
000101
 <<<	(the 101 is moved to the left by 2 bits)
------
010100
Right Shift (>>)
Shifts the bits of the left operand by the number in the right operand to the right, underflow is discarded
22 >> 3 (10110 >> 00011) = 2
10110
   >>>>>(the 10110 is move to the right by 3 bits)
-----
00010
Now that those are understood, let's take a look at some bit manipulation, I'm going to use C bitwise operators from now on.

First note, counting bits starts from the least significant bit (right-most), which is the 0th bit.
Setting a bit
To set a bit to 1 OR the number with 1 left shifted with the desired bit to set
number |= (1 << bit)
number = 5, bit = 3
5 | (1 << 3) = 13 

1 << 3 = 8
 0001
 <<<
-----
 1000

5 | 8 = 13
  0101
| 1000
------
  1101
  ^
  our 3rd bit (0,1,2,3) is now set
Clearing a bit
To clear a bit AND the number with the one's complement of 1 left shifted with the desired bit to clear
number &= ~(1 << bit)
number = 9, bit = 3
9 & ~(1 << 3) = 1

1 << 3 = 8
0001
<<<
----
1000

~8 = 7
  1000
~
------
  0111

9 & 7 = 1
  1001
& 0111
------
  0001
  ^
  our 3rd bit (0,1,2,3) is now 0
Getting a bit
To get a bit right shift the number by the desired bit to retrieve, then AND it by 1
result = (number >> bit) & 1
number = 14, bit = 3
result = (14 >> 3) & 1
result = 1

14 >> 3 = 1
1110
>>>
-----
0001

1 & 1 = 1
01
01
--
01
Toggling a bit
To toggle a bit XOR the number by 1 left shifted by the desired bit
number ^= 1 << bit
number = 12, bit = 4
12 ^= 1 << 4 = 28

1 << 4 = 16
00001
<<<<
-----
10000

12 ^ 16 = 28
  01100
^ 10000
-------
  11100
  ^
  our 4th bit (0,1,2,3,4) is now set
Let's do something with this knowledge and use the bits of a char as individual flags that we can use. Using an enum is a much better way to do this.
#include <stdio.h>

int
dectobin(int dec)
{
	/* return the binary representation of dec */
	if (dec == 0) return 0;
	if (dec == 1) return 1;
	return (dec % 2) + 10 * dectobin(dec / 2);
}

int
getbit(int num, int bit)
{
	return (num >> bit) & 1;
}

int
togglebit(int number, int bit)
{
	return number ^= 1 << bit;
}

int
setbit(int number, int bit)
{
	return number |= 1 << bit;
}

int
clearbit(int number, int bit)
{
	return number &= ~(1 << bit);
}

int
main(void)
{
	unsigned char flags = 0; /* char because it is 1 byte, unsigned because we don't need negatives */
	printf("flag has value %d, in binary: %08d\n", flags, dectobin(flags));
	puts("let's make the number 145");
	puts("we need to toggle bits: 0, 4, 7");
	flags = setbit(flags, 0);
	puts("set bit 0");
	printf("flag has value %d, in binary: %08d\n", flags, dectobin(flags));
	flags = setbit(flags, 4);
	puts("set bit 4");
	printf("flag has value %d, in binary: %08d\n", flags, dectobin(flags));
	flags = setbit(flags, 7);
	puts("set bit 7");
	printf("flag has value %d, in binary: %08d\n", flags, dectobin(flags));

	puts("let's check the value of bit 7");
	printf("bit 7 is %d\n", getbit(flags, 7));
	puts("we don't want bit 7 set anymore");
	flags = clearbit(flags, 7);
	puts("cleared bit 7");
	printf("flag has value %d, in binary: %08d\n", flags, dectobin(flags));
	return 0;
}

output:
flag has value 0, in binary: 00000000
let's make the number 145
we need to toggle bits: 0, 4, 7
set bit 0
flag has value 1, in binary: 00000001
set bit 4
flag has value 17, in binary: 00010001
set bit 7
flag has value 145, in binary: 10010001
let's check the value of bit 7
bit 7 is 1
we don't want bit 7 set anymore
cleared bit 7
flag has value 17, in binary: 00010001

direct link

string replacement in files

14/10/2018

I'm slowly rewriting my website generator in C and for it I required an easy way to write an input file to an output file replacing every instance of something with another string. So I made this convenience function that might be useful elsewhere one day.

int replaceinfile(char *originalfile, char *destinationfile, char *placeholder,
								char *replacement);
// 0 = failure, 1 = success

replaceinfile("input.txt", "output.txt", "%PLACEHOLDER%", "replacement string");
input.txt
Normal text file

%PLACEHOLDER%
%PLACEHOLDER%
%PLACEHOLDER%

With lines and whatever, %PLACEHOLDER%

123%PLACEHOLDER%123	
ok %PLACEHOLDER% ok
output.txt
Normal text file

replacement string
replacement string
replacement string

With lines and whatever, replacement string

123replacement string123	
ok replacement string ok
The source is available in my git repository.

direct link

startup scripts

6/10/2018

I setup and open a few things when I log into my computer through .bash_profile, these include two instances of irssi, mutt, a slack-to-irc gateway and cmus.

This is achieved through three shell scripts:

main.sh - this script launches my other two scripts. It checks for the file /tmp/started, if it exists nothing is done and the script exits. If the file does exist the file /tmp/started is created (empty) and my two other scripts start. /tmp is wiped on boot.

#!/bin/bash

cd "$HOME/programming/bash/autostart"

if [ ! -f /tmp/started ]; then
	TMPFILE="/tmp/started"
	touch $TMPFILE
	./startup.sh
	./slack.sh
fi
cd $HOME
startup.sh - This script is responsible for launching irssi, cmus and mutt. I open these in a tmux session named main. It has 3 windows named irssi. cmus and mutt respectively.
#!/bin/bash

tmux new-session -d -s main -n irssi
tmux new-window -t main:1 -n cmus
tmux new-window -t main:2 -n mutt

tmux send-keys -t main:0 "firejail irssi" C-m
tmux send-keys -t main:1 "cd $HOME/music/japanese;cmus" C-m
tmux send-keys -t main:2 "firejail mutt" C-m
cd $HOME
slack.sh - This script is responsible for launching my slack-to-irc gateway and an irssi instance that automatically connects to it. Once again a tmux session is created with two windows named lsirc and irssi.
#!/bin/bash
BASE="$HOME/programming/python/localslackirc"
cd $BASE
tmux new-session -d -s slack -n lsirc
tmux new-window -t slack:1 -n irssi

tmux send-keys -t slack:0 "cd $BASE;python3 ./irc.py -j" C-m
sleep 1
tmux send-keys -t slack:1 "firejail irssi --home $HOME/.slackirssi -c 127.0.0.1 -p 9007" C-m
While simple, its really effective and saves a lot of effort every boot. Is there a better way to run a script once when you initially log in? Probably but the effort to search was more than it was worth.

direct link

cfg parser

5/10/2018

I wanted to roll my own cfg/ini-like parser in C, it started off somewhat elegant to my low standards, however it quickly became a mess. It isn't great, but it "works".

It handles simple cfg/ini like files with 'key=value' pairs.

It's really hacky, I wouldn't really trust it for anything.

Comments are ignored, they're denoted by the '#' character.

Spaces are currently treated literally. test=123 and test = 123 are DIFFERENT, the key has a space appended and the value a space prepended in the second example.

Duplicate keys are not handled currently, the cfggetvalue function will only give you the first value.

See settings.cfg for an example cfg file.

To compile type make. Run the example program by issuing ./example.

All functions return 0 on failure, this goes for getting a value, always check 0 isn't returned. If it is, that means the key was not found.

Writing back to the cfg file is NOT implemented.

main.c has a usage example, however the usage structure is like this:

// allocate memory for the cfg structure
struct cfgfile *cfg = malloc(sizeof(struct cfgfile));
// setup cfg structure passing it the cfg structure we just made and a file name
cfgsetup(cfg, "filename.cfg");

// create a buffer for our value
char buffer[256];
// collect our value, the function returns 0 on a failure (when it cannot find the key)
int val;
val = cfggetvalue(cfg, "key", buffer, sizeof(buffer));
if (val != 0)
	puts(buffer); // print our value
else
	puts("cannot find the key!");

// free all of our memory (including cfg itself)
cfgfree(cfg);
The source is available in my git repository.

prev 1 2 3 4 5 6 7 8 9 10 next

page generated 10-02-2019 using sitegenerator