Thought process of a programming problem

3/11/2024

For my LED matrix sign project I store all the customisations and text in the ESP32's NVRAM. I do this in a very naive way of writing raw memory directly to the NVRAM packed into two data structures. This has one very big flaw, if I change the internal layout of these structures in any way, the memory in NVRAM will no longer be valid.

Before making any changes to the code, this is the layout of the structure in question:

struct
{
	int pos;
	int brightness;
	char cssid[256];
	char cpassword[256];
	char date[32];

} globalconf;

This struct contains all the configuration settings that may be used globally.
Specifically for this, we're interested in the brightness integer.

I wanted to add automatic brightness adjustment depending on the rooms current light level. I do this using an LDR. to achieve this, I needed a way to store whether or not the auto brightness feature is enabled or not.

My first naive implementation seen in this commit simply added more fields to the globalconf structure.

This worked, however it made my the current settings stored in the NVRAM incompatible. This is because the memory size of the structure changed by 16 (or so) bytes. When this memory is read into the new structure from NVRAM, it is incorrect.

I had to come up with a new solution, so I did some brainstorming and made notes in chat. I am going to paste those logs here and explain at the end the final solutions.

I could just pack that autobrightness bool into the brightness int as it is literally 1 bit required
then i'd not break the globalconf structure
the value of brightness is literally 0-15
so i could use the MSB to store the bool
i could do:

brightness |= autoBrightness ? (brightness | 0b10000000) : (brightness & 0b01111111);

set the MSB if autobrightness is true, otherwise unset the MSB
then when i read the brightness later i need to read it like:

(brightness & 0b01111111)

to ensure it is unset
or i KISS and leave it as is
OR i do the smarter thing actually
create a bit field with the same size as an int and do this built in
a bitfield lets you assign certain bits to certain values in a single packed bit of memory

struct brightness
{
    unsigned int autoBrightness : 1;
    unsigned int brightness : 4;
    unsigned int RESERVED : 3;
}

this will give me 3 bits to play with later
this will use 1 byte of memory
brightness is 0-15 so 4 bits is plenty (can count to 15)
so i get MORE features and use 3 bytes less (as an int is 4 bytes)
but actually... i need to use all 4 bytes to not break my current memory structure
so really i need to do:

struct brightness
{
    unsigned int autoBrightness : 1;
    unsigned int brightness : 4;
    unsigned int RESERVED : 27;
}

to use all 32 bits of an int

Essentially, as seen in this commit, I simply added a struct bitfield the same size as the fields it replaces.

struct brightness
{
    unsigned int autoBrightness : 1;
    unsigned int brightness : 4;
    unsigned int RESERVED : 27;
};

struct
{
	int pos;
	struct brightness brightness;
	char cssid[256];
	char cpassword[256];
	char date[32];

} globalconf;

and gives me 27 bits later on to play with.

---

Downlights, Matter, Thread and Home Assistant

6/10/2024

I recently purchased a couple nanoleaf Essentials downlights to replace the ones in my room. I specifically chose these downlights as they support Matter over Thread. Matter is a communication protocol currently being implemented as another attempt at a standard for communication between hardware devices. Thread is a wireless networking technology being implemented to achieve the saame result. Matter can communicate over other networks as it is just a protocol, but a thread network requires special hardware.

I chose to purchase these from smarthome.com.au who happen to be relatively local. I went to their office to pick up my online purchase and found them very welcoming and helpful. The office felt like their loungeroom.

Ultimately, my plan was to finally setup Home Assistant and start using so called smart devices in a free and local way. There is not much in the way of online documentation for a lot of Matter/Thread devices, especially a relatively niche downlight, so most information was about nanoleaf's other products. Others require the device first be initialised in their Android software. They are setup via Bluetooth first, then Thread network support is enabled through the application.

Setting the devices up in the nanoleaf software did not go well. Recently a friend of mine gave me a Pixel 8a, finally a decent device with a modern bluetooth implementation. But the nanoleaf software refused to find any Thread networks on this phone. This is weird as the Thread networking is done on the downlight itself. It worked on a family members Samsung phone, which is much older than my device. No Matter, it worked in the end.

Now I needed Home Assistant running. I am currently running their provided OS image in kvm. To provide the VM direct access to the LAN without NATing, I gave the VM a Macvtap interface and I use the NAT interface for host->guest communication. Home Assistant conveniently listens on all interfaces. Obviously, as the Thread network requires special hardware, I also needed a device the VM can use to communicate. I chose to purchase the Home Assistant Connect. This USB dongle supports both Zigbee and Thread, however without very early firmware, only one or the other can operate at a time. I figured a Home Assistant specifically developed dongle would be the safer bet, even though it probably wont work on both networks yet. I only have Thread devices right now. I also gave it a Bluetooth dongle for a temperature sensor I planned to add as well.

I set Home Assistant up to be a Thread border router using the Open Thread Border Router integration, then setup both the Matter and Thread integrations following their documentation. Now it was time to add the devices to Home Assistant. This was a huge pain as Home Assistant refused to find the devices anywhere and I knew of no way to ensure my border router was working correctly and the other integrations were operating too. after hours and many threads read, I found I needed to sync Thread credentials by going into Settings->Companion app->Troubleshooting(!!)->Sync Thread credentials. After this the devices appeared and I was able to add them. Also, by the way, this all has to be done on a phone running the Home Assistant application. It cannot be done with the web UI. Currently I have my lights and a Bluetooth Xiaomi Mijia temperature sensor hooked up to Home Assistant. I also have my LED Matrix sign controller software available in Home Assistant to keep it all in one place. That Matrix sign is in deserving of its own post, but you can see it in the image below. I am also utilising the Home Assistant API too get data from the temperature sensor and display it on my Matrix display.

I have moved house 4 times since last posting images, so here is a current one with the lights in action.

---

IBM Model F XT

27/09/2024

A little bit ago I acquired an IBM Model F XT in quite good condition. It has no damage internal or external, both feet and handles are present. I just had to replace the feet with some of my own cork.

I also had to create an adapter to convert the PS/2 XT format to something a modern system can understand. This cannot be done with most active or passive converters. So I made one using a Pro Micro microcontroller running Soarer's Converter firmware and a female PS/2 connector.

I use this as my main keyboard for everyday tasks and games. As this keyboard is not made using a matrix there is no key rollover issues, so unlike the Model M, it is great for gaming (in my opinion). The layout is a bit hard to get used to though.

---

Inside a Powerpal

7/09/2024

I have a spare Powerpal sitting around. Powerpal is a simple energy monitoring device you install on your meter box, it works by counting the flashing rate of an LED. Simple and effective.

I thought it'd be cool to see what is inside.

This casing is welded shut and has non-replacable batteries. So opening it is destructive to the casing, but it still slides back together.





The device is powered by 2 double A batteries that have spot welded metal strips on them, and are glued together. These are easy to replace for someone comfortable with a soldering iron. I question why they didn't make them user replacable. My two guesses: they want you to buy more (they're over $100 each) and/or reduce tech support calls when the units batteries are replaced, as it seems to not have any non-volatile memory for user configurations.

The SOC is a nRF52840 with a 64MHz Cortex M4, 1MB flash and 256KB RAM, BLE and a ton of other peripherals that are not used. This seems very powerful for what it is.

I noticed the soldering of the sensor was very poor from the factory, so I fixed it while I was inside.





I'm integrating this into Home Assistant as the data is free to access from any bluetooth device.

---

Service monitoring on Googles Cloud Platform

21/08/2024

I had a reason to run an instance of uptime-kuma, which is a service monitor, so I put together a little guide to setting this up on Googles Cloud Platform free tier.

---

Program/script launching keypad

1/08/2022

I have various cameras around inside/outside my house. Originally to view them I was displaying them on a 10 second rotation, however this gave me no control and often i'd find myself looking at something happening on a camera only for it to switch.

I have a keypad and arduino laying around so I thought it'd be neat to add in controls.
I have 5 cameras in this view (I have more inside my house in areas I don't want to always look at), so the keypad buttons 1-A will swap whichever camera is "minimised" to the main view. I use motion to manage and record my cameras, the minimised cameras are substreams, so use less bandwidth than the main view. Each camera is also capable of IR mode and switch when it gets dark. I also have IR floodlights on the outside of the house for bright night viewing (the cameras have IR LEDs as well, however outside it is nice to have more).

To listen via serial and actually execute programs/scripts, I wrote a C program. It connects over serial, receives the key pressed and takes action according to what is assigned to each key. The assignments are controlled via a .ini file, the key value can either be the word "cam" which means a camera command, or a program name/script that is to be executed. The program then forks off and exectutes whatever you desire. Camera commands are different - I use wServer as a very simple websocket server that accepts connections from a websocket client (the viewer), the server then sends the client whatever key I press if it is assigned as a camera command.
To view/control the cameras and receive commands I use a basic html/javascript pairing.


I also recently made a laptop vesa mount tray which I have my laptop sitting on now to save desk space. Commercial laptop tray offerings were all too thin to support my girthy x200 and ultrabase. Also I'm using my x200 tablet temporarily while I figure out issues with my plain x200's.

---

Serial console via rpi zero

14/6/2022

The other day I was doing some networking changes on a server upstairs. Without thinking of the consequences I took the ethernet interface down with the intention of bringing it back up - however I was connected via ssh on that interface so I lost connection. I had to do a shameful walk upstairs to recover.

I have a spare rpi zero and a USB to rs232 cable and thought it'd be neat to get a recovery console using these (accessible over ssh to the pi). However I wasn't sure about the voltages and if i'd need extra hardware to deal with this, but while researching I found it can be done via the data USB port, called "USB Gadget Serial".

Setting this up was easy - connect the rpi zero to the server via USB (I also am powering the pi from a UPS), have it run an ssh server and do some simple config changes:
- on the pi add dtoverlay=dwc2 to /boot/config.txt
- on the pi add modprobe g_serial above exit 0 in /etc/rc.local
- on the server enable getty on ttyACM0 systemctl enable --now serial-getty@ttyACM0.service

Now you can open a connection to the server using screen /dev/ttyGS0, even when the networking is broken.

---

IBM Model M restoration

17/2/2022

I first purchased an IBM Model M a few months ago, recently I bought another one from ebay. The condition seemed ok visually - some missing key caps and a scratch or two. It is a 1390120 from 1986:



However when I got it, the keyboard felt like typing on a sponge, it obviously needed to have a bolt modification performed on it. This modification involves dismantling the keyboard and replacing plastic rivets holding the membrane together with bolts.

to perform this modification you need a few tools and bolts. Places on the internet will claim you need ~80 M2 2mm machine screws/bolts and ~80 matching M2 nuts and matching washers. This isn't true. I used maybe 20 bolts and nuts, and no washers (even though I ordered them), there just isn't room to put them on.

Another tool you will need is a 5.5mm driver. This is quite a unique driver size, however I happened to have one. Unfortunately though it was too fat to fit into the recessed holes the case bolts are fitted into. So I ended up grinding my 5.5mm driver bit down on a bench grinder.



With the case removed, the damage and dirt can be inspected:





It took me awhile to notice (and for it to snap in half) for me to notice the barrel plate had cracks, and was prone to snapping. (see the arrows) These cracks extend along the whole length of the plate:



I found epoxy resin to be incredibly good for fixing this problem, you can see i glued it back together and filled in the crack on both sides:



I proceeded to clean all the keys, caps, plate, case, sand away the rust etc - that isn't interesting.

Pictured below the back of the metal plate that holds the key membrane sandwich. The Model M has a major design flaw in that the rivets (the black circles below) used to hold this sandwich together is made of plastic that over time degrades and breaks. It only takes a handful of these to break for the membrane to no longer be held together tight enough for keys to register and feel correct, this is why we replace them with bolts. Mine doesn't look that bad, but it felt bad and did not register keys properly.



I cut off the rest of the rivets using a small craft knife. This let the membrane sandwich come apart into 3 pieces: the barrel plate, a cloth protector and the back metal plate. My model M seems unique as only a few others online have their membrane physically glued to the backplate. It is usually 5 pieces. The posts that were left over by the rivets needed to be cut flat, I did that using nail clippers.



The next step is to drill holes through the barrel plate where the old rivets were located. To do this you use a 1/16" drill bit. To help with the drilling I first marked each post I would have to drill through using some nail polish:



Then, using a soldering iron I don't mind destroying the tip on, I slightly melted through the top of the posts as a guide for the drill bit:



Now I started drilling. This was scary because I thought accuracy was important. It is not, I was sloppy and inaccurate but only a couple holes didn't line up in the end, and like I said, you do not need all rivets replaced. It also makes a mess:





I then reassembled the membrane sandwich, installed the screws and nuts, put it all back together and I had a finished Model M.

---

Userspace driver for Microsoft serial mouse

9/1/2022

To learn the basics of serial programming in UNIX I created a simple userspace driver for a Micrsoft serial mouse.

I use uinput to emulate an input device.

/*
 * Microsoft mouse
 */

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <fcntl.h>
#include <errno.h>
#include <termios.h>
#include <ctype.h>
#include <stdint.h>
#include <linux/uinput.h>

#define DOWN 		1
#define UP 		0

#define PORT 		"/dev/ttyUSB0"
#define BAUD 		B1200
#define LBIT 		1 << 5
#define RBIT		1 << 4
#define SYNCBIT 	1 << 7
#define XBYTE0MASK	0x03
#define XBYTE1MASK 	0x7F
#define YBYTE0MASK	0x0C
#define YBYTE2MASK 	0x7F

void process(void);
void emit(int fd, int type, int code, int val);

uint8_t packet[3];

struct 
{
	uint8_t left;
	uint8_t right;
} mouse;

struct uinput_setup usetup;
int ud;

int
main(void)
{
	int fd, flags, onbyte;
	ssize_t byte;
	struct termios options;
	fd = open(PORT, O_RDWR | O_NOCTTY);
	if (fd == -1)
	{
		perror("cannot open serial device");
		exit(EXIT_FAILURE);
	}
	flags = fcntl(fd, F_GETFL);			// get current flags
	fcntl(fd, F_SETFL, flags & ~O_NONBLOCK);	// mask out O_NONBLOCK = blocking

	// get port options and set flags
	tcgetattr(fd, &options);
	cfsetispeed(&options, BAUD);
	cfsetospeed(&options, BAUD);
	options.c_cflag |= (CLOCAL | CREAD);	// local line, enable read
	// set character size
	options.c_cflag &= ~CSIZE;		// mask out the character size bits
	options.c_cflag |= CS7;			// 7 bits
	// no parity
	options.c_cflag &= ~PARENB;		// mask out parity bit
	options.c_cflag &= ~CSTOPB;		// mask out stop bit (unset = 1 bit)
	// raw input
	options.c_lflag &= ~(ICANON | ECHO | ECHOE | ISIG);	
	options.c_iflag &= ~INPCK; 		// mask out parity checking
	options.c_oflag &= ~OPOST;		// mask out postprocessing - raw output

	options.c_cc[VMIN] = 1;			// minimum of 1 byte read
	options.c_cc[VTIME] = 0;		// wait indefinitely
	// apply options
	tcsetattr(fd, TCSANOW, &options);

	// setup uevent device

	ud = open("/dev/uinput", O_WRONLY | O_NONBLOCK);
	if (ud == -1)
	{
		perror("cannot open /dev/uinput");
		exit(EXIT_FAILURE);
	}


	// enable mouse left/right and relative events
	ioctl(ud, UI_SET_EVBIT, EV_KEY);
	ioctl(ud, UI_SET_KEYBIT, BTN_LEFT);
	ioctl(ud, UI_SET_KEYBIT, BTN_RIGHT);

	ioctl(ud, UI_SET_EVBIT, EV_REL);
	ioctl(ud, UI_SET_RELBIT, REL_X);
	ioctl(ud, UI_SET_RELBIT, REL_Y);

	memset(&usetup, 0, sizeof(usetup));
	usetup.id.bustype = BUS_USB;
	usetup.id.vendor = 0x1234; 
	usetup.id.product = 0x5678;
	strcpy(usetup.name, "Userspace Microsoft serial mouse");

	ioctl(ud, UI_DEV_SETUP, &usetup);
	ioctl(ud, UI_DEV_CREATE);

	mouse.left = 0x00;
	mouse.right = 0x00;
	onbyte = 0;
	byte = 0x00;

	while(1)
	{
		read(fd, &byte, 1);
		if (byte == 0xcd) 		// some kind of init?
			continue;
		// is byte first of packet?
		if (byte & SYNCBIT && onbyte == 0)
		{
			packet[0] = byte;
			onbyte = 1;
		}
		else if (onbyte == 1)
		{
			packet[1] = byte;
			onbyte = 2;
		}
		else if (onbyte == 2)
		{
			packet[2] = byte;
			onbyte = 0;
			process();
		}
		fflush(stdout);
	}
	close(fd);
	ioctl(ud, UI_DEV_DESTROY);
	close(ud);
	return 1;
}

void
process(void)
{
	int8_t relx, rely;
	printf("0x%02x\t", packet[0]);
	printf("0x%02x\t", packet[1]);
	printf("0x%02x\t\n\n", packet[2]);
	if (packet[0] & LBIT && !mouse.left)
	{
		mouse.left = DOWN;
		puts("left down");
		emit(ud, EV_KEY, BTN_LEFT, 1);
		emit(ud, EV_SYN, SYN_REPORT, 0);
	}
	else if (mouse.left && !(packet[0] & LBIT))
	{
		mouse.left = UP;
		puts("left up");
		emit(ud, EV_KEY, BTN_LEFT, 0);
		emit(ud, EV_SYN, SYN_REPORT, 0);
	}
	if (packet[0] & RBIT && !mouse.right)
	{
		mouse.right = DOWN;
		puts("right down");
		emit(ud, EV_KEY, BTN_RIGHT, 1);
		emit(ud, EV_SYN, SYN_REPORT, 0);
	}
	else if (mouse.right && !(packet[0] & RBIT))
	{
		mouse.right = UP;
		emit(ud, EV_KEY, BTN_RIGHT, 0);
		emit(ud, EV_SYN, SYN_REPORT, 0);
	}

	relx |= ((packet[0] & XBYTE0MASK)<<6) | (packet[1] & XBYTE1MASK);
	rely |= ((packet[0] & YBYTE0MASK)<<4) | (packet[2] & YBYTE2MASK);
	emit(ud, EV_REL, REL_X, relx);
	emit(ud, EV_REL, REL_Y, rely);
	emit(ud, EV_SYN, SYN_REPORT, 0);


}

void
emit(int fd, int type, int code, int val)
{
	struct input_event ie;
	ie.type = type;
	ie.code = code;
	ie.value = val;
	/* timestamp values below are ignored */
	ie.time.tv_sec = 0;
	ie.time.tv_usec = 0;

	write(fd, &ie, sizeof(ie));
}

---

Various updates

1/1/2022

I haven't made a post in over 6 months because my website generator has had a bug with pagination and I haven't bothered to do anything about it.

I've become very interested in the hobby of soldering and have spent a lot of time practicing. I acquired a Pinecil iron, a holder, hotair station, DC power supply, cheap microscope, extra ts100 soldering tips, flux/solder/alcohol dispenser/good precision tweezers etc. I've been taking things apart, removing components and soldering them back on, mostly laptop motherboards. I've repaired a few broken items of mine so far (a wire breaking in a servo motor, ch341a voltage mod, a "smart" light bulb for a family member with an internal trace ripped), so these skills have come into use.

I purchased another Thinkpad x200, this time a Japanese import with a JIS keyboard, I think it's pretty neat.

I purchased two IBM Model M keyboards from ebay (1391401 from 1988 and a 1390120 from 1986). One keyboard was in great mechanical and electrical condition, I just disassembled the shell and cleaned each key and the shell, it works perfectly and I am using it now. The other, the 1390120 with a square medal badge from 1986 felt like typing on a sponge. So I have completely cleaned the shell/keys and am waiting on hardware to arrive to bolt mod it.

I'm going to start putting out more posts.

Here is a couple of photos of my soldering/desk setup and the x200 and keyboards.

---