If you wan’t em…
Category Archives: Hardware
Data paranoia, Drobos and Synology NAS
I am in the slow process of upgrading five WD 6TB hard drives in my Synology DS1522+ NAS to Seagate 8TB drives. While folks I asked overwhelming say the larger drives (16TB, 20TB, etc.) “have not had any issues,” I am old school and do not want to make that large of a storage jump just yet. For those as data paranoid as I am, some tips:
- Do a low-level format (or “secure erase”) on each new drive first. This will write to every sector, and can catch issues. I have only caught one (or maybe two) drive with issues of the years, but the time spent is worth it to me. I’d rather catch a problem before I install and send them back for a replacement, rather than having an issue show up much later (possibly when the drive is out of warranty).
- Until someone can say that a “20TB” drive is as reliable as a smaller drive, upgrage to the next size up that gives you enough storage. The less dense the data, the safer it “should” be. Also, if a 6TB drive fails, the rebuild time to replace it will be significantly faster than replacing a 20TB drive. And, during the rebuilt time, your data is at risk. I run dual drive redundancy so during the 12 hours my NAS rebuilds, if I have a second drive fail, I am still okay… but if that happens I have no protection from a third failure. Doing 20 hours rebuilds creates a much larger window for data loss if something goes terribly wrong.
- And, of course, make sure anything important is on a backup drive (I use a standalone 10TB just to clone my most important “can’t live without” data), and have an offsite backup of that (I then have that entire drive backed up to a cloud backup service).
If my home burns down, I should at least be able to get back the 10TB of “can’t live without” data from my offsite backup.
Hardware Redundancy is Better
Sadly, Synology units are much more expensive than my Drobos were. My Drobo 5C was $300 retail. I had two that I paid maybe $250 each for. That let me have two 5-bay units for hardware redundancy. This means if a Drobo suddenly died, I still had a second unit with duplicate data (I would sync the two drives). Spending $500 for two 5-drive units was an easier investment than the $1400 it would take for me to buy two DS1522+.
Eventually I do plan to have a duplicate Synology unit. It doesn’t matter what features the device has if one morning it has died. I would have zero access to my data until the unit is replaced or repaired. Having backup hardware is what I prefer.
But I am data paranoid.
How about you? Are you unlucky, like I am, and have had drives “die suddenly” over the years? After that happens enough, the paranoia sets in. I can’t think of a time in the past decades where I didn’t have three backups of everything important ;-)
Teensy used to emulate a 6809!
Hat tip to Boisy P. for posting this link in one of the Facebook CoCo groups.
NEAT!!!!
https://microcorelabs.wordpress.com/2024/07/16/mcl6809-drop-in-motorola-6809e-emulator/
Wokwi online Arduino/ESP32 simulator
Oh, Wokwi! Where have you been all my life? Or at least where were you a few years ago when I was working on Arduino projects?
I initially started using this Sub-Etha Software blog for Arduino projects. I did crazy things like porting a Color BASIC program to Arduino C, and fun things like figuring out how to write Pac-Man using the Arduio TVOut library. Eventually, I merged my interests in the old Radio Shack Color Computer (CoCo) with Arduino (and ESP8266/ESP32) and experimented with a serial port sound player and WiFi modem.
So … many … wires.
At the time, I was hoping to find some kind of Arduino emulator so I could write and test code without hooking up hardware. I found nothing.
But that seems to have changed. I just learned about Wokwi which allows one to “simulate IoT projects in your browser.” In a nutshell, it’s a website that has a code editor (which appears to be Microsoft Visual Studio Code), compiler, and virtual target hardware like Arduino and ESP32 devices. It even supports some add-on hardware, like buttons, LCD displays, LEDs and more.
Here’s a project someone made that simulates an Arduino hooked to a numeric keypad and LCD display:
And you can build and run it right there!
There is a library of devices that are supported, and you can add them to your project and wire them up to the computer’s I/O pins. For example, as I write this blog post, I opened up a starter project that is an Arduino and a two-line LCD display. I then added a pushbutton to it.
I could then move the button to where I wanted it, then click on the connectors and draw wire lines between it and I/O pins on the Arduino. By hooking one side to an I/O pin, and the other to ground, I could then modify the program to read that button and, for this example, increment a counter while the button is being held done.
It’s just that easy! I had no idea!
The files can be downloaded and used on real hardware, or you can make an account and log back in to continue working on them. (It has an unusual way to log in — it sends you an e-mail and you click a link to log in, rather than having a username and password. This seems to mean I cannot log in from any system that I don’t have my e-mail account configured on, but I do see options for using a Google or Github login.)
I hope you find this as useful as I already have.
Happy coding!
Arduino, ZigBee and motion sensor help needed.
For a future project, I need to make use of remote triggers. These could be motion sensors, beam sensors, pressure mats, etc.
The ZigBee standard seems to be the way to go since I can find cheap consumer motion sensors that run on batteries. There also seems to be ZigBee repeaters, which allow giving the distance I need simply by plugging them in from place to place to create a mesh network.
XBee might be another option, if cheap motion sensors and repeaters are also available.
The goal is to have a central location be able to read the motion sensor status for many sensors, that could be spread out beyond walls hundreds of feet away.
Any pointers to where I might get started would be appreciated. Ideally I’d drive this all by a low-cost Arduino since the device will be used in an area where power might not be stable (and I wouldn’t want to corrupt the Linux file system on a Raspberry Pi).
Thanks…
When there’s not enough room for sprintf…
Updates:
- 2022-08-30 – Corrected a major bug in the Get8BitHexStringPtr() routine.
“Here we go again…”
Last week I ran out of ROM space in a work project. For each code addition, I have to do some size optimization elsewhere in the program. Some things I tried actually made the program larger. For example, we have some status bits that get set in two different structures. The code will do it like this:
shortStatus.faults |= FAULT_BIT; longStatus.faults |= FAULT_BIT;
We have code like that in dozens of places. One of the things I had done earlier was to change that in to a function. This was primarily so I could have common code set fault bits (since each of the four different boards I work with had a different name for its status structures). It was also to reduce the amount of lines in the code and make what they were doing more clear (“clean code”).
void setFault (uint8_t faultBit) { shortStatus.faults |= faultBit; longStatus.faults |= faultBit; }
During a round of optimizing last week, I noticed that the overhead of calling that function was larger than just doing it manually. I could switch back and save a few bytes every time it was used, but since I still wanted to maintain “clean code”, I decided to make a macro instead of the function. Now I can still do:
setFault (FAULT_BIT);
…but under the hood it’s really doing a macro instead:
#define setFault(faultBit) { shortStatus.faults |= faultbit; longStatus.faults |= faultBit; }
Now I get what I wanted (a “function”) but retain the code size savings of in-lining each instance.
I also thought that doing something like this might be smaller:
shortStatus.faults |= FAULT_BIT; longStatus.faults = shortStatus.faults;
…but from looking at the PIC24 assembly code, that’s much larger. I did end up using it in large blocks of code that conditionally decided which fault bit to set, and then I sync the long status at the end. As long as the overhead of “this = that” is less than the overhead of multiple inline instructions it was worth doing.
And keep in mind, this is because I am 100% out of ROM. Saving 4 bytes here, and 20 bytes there means the difference between being able to build or not.
Formatting Output
One of the reasons for the “code bloat” was adding support for an LCD display. The panel, an LCD2004, hooks up to I2C vie a PCF8574 I2C I/O chip. I wrote just the routines needed for the minimal functionality required: Initialize, Clear Screen, Position Cursor, and Write String.
The full libraries (there are many) for Arduino are so large by comparison, so often it makes more sense to spend the time to “roll your own” than port what someone else has already done. (This also means I do not have to worry about any licensing restrictions for using open source code.)
I created a simple function like:
LCDWriteDataString (0, 0, "This is my message.");
The two numbers are the X and Y (or Column and Row) of where to display the text on the 20×4 LCD screen.
But, I was quickly reminded that the PIC architecture doesn’t support passing constant string data due to “reasons”. (Harvard architecture, for those who know.)
To make it work, you had to do something like:
const char *msg = "This is my message"; LCDWriteDataString (0, 0, msg);
…or…
chr buffer[19]; memcpy (buffer, "This is my message"); LCDWriteDataString (0, 0, msg);
…or, using the CCS compiler tools, add this to make the compiler take care of it for you:
#device PASS_STRINGS=IN_RAM
Initially I did that so I could get on with the task at had, but as I ran out of ROM space, I revisited this to see which approach was smaller.
From looking at the assembly generated by the CCS compiler, I could tell that “PASS_STRINGS=IN_RAM” generated quite a bit of extra code. Passing in a constant string pointer was much smaller.
So that’s what I did. And development continued…
Then I ran out of ROM yet again. Since I had some strings that needed formatted output, I was using sprintf(). I knew that sprintf() was large, so I thought I could create my own that only did what I needed:
char buffer[21]; sprintf (buffer, "CF:%02x C:%02x T:%02x V:%02x", faults, current, temp, volts); LCDWriteDataString (0, 0, buffer); char buffer[21]; sprintf (buffer, "Fwd: %u", watts); LCDWriteDataString (0, 1, buffer);
In my particular example, all I was doing is printing out an 8-bit value as HEX, and printing out a 16-bit value as a decimal number. I did not need any of the other baggage sprintf() was bringing when I started using it.
I came out with these quick and dirty routines:
char GetHexDigit(uint8_t nibble) { char hexChar; nibble = (nibble & 0x0f); if (nibble <= 9) { hexChar = '0'; } else { hexChar = 'A'-10; } return (hexChar + nibble); } char *Get8BitHexStringPtr (uint8_t value) { static char hexString[3]; hexString[0] = GetHexDigit(value >> 4); hexString[1] = GetHexDigit(value & 0x0f); hexString[2] = '\0'; // NIL terminate return hexString; }
The above routine maintains a static character buffer of 3 bytes. Two for the HEX digits, and the third for a NIL terminator (0). I chose to do it this way rather than having the user pass in a buffer pointer since the more parameters you pass, the larger the function call gets. The downside is those 3 bytes of variable storage are reserved forever, so if I was also out of RAM, I might rethink this approach.
I can now use it like this:
const char *msgC = " C:"; // used by strcat() const char *msgT = " T:"; // used by strcat() const char *msgV = " V:"; // used by strcat() char buffer[20]; strcpy (buffer, "CF:"); // allows constants strcat (buffer, Get8BitHexStringPtr(faults)); strcat (buffer, msgC); strcat (buffer, Get8BitHexStringPtr(current)); strcat (buffer, msgT); strcat (buffer, Get8BitHexStringPtr(temp)); strcat (buffer, msgV); strcat (buffer, Get8BitHexStringPtr(volts)); LCDWriteDataString (0, 1, buffer);
If you are wondering why I do a strcpy() with a constant string, then use const pointers for strcat(), that is due to a limitation of the compiler I am using. Their implementation of strcpy() specifically supports string constants. Their implementation of strcat() does NOT, requiring me to jump through more hoops to make this work.
Even with all that extra code, it still ends up being smaller than linking in sprintf().
And, for printing out a 16-bit value in decimal, I am sure there is a clever way to do that, but this is what I did:
char *Get16BitDecStringPtr (uint16_t value) { static char decString[6]; uint16_t temp = 10000; int pos = 0; memset (decString, '0', sizeof(decString)); while (value > 0) { while (value >= temp) { decString[pos]++; value = value - temp; } pos++; temp = temp / 10; } decString[5] = '\0'; // NIL terminate return decString; }
Since I know the value is limited to what 16-bits can old, I know the max value possible is 65535.
I initialize my five-digit string with “00000”. I start with a temporary value of 10000. If the users value is larger than that, I decrement it by that amount and increase the first digit in the string (so “0” goes to “1”). I repeat until the user value has been decremented to be less than 10000.
Then I divide that temporary value by 10, so 10000 becomes 1000. I move my position to the next character in the output string and the process repeats.
Eventually I’ve subtracted all the 10000s, 1000s, 100s, 10s and 1s that I can, leaving me with a string of five digits (“00000” to “65535”).
I am sure there is a better way, and I am open to it if it generates SMALLER code. :)
And that’s my tale of today… I needed some extra ROM space, so I got rid of sprintf() and rolled my own routines for the two specific types of output I needed.
But this is barely scratching the surface of the things I’ve been doing this week to save a few bytes here or there. I’d like to revisit this subject in the future.
Until next time…
Drobo “saves the day”?
I hate it when this happens… It looks like a 4TB drive in my 5-bay Drobo has gone out. Drobo cannot detect it. I have dual-drive redundancy enabled, so two drives can fail and I’d still have my data… Fortunately.
Hopefully, I won’t have two drives fail between now and the time my replacement drive arrives. :)
On the plus, drive prices have dropped since I bought these drives in 2019. I’ll begin the process of upgrading drives to 6TB models over coming months, money permitting.
16-bits don’t always add up.
Consider this simple program:
#include <stdio.h>
#include <stdlib.h>
#include <stdint.h>
int main(int argc, char **argv)
{
uint16_t val1;
uint16_t val2;
uint32_t result;
val1 = 40000;
val2 = 50000;
result = val1 + val2;
printf ("%u + %u = %u\n", val1, val2, result);
return EXIT_SUCCESS;
}
What will it print?
On my Windows PC, I see the following:
40000 + 50000 = 90000
…but if I convert the printf() and run the same code on an Arduino:
void setup() {
// put your setup code here, to run once:
Serial.begin(9600);
uint16_t val1;
uint16_t val2;
uint32_t result;
val1 = 40000;
val2 = 50000;
result = val1 + val2;
//printf ("%u + %u = %u\n", val1, val2, result);
Serial.print(val1);
Serial.print(" + ");
Serial.print(val2);
Serial.print(" = ");
Serial.println(result);
}
void loop() {
// put your main code here, to run repeatedly:
}
This gives me:
40000 + 50000 = 24464
…and this was the source of a bug I introduced and fixed at my day job recently.
Tha’s wrong, int’it?
I tend to write alot of code using the GCC compiler since I can work out and test the logic much quicker than repeatedly building and uploading to our target hardware. Because of that, I had “fully working” code that was incorrect for our 16-bit PIC24 processor.
In this case, the addition of “val1 + val2” is being done using native integer types. On the PC, those are 32-bit values. On the PIC24 (and Arduino, shown above), they are 16-bit values.
A 16-bit value can represent 65536 values in the range of 0-65535. If you were to have a value of 65535 and add 1 to it, on a 16-bit variable it would roll over and the result would be 0. In my example, 40000 + 50000 was rolling over 65535 and producing 24464 (which is 90000 – 65536).
You can see this happen using the Windows calculator. By default, it uses DWORD (double word – 32-bit) values. You can do the addition just fine:
You see that 40,000 + 50,000 results in 90,000, which is 0x15F90 in hex. That 0x1xxxx at the start is the rollover. If you switch the calculator in to WORD mode you see it gets truncated and the 0x1xxxx at the start goes away, leaving the 16-bit result:
Can we fix it?
The solution is very simple. In C, any time there is addition which might result in a value larger than the native int type (if you know it), you simply cast the two values being added to a larger data type, such as a 32-bit uint32_t:
void setup() {
// put your setup code here, to run once:
Serial.begin(9600);
uint16_t val1;
uint16_t val2;
uint32_t result;
val1 = 40000;
val2 = 50000;
// Without casting (native int types):
result = val1 + val2;
//printf ("%u + %u = %u\n", val1, val2, result);
Serial.print(val1);
Serial.print(" + ");
Serial.print(val2);
Serial.print(" = ");
Serial.println(result);
// Wish casting:
result = (uint32_t)val1 + (uint32_t)val2;
Serial.print(val1);
Serial.print(" + ");
Serial.print(val2);
Serial.print(" = ");
Serial.println(result);
}
void loop() {
// put your main code here, to run repeatedly:
}
Above, I added a second block of code that does the same add, but casting each of the val1 and val2 variables to 32-bit values. This ensures they will not roll over since even the max values of 65535 + 65535 will fit in a 32-bit variable.
The result:
40000 + 50000 = 24464 40000 + 50000 = 90000
Since I know adding any two 16-bit values can be larger than what a 16-bit value can hold (i.e., “1 + 1” is fine, as is “65000 + 535”, but larger values present a rollover problem), it is good practice to just always cast upwards. That way, the code works as intended, whether the native int of the compiler is 16-bits or 32-bits.
As my introduction of this bug “yet again” shows, it is a hard habit to get in to.
Until next time…
PiZ SuperCap safe shutdown for Raspberry Pi Zero
Last year, I learned of a new Kickstarter that solved a significant problem with using a Raspberry Pi for embedded “turn key” projects. The Raspberry Pi is a disk-based Linux system, using a microSD card in place of a hard drive. If you kill power to a Raspberry Pi without safely shutting down Linux first, file system corrupt can occur. I have seen this dozens of times over the years on my devices. When it happens, I just reformat the memory card and re-image it and continue.
But now we don’t have to — at least not if we are using the Pi Zero.
Abhinav Shukla created the PiZ SuperCap and launched it as a Kickstarter in 2021. The device is a small circuit board with a capacitor. It connects to a Pi Zero via the I/O header. Instead of plugging the USB power cable directly in to the Pi, you plug it in to the PiZ SuperCap board. This charges the capacitor then begins powering the Pi.
If power is disconnected, the capacitor has enough power to run the Pi Zero for a short amount of time (about 15 seconds). It also toggles a GPIO pin to indicate that power has been lost. By running a simple Python script on the Pi, you can now detect when power has been lost and, if it is not resumed in a set amount of seconds, safely shut down the Pi Zero by issuing a “shutdown” command.
I backed ten of these units, and I am glad I did. They work great! Now I can use a Pi Zero for any type of embedded project I want and just kill power when I want to shut down.
How it works
Here are some things to be aware of:
- When you first apply power, the Pi Zero will not immediately power up like you are used to. It must first charge the capacitor. The Pi Zero won’t actually start up until about 8 seconds after you apply power.
- When you turn off power, if no shutdown script is installed, the Pi won’t turn off until the capacity runs out. On my Pi Zero W (first version), I timed it at 75 seconds when just sitting at a text console login prompt. If you are running a graphical desktop or any programs using the CPU, it won’t last that long.
- The PiZ SuperCap will toggle GPIO pin 4 to indicate a power loss. To enable safe shutdown, you need some form of program or script that will monitor GPIO pin 4 and shut the system down when power goes away. The sample code provided uses a 9 second delay before deciding to power off, making the unit act like a mini UPS rather than shutting down on any temporary power blip. Clever.
Shutdown script
The designer provides some sample Python code on the project page, but here is a shorter one I came up with. I am not a Python programmer, so I have no idea if my technique is good. I just wanted it to block until it sees a power loss, and either shut down (after a delay) or continue:
supercap_shutdown.py
#!/usr/bin/python3
import RPi.GPIO as GPIO
import time
import os
GPIO.setmode(GPIO.BCM)
GPIO.setup(4, GPIO.IN)
while True:
# Wait for power to drop.
print ("Waiting for power loss...");
GPIO.wait_for_edge(4, GPIO.FALLING)
print ("Power loss")
# Give user 9 seconds to restore power.
print ("Waiting to see if power is restored...")
time.sleep(9)
if GPIO.input(4) == 0:
print ("Power not restored.")
os.system("sudo shutdown -h now")
break;
else:
print ("Power restored.")
The above script has print statements in it so you can run it from the console to verify it is working. Those can be removed (or commented out) once you are sure it works.
You will need to do a “chmod +x supercap_shutdown.py” to make it executable.
If you want to make it run on startup, edit the /etc/rc.local file:
sudo pico /etc/rc.local
…and add this line at the end before “exit 0”:
# Pi Z SuperCap monitor script: python /home/pi/supercap shutdown.py
You can then restart the system (“sudo restart now”) and when it reboots, the script should be running. Disconnect power and after 9 seconds (or whatever time you modify the script to use) it will issue a safe “shutdown” command.
3-D printed enclosure
I have created a very simple 3-D printer enclosure that holds a Raspberry Pi Zero and the PiZ SuperCap. Let me know if this is something you might want.
Support the designer
If one of these might be of interest to you, consider backing the project at:
Arduino Serial output C macros
Here is a quickie.
In Arduino, instead of being able to use things like printf() and puchar(), console output is done by using the Serial library routines. It provides functions such as:
Serial.print(); Serial.println(); Serial.write();
These do not handle any character formatting like printf() does, but they can print strings, characters or numeric values in different formats. Where you might do something like:
int answer = 42;
printf("The answer is %d\r\n", answer);
…the Arduino version would need to be:
int answer = 42; Serial.print("This answer is "); Serial.print(answer); Serial.println();
To handle printf-style formatting, you can us sprintf() to write the formatted string to a buffer, then use Serial.print() to output that. I found this blog post describing it.
I recently began porting my Arduino Telnet routine over to standard C to use on some PIC24 hardware I have at work. I decided I should revisit my Telnet code and try to make it portable, so the code could be built for Arduino or standard C. This would mean abstracting all console output, since printf() is not used on the Arduino.
I quickly came up with these Arduino-named macros I could use on C:
#include <stdio.h>
#include <stdlib.h>
#define SERIAL_PRINT(s) printf(s)
#define SERIAL_PRINTLN(s) printf(s"\r\n")
#define SERIAL_WRITE(c) putchar(c)
int main()
{
SERIAL_PRINT("1. This is a line");
SERIAL_PRINTLN();
SERIAL_PRINTLN();
SERIAL_PRINTLN("2. This is a second line.");
SERIAL_PRINT("3. This is a character:");
SERIAL_WRITE('x');
SERIAL_PRINTLN();
SERIAL_PRINTLN("done.");
return EXIT_SUCCESS;
}
Ignoring the Serial.begin() setup code that Arduino requires, this would let me replace console output in the program with these macros. For C, it would use the macros as defined above. For Arduino, it would be something like…
#define SERIAL_PRINT(s) Serial.print(s) #define SERIAL_PRINTLN(s) Serial.println(s) #define SERIAL_WRITE(c) Serial.write(c)
By using output macros like that, my code would still look familiar to Arduino folks, but build on a standard C environment (for the most part).
This isn’t the most efficient way to do it, since Arduino code like this…
Serial.print("["); Serial.print(val); Serial.println("]");
…would be one printf() in C:
printf ("[%d]\n", val);
But, if I wanted to keep code portable, C can certainly do three separate printf()s to do the same output as Arduino, so we code for the lowest level output.
One thing I don’t do, yet, is handle porting things like:
Serial.print(val, HEX);
On Arduino, that outputs the val variable in HEX. I’m not quite sure how I’d make a portable macro for that, unless I did something like:
#define SERIAL_PRINT_HEX(v) Serial.print(v, HEX) #define SERIAL_PRINT_HEX(v) printf("%x, v)
That would let me do:
SERIAL_PRINT("["); SERIAL_PRINT_HEX(val); SERIAL_PRINTLN("]");
I expect to add more macros as-needed when I port code over. This may be less efficient, but it’s easier to make Arduino-style console output code work on C than the other way around.
Cheers…