In a recent article, I discussed methods to copy variables (or structure elements) in to a packed buffer, as well as parse buffer bytes back in to variables.
Doing it manually was clunky. Using my GetPutData routines made it easier to be clunky.
But today, we’ll make it super simple and far less clunky.
Defining the message bytes
Let’s start with a simple message. It will be represented by a C structure that we’d use in the program. For example, here are the elements that might be part of a “Set Date and Time” message:
typedef struct
{
uint16_t year;
uint8_t month;
uint8_t day;
uint8_t hour;
uint8_t minute;
uint8_t second;
uint8_t timeZone;
bool isDST;
} SetDateTimeMessageStruct;The packed message would be nine bytes and look like this:
[ 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 ] \ / | | | | | | | year mon day hr min sec tz dst
To automate this process, a lookup table is created that represents the size of each element and the offset to where it is located within the structure.
offsetof()
Inside stddef.h is a macro that is used to calculate where in the structure’s memory a particular element exists. It is a relative offset from wherever the structure variable starts in memory. Here is an example showing offsetof():
#include <stdio.h>
#include <stdlib.h> // for EXIT_SUCCESS
#include <stdint.h>
#include <stdbool.h>
#include <stddef.h> // for offsetof()
typedef struct
{
uint8_t a;
uint16_t b;
bool c;
uint32_t d;
float e;
double f;
} MyStruct;
int main()
{
MyStruct test;
printf ("sizeof(test) = %ld\n", sizeof(test));
printf ("a is %lu bytes at offset %lu\n",
sizeof(test.a), offsetof(MyStruct, a));
printf ("b is %lu bytes at offset %lu\n",
sizeof(test.b), offsetof(MyStruct, b));
printf ("c is %lu bytes at offset %lu\n",
sizeof(test.c), offsetof(MyStruct, c));
printf ("d is %lu bytes at offset %lu\n",
sizeof(test.d), offsetof(MyStruct, d));
printf ("e is %lu bytes at offset %lu\n",
sizeof(test.e), offsetof(MyStruct, e));
printf ("f is %lu bytes at offset %lu\n",
sizeof(test.f), offsetof(MyStruct, f));
return EXIT_SUCCESS;
}Try it: https://onlinegdb.com/GDvhi9TLq
It produces the following output:
sizeof(test) = 24 a is 1 bytes at offset 0 b is 2 bytes at offset 2 c is 1 bytes at offset 4 d is 4 bytes at offset 8 e is 4 bytes at offset 12 f is 8 bytes at offset 16
This is an example of how elements in a structure may be padded to ensure each one starts on an even-byte in memory. ‘a’ is one byte starting at offset 0, but ‘b’ doesn’t start at 1. As a 16-bit value, it has to skip a byte and start at offset 2. There are more skipped bytes after ‘c’ at offset 4. Since the next value is a 32-bit value, it skips three bytes.
If we just wrote the structure memory out (in this architecture), it would look like this:
[ 0| 1| 2| 3| 4| 5| 6| 7| 8| 9|10|11|12|13|14|15|16|17|18|19|20|21|22|23] | | | | | | a b--b c d--d--d--d e--e--e--e f--f--f--f--f--f--f--f
That would waste 4 bytes in that 24 byte message. Plus, if the message were parsed by a system that had different padding/alignment requirements, they wouldn’t be able to read the buffer back in to their structure variable and get the correct data.
Using offsetof() to determine where in a structure an element exists, a lookup table could be created. A C function could be written to use that table to know where to copy the data. Here’s a rough concept:
typedef struct
{
size_t size; // size of structure element
size_t offset; // offset from start of structure to that element
} ElementOffsetStruct;
MyStruct test;
ElementOffsetStruct testTable[] =
{
{ sizeof(test.a), offsetof(MyStruct, a) },
{ sizeof(test.b), offsetof(MyStruct, b) },
{ sizeof(test.c), offsetof(MyStruct, c) },
{ sizeof(test.d), offsetof(MyStruct, d) },
{ sizeof(test.e), offsetof(MyStruct, e) },
{ sizeof(test.f), offsetof(MyStruct, f) },
{ 0, 0 } // end of table
};
for (int idx=0; testTable[idx].size != 0; idx++)
{
printf ("? is %u bytes at offset %u\n",
testTable[idx].size, testTable[idx].offset);
}Knowing where items are in the structure allows the specific element bytes to be copied to a buffer. Something like this:
uint8_t buffer[80];
unsigned int offset;
offset = 0;
for (int idx=0; testTable[idx].size != 0; idx++)
{
printf ("Copy %lu bytes at structure offset %lu to buffer offset %u\n",
testTable[idx].size, testTable[idx].offset, offset);
memcpy (buffer + offset,
(uint8_t*)&test + testTable[idx].offset,
testTable[idx].size);
offset = offset + testTable[idx].size;
}“And it’s just that easy.” Running the following test program dumps the memory contents of the C structure “test” and then writes them out to a buffer using this code and then dumps the memory contents of the buffer.
#include <stdio.h>
#include <stdlib.h> // for EXIT_SUCCESS
#include <stdint.h>
#include <stdbool.h>
#include <stddef.h> // for offsetof()
#include <string.h> // for memcpy()
typedef struct
{
uint8_t a;
uint16_t b;
bool c;
uint32_t d;
float e;
double f;
} MyStruct;
typedef struct
{
size_t size; // size of structure element
size_t offset; // offset from start of structure to that element
} ElementOffsetStruct;
int main()
{
MyStruct test = { 0x11, 0x2222, true, 0x33333333, 42.42, 42.42 };
ElementOffsetStruct testTable[] =
{
{ sizeof(test.a), offsetof(MyStruct, a) },
{ sizeof(test.b), offsetof(MyStruct, b) },
{ sizeof(test.c), offsetof(MyStruct, c) },
{ sizeof(test.d), offsetof(MyStruct, d) },
{ sizeof(test.e), offsetof(MyStruct, e) },
{ sizeof(test.f), offsetof(MyStruct, f) },
{ 0, 0 } // end of table
};
for (int idx=0; idx<sizeof(test); idx++)
{
printf ("%02x ", ((uint8_t*)&test)[idx]);
}
printf ("\n");
uint8_t buffer[80];
unsigned int offset;
offset = 0;
for (int idx=0; testTable[idx].size != 0; idx++)
{
printf ("Copy %lu bytes at structure offset %lu to buffer offset %u\n",
testTable[idx].size, testTable[idx].offset, offset);
memcpy (buffer + offset,
(uint8_t*)&test + testTable[idx].offset,
testTable[idx].size);
offset = offset + testTable[idx].size;
}
for (int idx=0; idx<offset; idx++)
{
printf ("%02x ", buffer[idx]);
}
printf ("\n");
return EXIT_SUCCESS;
}I get the following output:
11 00 22 22 01 00 00 00 33 33 33 33 14 ae 29 42 f6 28 5c 8f c2 35 45 40 Copy 1 bytes at structure offset 0 to buffer offset 0 Copy 2 bytes at structure offset 2 to buffer offset 1 Copy 1 bytes at structure offset 4 to buffer offset 3 Copy 4 bytes at structure offset 8 to buffer offset 4 Copy 4 bytes at structure offset 12 to buffer offset 8 Copy 8 bytes at structure offset 16 to buffer offset 12 11 22 22 01 33 33 33 33 14 ae 29 42 f6 28 5c 8f c2 35 45 40
As I discovered when I figured this out, there are a few gotchas. First, the sample code here requires the structure variable to exist. This is because the macro sizeof() needs to have something to work on to determine the size:
sizeof(test.a)I asked ChatGPT about this, and it provided me a clever workaround that worked on the structure itself rather than a variable:
sizeof(((MyStruct*)0)->a)
Looking at that, I think I get how it works. First, it is taking a memory location of 0 and casting that to be a pointer to a MyStruct. No such structure exists at memory 0, but you can cast a pointer to anywhere (even 0) as any kind of pointer you want. (Actually using a pointer that point to 0 would be bad, however.)
Then, it gets the offset of the “a” element from the structure we pretend is located at memory location 0. Let’s break that apart a bit:
sizeof( )
( )->a
(MyStruct*)0
And this is enough to allow creating a table based on a structure, and not on a variable declared using that structure.
typedef struct
{
uint8_t u8;
uint32_t u32;
} SmallStruct;
ElementOffsetStruct smallTable[] =
{
{ sizeof(((SmallStruct*)0)->u8), offsetof(SmallStruct, u8) },
{ sizeof(((SmallStruct*)0)->u32), offsetof(SmallStruct, u32) },
{ 0, 0 }
};This would allow an easy way to define the structure and a table of offsets for later use. I’d probably make the table structure const since it won’t need to change.
With this framework figured out, a reverse function could be done that would copy bytes from a buffer in to the correct offset inside the structure.
unsigned int offset;
offset = 0;
for (int idx=0; testTable[idx].size != 0; idx++)
{
printf ("Copy %lu bytes at buffer offset %u to structure offset %lu\n",
testTable[idx].size, offset, testTable[idx].offset);
memcpy ((uint8_t*)&test + testTable[idx].offset,
buffer + offset,
testTable[idx].size);
offset = offset + testTable[idx].size;
}And now, by creating a structure and a table of size/offsets, any structure can be written to a buffer, or decoded from a buffer.
Too. Much. Typing.
For the final touch, macros can be made to simplify building the table.
#define ENTRY(s, v) { sizeof(((s*)0)->v), offsetof(s, v) }
#define ENTRYEND {0, 0}Now, instead of having to type all that gibberish out, the table can be simplified:
ElementOffsetStruct testTable[] =
{
ENTRY(MyStruct, a),
ENTRY(MyStruct, b),
ENTRY(MyStruct, c),
ENTRY(MyStruct, d),
ENTRY(MyStruct, e),
ENTRY(MyStruct, f),
ENTRYEND
};And this is just what I have posted to my GitHub:
https://github.com/allenhuffman/StructureToFromBuffer
You can find a short README there that explains the two functions and how to use them. It greatly simplifies the process of creating message buffers or parsing them in to C structures that can easily be used in a program.
If you try it and find it useful, please let me know.
Until then…
