nerd1951.com

Let’s face it.  Software design falls into two extremes in the majority of software development organizations. Either it is hardly done at all or it is done to the point of dimensioning returns. You know I’m talking about you, most of you anyway.

The “hardly done at all” camp usually sees little value in doing design. Our managers want to see code! So, we will do the minimum documentation to satisfy our customers, clients or the process police. We even have a name for this approach: Agile Programming. To quote a Dilbert comic, “We just start coding and complaining.”

The design to death crowd has usually been sold some expensive tool or trained in some complex process. These days it’s almost always done in UML. Use cases must be reviewed by all of the stake-holders in painfully long meetings. Then all the varieties of UML diagrams are produced and painstakingly reviewed before a line of code is written. And this is still supposed to be an iterative process.

In the dozen or so years since I abandoned Structured Analysis and Design, I’ve learned two things: Object Oriented Programming really works well and the Waterfall Method of software development does not. Agile Development is a direct response to the old Waterfall model but most “agile” development organizations neither understand nor follow the Agile Process. What I haven’t learned though is a process better than Structured Analysis and Design for getting from the user’s requirements to code.

UML has a host of problems. My biggest problem with UML is that it is mistaken for a development process. There are no well defined processes for Object Oriented design. The analysis phase is well defined with Use Cases and Interaction Diagrams. But beyond this I find most of the process gets muddled. Where is the step-wise refinement that allows you to drill down from the high level architecture to the individual classes and objects?

I don’t even think UML is good notation. It’s just not intuitive. How can any modeling language that professes to be “Universal” address software development in a concise fashion? When I learned how to draw data flow diagrams in the 1980s, I knew intuitively what I was doing. The hardest concept was separating control flow from data flow. Real-time Structured Analysis even solved this problem by introducing control flow diagrams.

But I’m just ranting. There is a root cause is that we as front line developers have let this happen. We have allowed these problems to exist for so long that the quality of our work and our productivity have slipped.

When Object Oriented Analysis and Design were introduced we conviced ourselves and our managers that it would produce big productivity gains. When it didn’t live up to the hype we were told it would take a while for us to “get it.” We had been brought up on a diet of Structured Analysis and procedural programming and would need to unlearn those habits. Another problem early on was the lack of standard notation and processes. UML was supposed to fix that. So we struggled and blamed ourselves for not getting it and waited for better tools.

But now I have to say, I get Object Oriented Programming. I understand UML even though I find it hard to use. Yet I’m still dissatisfied with the current state of software design. Now development processes are often defined by  managers or process specialists who don’t actually develop software. Or is management is sold on Agile Development we skip most of the design phase. We have capitulated and gone along with the situation and have been afraid to admit that “The emperor has no clothes.”

As engineers we need to take control of the process again. We need to step up and own the process and put in the effort to make the process and the tools useful again. To paraphrase Shakespeare: “The problem is not in our tools but in ourselves.”

A couple of years ago I gave a presentation at the Embedded Systems Conference about implementing communications protocols in C++. One of the main themes in my talk was avoiding context switch because of its high cost. This is especially important in embedded Linux. The computational costs for context switching are high whether you are using processes or pthreads.

The primary reason for using embedded Linux is for network centric applications. One of the Linux Kernel’s biggest advantages is its reliable and efficient TCP/IP network stack. If you’re using Linux in an embedded system and networking isn’t a big part of your application you need to make sure you have another compelling reason for incurring the costs of a Linux kernel.

Linux provides an elegant mechanism for simulating multitasking for network applications. This is the select() function. For those who are not familiar with the select() function, it allows you to wait on events on multiple file descriptors. This might not seem too interesting unless you realize that network sockets are file descriptors. You can also write drivers for your specialized hardware that emulate file I/O so that you can create file descriptors for your application specific hardware devices.

The select() function works on three sets of file descriptors: read, write, and exception. You can also pass a timeout value to select allowing execution of periodic tasks. The main loop for an application that uses select() looks something like this:

Do forever:

Initialize the timeout
Initialize the File Descriptor Lists

Call select

Check for file descriptors in each list that are ready for servicing.
Determine if the time interval for periodic tasks has elapsed.

End do

In addition to the select() function, Linux provides macros for clearing, building file descriptor lists. There is also a macro to check which of the file descriptors are ready after select() returns.

The select() function is commonly used in implementing all kinds of network servers from Web servers to SNMP. The Linux man pages select(2) and select_tut(2) are very complete if a little overwhelming. On the Web, the world of select is a good starting point. If you want to look at a real world example that’s not too complicated, the boa web server is a good example.

I goofed yesterday on one of the code examples as one reader, Michael, pointed out.  I was trying to show how to define a hardware register as a cosnt pointer to volatile data and I wrote:

static const uint8_t* volatile dataOut = 0x20000010;

which is a volatile pointer to const data.  This may be useful in some bizzarre application but it’s certainly not what I was trying to do.  Here is what I meant to do:

static volatile uint8_t* const dataOut = 0x20000010;

Fortunately, I had the code correct in the driver I’ve been working on and I hope I didn’t confuse anyone.  Actually if it had been real code instead of a web page, I’m sure the compiler would have complained the first time I tried to write to the register.  That’s one reason to have all of these qualifiers on your data.  Error messages are a great help, especially when you’ve been (trying to) code for eighteen hours.

Thanks for pointing this out Michael.

By convention, at this point in time you can assume that a char in C is 8 bits, an int or a long is 32 bits and a short is 16 bits but not always. That can be a problem in embedded system programming since we often need to know the exact size of a variable. We also tend to use unsigned variables a lot and typing out unsigned gets tiresome. So you often see people inventing their own coding conventions such as ubyte, ushort, and ulong, etc. I work on code every day that may use two or three different conventions depending on who worked on the code last.

There is a standard way of expression variable sizes and whether they are signed or unsigned in the GNU and POSIX worlds. It is to use the header file stdint.h which defines a number of types of specific sizes: int8_t, uint8_t, int16_t uint16_t, int32_t, uint32_t. The stdint.h file is customized for the processor that you are working with so you can depend on the sizes. This header file also specifies things like the minimum and maximum values that can be represented by a type such as INT32_MAX, UINT32_MAX, and INT32_MIN, etc. All of this is defined in the man page for stdint.h if you are using Linux or Unix.

Sometimes you don’t really care about the exact width of a variable but you want to use a data width of some minimum size that is the most efficient for your processor. Stdint.h provides representations for that too, like: uint_fast8_t and int_fast32_t. There is also a definition for the widest types supported: int_max_t and unit_max_t. You could find out how many bytes wide these are by using sizeof(int_max_t) for example.

stddef.h is another useful header file. It defines things like NULL (though according to Stroustrup you should just use 0 for null pointers) and size_t. size_t is defined as the type returned by the sizeof() function. size_t is useful for times when you don’t really care about the size of a variable, as long as is large enough to represent the size of a data object in bytes. For example, suppose you want the binary inverse of every byte in and array:

for(size_t i = 0; i < sizeof(array); i++)
{
array[i] = ~array[i];
}

Finally, do you know what this represents?

static const uint8_t* volatile dataOut = 0x20000010;

This is a const pointer to a volatile byte at location 0×20000010. A hardware register.

You want a const pointer because the hardware registers shouldn’t be moving around. Declaring the pointer is const allows the compiler to catch you if you forget to dereference the pointer when writing to the location.

The value stored at location 0×20000010 could change on its own without the software changing it. Volatile tells the compiler not to optimize away any reads or writes to this location. For example if you only ever write to the register and never read it, the compiler might remove what seems to be unnecessary writes to this location. The optimizer might completely remove the variable from the object code. Volatile lets the compiler know that this location is special.

I’ve been doing a bit of work lately on a project that uses the Analog Devices Blackfin processor. The application is very ‘net-centric so we decided to use Linux, specifically uCLinux for the operating system. Of course this means that our development tools are the GNU compiler collection and all of its related applications.

Often, one of the hardest parts of a project like this is getting Linux up and running on your target hardware. The Blackfin uCLinux web page is very well organized and has a lot of good information. I downloaded the tools and the Linux distribution from their site and had Linux up and running on a development board in a matter of hours.

One thing that is an absolute necessity is to have a JTAG interface for programming FLASH through the processor. Once you have the boot program running you can program the FLASH using commands that the bootstrap provides but you need to get the bootstrap in FLASH first. An Austrian company, Blue Technix sells a USB JTAG ICE for the Blackfin called the Ice bear. It’s about $320 (US) and since they are in Austria, it could take a couple of weeks to get one so you need to plan ahead. The ICE worked just as advertised and was critical to the success of my project. Blue Technix also has some low cost eval boards too but given the Euro/Dollar exchange rate you might do better getting something from Analog Devices or one of their distributors.

As I was saying, the hard part is often getting Linux up on your own hardware. My target had some fundamental differences from the Eval board that I started working with. Like most micro controllers, the Blackfin has a bunch of multi-function pins. It also has two UARTS. Well our design used UART0’s data out pin as a software controlled hard reset. The other fundamental difference was that we were using SPI serial FLASH instead of the parallel NAND FLASH used on the eval board.

Porting the bootstrap went pretty smoothly. After about a days work, I had the bootstrap working. But it was different with the Linux kernel. I disabled UART0 and enabled UART1 using make menuconfig. But every time the board started to boot – bam! It reset almost immediately. I knew that somewhere the function for that pin was being set up to be UART0’s output. It took about three days of searching the code but I finally found it. The I/O pin setup for both UART0 and UART1 was hard coded deep in the init code. After I fixed that, I could boot Linux using TFTP.

The next step was getting Linux to work out of the SPI FLASH. There are two approaches to this. You can create a complete compressed file system image for a RAM disk system. Then you have the boot program copy the image from FLASH to RAM, decompress it and go. This approach is very simple to configure because it’s the stock build that comes with the distribution. The downside is that you don’t have any non-volatile storage without jumping through hoops.

The second approach took me a couple of more days to get running. You build a file system for FLASH and a kernel image. There’s more to configure to get this to work. One thing you need to keep in mind is that you can’t use any loadable kernel modules to access the file system since you need the file system to load them. The nice part about this is that now you have a file system that’s almost like a disk. If you make changes to your application you can use FTP to load them on the target. You can also use the system logger to log to syslog (they actually call the log /var/log/messages).

Throughout this whole process I was able to get very good support from their web site and forums. If an answer wasn’t in the documentation wiki, I would most likely find it by searching the forums. When I posted questions to the forum, I often had an answer within an hour or two. The documentation and support were better than many commercial RTOSs and cross-compilers that I’ve used.

What really amazed me was how well the tools all work. I wrote a pretty complex application in C++ using many of the C++ library classes, Pthreads, sockets, and all kinds of system facilities. I got it all working on my desktop Linux machine. Then I just recompiled my application using the Blackfin version of the tools, loaded it on my Blackfin uCLinux system and it just worked.

The one thing though that you have to remember is that if you’re messing with the kernel then according to the GPL, anyone who buys your product is entitled to the source code. Then they’re also allowed to re-distribute that code. If you’re working with custom hardware then you’re going to be messing with the kernel. I don’t think that this requirement ever hurt sales of the Linksys 54G routers though. If the project is for a government agency, they may require full source code anyway.

Some things that help make a normal life pleasant can get to be distractions when you’re way behind schedule on a project. Nerds are legendary for ignoring these distractions when a technical challenge requires their full attention. Details like hygiene and nutrition are the first casualties in battles against bugs and deadlines. It’s really hard to be fresh smelling and perky looking when you’re in the middle of marathon systems integration problems and have been working for eighteen hours straight.

Over the last couple of weeks, I’ve really noticed a decline in my eating habits. I usually try to prepare my food from fresh unadulterated ingredients; lots of vegetables, beans, salad and grains and a good bit of meat too. But I cook from scratch and watch the carbs and fat. That is until I hit systems integration.

I started out last week with salads for lunch and home made chili for dinner. When the chili and the lettuce ran out, I switched to carry out food. I tried sticking to wholesome stuff like the local kabob place, easy on the rice and Chipotle which is quite healthy and tasty without the rice and tortillas.

Then I started eating at odd hours, late at night or very early in the morning. I switched to the diet of the legendary first generation of hackers at MIT, like Richard Stallman. I started alternating between Chinese carry out and pizza washed down with lots of Coke.

I knew I hit bottom this morning. Taco Bell is open late around here. They call the time between midnight and two AM, “The Forth Meal.” I found myself driving to the Taco Bell to get there before closing so I could get mine. A few hours later I was at McDonalds’ getting a sausage, egg, and cheese McGriddle and another Coke.

I’m lost now and I’ll admit it. I’m not eating again until I can cook something for myself. Right after a shower and a twelve hour nap.

We embedded systems programmers have some special challenges in our work.  One of the major problems we face is that the hardware we work with is often not fully exercised until our software exercises it.  So, it is tempting to blame the hardware when things don’t work right and the cause is not obvious.  I’ve done my share of blaming the hardware, sometimes to the point of embarrassment when it turned out I was wrong.

Over the years I’ve learned to work with the hardware engineers to solve a problem rather than point my finger.  I’ve learned the hard way to devise some low level tests to isolate a suspected hardware problem before I go bother the hardware designer.  Sometimes a ’scope or a logic analyzer are the best software debugging tools and embedded systems programmer can have.  But sometimes you run across a bizarre software problem that masquerades as an obvious hardware problem.  A couple of these kinds of bugs will change your approach to debugging forever.

Will Rogers once said “There are three kinds of men. The one that learns by reading. The few who learn by observation. The rest of them have to pee on the electric fence for themselves.”  For those who learn by reading or observation here are a couple of war stories about obvious hardware problems that weren’t.

The first story involves a very successful piece of test equipment.  If you ever worked with T1s and I told you the model number you would probably recognize it.  I worked on this product toward the end of its life cycle.  I did some of the last feature upgrades and became responsible for software maintenance on it.  This test set used a popular Intel counter-timer chip.  The same chip was used in the original IBM PC and there are incarnations of it in the system chips of personal computers to this day.  NEC was a second source for this part but for some reason this test set would not work with the NEC version of the timer chip.  There was even a note on the BOM (Bill of Materials) that only the Intel part could be used.

Well, one day, something fell through the cracks and a batch of these units were built with the NEC part.  I got a call from production test that the software was failing in these units.  So, I went over to the factory and picked up one of the failing units to look at with an ICE.  As soon as I took the unit apart, I saw the NEC timer and knew that was the problem but when I called the production manager, he insisted that I should take a second look at the software.  We used the same NEC time in several other products with no problems and it was getting harder to get the Intel part.  I was certain that it was not a software problem.  There were literally hundreds of these test sets in the field with this software, working just fine.  I couldn’t very well say no so I set up a a couple of breakpoints and ran the unit trough its paces.  As I single-stepped through the ISR for the timer for about the tenth time, I noticed that it pushed one more register on the stack than it popped off at the end.  There it was in x86 assembly code, a function that should have crashed every time no matter what timer chip was used.  I fixed the routine and the test set worked just fine with the NEC part from then on.  To this day I have no idea why the problem never showed up with the Intel timer chip but if it weren’t for the persistence of that production manager, that bug might have showed up at another time in another way.

The second story is about another very popular product from the same company.  I never worked on that product but a coworker told me about this one.  Test equipment tends to have a very long product lifetime.  This product had undergone so many upgrades that they decided that it was time for a major software rewrite.  The project went well until system testing.  It seemed that the units in the lab worked just fine but when they were buttoned up with the covers on, the software crashed.  How could the presence of the cover affect the software?  It seemed like a classic hardware problem.  Perhaps it was a problem with noise or temperature.  The mystery was that the previous software release worked just fine, with or without the covers on.  Debugging this was a nightmare.  You couldn’t hook up an ICE with the cover on the unit so they had to write test code and install it and put the cover back on.  There didn’t seem to be any rhyme or reason to the software crashes.  Meanwhile they were also pursuing another seemingly unrelated problem.  If two of the pins on the RS-232 interface were shorted together (like RTS to CTS) the software would crash as soon as the test set was powered up.  It turned out that in the start up code in the new version, someone enabled interrupts before the rest of the hardware was initialized.  Shorting the RS-232 pins together or putting the cover on the unit added just enough noise on a floating interrupt line to cause it to trigger an interrupt.  Since the interrupt vector table had not been initialized at this point, the interrupt vectored to an invalid address and crashed the system.  Yep, another classic hardware problem that turned out to be software.

So what should you take away from these stories?  Never assume the problem is hardware.  If it seems like it is hardware, work with the hardware engineer to solve the problem.  Don’t just point your finger.  Use a ’scope or a logic analyzer if you know how and if you don’t get an EE to drive for you.  Some very bizarre software bugs can disguise themselves as hardware problems.

One of the most common uses of threads in embedded systems is to serialize usage of shared resource such as data, I/O ports, sockets, or files. A typical implementation is to use a queue and a thread. Threads accessing the shared resource place requests or operations on the queue. A thread waits on the queue, dequeuing requests and performing the operations on the shared resource. Structurally, it looks something like this:

Usually it’s best to hide all of this complexity from the resource users. I usually place the thread, the queue and their related components in a single class. The resource user sees only one public function; queueRequest(Request&). This helps insure that users only access the resource through the access queue/thread. The name of the function reminds the user that the request may not be executed immediately. Now let’s look at an implementation in C++ using POSIX threads:

First the class declaration:

class SharedResource
{
public:
    SharedReasouce(Resource& resource);
    ~SharedResource();

    queueRequest(Request&);

private:
    static void* accessThread(void* params);

    Resource&             _resource;
    std::queue<request&>  _requestQueue;
    pthread_t             _accessThreadId;
    pthread_cond_t     _queueSignal;
    pthread_mutex_t       _queueMutex;  // Required to use pthread condition code
};

We pass a reference to the resource to the constructor. This way if we have several resources of the same type, we can create an object using this class for each of the resources.

The thread function must be static in order to use it with the POSIX create_thread function. Thread functions take a single parameter, a void pointer. We will use this pointer to pass the this pointer to the thread.

The queueSignal condition code is used to signal accessThread when a request has been placed on the queue. Pthreads requires that you use a mutex to signal a condition code and this is what queueMutex is used for.

Now let’s look at the constructor:

SharedResource::SharedResource(Resource& resource)
{
    _resource = resource;
    pthread_mutex_init(&_queueMutex, 0);
    pthread_cond_init(&_queueSignal, 0);
    pthread_create(&_accessThreadId, 0, SharedResource::accessThread, this);
}

The constructor makes a copy of the resource reference and creates the pthread condition code and mutex.  Then the access thread is created.  This starts execution of the function  SharedResources::accessThread.  A pointer of this instance of class SharedResource, the this pointer, is passed to the thread function as a parameter.

The queueRequest function is streight-forward.  It simply places a request reference in the queue then signals the accessThread that a request is in the queue:

void SharedResource::queueRequest(Request& request)
{
    _commandQueue.push(request);
    pthread_cond_signal(&_queueSignal);
}

Finally let’s take a look at the thread itself:

void* SharedResource::accessThread(void* param)
{
    // Convert the void* parameter back to a SharedResource reference
    SharedResource& sharedResource = reinterpret_cast<SharedResource&>(param);

    while(1)
    {
        // Wait on the request queue
        pthread_mutex_lock(&sharedResource._queueMutex);
        pthread_cond_wait(&sharedResource._queueSignal,
            &sharedResource._queueMutex);

        // Dequeue any requests and process them
        while(sharedResource._requestQueue.size() > 0)
        {
            request = sharedResource._requestQueue.front();
            sharedResource._requestQueue.pop();

            // process the request on resource
        }
    }
}

A couple of things to note.  In an actual program, you would check the return codes of each of the function calls in the constructor that allocate resources.  If one of these functions fails you need to clean up any allocated resources and probably throw an exception.

I’ve become a big Ubuntu Linux fan. It’s not that I hate Red Hat Linux. I used Fedora Core Linux all the time until it wouldn’t run on my shiny new computer.  After that I found the Ubuntu package manager to be a real joy to use. I also like the fact that Ubuntu doesn’t install hundreds of extra packages when you install it.

But, what impressed me most was the last major Ubuntu upgrade. The update manager showed a button one day indicating that I could upgrade from version 7.10 to 8.04. I carefully backed up all my critical files and then clicked on the upgrade button. About a half hour later, I was running Ubuntu 8.04 without having to configure anything or restore any of my files.

There are some software packages though that are not available as Debian packages, which is what Ubuntu uses. This is especially true for cross development versions of GCC which we commonly use in the embedded systems world. Often our choices are an rpm file or a tar file. While tar files are usually fine, the rpm files, like Debian deb files contain some additional configuration information. If you simply use the tar files then you have to figure out where the files should reside and what environment variables you need to define. Both rpm and deb package managers take care of these details for you when you install from the package files. The problem arises when you are using a Debian based Linux such as Ubuntu and the package you want is only available as an rpm. The rpm packages are designed to be installed on Red Hat or Fedora Linux and can’t be natively installed on Ubuntu.

This is where the alien command comes in. The alien command converts rpm packages to deb packages. If you invoke alien with the -i option it will then install the deb package. Also remember that you need super user privileges to install software packages so the command to install an rpm on Ubuntu would be something like this:

$ sudo alien -i blackfin-toolchain-08r1-8.i386.rpm

In this case, installing a GNU tool chain for Analog Devices’ Blackfin processor.

Most likely, alien is not installed on your computer but it is easy to get using apt-get:

$ sudo apt-get install alien


Linux has become a very popular operating system for embedded systems.  I’m currently working on two projects that use embedded Linux.  Linux pops up on alls kinds of devices; routers, network-attached disk drives, and digital video recorders to name a few.

There has been a lot of discussion about the limitations of Linux as a real time operating system.  When embedded software engineers adopt Linux for a project they are usually aware of these limitations.  Either the engineers have a work-around for time critical tasks or they have decided that Linux can meet their real time requirements.

One problem that I’ve had using embedded Linux is not really a technical issue.  It’s the expectations of users, managers and clients.  When a manager or someone in marketing finds out that you are using Linux, they start dreaming up all the extra things the device should do because it “comes free” with Linux.  Suddenly features like extra layers of security and web-based user interfaces become must-haves.

People need to realize, or be educated to realize that a device running embedded Linux is still an embedded system with the limited resources that embedded systems have.  The art of embedded Linux is pairing down a standard distribution and even its kernel.  Components like OpenSSL and the Apache web server take a lot of storage space and processor time.  In addition, every extra feature that is added means more complexity, more interactions between components and more ways for a system to fail.

Embedded Linux brings many advantages to embedded developers.  Using Linux makes unit testing on a desk top computer easier.  Linux provides a rich and well understood API and the development environment is the GNU development tools.  The Linux IP stack makes it a natural choice for many network attached devices.  While these are big advantages, your customers and managers need to be reminded that you don’t get everything that “comes for free” with Linux in an embedded system.