nerd1951.com

Let’s face it.  Software design falls into two extremes in the majority of software development organizations. Either it is hardly done at all or it is done to the point of dimensioning returns. You know I’m talking about you, most of you anyway.

The “hardly done at all” camp usually sees little value in doing design. Our managers want to see code! So, we will do the minimum documentation to satisfy our customers, clients or the process police. We even have a name for this approach: Agile Programming. To quote a Dilbert comic, “We just start coding and complaining.”

The design to death crowd has usually been sold some expensive tool or trained in some complex process. These days it’s almost always done in UML. Use cases must be reviewed by all of the stake-holders in painfully long meetings. Then all the varieties of UML diagrams are produced and painstakingly reviewed before a line of code is written. And this is still supposed to be an iterative process.

In the dozen or so years since I abandoned Structured Analysis and Design, I’ve learned two things: Object Oriented Programming really works well and the Waterfall Method of software development does not. Agile Development is a direct response to the old Waterfall model but most “agile” development organizations neither understand nor follow the Agile Process. What I haven’t learned though is a process better than Structured Analysis and Design for getting from the user’s requirements to code.

UML has a host of problems. My biggest problem with UML is that it is mistaken for a development process. There are no well defined processes for Object Oriented design. The analysis phase is well defined with Use Cases and Interaction Diagrams. But beyond this I find most of the process gets muddled. Where is the step-wise refinement that allows you to drill down from the high level architecture to the individual classes and objects?

I don’t even think UML is good notation. It’s just not intuitive. How can any modeling language that professes to be “Universal” address software development in a concise fashion? When I learned how to draw data flow diagrams in the 1980s, I knew intuitively what I was doing. The hardest concept was separating control flow from data flow. Real-time Structured Analysis even solved this problem by introducing control flow diagrams.

But I’m just ranting. There is a root cause is that we as front line developers have let this happen. We have allowed these problems to exist for so long that the quality of our work and our productivity have slipped.

When Object Oriented Analysis and Design were introduced we conviced ourselves and our managers that it would produce big productivity gains. When it didn’t live up to the hype we were told it would take a while for us to “get it.” We had been brought up on a diet of Structured Analysis and procedural programming and would need to unlearn those habits. Another problem early on was the lack of standard notation and processes. UML was supposed to fix that. So we struggled and blamed ourselves for not getting it and waited for better tools.

But now I have to say, I get Object Oriented Programming. I understand UML even though I find it hard to use. Yet I’m still dissatisfied with the current state of software design. Now development processes are often defined by  managers or process specialists who don’t actually develop software. Or is management is sold on Agile Development we skip most of the design phase. We have capitulated and gone along with the situation and have been afraid to admit that “The emperor has no clothes.”

As engineers we need to take control of the process again. We need to step up and own the process and put in the effort to make the process and the tools useful again. To paraphrase Shakespeare: “The problem is not in our tools but in ourselves.”

I’m sure you’ve heard the news by now that Intel has purchased WindRiver, one of the largest providers of Real Time Operating Systems and embedded systems development tools.

Intel’s purchase of WindRiver for 884 million dollars is the end of VxWorks as a viable embedded RTOS.  Actually VxWorks has been losing ground in the embedded systems market for years.  Back in the late ’90s they changed their sales strategy to focus on selling to upper management rather than engineers.  They had to talk to upper management at their price.  As my manager quipped, “Merger?  I thought that the $884 million was for a developer’s license and that’s only valid in the state of California.”  WindRiver’s prices and license policies limited their appeal to everyone except large corporations that develop high quantity devices or companies in very high margin markets such as defense or heavy industrial machinery.  Admittedly, that’s where the money is.  The smaller players in the embedded systems market are not a lucrative customer based even though they employ the largest number of developers.

Intel’s motive is to penetrate the portable consumer device market.  This market is growing much faster than the desktop PC, laptop PC and server markets that Intel dominates.  These markets have all matured and will never experience the explosive growth of a new portable gadget.  Intel’s processors have not been as successful in the portable market and Intel has abandoned its embedded processor products more than once.  Their hope is that by providing the whole package, processor and operating system, they can capture this growth market that has slipped away from them.

Ironically, Intel had one of the first RTOS’s on the market and it was reasonably successful in the 1980s and early 1990s.  Intel’s RMX operating system was the first RTOS that I ever used on a microprocessor.  But by 2000, Intel was so focused on WinTel products that they no longer wanted the cost of supporting RMX86.

Technically, VxWorks is a good product and WindRiver has a history of innovation.  VxWorks was one of the first RTOS’s to provide Internet Protocol support and remote debugging.  They were also an early adopter of the GNU tool chain though they bent the rules quite a bit in redistributing the GNU development tools.  But their corporate policies alienated a lot of engineers.  In the late 1990’s WindRiver went on a shopping spree, purchasing several companies in the embedded systems developer’s market.  They purchased and marginalized pSOS.  They bought a couple of cross compiler companies and then jacked up the prices on their products. Finally there was the new sales model which tried to bypass the engineers that had to use their products.

So, why do I think this is then end of VxWorks as an embedded systems platform?  Intel will focus on portable consumer devices because they want to sell large quantities of processors into that market.  The rest of the embedded developer’s community, those of us who build communications, industrial and other embedded systems products, will become second class citizens.  VxWorks is also a popular OS for Freescale’s embedded PowerPC processors.  Will an Intel subsidiary continue to support a competitor’s products?  But the landscape has changed quite a bit in the last ten years and now we have plenty of alternatives for embedded real time operating systems.

I’ve been doing a bit of work lately on a project that uses the Analog Devices Blackfin processor. The application is very ‘net-centric so we decided to use Linux, specifically uCLinux for the operating system. Of course this means that our development tools are the GNU compiler collection and all of its related applications.

Often, one of the hardest parts of a project like this is getting Linux up and running on your target hardware. The Blackfin uCLinux web page is very well organized and has a lot of good information. I downloaded the tools and the Linux distribution from their site and had Linux up and running on a development board in a matter of hours.

One thing that is an absolute necessity is to have a JTAG interface for programming FLASH through the processor. Once you have the boot program running you can program the FLASH using commands that the bootstrap provides but you need to get the bootstrap in FLASH first. An Austrian company, Blue Technix sells a USB JTAG ICE for the Blackfin called the Ice bear. It’s about $320 (US) and since they are in Austria, it could take a couple of weeks to get one so you need to plan ahead. The ICE worked just as advertised and was critical to the success of my project. Blue Technix also has some low cost eval boards too but given the Euro/Dollar exchange rate you might do better getting something from Analog Devices or one of their distributors.

As I was saying, the hard part is often getting Linux up on your own hardware. My target had some fundamental differences from the Eval board that I started working with. Like most micro controllers, the Blackfin has a bunch of multi-function pins. It also has two UARTS. Well our design used UART0’s data out pin as a software controlled hard reset. The other fundamental difference was that we were using SPI serial FLASH instead of the parallel NAND FLASH used on the eval board.

Porting the bootstrap went pretty smoothly. After about a days work, I had the bootstrap working. But it was different with the Linux kernel. I disabled UART0 and enabled UART1 using make menuconfig. But every time the board started to boot – bam! It reset almost immediately. I knew that somewhere the function for that pin was being set up to be UART0’s output. It took about three days of searching the code but I finally found it. The I/O pin setup for both UART0 and UART1 was hard coded deep in the init code. After I fixed that, I could boot Linux using TFTP.

The next step was getting Linux to work out of the SPI FLASH. There are two approaches to this. You can create a complete compressed file system image for a RAM disk system. Then you have the boot program copy the image from FLASH to RAM, decompress it and go. This approach is very simple to configure because it’s the stock build that comes with the distribution. The downside is that you don’t have any non-volatile storage without jumping through hoops.

The second approach took me a couple of more days to get running. You build a file system for FLASH and a kernel image. There’s more to configure to get this to work. One thing you need to keep in mind is that you can’t use any loadable kernel modules to access the file system since you need the file system to load them. The nice part about this is that now you have a file system that’s almost like a disk. If you make changes to your application you can use FTP to load them on the target. You can also use the system logger to log to syslog (they actually call the log /var/log/messages).

Throughout this whole process I was able to get very good support from their web site and forums. If an answer wasn’t in the documentation wiki, I would most likely find it by searching the forums. When I posted questions to the forum, I often had an answer within an hour or two. The documentation and support were better than many commercial RTOSs and cross-compilers that I’ve used.

What really amazed me was how well the tools all work. I wrote a pretty complex application in C++ using many of the C++ library classes, Pthreads, sockets, and all kinds of system facilities. I got it all working on my desktop Linux machine. Then I just recompiled my application using the Blackfin version of the tools, loaded it on my Blackfin uCLinux system and it just worked.

The one thing though that you have to remember is that if you’re messing with the kernel then according to the GPL, anyone who buys your product is entitled to the source code. Then they’re also allowed to re-distribute that code. If you’re working with custom hardware then you’re going to be messing with the kernel. I don’t think that this requirement ever hurt sales of the Linksys 54G routers though. If the project is for a government agency, they may require full source code anyway.

Some things that help make a normal life pleasant can get to be distractions when you’re way behind schedule on a project. Nerds are legendary for ignoring these distractions when a technical challenge requires their full attention. Details like hygiene and nutrition are the first casualties in battles against bugs and deadlines. It’s really hard to be fresh smelling and perky looking when you’re in the middle of marathon systems integration problems and have been working for eighteen hours straight.

Over the last couple of weeks, I’ve really noticed a decline in my eating habits. I usually try to prepare my food from fresh unadulterated ingredients; lots of vegetables, beans, salad and grains and a good bit of meat too. But I cook from scratch and watch the carbs and fat. That is until I hit systems integration.

I started out last week with salads for lunch and home made chili for dinner. When the chili and the lettuce ran out, I switched to carry out food. I tried sticking to wholesome stuff like the local kabob place, easy on the rice and Chipotle which is quite healthy and tasty without the rice and tortillas.

Then I started eating at odd hours, late at night or very early in the morning. I switched to the diet of the legendary first generation of hackers at MIT, like Richard Stallman. I started alternating between Chinese carry out and pizza washed down with lots of Coke.

I knew I hit bottom this morning. Taco Bell is open late around here. They call the time between midnight and two AM, “The Forth Meal.” I found myself driving to the Taco Bell to get there before closing so I could get mine. A few hours later I was at McDonalds’ getting a sausage, egg, and cheese McGriddle and another Coke.

I’m lost now and I’ll admit it. I’m not eating again until I can cook something for myself. Right after a shower and a twelve hour nap.

I keep both a Windoze machine and an Ubuntu Linux machine on my desk.  There are a few applications that I need to use for embedded development that just aren’t available on Linux so I need to keep the Windoze machine around.  It’s actually been months since I’ve used it though.  I use Ubuntu exclusively for office stuff, web surfing, email, and entertainment like listening to music or watching DVDs.  (I have 4 computers and no working TV).

When I first turn on the computer in the morning there is often a notification for software updates.  These usually go pretty fast so I let the computer update itself while I fix my coffee and oatmeal.

A few months ago, the most amazing thing happened.  There was a notification that an entirely new revision of Ubuntu was available for download.  Being cautious, I deferred doing that update until after work.  I spent an hour or so backing up some critical files to the computer that I use as a CVS and Subversion server.  Then I started the update which only took about half an hour.  A lot of software, including the Linux Kernel was changed.  When I rebooted the computer, I was prepared for the worst but everything just worked!  Some applications were even a little faster!

Isn’t that the most amazing thing?

I’ve become a big Ubuntu Linux fan. It’s not that I hate Red Hat Linux. I used Fedora Core Linux all the time until it wouldn’t run on my shiny new computer.  After that I found the Ubuntu package manager to be a real joy to use. I also like the fact that Ubuntu doesn’t install hundreds of extra packages when you install it.

But, what impressed me most was the last major Ubuntu upgrade. The update manager showed a button one day indicating that I could upgrade from version 7.10 to 8.04. I carefully backed up all my critical files and then clicked on the upgrade button. About a half hour later, I was running Ubuntu 8.04 without having to configure anything or restore any of my files.

There are some software packages though that are not available as Debian packages, which is what Ubuntu uses. This is especially true for cross development versions of GCC which we commonly use in the embedded systems world. Often our choices are an rpm file or a tar file. While tar files are usually fine, the rpm files, like Debian deb files contain some additional configuration information. If you simply use the tar files then you have to figure out where the files should reside and what environment variables you need to define. Both rpm and deb package managers take care of these details for you when you install from the package files. The problem arises when you are using a Debian based Linux such as Ubuntu and the package you want is only available as an rpm. The rpm packages are designed to be installed on Red Hat or Fedora Linux and can’t be natively installed on Ubuntu.

This is where the alien command comes in. The alien command converts rpm packages to deb packages. If you invoke alien with the -i option it will then install the deb package. Also remember that you need super user privileges to install software packages so the command to install an rpm on Ubuntu would be something like this:

$ sudo alien -i blackfin-toolchain-08r1-8.i386.rpm

In this case, installing a GNU tool chain for Analog Devices’ Blackfin processor.

Most likely, alien is not installed on your computer but it is easy to get using apt-get:

$ sudo apt-get install alien


I have often seen obscure code written in a misguided attempt at “hand optimizing.” The irony is that I have also noticed in the make files for the same code that no optimization options were used when compiling the code. What a waste of time; human and computer.

The default level of optimization for most C/C++ compilers is none. The compiler spits out machine code that conforms very closely to the code you have written. This is good for debugging but is not efficient when running the code.

Let’s take a look at some code for a couple of simple functions in C and the assembly language that GCC generates for the ARM processor. Here is the C code:

int adder(int a, int b)
{
    int c;
    c = a + b;
    return c;
}

main(int argc, char* argv[])
{
    int a, b, c;
    a = atoi(argv[1]);
    b = atoi(argv[2]);
    c = adder(a, b);

    printf("nsum = %dn", c);
}

Now let’s look at the assembler generated for the function adder. The assembly was generated with the option -fverbose-asm. This option tells the compiler to put some comments in the assembly language about what it’s doing. The option is generally only used by compiler developers but it’s also useful for our purpose, trying to understand how optimizations effect the generated code.

adder:
        @ args = 0, pretend = 0, frame = 12
        @ frame_needed = 1, uses_anonymous_args = 0
        mov     ip, sp  @,
        stmfd   sp!, {fp, ip, lr, pc}   @,
        sub     fp, ip, #4      @,,
        sub     sp, sp, #12     @,,
        str     r0, [fp, #-20]  @ a, a
        str     r1, [fp, #-24]  @ b, b
        ldr     r2, [fp, #-20]  @ a, a
        ldr     r3, [fp, #-24]  @ b, b
        add     r3, r2, r3      @ tmp105, a, b
        str     r3, [fp, #-16]  @ tmp105, c
        ldr     r3, [fp, #-16]  @ D.1727, c
        mov     r0, r3  @ (result), (result)
        sub     sp, fp, #12
        ldmfd   sp, {fp, sp, pc}

The @ symbol begins a comment. The first four lines of code save some state information then create space on the stack for the function’s parameters. In C, local variables and function parameters are traditionally passed on the stack. The compiler makes room for two four byte words on the stack with the instructions:

    sub    fp, ip, #4
    sub    sp, sp, #12

What does it do with this space? It takes the two parameters that are already loaded in registers and stores them in memory on the stack; exactly what the C code says to do. Then the same two values are loaded into registers so we can do the real work of the function, add a and b. The result is moved to R0 to pass it back to the caller. Again, the traditional implementation in C is to return results in R0.

Why would the compiler generate code that wastes time moving things in and out of memory like this? It’s actually to make the code easier to debug. You might not have access to the processor’s registers due to the limitations of your debug tools and writing the parameters out to memory might be your only chance to see what is being passed to a function.

So now lets look at what the first level of optimization does to our function (-O1 for gcc).

        @ link register save eliminated.
        @ lr needed for prologue
        add     r0, r0, r1      @ (result), a, b
        mov     pc, lr  @

Quite a bit smaller, isn’t it? The first thing the compiler has told us that it doesn’t need to save the link register. Actually the compiler generated code doesn’t save a lot of state information that it did it the previous case. Next the parameters are left in the registers. The compiler was even smart enough to put one of the parameters, a, in R0. Since we are about to clobber R0 to pass back the result, this saves using another register. So the first level of optimization saves about a dozen instructions without doing anything exotic.

At the highest level of optimization, -O3 for gcc, the compiler uses the option -finline-functions-called-once. This tells the compiler to inline any functions that are only used once. So the entire function shrinks to a single line of code in main():

     add     r4, r4, r0      @ tmp109, a,

This is an important optimization to know about. Often you may have a long function that should be broken up into several functions for readability but you may be reluctant to do this because you don’t want to incur the overhead of several function calls. With the -finline-functions-called-once optimization you don’t have to worry about the function call overhead and you can focus on writing readable code.

So when you’re done debugging your code, don’t forget to turn on the optimization options and recompile. Then test it one last time. Compilers are smart but not infallible.

The Web is a wonderful place to find software documentation; how-to’s, tutorials, and user’s manuals. The problem is like most resources on the Web, you need to sift through a lot of stuff to find what you’re looking for – or what actually works.

I’m working on a project right now where I need to develop a GUI for a Linux application. In the past I would have used Java Swing but it’s been so long since I’ve done this kind of work that it’s like starting from scratch. So, I decided to learn a GUI programming library and development tool that I could use from C/C++ as well as a scripting language or Java. I decided on GTK+ and Glade after doing some research. GTK+ is the GIMP Tool Kit used to develop the GUI for GIMP, the GNU Image Manipulation Program. It’s kind of like Photoshop without the price tag. GTK+ has a C API but a library, libgtkmm, provides C++ wrappers for GTK+. Glade is a graphical GUI development tool. It allows you to build up a GUI by dragging and dropping the graphical objects you need to build up you GUI.  Finally, a library, libglademm, hooks up the output from Glade to libgtkmm.

The next step was to find some documentation, tutorials, and examples on the Web. I found some easy “Hello World” examples that didn’t really help much. I also found some very complete documentation but the author was really intent on me learning autoconfig, automake, and libtool. Now I’m sure learning these tools would be good for me in the long run but right now I’m on a deadline. Researching these tools I found a bunch of outdated examples that wouldn’t work on my up-to-date Linux environment. I finally found a quick how-to that was complex enough to be useful but didn’t require me to learn a bunch of other tools: A simple libglademm application. The example application is a simple calculator. It demonstrates the basics of building a GUI using Glade and then writing the C++ code behind it that makes it work.

I ran into some other outdated how-to’s this weekend. I was helping a friend set up Embedded Linux on an ARM evaluation board. The manufacturer actually provided a very complete build of GNU tools for their processor and Linux 2.6 build source for their development board. The problem was that there were too many how-to’s on the site – mostly outdated. Some of the older ones were just plain wrong or referred to links on their web site that no longer exist. The most up-to-date and correct documentation was buried in their file downloads and you had to hunt around to find everything. Once we found the right downloads and documentation, the build went smoothly with the exception of one minor glitch.

By the way, this is not to say that commercial software is better. Commercial software is often worse with even harder to find documentation and even more outdated material. Often you need to buy a book to make up for the lack of documentation that comes with the product.

So here’s my message to all of you great people who provide open source software: please delete your obsolete documentation, and to paraphrase Einstein; “Keep it a simple as possible but no simpler.”

Finally, I have to say, that I think Glade, Gtk+, libglademm, and gtkmm are great!

I’ve been using Ubuntu Linux for about a year now and have been quite happy with it. I use to use Fedora but I have had problems with the recent releases running on new hardware. Ubuntu seems to have better hardware support. The other big difference is that Ubuntu takes a minimalist approach in their distribution, providing the minimum useful system on the distribution CD. All the optional packages you may want are provided on Ubuntu and Debian Linux web sites. Ubuntu provides a software package manager that takes care of downloading and installing what ever software packages you may select.

The one problem that I have had with Ubuntu is that I could not successfully build GCC cross-compilers on the latest release, Ubuntu 7.10. I use and excellent script called crosstool to build cross-compilers. Crosstool has always worked fine before for me, even on earlier releases of Ubuntu. Finally, in desperation, I checked the crossgcc mailing list (reminder to self: RTFM!). The answer was there and was quite simple. It seems that on Ubuntu 7.10, /bin/sh is linked to /bin/dash. I didn’t realize that there were so many shell programs out there. I simply linked /bin/sh to /bin/bash and crosstool worked perfectly, allowing me to build cross compilers for the ARM and PowerPC processors.

It’s been a busy week around here in nerd1951 land. I’m moving which just makes it all so much more exciting but on to the techie stuff:

I’m going out to Silicon Valley for some advanced FPGA training next week. Actually it’s about using a particular FPGA core that’s advanced. My Verilog skills are still pretty rudimentary. I actually downloaded a design to a real FPGA for the first time today. That was fun, made some LEDs blink. The way I figure it, if I want to stay in the computer networking world then I need to become a logic designer at least part time. I want to become as fluent in Verilog as I am in Java (note I didn’t say C because I’ve been using C for almost three decades). I use to do logic design back in the days when a soldering iron was the only logic programming tool. Then I learned Able which is a lot like learning assembly compared with Verilog.

We’ve started a little embedded Linux informal study group at work. We all want to learn more of the nitty-gritty details of how to bring up Linux on our own devices. It’s becoming obvious to me that in-depth knowledge of building custom Linux kernels is becoming a requirement for embedded systems programming. We choose the Linksys WRT-54G as our first target. You can’t buy any kind of processor eval board for what the Linksys router costs. So this week, I set up a Linux build machine at work with the OpenWrt build environment and kernel. I built the Kernel. It’s not that complicated. Just typing something like make <target>. Then I downloaded it to my WRT-54GL. It was a thrill to Telnet into my router and see Linux come up. I think the next step will be to build some kind of Kernel module or change a driver and dig into the boot code.

On my real job, I’m using ucLinux with an Analog Devices Blackfin processor. Again the tools and kernel distribution on the Web were first rate. I was able to download the sources, build the toolchain and the kernel and boot Linux on an eval board in just a couple of days of work. I’ll have to write some drivers as well as the applications code for this project.

Another part of the same project is a sharable library that runs on an X86 Linux host. The library provides an API for communicating with the embedded device that we are building. In the past I’ve always written API documents using Word, then written the C or C++ header files, cutting text from the Word document and pasting the text into comments in the header files. I decided that this was stupid since I know that Doxygen will create documentation from a well documented header file. So I forced myself to learn Doxygen which wasn’t too hard since it can work just like JavaDoc which I’ve used in the past. There’s a real productivity gain from only writing things once. It reduces errors and makes it easier to keep the documentation in sync with the code. But I think the biggest advantage is the discipline that this process imposes. It forces a consistent style for data and function headers. Since you know other people will actually see your comments in a document, it makes you do a better job at writing useful comments and quality code.