nerd1951.com

I hope that no one is waiting anxiously for completion of my TCP/IP stack implementation.  There are two reasons why progress has been so slow.  The first reason is the obvious one; my day job has been quite demanding the last few months.  This has left me with little free time to work on Cannonball.  I’m also in the midst of updating my slides for my Embedded Systems Conference presentation which is in April.

The less obvious reason that progress on Cannonball has been so slow is that it has changed from a straight forward development project into more of a research project.  I started out building the TCP/IP stack as a demonstration of using C++ for a real-time application.  As I started the design, I decided that I wanted to build a class library for constructing network protocol software.  While researching existing C++ libraries for existing implementations of the components that I need I discovered generic programming.

The problem in working on real-time embedded systems is that we tend to be a bit disconnected from main stream software development.  We become domain experts and hardware/software integration experts but if we are not careful we can fall behind in software development technology itself.  So one motivation for developing Cannonball is to learn to apply generic programming to network protocol software.  Of the final products, I believe that the Network Protocol C++ library will be more important than the TCP/IP stack itself.

My previous posts about efficiency and generic programming were wrong.  In addition there is no conflict between the concepts of generic programming and the design principle in Network Algorithmics to “Avoid Unnecessary Generality.”  Generic programming allows for specialization and customization where it is required.

So please forgive my errors in previous posts due to my misunderstanding of generic programming.  Also please be patient with the progress of Cannonball as I do the research and struggle to catch up with modern C++ programming techniques.  Thank you.

There are four core issues that make embedded systems programming different from other types of software development:

1. Resource constraints including time, processor bandwidth, and memory.
2. Hardware Interfaces.
3. Multithreaded programming
4. The target execution environment is not the same as the development environment

To some extent these issues are not unique to embedded systems software but all four of these issues are always present in embedded development.  Two of these issues: multithreading and hardware interfaces could be supported by our programming languages but there are no standards as of yet.  We are at the mercy of our tool vendors who find wonderfully different ways at times to provide similar functionality.  But this may be changing. 

I recently read an ISO technical report: ISO/IEC TR 18015:2006(E) Technical Report on C++ Performance that includes a discussion of headers for C and C++ to provide standardized access mechanisms for hardware registers.

The C++ standards commitee seems even more active in the area of concurency and multi-threading. Trip Report: Ad-Hoc Meeting on Threads in C++ provides a good overview of this activity while Hans Boehm provides a comprehensive list of proposals and reports on his web site: Threads and memory model for C++.

My Network Protocol Object Framework is a piece of real world software engineering and on the nerd1951 blog you have a front row seat to this work in progress.  You stand witness to all my flip-flopping on decisions, changes in approach both major and minor and all the flaws in my thinking.

I have one reader, Tommy who has been following this development closely.  His comment about my post: “The Costs of Generic Programming vs. Execution Efficiency” questioned if I saw a conflict between the requirements of high performance networking code and generic programming.  His comment got me thinking and exploring the Boost libraries and the standard C++ libraries in greater detail.  In turn this led to me questioning my design approach for the Cannonball Project.  How do I know that my FIFO Queue design is really more efficient than those available in existing libraries?  Do I know if there is a run-time cost to using smart pointers?  Can I really say that any of my specialized designs are more efficient than implementations available in existing libraries?

I’ve ignored some of the oldest advice in computer programming.  As Donald E Knuth said: “Premature optimization is the root of all evil (or at least most of it) in computer programming.”  In The Elements of Programming Style circa 1978, Brian Kernighan admonished: “Make it right before you make it faster.” and “Instrument your programs.  Measure before making efficiency changes.”

So, for now, I’m going to ignore George Varghese’s principle of “Avoid unnecessary generality” from his book Network Algorithmics.  I’ll use the STL, the Boost libraries and generic programming approaches.  Once I have a working system, I’ll instrument it and measure it.  Then I’ll look at designing specialized data structures and algorithms to improve performance.  This is how we do engineering and science.

Thanks to Larry for the update on Netburner’s $99 Eclipse Ethernet Development Kit. This is the development target that I originally wanted to use for the Network Protocol Object Framework but at $599 it was out of my price range.  Netburner’s new $99 development kit uses open source development tools such as the Eclipse IDE.  This is especially interesting to me because I use Eclipse on my Linux box for prototyping and unit testing my C++ code and now I’ll have the same environment for embedded development.

The other exciting part is that the kit is based on the Freescale MCF5270 ColdFire processor.  ColdFire is a 32 bit reduced instruction set derivitive of the 68000 that is optimized for embedded systems applications.  The MCF5270 is a pretty hot version of this family running at clock rates up to 150 MHz.  The chip has an 8K cache and a 32 bit multiply/accumulate unit that also can do a 32 bit divide.  The on-chip peripherals include a 10/100 BaseT Ethernet controller, timers, serial ports, I2C, and SPI interfaces.

So, while I’m still starting the Cannonball development on my ARM-based AME-51 evaluation kit, I’ll probably migrate to the Netburner MCF5270 development kit pretty soon.  I actually want to bring up Cannonball on both boards as a test of portability.

As I flesh out the design for the Protocol Object Framework, I’ve been referring to several sources for implementation ideas.  Many C++ programming references, such as boost.org and Modern C++ Design, focus on generic programming principles.  Networking implementation references such as the Linux TCP/IP stack and Network Algorithmics tend to focus on specialized software approaches that emphasize efficiency.  In fact, one of the fifteen design principles in Network Algorithmics is “Avoid Unnecessary Generality.”

Engineering is often the art of compromise and that is one of my tasks for this design.  I want to develop a framework that is generic and flexible enough to implement a full spectrum of communications protocols.  At the same time I cannot lose site of the need for execution efficiency in networking software.  I’ll try to document the compromises as I document the design.

I’ve been taking a break from all things digital for a few days but I have posted a new Rant. 

I’ve also copied a bit more of my design notebook for the Protocol Object Framework to the Design Notes pages.  Right now I’m digging in to the subtleties of Smart Pointers in Modern C++ Design.  The concept of smart pointers seems simple enough: overloading operator-> and operator* for a class but like many things in C++ they are much more complex than they appear.  This is the C++ equivalent of how the Linux TCP/IP stack manages multiple pointers to a Socket Buffer (Linux Network Architecture) with a reference count in the buffer itself.

Expect copious updates of the design notes next week.

As I noted in a previous rant, Software Design Documents, Reviews, and Agile Methods, software design documents become obsolete once coding is started.  So, what kind of documentation should be available for the newbie that has to enhance or maintain a product?  The answer is a document that is generated during system testing or even after the product release.  This document is an opportunity for the developers to do a “brain dump” once coding is complete.  I call this document the Software Maintenance Manual.

The Software Maintenance Manual is not a Design Document

Design documents are a product of the design process. They written are before the code is written and are requirements-centric. That is the design documents explain how requirements are to be implemented in code. By contrast, the maintenance manual is code-centric. It explains how the code is structured. The maintenance manual must be written after the code is complete or nearly complete so that it is an accurate reference to the implementation. It is also a living document which must be updated as code is modified.

The problem with using design documents as a reference during maintenance is that they are an abstraction of the implementation. As lessons are learned during the implementation, the actual code diverges from the documented design. No one wants to interrupt the momentum of coding to update design documents so that by the end of the implementation, the design documents are no longer relevant. The code itself becomes the most relevant documentation of the design.

The problem with using the source code as a reference is that it is too detailed. An engineer can spend hours or even days just finding the modules relevant to a problem. The usual solution is to ask someone, who is already familiar with the code, for pointers for getting started. What is needed is a guideline or road map for the source code. The Software Maintenance Manual should provide this road map.

Finally, creating a Software Maintenance Manual should not be burdened with a lot of process. Requirements and functionality are already solidified by the time the maintenance manual is being written. The manual is internal to the software development group. So, input from marketing, manufacturing, and system testing is not required. The document’s style should be lean and informal.

What is in the Software Maintenance Manual?

A Software Maintenance Manual consists mainly of “How To’s”, Frequently Asked Questions (FAQs) and system specific data.

There are a few “How To’s that are common to most projects. Just documenting these common items can save hours of a developer’s time. Some common “How To’s” include:

• How to set up the development environment.
• How to build the system.
• How to do a software release

There are also system specific “How To’s”. They usually explain how to enhance the system. These “How To’s” should include pointers to specific source files and even specific data structures or functions. System-specific “How To’s” are obviously tied to the system’s applications. Here are some examples from previous projects:

• How to add a new menu.
• How to add a new item to a menu.
• How to add a new command to the Command Line Interface.
• How to add a new SNMP tables to the system.

There are other important items that are not obvious from reading the code that are not really “How To’s”. These items can be addressed by Frequently Asked Questions lists. Some examples of FAQs from previous systems are:

• How many Virtual Circuits can the system process
• Why is the system limited to 1024 routes?
• Why are the objects of class OpticalChannel always statically constructed?

The system specific data consists of tables and lists important to system maintenance. Some of the items that may be in the system data include:

• Physical memory and I/O maps
• Functions assigned to each module
• List of files that make up a module

Writing a “How To”

“How To’s” should be specific, focused, and detailed. They should focus on solving one problem or document one process. A “How To” consists of a detailed step by step procedure for a specific task.

“How To’s” must be accurate and complete to be useful. An inaccurate or incomplete “How To” may be worse than none at all. If an engineer using the “How To” finds problems with it, they may skip using the document the next time and go directly to the more experienced engineer. The whole purpose of the document is to prevent this.

It is easy to forget details and steps in a process when you are familiar with it. The best way to start a “How To”, is to document a process as you are doing it. Try to perform the process on someone else’s computer so that you will find key files or environment variables that may not be present. The next step is to give the document to someone else who is not familiar with the process you are documenting. Observe them performing the process and carefully note any questions that they have, or any problems that they run into.

Producing an accurate, complete “How To” is time consuming but will save you and your team hours of work in the future.

Maintaining a FAQ List

The first step in creating a FAQ list is to get a list of questions. This is easy to do. Keep a log or notebook of questions that arise late in development, during system integration and system test. New questions can be added when ever there is an enhancement project or a bug fix. If a list gets too long it becomes less useful. When this happens, the FAQ list should be broken up by topic to make specific questions easier to find.

FAQ List answers must be accurate and understood to be useful. One strategy for writing useful answers is to have the person asking the question write the answer. Then the engineer who is familiar with the system can review the answer. This insures that the answer was really understood by the person who asked the question and that others who read the FAQ will have a better chance at getting a useful answer.