nerd1951.com

When a system is well designed you can use it without even realizing the complexity of the technology behind it. This is true of the Zipcar car sharing service.

For those of you who are not familiar with the service, Zipcar allows you to rent cars by the hour or the day. In the cities where Zipcar operates, their cars are strategically located around the area, especially around public transportation hubs. You can search for available cars on the Zipcar web site then reserve the car you want. If you need to extend your reservation at the last minute you can do so using your cell phone.

Cars are accessed using a membership card that uses an RFID technology to unlock the car. When the car has been returned and your reservation is over, Zipcar automatically bills your credit card.

Zipcar employs a range of technologies that must be integrated seamlessly in order to provide an easy-to-use service. First there is the RFID technology employed in the membership card. The car must communicate with the back office to determine if you have a reservation. I assume that Zipcar uses satellite communications for that purpose. The back office must track which cars are available and which cars have need reserved. They also track when the cars have been picked up and returned. Finally the back office operation has to take car of the billing to your credit card.

The Zipcar website does an excellent job of presenting the member with all of the information that they need. You can see where available cars are located and the cost of using them. The web site also displays your future reservations and billing history.

All of these technologies spanning from the RFID on the membership card to the communications with the back office and the web site have been expertly woven together. The most impressive part is that the user never experiences the complexity of these technologies interacting. I use the service often without even thinking about it. This is the best indication of a well designed system.

Nerds like us don’t get much love but obviously someone loves nerds.  If you have a special friend that likes Hello Kitty she might like this.

I’m sure you’ve heard the news by now that Intel has purchased WindRiver, one of the largest providers of Real Time Operating Systems and embedded systems development tools.

Intel’s purchase of WindRiver for 884 million dollars is the end of VxWorks as a viable embedded RTOS.  Actually VxWorks has been losing ground in the embedded systems market for years.  Back in the late ’90s they changed their sales strategy to focus on selling to upper management rather than engineers.  They had to talk to upper management at their price.  As my manager quipped, “Merger?  I thought that the $884 million was for a developer’s license and that’s only valid in the state of California.”  WindRiver’s prices and license policies limited their appeal to everyone except large corporations that develop high quantity devices or companies in very high margin markets such as defense or heavy industrial machinery.  Admittedly, that’s where the money is.  The smaller players in the embedded systems market are not a lucrative customer based even though they employ the largest number of developers.

Intel’s motive is to penetrate the portable consumer device market.  This market is growing much faster than the desktop PC, laptop PC and server markets that Intel dominates.  These markets have all matured and will never experience the explosive growth of a new portable gadget.  Intel’s processors have not been as successful in the portable market and Intel has abandoned its embedded processor products more than once.  Their hope is that by providing the whole package, processor and operating system, they can capture this growth market that has slipped away from them.

Ironically, Intel had one of the first RTOS’s on the market and it was reasonably successful in the 1980s and early 1990s.  Intel’s RMX operating system was the first RTOS that I ever used on a microprocessor.  But by 2000, Intel was so focused on WinTel products that they no longer wanted the cost of supporting RMX86.

Technically, VxWorks is a good product and WindRiver has a history of innovation.  VxWorks was one of the first RTOS’s to provide Internet Protocol support and remote debugging.  They were also an early adopter of the GNU tool chain though they bent the rules quite a bit in redistributing the GNU development tools.  But their corporate policies alienated a lot of engineers.  In the late 1990’s WindRiver went on a shopping spree, purchasing several companies in the embedded systems developer’s market.  They purchased and marginalized pSOS.  They bought a couple of cross compiler companies and then jacked up the prices on their products. Finally there was the new sales model which tried to bypass the engineers that had to use their products.

So, why do I think this is then end of VxWorks as an embedded systems platform?  Intel will focus on portable consumer devices because they want to sell large quantities of processors into that market.  The rest of the embedded developer’s community, those of us who build communications, industrial and other embedded systems products, will become second class citizens.  VxWorks is also a popular OS for Freescale’s embedded PowerPC processors.  Will an Intel subsidiary continue to support a competitor’s products?  But the landscape has changed quite a bit in the last ten years and now we have plenty of alternatives for embedded real time operating systems.

OK – this is not technical in any way – just a pure rant.

When did we go from being people to being resources? Was it when everyone started using Microsoft Project? I hate seeing my name listed under “resources”.I never see anything but people listed on a project as resources. I’ve been on projects where test-beds or prototype hardware were the critical resources during integration. Yet they were never resources on the Microsoft Project schedule.

The other thing I hate is the Human Resources Department; now days just H.R. When I first started working this was called the Personnel Department as in “person.” If we have a Human Resources Department should metropolitan police departments have Canine Resources Departments and Equine Resources Departments? Actually I’ve worked for some tech companies that should have Equine Resources Departments for some of the managers, marketing folks and even some engineers but that’s a different story.

I guess another part of it is that you never lay-off resources. You just downsize. Somehow downsizing your resource head count sounds better than laying-off people. And no one gets fired anymore; just terminated. Well that’s all for now. I have to go see to my terminated bovine resources.

A couple of years ago I gave a presentation at the Embedded Systems Conference about implementing communications protocols in C++. One of the main themes in my talk was avoiding context switch because of its high cost. This is especially important in embedded Linux. The computational costs for context switching are high whether you are using processes or pthreads.

The primary reason for using embedded Linux is for network centric applications. One of the Linux Kernel’s biggest advantages is its reliable and efficient TCP/IP network stack. If you’re using Linux in an embedded system and networking isn’t a big part of your application you need to make sure you have another compelling reason for incurring the costs of a Linux kernel.

Linux provides an elegant mechanism for simulating multitasking for network applications. This is the select() function. For those who are not familiar with the select() function, it allows you to wait on events on multiple file descriptors. This might not seem too interesting unless you realize that network sockets are file descriptors. You can also write drivers for your specialized hardware that emulate file I/O so that you can create file descriptors for your application specific hardware devices.

The select() function works on three sets of file descriptors: read, write, and exception. You can also pass a timeout value to select allowing execution of periodic tasks. The main loop for an application that uses select() looks something like this:

Do forever:

Initialize the timeout
Initialize the File Descriptor Lists

Call select

Check for file descriptors in each list that are ready for servicing.
Determine if the time interval for periodic tasks has elapsed.

End do

In addition to the select() function, Linux provides macros for clearing, building file descriptor lists. There is also a macro to check which of the file descriptors are ready after select() returns.

The select() function is commonly used in implementing all kinds of network servers from Web servers to SNMP. The Linux man pages select(2) and select_tut(2) are very complete if a little overwhelming. On the Web, the world of select is a good starting point. If you want to look at a real world example that’s not too complicated, the boa web server is a good example.

There can be a big difference between the cost of choosing a processor or micro-controller and its price. That might sound counterintuitive but the total cost of using a particular processor includes much more than its piece-price. These additional costs include the cost of development tools, the RTOS if you are using one, and software especially development time. The software development time can also have a significant impact on a product’s time-to-market. I’ve work on many products where the cost of time-to-market in terms of lost sales far out-weighed component costs. The product quantities, tool costs, time-to-market, and software complexity must all be considered in determining the most cost effective processor.

The software team needs to be involved in processor selection as much as the hardware engineers. Some hardware engineers have a narrow view of processor costs; limited to the component cost for the processor and support components. I have seen projects where choosing the wrong micro-controller cost man-months of development time because the processor had a small address space and the memory had to be bank-switched, or the internal RAM was too small and variables had to be copied in and out of the internal RAM as needed.

Sometimes the hidden costs are in the development tools. Some processors are only supported by expensive proprietary tool sets. In others cases the only tools available provided a nonstandard subset of the C language. One well known micro-controller family is supported by a C compiler that can’t even link multiple object files. If you want to write modular code you have to divide your code into several include files that are basically compiled and linked as one single source file.

Another risk is thinking too small. You should be very careful about using small eight bit micro-controllers unless you are developing a product that will be produced in quantities of thousands per year and every penny of component costs is critical. There are so many low cost low power 16 bit and 32 bit processors out there that there’s no excuse for using under-powered processors for short-run or one-off embedded systems. Some examples are the ARM families, TI’s 403, Atmel’s 32 bit AVR, Frescale’s ColdFire, and Analog Devices’ Blackfin. Most of these processors have wide support from a number of software tool vendors as well as GNU open source tool support.

The tradeoffs in choosing the best processor for a product are more complex than the component costs. Engineering is often the art of compromise and choosing the most cost effecting processor is a compromise between component prices and development costs.

I goofed yesterday on one of the code examples as one reader, Michael, pointed out.  I was trying to show how to define a hardware register as a cosnt pointer to volatile data and I wrote:

static const uint8_t* volatile dataOut = 0x20000010;

which is a volatile pointer to const data.  This may be useful in some bizzarre application but it’s certainly not what I was trying to do.  Here is what I meant to do:

static volatile uint8_t* const dataOut = 0x20000010;

Fortunately, I had the code correct in the driver I’ve been working on and I hope I didn’t confuse anyone.  Actually if it had been real code instead of a web page, I’m sure the compiler would have complained the first time I tried to write to the register.  That’s one reason to have all of these qualifiers on your data.  Error messages are a great help, especially when you’ve been (trying to) code for eighteen hours.

Thanks for pointing this out Michael.

By convention, at this point in time you can assume that a char in C is 8 bits, an int or a long is 32 bits and a short is 16 bits but not always. That can be a problem in embedded system programming since we often need to know the exact size of a variable. We also tend to use unsigned variables a lot and typing out unsigned gets tiresome. So you often see people inventing their own coding conventions such as ubyte, ushort, and ulong, etc. I work on code every day that may use two or three different conventions depending on who worked on the code last.

There is a standard way of expression variable sizes and whether they are signed or unsigned in the GNU and POSIX worlds. It is to use the header file stdint.h which defines a number of types of specific sizes: int8_t, uint8_t, int16_t uint16_t, int32_t, uint32_t. The stdint.h file is customized for the processor that you are working with so you can depend on the sizes. This header file also specifies things like the minimum and maximum values that can be represented by a type such as INT32_MAX, UINT32_MAX, and INT32_MIN, etc. All of this is defined in the man page for stdint.h if you are using Linux or Unix.

Sometimes you don’t really care about the exact width of a variable but you want to use a data width of some minimum size that is the most efficient for your processor. Stdint.h provides representations for that too, like: uint_fast8_t and int_fast32_t. There is also a definition for the widest types supported: int_max_t and unit_max_t. You could find out how many bytes wide these are by using sizeof(int_max_t) for example.

stddef.h is another useful header file. It defines things like NULL (though according to Stroustrup you should just use 0 for null pointers) and size_t. size_t is defined as the type returned by the sizeof() function. size_t is useful for times when you don’t really care about the size of a variable, as long as is large enough to represent the size of a data object in bytes. For example, suppose you want the binary inverse of every byte in and array:

for(size_t i = 0; i < sizeof(array); i++)
{
array[i] = ~array[i];
}

Finally, do you know what this represents?

static const uint8_t* volatile dataOut = 0x20000010;

This is a const pointer to a volatile byte at location 0×20000010. A hardware register.

You want a const pointer because the hardware registers shouldn’t be moving around. Declaring the pointer is const allows the compiler to catch you if you forget to dereference the pointer when writing to the location.

The value stored at location 0×20000010 could change on its own without the software changing it. Volatile tells the compiler not to optimize away any reads or writes to this location. For example if you only ever write to the register and never read it, the compiler might remove what seems to be unnecessary writes to this location. The optimizer might completely remove the variable from the object code. Volatile lets the compiler know that this location is special.

I’ve been doing a bit of work lately on a project that uses the Analog Devices Blackfin processor. The application is very ‘net-centric so we decided to use Linux, specifically uCLinux for the operating system. Of course this means that our development tools are the GNU compiler collection and all of its related applications.

Often, one of the hardest parts of a project like this is getting Linux up and running on your target hardware. The Blackfin uCLinux web page is very well organized and has a lot of good information. I downloaded the tools and the Linux distribution from their site and had Linux up and running on a development board in a matter of hours.

One thing that is an absolute necessity is to have a JTAG interface for programming FLASH through the processor. Once you have the boot program running you can program the FLASH using commands that the bootstrap provides but you need to get the bootstrap in FLASH first. An Austrian company, Blue Technix sells a USB JTAG ICE for the Blackfin called the Ice bear. It’s about $320 (US) and since they are in Austria, it could take a couple of weeks to get one so you need to plan ahead. The ICE worked just as advertised and was critical to the success of my project. Blue Technix also has some low cost eval boards too but given the Euro/Dollar exchange rate you might do better getting something from Analog Devices or one of their distributors.

As I was saying, the hard part is often getting Linux up on your own hardware. My target had some fundamental differences from the Eval board that I started working with. Like most micro controllers, the Blackfin has a bunch of multi-function pins. It also has two UARTS. Well our design used UART0’s data out pin as a software controlled hard reset. The other fundamental difference was that we were using SPI serial FLASH instead of the parallel NAND FLASH used on the eval board.

Porting the bootstrap went pretty smoothly. After about a days work, I had the bootstrap working. But it was different with the Linux kernel. I disabled UART0 and enabled UART1 using make menuconfig. But every time the board started to boot – bam! It reset almost immediately. I knew that somewhere the function for that pin was being set up to be UART0’s output. It took about three days of searching the code but I finally found it. The I/O pin setup for both UART0 and UART1 was hard coded deep in the init code. After I fixed that, I could boot Linux using TFTP.

The next step was getting Linux to work out of the SPI FLASH. There are two approaches to this. You can create a complete compressed file system image for a RAM disk system. Then you have the boot program copy the image from FLASH to RAM, decompress it and go. This approach is very simple to configure because it’s the stock build that comes with the distribution. The downside is that you don’t have any non-volatile storage without jumping through hoops.

The second approach took me a couple of more days to get running. You build a file system for FLASH and a kernel image. There’s more to configure to get this to work. One thing you need to keep in mind is that you can’t use any loadable kernel modules to access the file system since you need the file system to load them. The nice part about this is that now you have a file system that’s almost like a disk. If you make changes to your application you can use FTP to load them on the target. You can also use the system logger to log to syslog (they actually call the log /var/log/messages).

Throughout this whole process I was able to get very good support from their web site and forums. If an answer wasn’t in the documentation wiki, I would most likely find it by searching the forums. When I posted questions to the forum, I often had an answer within an hour or two. The documentation and support were better than many commercial RTOSs and cross-compilers that I’ve used.

What really amazed me was how well the tools all work. I wrote a pretty complex application in C++ using many of the C++ library classes, Pthreads, sockets, and all kinds of system facilities. I got it all working on my desktop Linux machine. Then I just recompiled my application using the Blackfin version of the tools, loaded it on my Blackfin uCLinux system and it just worked.

The one thing though that you have to remember is that if you’re messing with the kernel then according to the GPL, anyone who buys your product is entitled to the source code. Then they’re also allowed to re-distribute that code. If you’re working with custom hardware then you’re going to be messing with the kernel. I don’t think that this requirement ever hurt sales of the Linksys 54G routers though. If the project is for a government agency, they may require full source code anyway.

Some things that help make a normal life pleasant can get to be distractions when you’re way behind schedule on a project. Nerds are legendary for ignoring these distractions when a technical challenge requires their full attention. Details like hygiene and nutrition are the first casualties in battles against bugs and deadlines. It’s really hard to be fresh smelling and perky looking when you’re in the middle of marathon systems integration problems and have been working for eighteen hours straight.

Over the last couple of weeks, I’ve really noticed a decline in my eating habits. I usually try to prepare my food from fresh unadulterated ingredients; lots of vegetables, beans, salad and grains and a good bit of meat too. But I cook from scratch and watch the carbs and fat. That is until I hit systems integration.

I started out last week with salads for lunch and home made chili for dinner. When the chili and the lettuce ran out, I switched to carry out food. I tried sticking to wholesome stuff like the local kabob place, easy on the rice and Chipotle which is quite healthy and tasty without the rice and tortillas.

Then I started eating at odd hours, late at night or very early in the morning. I switched to the diet of the legendary first generation of hackers at MIT, like Richard Stallman. I started alternating between Chinese carry out and pizza washed down with lots of Coke.

I knew I hit bottom this morning. Taco Bell is open late around here. They call the time between midnight and two AM, “The Forth Meal.” I found myself driving to the Taco Bell to get there before closing so I could get mine. A few hours later I was at McDonalds’ getting a sausage, egg, and cheese McGriddle and another Coke.

I’m lost now and I’ll admit it. I’m not eating again until I can cook something for myself. Right after a shower and a twelve hour nap.