A certain degree of frustration

Now on iteration…mmmmm….four, I think, of pushing IP v6 into CygNet, and am making invisible progress, I’d say.  That’s the kind where you seem to be doing a lot and accomplishing little of substance.  But, it leads to good situations like today, where I realized trying to keep the v4 infrastructure and shovel in the v6 isn’t going to be a good idea, even for a prototype.

Part of the problem is that CygNet is very nearly perfect in its architecture, in the sense that it works without requiring enormous computational overhead, and that it’s been pruned by expert code gardeners wielding sharp and precise tools.  There’s no fat, and there’s the source of one of my problems.  In many systems it’s possible to exploit of inherent inefficiencies in objects/classes/entities.  We don’t have those, in any meaningful way.

The other issue, and the one that’s been stymieing me is that the network world has changed.  When CygNet rolled out, most non-gateway(router) machines had a single NIC and a single address, which rarely changed.  Today, though, it’s hard to even define what “computer” or “host” means.  The research server has six IP addresses scattered across 3 physical NICs and 3 virtual (for VMware).  It has inward and outward facing names/addresses (frith.research.dom and research.cygnetscada.com).

It has at least 4 different copies (in 2 versions) of CygNet running on it: one on the main OS and the others on simultaneously executing VMs.

Now, you tell me, does it make sense to say of CygNet, “It’s running on the research server.”?  Well, yes, it does, if you’re willing to accept that “server” no longer means a physical computer, but instead refers to a logical locus of execution.  This is, I should point out, part of the premise of cloud computing.  Unfortunately for us, simply pushing CygNet onto v6 without any other changes wouldn’t accomplish much, in my current opinion.

And then there’s NAT: we already have problems with NAT, because NAT is the Wrong Thing, poorly implemented.  It broke the fundamental design of end-to-end connectivity upon which the ‘net (and CygNet) was predicated.

It’s not that we should stop pushing v6 into the product, but simply swapping out the v4 for v6 addresses isn’t going to teach us anything, and will cost us in the long run because we’ll need to come to terms with the v6 world and the coming distributed environments.

Architecture Notes

Imagine I wanted to run Cygnipede services on the Research server.  What I’m saying by this is that there is a business domain (Research) in which computing resources are [probably] available to execute “business” (logical) transactional activities.

Consequence:  rather than being host centric (and literally tied to a single IP address), a v6 CygNet could use a scope id to define a logical set of addresses from which services could be requested.  Changes to CygNet, beginning at the bottom (CAddressBytes()), the v6 changes need to accommodate a logical-locus-of-execution model.  Entities which currently use CAddressBytes() as a way to access services will need to move to using a new entity like “CLogicalDomain())”, which itself is a list (ideally dynamic) of CServiceHosts which are lists of 1 or more service/host names and corresponding addresses.  Note that IP is clever in that you can have multiple addresses associated with multiple names.

More later after I try some more mods to CAddressBytes().


A Certain Degree of Frustration, Part II

Scene:  the following day, during which our protagonist has an insight that might perhaps not be as stupid as it initially appears


A few weeks ago, I asked Jeff & Derek, “Why is CAddressBytes() a CByteVector()?”

The answer was, “It seemed like and was a good idea at the time.”  They’re fast, they’re lightweight.  The name is a bit misleading, since it holds more than just the address octets.  It also knows about the port number. 

For v6, from an architectural POV, it will also need to know about the ADDRESS_FAMILY (AF_INET, AF_INET6, …), the flow and the scope id (if it’s an IPv6 address).  So…is there some reason it shouldn’t just be an instance of something like sockaddr_storage?

How about an addrinfo?  Answer:  bad idea, because it uses a sockaddr, and Microsoft explicitly recommends the use of sockaddr_storage instead.  Is it my imagination, or does it look as though there were two different but simultaneous approaches to dealing with IPv6?  First the Berkeley/RFC people came up with stuff like sockaddr_in6, and then the sockaddr_storage class to rule them all.

A sockaddr_storage already knows about alignment and has enough space to store any kind of “actual” (v4/v6") address we might use.  It’s 128 bytes

The Short Version:

CAddressBytes may have reached the end of its useful life insofar (and only insofar) as IPv6 is concerned.  I know there are higher-level entities in the current framework that aggregate the “IP address”, but perhaps we should look at moving towards a more complex mechanism for representing hosts. 

My shot at the v2 IPv6 implementation (Now with CLOUDS!) would be for CAddressBytes to be a smart, possibly stateful object that resembles a container of identifiers capable of being resolved to physical hosts in some fashion.  Simplest case:  create a turbo sockaddr_storage named CSockAddr_Storage and make CAddressBytes a container of them.  IOW, promote it to being more than just a physical address.  OTOH, there may well be higher level abstractions in the current framework where this could happen with greater ease and/or more sense.  I.e.  ARS could be combined with a replicator, federation manager, DNS, and LDAP (including Microsoft) to take care of this.

Huh, I think I just reinvented the VMS logical name table.  Steal from the best, I’d say, but someone already said it.  Oh, wait, I mean, “Yes, that’s what I say.”

In the meantime, I’m going to push ahead on converting CAddressBytes to use sockaddr_storage

Gumption Traps

I enjoy working here more than any place I can think of, but that doesn’t mean it’s always hunky dory.  The last couple of months have been filled with gumption traps.  I first read about these in Zen and the Art of Motorcycle Maintenance.  If you haven’t read it, it’s an allegory about the nature of fitness and the design of functionality disguised as a personal account of a motorcycle, all couched in a modern version of a Platonic dialogue. 


Here are the important things to know for this blog entry.  All quotations are from the book unless otherwise noted.


Quality—you know what it is, yet you don’t know what it is. But that’s self-contradictory. But some things are better than others, that is, they have more quality. But when you try to say what the quality is, apart from the things that have it, it all goes poof! There’s nothing to talk about. But if you can’t say what Quality is, how do you know what it is, or how do you know that it even exists? If no one knows what it is, then for all practical purposes it doesn’t exist at all. But for all practical purposes it really does exist. What else are the grades based on? Why else would people pay fortunes for some things and throw others in the trash pile? Obviously some things are better than others—but what’s the "betterness"? — So round and round you go, spinning mental wheels and nowhere finding anyplace to get traction. What the hell is Quality? What is it?


I like the word gumption because it’s so homely and so forlorn that it looks out of style it looks as though it needs a friend and isn’t likely to reject anyone who comes along. It’s an old Scottish word, once used a lot by pioneers, but which, like "kin," seems to have all but dropped out of use. I like it also because it describes exactly what happens to someone who connects with Quality. He gets filled with gumption.

The Greeks called it enthousiasmos, the root of "enthusiasm." which means literally "filled with theos," or God, or Quality. See how that fits?

A person filled with gumption doesn’t sit around dissipating and stewing about things. He’s at the front of the train of his own awareness, watching to see what’s up the track and meeting it when it comes. That’s gumption.

Gumption Traps

But there’s another kind of detail that no shop manual goes into but that is common to all machines and can be given here. This is the detail of the Quality relationship, the gumption relationship, between the machine and the mechanic, which is just as intricate as the machine itself. Throughout the process of fixing the machine things always come up, low-quality things, from a dusted knuckle to an accidentally ruined "irreplaceable" assembly. These drain off gumption, destroy enthusiasm and leave you so discouraged you want to forget the whole business. I call these things "gumption traps."

The Types of traps

There are two main types of traps:

Externally-originated setbacks – the best example is managing to leave out a step in a complex series of instructions.  This will always be a required item that’s not in the critical-path, otherwise you wouldn’t be able to skip it.  My personal favorite in this regard is gaskets for mechanical work, very small cables and DIP switches in computer hardware, and overlooking a version-critical sub-component in software.

Internally-originated “hang ups”.  I’m quoting Pirsig here, and I have to stop and think this one out every time I read it.  Example would be the inability to separate yourself (your ego) from your conception of both the problem and how you’re trying to solve it.  Oftentimes, when you’re hammering something together, you won’t and/or can’t ask, “Do I need a hammer?  Do I even need a tool?  Should this job even be done?”  This can also be the result of applying one problem’s “solution mindset” to another apparently similar problem.

The CygNet Motorcycle

The first [problem] is that a motorcycle, so described, is almost impossible to understand unless you already know how one works. The immediate surface impressions that are essential for primary understanding are gone. Only the underlying form is left.

Like airplanes, motorcycles tend to be composed only of the parts required to provide two functions:  acceleration (positive and negative) and operator affordances (seating and controls).  Everything else is strictly unnecessary.  As such, it’s next to impossible to build a causal model based on overall impressions.  There’s nothing a motorcycle is like except a motorcycle.

You need to understand it before you can understand it.  That’s the hardest part of CygNet for me,is  that it’s spent so many years being turned into a perfectly functional object that it’s not like anything except itself.  Since there’s a natural tendency, for me at least, to casually build narrative model shells around other people’s tools and then build a causal model, I’ve tried to do the same with CygNet and it’s been a constant struggle to undo this approach as soon as I subconsciously veer into it.

I’ve spent a couple years building, tearing down, and rebuilding my understanding of what CygNet is and how it functions.  I feel as though I understand it as well as anyone, although certainly not as deeply, and that’s where the gumption traps come in. 

Why is this important?

BECAUSE – IPv6 makes bunny cry


Instructions, tools, and products shouldn’t make people feel stupid, which I might carefully restate as, “It shouldn’t make thoughtful and smart customers with paid-up licenses feel stupid.”

Primary sources of my IPv6 gumption traps

  • It’s like IPv4, except when it’s not. 
  • They’ve obsoleted “standards”, e.g. 6TO4->ISATAP->RANGER (real soon now) on the fly.  Each of these is a way to tunnel v6 over v4.
  • Vendors are waiting for every other vendor to commit.
    Customer – “Does your device support IPv6?”
    Sales rep – “It depends…”
    C – “Depends on what?”
    SR – “What you mean by ‘supports’?”
    C – <facepalms>

    They’re serious.  At one level, Ethernet packets will get passed around by routers, but at another level, IP packets might get routed and might not.  If you’re using what appears to be a “flat” ethernet link, your packets will get where they’re going, regardless of IP version.  OTOH, IP packet routing these days on everything but carrier-grade equipment, means IPv4 stuff gets routed, which leaves you with the option of tunneling, using carrier-grade equipment, or using open-source stuff like DD-WRT and even there the support is tricky, marginal, and incorrectly documented.

  • ISPs are waiting for demand.  Customers are waiting for ISPs.  When they ask, and I have, we’re told, “No one wants it.”  As bad as that is, the reality is worse, because simply trying to find someone who knows what you’re asking about is difficult.  It might be different for corporate customers, but I doubt it, because my office uses business-class Time Warner, and while their sales staff and support droids are very helpful, even for obscure routing and IPv4 issues, IPv6 isn’t on their radar.
  • People in “developed” countries are in much less danger of IPv4 address space exhaustion than places like China and India, who seem to be driving a lot of the demand.  Training dollars for IT personnel have (in my experience) always been the lowest priority in any “normal” corporation, which means that networking personnel who say, “We need to go learn about a protocol that we don’t use and which isn’t fully baked in the market and whose benefits we’re incapable of describing in business terms,” are going to get laughed out of the budget meetings.  Result:  lots of us know a little, and the people who know a lot are either in hiding or categorically incapable of writing a declarative English sentence.
  • I think I understand IPv4.  That’s not a typo.  IPv4 is like IPv6 the way a bobcat is like a small tiger.  Superficially sleek and attractive, but not really user friendly, and subject to more-or-less random fits of violence.  With a bobcat, you’re hurt.  With a tiger, even a small one, you’re supper.
  • Documentation isn’t.  A lot of the so-called documentation is descriptions of standards.  Even with my subscription to Safari books, I’m finding it rough going.  There’s a crying need here for some books based on actual implementations, and that include lots and lots and LOTS of examples.


More on specific technical issues in my next post on this subject



You Should Never Make Normative Statements

Finally, something works

It’s been a crappy few months, frankly.  One of the problems with “research” is that by normal standards, most projects going to be “failures” in the sense that you fail to achieve something.  In the scientific and commercial sense, though, it’s not really a failure if and only if you learn something from your actions.  This blog is here for precisely that reason, but for a couple months now I’ve been running into brick walls, several of my own construction, but at least we now know where a lot of walls are in the pitch-black Dungeon of IPv6.

Stephen H has been the other test monkey on the IPv6 infrastructure, and he’s gotten to experience vendors who don’t know what they’re talking about, vendors who ship the wrong things in the wrong order, and researchers (heh) who have eyes bigger than their stomachs, so to speak.

From the research point of view, the IPv6 problem is one coin with two sides:

  • Is IPv6 practical, useful, and economically viable?
  • Can we make our products IPv6 compatible in a cost-effective manner?
    • If so, how?

I’ve been been flipping from one question to the other as the frustration builds up to a point of intolerance.  I couldn’t get IPv6 infrastructure working so I did coding.  I couldn’t get coding to work so I did infrastructure.  It’s a tunnel with two ends, both of them dark to date, but this week I’ve seen a we spark at one end, and I’m reasonably sure it’s not an oncoming train.

See my previous post on gumption traps, because that’s what the last three months have been for me, one after another.

The Fail Saga to date

  • Two wrong ways to switch out the underlying CAddressBytes class – nothing stupid here, just experimentation, but since this particular style of work needs to result in running code, making a mistake, even if you learn from it, is expensive and annoying. 
  • A couple weeks of probably-not-so-sucky coding undone by a stupid VSS mistake (mine).  Originally I’d been using the development VSS, and wanted read-only access, but there was some problem with setting it up that way so I ended up with a normal account, which means I could write to it.  And I did.  <mentally insert emoticon for stupid-face here>  I have a completely separate VSS repository for Research projects, which I created a couple months ago when we started on the IPv6 stuff, and I’d replicated the entire dev VSS.  No problem.  Set up Visual Studio to talk to my research repository.

    Here’s where the stupid (aka learning) part comes in.  If you have multiple repositories, one of them is the default.  At some point, somehow, I managed to re-set my default back to the development VSS.  On top of that, VSS hasn’t been my native source control mechanism for a long time so I’m not used to saving projects back to it, and I did work on the SupportNet library without actually managing to put it in the repository.  Eventually I noticed I hadn’t been saving it, and saved it to VSS.  The wrong VSS.  <bad word><worse word>.  In a near-panic, I restored, I deleted, I reverted, I saved, and at some point, I lost the two files I’d been working on.  <more bad words>.  Fortunately I was able to revert my changes to the development VSS and then Derek did some magic to make it read-only for me, and my adventure was over. 

  • networks that mostly don’t – ongoing problems with VPN connections dropping in speed from 5Mbps to 3Mbps to 1.5Mbps (on a good day).  Some of this was due to upgrades to Windows 7, because Cisco doesn’t support their *good* VPN client software, and the rest of it was just a freakin’ mystery of the ages. 

    Finally, Stephen and Will, sick of my shadow darkening their door, shoved a metal box in a cardboard box and sent it to me saying, “Plug this in and go away.  Please.”  They’re very nice, and I wish I didn’t have to spend so much time bugging them, and this box will do that.  It’s an outer gateway/VPN router for my office network, which allows me to see the SLO offices transparently, without a VPN connection.

  • IPv6 that doesn’t seem to work.  Despite being pretty scrupulous about following various Microsoft documents, I haven’t been able to get IPv6 packets from one machine to another unless they were on the same physical network segment.  Barfy.  IPv6 makes bunny cry, some more.  Being able to send packets locally is pretty meaningless, since no network in existence is actually configured that way.  Very frustrating. 

Fail saga fails, finally

…and then the sun came out.  Admittedly, that meant it was –4 degrees F, but you can’t have everything.  Where would you put it?  In our case, we’d put it in the giantamous (it’s a word, really, I promise) IPv6 address space, and we did.

Note that the Appleton and SLO offices are connected over the public Internet.  I signed up with a “tunnel broker”, a firm who helps provide “tunnels” for routing IPv6-over-IPv4 over the public Internet.  At each end, a daemon pretends to be an IPv6 device and sends/receives IPv6 data by packing it into and unpacking it from IPv4 packets, which themselves are routed normally.  Needful to say, this means everything takes more than twice as long, but connectivity beggars can hardly be choosers.

I have two tunnels: one between SLO<->Santa Barbara, and one between Appleton<->Chicago.  The end points (on the Internet side) are public POPs provided gratis by various vendors/ISPs in an attempt to help foster IPv6 adoption.

Our tunnel broker is SixXS.net – After they approved the routing and rationale for my wanting these tunnels, I set up the virtual network devices (tunnel adapters) on frith’s external IP address and another on one of my development machines here in Appleton, taking care to route proto-41 (ports 41-44 and ports 58-60, actually, which are the IPv6-over-IPv4 canonical ports) to my dev machine.  After a few minutes of hyper-ventilating, it worked.  I could send IPv6 packets from Appleton to SLO and vice-versa, as confirmed by WireShark.

There were problems, though, and we need to keep them in mind.  With the VPN router in place, I have an outer router/firewall (NetGear VPN FVS-338) and an inner router/firewall (DD-WRT on a WRT54G-TM).  That combination didn’t work.  Packets got dropped at the outer routers despite firewall rules saying “pass them to port NN”.  This may have been because we’re having problems getting the outer router to see anything inside the inner router.

Other important things to note: 

  • both of the CygNet tunnel endpoints have fixed IP addresses.  You can do this with dynamic addresses, but it’s unreliable, a pain, and did I mention it’s unreliable.  Don’t do it.
  • this wouldn’t be necessary if we could get IPv6 Internet service, but neither Time Warner Business Cable (Appleton), or <mumble> (SLO) offers it.  In fact, they offer the Catch-22 version.  “No one asks for it so we don’t sell it.”  “We’re asking for it.”  See statement #1.

    It gets worse, in point of fact, because eventually, assuming ISPs switch to IPv6, you’ll need to run IPv4-over-IPv6 using similar tunneling mechanisms, although I suspect this will be a long time coming.  If we’re lucky, we’ll see IPv6 in the next 3-5 years with major ISPs in major markets.  ComCast announced just last summer (2009) that they’re going to be conducting tests of IPv6.

  • Documentation about tunneling technologies is both abundant, incorrect, obtuse, and scarce.  Bad combination.  There are several tunneling technologies in use (IPv6-to-IPv4, ISATAP, and soon Ranger) on MS platforms.  Docs are being written by different groups with differing levels of knowledge and authority.  Very frustrating.

I remember being this frustrated when I was learning about IPv4, after having spent 10 years learning DECnet, both at a personal and a professional level.  I know it will get better, but damn, I feel so stupid right now that it actually gives me headaches.  I’m convinced that IPv6 is the way to go, and I wasn’t [at this time] last year, so I’m feeling less like the Sisyphean router, endlessly pushing infinite TTL packets into a local loop.

<diagram goes here – working on it after realizing I didn’t have Visio installed and debugged – Lynne>


Actually, this is a real Visio diagram using a  stencil I got at http://www.visguy.com/, which also has lots of other useful Visio stuff (articles, stencils, tools).  We (and by “we” I mean “I”)  need (and by “need” I mean “want”) this for CygNet diagrams.  Actually, you know, a CygNet Visio stencil would rule.

And Another Thing

Our IT guys?  Inarguably the best generalist support team I’ve ever worked with anywhere, ever.  I know people who know more about single subjects, and I know people who are nicer (well, I now people who bring me sandwiches, at least), but in all seriousness, I’ve yet to meet any two people with more patience and knowledge.  Better, Stephen is getting a baptism-by-fire on IPv6 problems, which means he’s going to be valuable as a customer resource in all likelihood. Dudes, thanks.

Finally, something works – An update

As part of getting the IPv6 tunnel functioning, I mentioned I’d had to take out the NetGear VPN.  Unfortunately, it’s really needed for the VPN function, so I restored the the full outer-inner router arrangement yesterday. 

The VPN wouldn’t come up.  Stephen poked at it, but didn’t change anything.  Mysteriously, though, a few hours later, not only did the VPN come back up, but the NetGear firewall rules started magically working, meaning we now have a functioning SLO-Appleton IPv6 connection via the IPv4 tunnel.

While I’m able to ping6 SLO from Appleton, the reverse situation is a bit strange, because one of the two firewalls involved is dropping packets.  Now that we’ve demonstrated that it’s possible to get this system going, I’m not going to worry about this for the moment.  The important part was demonstrating that it’s possible to get IPv6 packets from one place to another over a non-trivial network path.


Here’s a screenshot from a Wireshark session on frith.vsi.dom showing the pinging from Appleton



Appleton pinging SLO



SLO –> Appleton (Chicago POP, actually)



Accessing Google IPv6 site.  Big deal?  Check out the URL


IMPORTANT NOTEUnlike IPv4 addresses, which can be used directly as URIs in Firefox and Internet Exploder, IPv6 addresses must be enclosed in square brackets: http://[2001:4860:800b::93]/  (this is one of the resolved addresses for IPv6.google.com).  Owing to some of the vagaries of the DNS configuration in Appleton, IPv6 names aren’t resolving correctly, but the packets are coming and going correctly.  Here it is from Appleton.




Yay!  Two things work!  Sure, it’s mysterious that routes that didn’t work yesterday are working now, but I’m not one to punch a gift horse in the mouth – it might bite me.

IPv6 – various

Diameter – http://en.wikipedia.org/wiki/Diameter_%28protocol%29

Upgrade from RADIUS

The name is a pun on the RADIUS protocol, which is the predecessor (a diameter is twice the radius). Diameter is not directly backwards compatible, but provides an upgrade path for RADIUS. The main differences are as follows:

  • Reliable transport protocols (TCP or SCTP, not UDP)
  • Network or transport layer security (IPsec or TLS)
  • Transition support for RADIUS, although Diameter is not fully compatible with RADIUS
  • Larger address space for attribute-value pairs (AVPs) and identifiers (32 bits instead of 8 bits)
  • Client-server protocol, with exception of supporting some server-initiated messages as well
  • Both stateful and stateless models can be used
  • Dynamic discovery of peers (using DNS SRV and NAPTR)
  • Capability negotiation
  • Supports application layer acknowledgements, defines failover methods and state machines (RFC 3539)
  • Error notification
  • Better roaming support
  • More easily extended; new commands and attributes can be defined
  • Aligned on 32-bit boundaries
  • Basic support for user-sessions and accounting


Stream Control Transmission Protocol


SCTP in battlefield networks

– 2001

(Citations: 7)

Phillip T. Conrad, Gerard J. Heinz, Armando L. Caro, Paul D. Amer, John Fiore

ABSTRACT The Stream Control Transmission Protocol (SCTP) is a new Internet standards track transport layer protocol. SCTP was originally designed to transport PSTN signaling messages over IP networks, but is also capable of serving as a general purpose transport protocol. As such, SCTP provides an alternative that may be better able to satisfy the requirements of future battlefield networks than the traditional transport protocols, TCP and UDP. Unlike traditional transport protocols, SCTP allows multiple streams of messages within a single connection (or, in SCTP terminology, a single association). As the results in this paper show, this ability is particularly helpful in reducing latency for streaming,multimedia in high loss environments. SCTP also provides features for multi-homing that may be helpful in highmobility environments,and additional security against denial-of-service attacks based on SYN flooding.

Sensor Data Collection Through Gateways in a Highly Mobile Mesh Network

(Citations: 2)

Andrew Jenkins, Daniel Henkel, Timothy X Brown

Abstract—Widely distributed sensors must discover paths back to data collection points possibly through sparsely connected and mobile networks. Current addressing and service discovery schemes in mobile networks are not well-suited to multihop disconnected networks. This paper describes an architecture and protocol for sensor data collection through highly mobile ad- hoc network (MANET) that may never experience end-to-end connectivity. Special gateway nodes are described which are responsible for intelligently routing messages to their intended destination(s). These gateway nodes qualify their links and announce their status to the MANET, a simple approach to service discovery that is effective in this implementation. The protocol is implemented and tested in a laboratory and outdoor environment.

IPv6 SCADA system for energy use monitoring

generic sensor node – http://www.archrock.com/products/ipsensor_node.php


Arch Rock products and technology enable new wireless sensor applications to be developed in familiar environments, delivered on a wide range of embedded hardware platforms, and integrated with the enterprise. Drawing together three major advancements: industry standard IP-based, multi-hop mesh networking, powerful embedded systems and flexible web services computing architectures, Arch Rock has not only created the foundation to network the physical world, but has now delivered a comprehensive product suite based upon that foundation – Arch Rock Energy Optimizer™ – to help meet the challenge of Green House Gas reduction.

I can’t find anything that says they’re actually producing and selling this technology, which seems odd for a company that’s gotten $15M in venture funding.

They’re doing IPv6 over Zigbee, basically, which is what 802.15.4 is