Computing in the real world, with giant spiders

This is exactly the sort of thing that many programmers wouldn’t really think about, that is, spiders interfering with data capture.  In this case, it’s just video from atop a hotel in Appleton, WI.  SCADA developers, on the other hand, would probably have already accounted for mammals, insects, and the occasional rain of frogs in their list of possible exceptions.

Giant Spider Menaces Appleton, Wisconsin

Advertisements

World’s cheapest SCADA device: source code

This is the source code (as of May 27, 2008) for the Parallax BasicStamp Recording light meter.  I apologize for the formatting, but in the exciting new world of Office 2007, there is *literally* no way to edit the HTML that Google or Help know about, so I can’t use the  <pre></pre> tag.

' {$STAMP BS2}
' {$PBASIC 2.5}

index VAR Nib

wStrength VAR Word
wCount VAR Word

soundTime VAR Word
bCmd VAR Byte

'  generic tmp variable
iTmp VAR Word
wTmp VAR Word

''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
'  '
' ''' Number of values to store in EEPROM
'  '
'
wMaxValuesToStore VAR Word
wMaxValuesToStore = 200
wCurrEntvalueLocation VAR Word
wStoredValuesCount VAR Word
wStoredValuescount = 0

''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
'  '
' ''' How long to wait (milliseconds) for command input before resuming the main loop
'  '
'
wCommandDelay VAR Word
wCommandDelay = 3000

OUTH = %00000000
DIRH = %11111111

UNUSED_DATA CON 65535
'
'  Commands that can be sent to the application
'
CMD_RESET CON 0
CMD_DUMP_LAST_VALUE CON 1
CMD_DUMP_ALL_VALUES CON 2
CMD_SLEEP CON 3
'  returns a fixed length buffer with known contents
CMD_DIAG CON 4
CMD_NOOP CON 254

bCmd = CMD_NOOP

''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
'   ''
' ''''''
' ''''''
'   ''
'
GOSUB Reset

DO
  GOSUB getRCTime
  GOSUB getCommand
  GOSUB Delay
  GOSUB updateDisplay
LOOP

''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''

''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
'
Reset:
  DEBUG "ResetStart",CR

  '  Make sure history memory ring buffer is set to a value that lets us later distinguish it from zero (0)
  DEBUG "Starting",CR
  DEBUG "Will save ", DEC wMaxValuesToStore, " values in EEPROM",CR

  DEBUG "Initializing history ring buffer",CR
  FOR iTmp = 0 TO ((wMaxValuesToStore -1) * 2) STEP 2
    WRITE iTmp, Word UNUSED_DATA
    'DEBUG DEC iTmp, "."
  NEXT
  DEBUG "Finished initializing history ring buffer",CR
  'DEBUG CLS, HOME
  wCount = 0
  wStrength = 0
  FOR iTmp = 0 TO ((wMaxValuesToStore -1) * 2) STEP 2
    WRITE iTmp, Word 0
  NEXT
  wStoredValuesCount = 0
  DEBUG "ResetFinish",CR
  RETURN

''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
'
getCommand:

  'DEBUG "cmd> "
'  SERIN 16,  84,getCommandBadData, 5000,getCommandTimeout, [WAITSTR cmdBuffer9]
  SERIN 16,  84, wCommandDelay, getCommandTimeout, [DEC1 bCmd]
  'SERIN 16,  84, [DEC bCmd]
  'DEBUG DEC bCmd
  GOSUB processCommand
  RETURN

''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
'
processCommand:
  'DEBUG "Got command: ", DEC bCmd, CR

  SELECT bCmd

    '  reset
    CASE CMD_RESET
      GOSUB Reset

    '  send current value
    CASE CMD_DUMP_LAST_VALUE
      DEBUG "", DEC5 wStrength, "", CR

    '  send last N values
    CASE CMD_DUMP_ALL_VALUES
'      DEBUG CLS, HOME
      DEBUG "" ,CR
      DEBUG TAB, "", DEC wMaxValuesToStore, "",CR
      DEBUG TAB, "", DEC wStoredValuesCount, "", CR
      DEBUG TAB, "", DEC (wCurrentValueLocation / 2), "",CR
      DEBUG TAB, "", DEC wCommandDelay, "", CR

'      FOR iTmp = 0 TO ((wMaxValuesToStore -1) * 2) STEP 2
      FOR iTmp = 0 TO ((wStoredValuesCount *2 ) + 1) STEP 2
        READ iTmp, Word wTmp
        IF wTmp = UNUSED_DATA THEN EXIT
        DEBUG TAB,"", CR
        DEBUG TAB, TAB,"", DEC iTmp, "", CR
        DEBUG TAB, "", CR
        wtmp = 0

      NEXT
      DEBUG "", CR

    CASE CMD_SLEEP
      DEBUG "Sleeping", CR
      SLEEP 1

    CASE CMD_DIAG
      DEBUG "deadbeef0",CR

    CASE CMD_NOOP
      'DEBUG DEC wCurrentValueLocation,"->",DEC wStrength,CR

      'DEBUG "Received NOOP command", CR
'      NAP 0
      DEBUG 0, CR

    CASE ELSE
      'DEBUG "Received unknown command", CR

    ENDSELECT

  RETURN

''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
'
getCommandBadData:
  'DEBUG "Got bad command", CR
  DEBUG "CommandError", CR
  RETURN

''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
'
getCommandTimeout:
  'DEBUG "No command received" ,CR
'     DEBUG DEC wCurrentValueLocation,"->",DEC wStrength,CR
      'DEBUG DEC wStrength,CR

  RETURN

''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
'
getRCTime:
  '  These next 3 lines are the actual application.  How long is it taking for a charged cap to drop to 0?
  HIGH 2   '  arbitrary short time to charge the cap (from the training example)
  PAUSE 3  '  arbitrary short time to let it "leak" down to zero (from the training example)
  RCTIME 2, 1, wStrength

  soundTime = wStrength * 3

  'DEBUG  "Raw strength: ", DEC wStrength ,CR
  '22wStrength = wStrength * wStrength

  wCurrentValueLocation = (wcount // wMaxValuesToStore) * 2
  WRITE wCurrentValueLocation, Word wStrength
  wStoredValuesCount = (wStoredValuesCount + 1) MAX wMaxValuesToStore

  wCount = wCount + 1

'  DEBUG "", DEC5 wStrength, "", CR

' DEBUG CLS, DEC5 sStrength, " soundTime: ", DEC5 soundTime
' FREQOUT 5,  soundTime MAX 250 , ABS (soundTime - 2640)

  RETURN

''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
'
Delay:
  PAUSE  wStrength MAX 250
  RETURN

''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
'
updateDisplay:
  IF index = 6 THEN index = 0
  LOOKUP index, [ %01000000,
    %10000000,
    %00000100,
    %00000010,
    %00000001,
    %00100000 ], OUTH

  index = index + 1
  RETURN

The world’s cheapest SCADA device?

As part of my learning about CygNet, SCADA, and the exciting world of Real World Data, I decided to build a SCADA-enabled device from scratch.

I wanted something that could be reconfigured to support any sort of sensor, was very small, and that didn’t require soldering.  A little bit of research (and by research I mean shopping) turned up the venerable BasicStamp: http://www.parallax.com/Default.aspx?tabid=295

It’s just about the simplest embedded system possible.  A processor with a built-in Basic (the language) interpreter, 16 I/O lines, a serial port, and a small breadboard for components.

For $80, you can pick up a version at Radio Shack that includes a complete training program.   (http://www.radioshack.com/product/index.jsp?productId=2117994&cp=&sr=1&origkw=parallax&kw=parallax&parentPage=search), or you can buy just the boards for anywhere from $10-$40 on eBay or the Parallax store.

I spent a little over a week going through the training and ended up with a photocell-based light meter that output readings on the serial port, and stored a small number of recent readings in its onboard EEPROM.  I hadn’t read a circuit diagram in about 30 years, so that was fun.

Useful tidbit:  the web has lots of resistor value calculators

ParallaxBasicStampSCADA002

Doug had already created a serial-device EIE for the Davis Vantage weather station, so it was a matter of hours for him to add a new “general” message that would allow communications with arbitrary devices

Protocol

The protocol is blindingly simple. Send a number (really a single byte with the ASCII code for a numeral) from 0-n, and the device sends back data and/or takes some action. The current command set:

  • 0 (48/0x30) – reset the device. Clear the ring buffer and all counters and restart the program
  • 1 (49/0x31) – return the most recent reading
  • 2 (50/0x32) – return all readings
  • 3 (51/0x33) – sleep (turn off power to the device for N seconds, where N is currently hardcoded)
  • 4 (52/0x34 – return a known diagnostic string (“deadbeef”). Why that word? It’s the default register & memory settings for the RS/6000 plus it’s all in hex. I didn’t say the commands made sense.
  • 254 – used in early testing

Example: Send “1” -> Receive “NNNNN” where N is a numeral

There is no way I could have gotten this running without help from everyone, but especially Doug, Pete, Sheri, and Derek.

World’s Cheapest SCADA Device: Conclusions

For the sake of brevity, I’m going to refer to this type of device as a General Purpose SCADA Device, i.e. GPSD.  No, wait, that’s already taken.  How about “GPD2”, for General Purpose Digital Device (the “2” is silent)?

What I learned:

  • OMFG this is painful.  Hooking up a random new device is hard.  My hat is off to the EIE team.
  • Having control of both ends of the conversation (app-level protocol) makes things much easier.
  • As a software person, did I mention the pain of talking to the real world?  Gahh….really, why is it this hard?
  • It was fun.  Honestly, I’m someone who enjoys their work, but this was just ridiculously interesting.  Jeff says it never goes away, that excitement when software makes something happen in the Real World.
  • Cheap.  This was insanely cheap.  I was using 2 out of 16 general-purpose I/O lines.  Parallax actually sells embeddable OEM versions of some of their more capable chips.  Their latest product is a single-chip 8-core machine programmable using real languages and tools.
  • Despite the feeling of being frustrated, the reality is that the majority of my time was spent learning about CygNet and relearning some elementary electronics.  The vast majority of time was spent doing two things:
    • Designing the protocol – my original design just streamed data out the serial port *and* responded to commands.  Since the original device was a beeping, science-fictiony kind of meter, I changed it to a request-response architecture (at Derek’s urging.)
    • Learning about the EIE stuff.  Doug was out when I initially tried to get the GenRawMsg working, but it only took Sheri (and me) a couple of hours to get it working.  I would also say that this is a testament to the fundamental sanity of the architecture
  • The market for GPD technology is insane.  Pricing makes no sense.  There are hundreds, maybe thousands of supplier of hardware, software, and services.  It’s like walking down 47th St. in New York, where all the random grey-market electronics vendors ply their trade.  Everyone sells everything for less than everyone else, and yet people end up spending lots of money.
    I’m assuming that there hasn’t been a shake-out because requirements/needs for IRL (In Real Life) data have been few and far between.  I would expect this market to follow the same patterns of all digital tech markets to date.  In the meantime, the more we know, the more likely it is that we’ll be able to pick some of the protocol/design/hardware winners in advance.
  • Some of the CygNet UIs are hard to use.  I can’t decide if they need to be this hard to use.  I’m thinking specifically of the DEID/datagroup/UIS commands.  Given the extremely high specificity of given devices, and the rarity of hooking up brand new devices, I’d be inclined to give this a pass, especially if we development some generic EIEs that allow protocol definition and specifications using simpler (and as yet undesigned and unimplemented) tools.
Conclusions

I haven’t used existing PLC/RTU’s except the Fisher 503 in training.  As a result, my perspective on SCADA is informed primarily by my experience as a software designer and developer.  It looks to me as though this is another one of those “push them off the cliff” situations where existing manufacturers feel that they’re in no danger of being supplanted by “toy” systems.  This is the exact same argument applied to the telephone, computers, personal computers, automobiles, etc.  I haven’t seen anything to suggest that this is any different, but I’m open to the idea.

There are dozens of protocols, at the very least.  Every time I look one up, I end up diving into a rathole that invariably leads to 1980 and a statement like, “Company Foo, finding existing technologies inadequate to the task of manufacturing anodized, left-handed smoke turners, decided to team up with 4 homeless electrical engineers and design a protocol optimized for that task.”  The result is the hodge-podge of protocols (hardware, software, application) currently in use.  I’m not saying, by any stretch of the imagination, that these are unnecessary, or bad, just that the sheer number is terrifying to someone used to markets where there tend to be only a few dominant architectures.

What I have seen of the protocols and capabilities suggests strongly that these dedicated devices have a very long tail due to their industrial construction and simple nature.  It’s not like they’re going away, certainly not in our market while petroleum costs are high.

At the same time, markets without existing or significant SCADA penetration will be more likely to turn to GPDs based on their lower cost to acquire and program.  More importantly, the utter lack of flexibility in existing designs will push penny-pinching small companies with novel SCADA requirements in the direction of software-based systems, rather than systems limited/constrained by decades-old hardware designs.

Questions

to be answered at a Research Directions meeting.  My responses are in blue.

  • Should I continue to develop this device so that it can serve as a complete, end-to-end demonstration?
    • Yes. In fact, I’d like to switch to a more sophisticated device.  Note that the cost increase for hardware might be as high (hah!) as $120 for a Cubloc device (w/training) that provides “real” features found in existing RTU/PLC, e.g. ladder logic, large numbers of I/O ports, MODBUS, and floating point calculations.
    • http://www.cubloc.com/product/01_01.php – products
    • http://www.cubloc.com/product/01_05.php – starter kits
  • Should we develop a “generic” EIE that supports cry-outs in addition to raw messages in both variable- and fixed-length versions?
    • If we believe (and I do) that the future of SCADA involves devices considerably smarter and cheaper than traditional PLC/RTUs, then we’re going to want to be ahead of the curve in understanding how to best make use of these “generic” SCADA (or anything) devices.
  • What about developing some “kits” that serve to demonstrate the wide applicability of SCADA-originated data?  E.g.  an environmental-monitoring “pod” containing sensors for temperature, GIS info, water quality, etc.
    • This could be done as a side-effect of the prior two items.  I know some environmental scientists who might interested in collaborating on the design and use of devices like this.
  • Do we want to begin pursuing publication opportunities in industry trade rags?  I have a hard time believing that any rational editor would pass up an article entitled, “The World’s Cheapest SCADA Device,” for instance.
    • I think we should, but I don’t know anyone in the right industries, I think.  I’ll be happy to write stuff.

Cross-tab data visualisation

Parallel Sets

Parallel Sets (ParSets) is a visualization application for categorical data, like census and survey data, inventory, and many other kinds of data that can be summed up in a cross-tabulation. ParSets provide a simple, interactive way to explore and analyze such data.

http://eagereyes.org/parallel-sets

123 Main Street, Anytown, USA – now with pictures

http://data.mashedworld.com/dualmaps/map.htm?x=-88.393290&y=44.262464&z=16&gm=0&ve=5&gc=0&bz=1&bd=0&mw=1&sv=1&svb=0&svp=0&svz=0&svm=1&svf=1

AppletonDatamashedWorld

This is the most interesting (to me, that is) free map application I’ve seen.  You’re looking at my neighborhood in Appleton in several ways, simultaneously.

I found it because the city of Appleton links to it from their "My Appleton" site:  http://my.appleton.org/Propdetail.aspx?PropKey=311043500  which is sitting on one of the canonical ESRI GIS systems (which I know only because I saw upgrade costs in the city budget docs.)

 

I’m updating Cygnipede and have been spending time looking at how this sort of data can be used.

Aging workforce

from http://www.prairiebizmag.com/articles/index.cfm?id=9659&section=News

The number of residents in North Dakota age 65 or older is forecasted to grow by 58 percent in the next two decades when it would represent 23 percent of the state’s population, according to projections from the North Dakota State Data Center at North Dakota State University. The number of Minnesota residents turning age 62 is also expected to jump by 30 percent in the next year.

North Dakota’s oil and gas industry, coal mining and power generation facilities are projected to lose about half of their workforce due to retirement in the next decade, according to Dave Farnsworth, power generation and engineering leader at Great River Energy’s Coal Creek Station near Underwood, ND.