Monday, November 19, 2018

Found It! For Borepatch...

I can't get the email thingy at Borepatch's profile site to work because, um, Luddite probably, but we had a brief "conversation" after one of his blog posts.  I thought he'd posted one of the examples but turns out it wasn't him and I actually had it buried on my own computer.  So this is sort of for him.

I spent 35 years crunching data.   At core, my former agency produces high accuracy data points on the earth's surface that create a network from which surveyors and mapmakers of every level of accuracy can hang their work.  For that you use data points, observational data, a certain amount of smoothing, and a sense that I can't explain, maybe it was experience, of what the numbers are telling you.  That sense was what had me worried about the agency's product when I left.  It was often clear that the younger data crunchers were seeing ONLY numbers, not information. There's a difference. They're not just numbers.  It's information that tells you something and sometimes what the numbers tell you is subtle.  The computer output from an algorithm doesn't flag that subtlety.  It's there in front of you but you have to recognize it. 

Anyway, I've seen this a couple places, and it pissed me off.

On the right, actual data locations, although I'm dubious about how much is really there.  There aren't a bunch of airports with weather kits close together in those areas - betcha.  On the left, complete misuse of the available data by maximum smoothing.  I suspect the computer program couldn't handle that little data over those large distances and did the computer program version of throwing up without crashing.  I've had it happen.  The output is useless because you drove the program crazy.

But gee whiz that computer barf creates an impression, doesn't it?.  An impression that guarantees employment.  Because that's how it works.  The primary purpose of a government agency is to continue its existence.  The work it was created for is gravy.  The primary job of every agency director is to get the money out of Congress needed to maintain the agency's existence with staff numbers at the highest possible levels.  This, of course, varies in degree and what the agency does and there are tons of government employees out there who just want to do the work and do it right.  I was one of them.  But you know those committees that are in charge of the purse strings for science agencies?  They probably know nothing about the subject matter they are doling out money for.  That's BEFORE you get to the "dumb as a post" part.

There's no validity to the map on the left.  They've manipulated data over huge distances in a way they should be beat with a stick for.

NOAA is a huge agency. My little agency is a part of it.  Most old-timers felt we were misplaced - DOT would be more appropriate.  When asked who we worked for, we never said NOAA.  But, whatever, I can guarantee that NOAA  has no intention of going away.  It does do many good things.  Good luck boating along our ever-changing coasts without their updated nautical charts or coming in and out of an airport without the surveyor who mapped the runways and the elevations of  the flight obstructions around it.  One of the face palms of one meeting with the purse holders of Congress was when one of them said that the Weather Service could be gotten rid of because people now had access to things like the Weather Channel.  None of the NOAA folks at the meeting said what they were thinking:  "WHERE DO YOU THINK ALL THE DATA FOR THINGS LIKE THE WEATHER CHANNEL COMES FROM YOU IDIOT?!!!!!!"  But most people don't know the details of things as deeply entwined with their daily lives as how the road they drive every day was laid out from Point A to Point B and so egregiously bad work like that map is how they are now keeping their staffing levels.  It is a dishonest and horrible misuse of data that makes me angry and gives me the desire to beat someone with a stick.


  1. (Here from the link on Quizikle)

    There used to be a saying on the wall in every computer lab back in the '70s, a little poem. I'm guessing you might remember this:
    "I really hate this damned machine
    I wish that they would sell it.
    It never does do what I want
    Only what I tell it."

    Which explains the software barf you describe as well as can be. The software didn't barf, it did exactly what it was told to do with the data it had. Barfing would writing out random strings of letters or making all of the display wrong.

    This is deliberate deception. It would have been more ethical to tell it to display nothing or something visually very different from the normal displays in areas where there are no thermometers, or if the spacing exceeded some limit.

    Which brings to mind another one of those 1970s computer lab sayings: "garbage in, gospel out". Where "in" includes the code as well as the data.

    1. Don't know the poem, but I could look at the output of the programs I used and tell there was too little data for the area. Things would skew. At which point I'd break the datasets up and specifically write up that there was too little data to process an area with the rest. I also always included a map of the data points used.

    2. There's a whole blog post in that story, too, about trying to see what the data says vs. trying to get a desired result. Honest inquiry vs. agendas.

      Raw data is sacrosanct, not something to be tortured into saying what we want.