Dear NASA and ESA: Thanks for Cassini-Huygens

Seeing Saturn as a child through a reasonably good amateur telescope was one of many formative experiences in my life. When the planetary 'seeing' was good (a still atmosphere is more important than how dark or clear the skies are when looking at anything in the solar system) and Saturn was well above the horizon, a 6 inch f8 Newtonian reflector showed the three main rings, some detail in the atmosphere, and as many as half a dozen moons. Though, really, the moons were just Titan and assorted specks.

Then, for the last 20 years, we've had Cassini-Huygens. Here's a composite image of Saturn eclipsing the Sun. It's one of my favorites, and definitely worth a click.

All day today NASA TV will be rerunning Casssini programming. At 7 a.m., Friday, September 15 (that is Eastern daylight, UTC -0400) they start End of Mission Commentary, etc. There will be new content appearing for a while, then they'll start rerunning it. The schedule is at https://www.nasa.gov/multimedia/nasatv/schedule.html.

It's amazing how much we're learned about the Saturn system my childhood days. There's a good summary of what NASA judges to be the Top 10 at https://saturn.jpl.nasa.gov/news/2892/cassini-10-years-at-saturn-top-10-discoveries/. Be sure to look at some of the links below the subsections. For instance, under #7 'Vertical structures in the rings imaged for the first time', there's a link to https://saturn.jpl.nasa.gov/news/2442/cassini-reveals-new-ring-quirks-shadows-during-saturn-equinox/. From that link,

The chunks of ice that make up the main rings spread out 140,000 kilometers (85,000 miles) from the center of Saturn, but they had been thought to be only around 10 meters (30 feet) thick in the main rings, known as A, B, C, and D. 

140E6 meters in diameter to 10 meters in thickness, or 14E7 to 1. That is a remarkable ratio. To look at 14 million to 1 on a more human scale, pack that ring diameter into the distance from Los Angeles to New York City, which is 3940 km or 2448 miles, as the crow flies. That gives us a reduction of 140,000 / 3940, a factor of about 35.5. Our thickness then becomes 10 / 35.5, or about .281 m, or 281 mm, or 11 inches. That seems pretty thin!

If it's the best data available, I would imagine a lot of people spent time arguing about the physics of how such a structure might have evolved and survived. But then came Cassini-Huygens, and an explosion of new knowledge about a dynamical system. From that same page,

In the new images, particles seemed to pile up in vertical formations in each of the rings. Rippling corrugations -- previously seen by Cassini to extend approximately 804 kilometers (500 miles) in the innermost D ring -- appear to undulate out to a total of 17,000 kilometers (11,000 miles) through the neighboring C ring to the B ring.

The heights of some of the newly discovered bumps are comparable to the elevations of the Rocky Mountains. One ridge of icy ring particles, whipped up by the gravitational pull of Saturn's moon Daphnis as it travels through the plane of the rings, looms as high as about 4 kilometers (2.5 miles). It is the tallest vertical wall seen within the rings. 

It seems likely to provide important clues to our understanding proto-planetary disks.  Exploring those links will be time well spent. Unfortunately for me, lunch is now over. But the yield from Cassini-Huygens is very far from over; these data are going to be the basis for important research results for decades. I look forward to it being an enormous time-sink over the coming years. More immediately, I'm going to catch some of that NASA TV coverage tomorrow, cup of coffee in hand. Because it will be 0400 here on the West coast, at the start of what's going to be a busy day.

An Opportunity Cost of Hyper-Partisanship

By 1978, this had become an obvious problem.  Thirty-nine years later, it's a lot worse.

I just heard about the Politico piece Why America Still Hasn’t Learned the Lessons of Katrina. How might that be good news? Because in one of the reddest states of the them all, where you can find entire communities that, according to CNN give little credence to anthropogenic climate change.

Congressman (Republican) Garret Graves, who served for six years as the head of Louisiana's coastal protection and restoration efforts in the wake of Katrina, and is now Chairman of the Water Resources and Environment Subcommittee that sets policy for the Army Corps and holds the power to investigate the agency.

What's more, Politico goes on to say that Louisiana’s approach is seen as a model for states facing looming coastal crises, and its experts regularly host visitors from around the world seeking to learn from its resiliency efforts. And that the most powerful industry in the state, the energy industry, is on Graves side.

It's timely, given that the former Hurricane Harvey is still pounding Houston, promising record or near-record damages, and it reminds everyone of the fear in with Hurricane Ike, in 2008:

While meteorologists have described Harvey’s dynamics as a “worst-case scenario,” with the potential to dump massive amounts of rain for days, it’s not the storm that Texas leaders fear the most. Emergency managers in the Lone Star State got a hint of what their worst-case scenario could look like in 2008, before Hurricane Ike shifted course. That storm still rang in as the third-costliest disaster in U.S. history, but had it shoved the predicted, massive storm surge through Galveston Bay and up the Houston ship channel, hitting the country’s fourth-largest city and the nation’s largest petrochemical and refining complex, it would have been far worse.

Many years ago, I lived in Louisiana. I'm no fan of the place (except for the food!), and I am a life-long believer in science. But neither am I fan of political wing-nuts, on either end of the spectrum. Oregon is fortunate in being one of the states least affected by changes in sea level -- or climate change in general. I live in a conservative county, but just a few miles from one of the most liberal areas of the state. I hear intolerance and wing-nuttery from both sides, and used to occasionally indulge in it myself, particularly when stoked by some particularly heart-felt bit of lunacy. It's possible that I still do it; nobody is always aware of how they come across to others. OK, that's wrong, and it's on me.

Until the general population learns to stop shouting past each other, find common ground, and just possibly rediscover civility, we will continue to pay opportunity costs in the billions of dollars, and in too many lives lost or irreparably damaged.

Can that actually happen? I have no idea, and most days, I doubt it.  But I also wonder how many more encouraging stories, such as this Politico piece, are out there, but never draw an audience.

So, what's with that barrier island image? It wasn't in the Politco Piece.

Why no. It wasn't. Politico supplied an image of disappearing wetlands, but in one of the rare weak bits of the piece, there nothing that plainly stated that it was a before-and-after image of the same place, much less mentioning the dates. And they do cover ballooning costs "to protect highly vulnerable bayou communities and a major oil and gas hub in Terrebonne Parish".

The image I supplied is about barrier islands, which are also part of the Army Corps of Engineers palette of solutions, part of the reason for that Terrebonne Parish problem, and it came from The United States Geological Survey, which I have long regarded as one of the most science-based organizations in all of government. Including the loss of barrier islands adds information, so it's a win. Especially since now I can point to more information by giving the source as Louisiana Coastal Wetlands: A Resource At Risk.

You Can Never Get Enough Corona Time

1m11.4s was the duration, of which I missed the first second or so looking for, and failing to spot,  shadow bands.

But wow, did I get a great binocular look at prominences. After it was over, I caught a broadcast from Nebraska  at https://www.nasa.gov/eclipselive/#NASA+TV+Public+Channel. I was fortunate enough to see far more than were seen there. That pure pink Hydrogen-alpha light is beautiful. Pure like laser light, but without that speckled quality. And I did see a great Diamond Ring at the end of totality.

The 1979 eclipse was better, overall. Longer, and I saw it from a location with long views, and fantastic shadow bands. This time, no bands, and without long views from a high point, I didn't see the moon's shadow racing across the ground. That provides the visual cue that allows you to realize that you are seeing the clockwork' of the solar system. Maybe west Texas in 2024?

OTOH, no racing around trying to find a blue hole to see it through: it was nice and clear. No worries about forest fire smoke, and all I had to do was walk outside and have a seat. If you are ever given a choice of locations, pick the one that overlooks surrounding terrain. It was just unfortunate that I didn't have that opportunity this time.

That said, this is is very far from anything like an expression of disappointment. It was awesome --  a word I don't use casually. I had forgotten about that rapid decrease in light during the last minute or so before totality. Very dramatic, and having forgotten about it, it was pretty much a New Thing for me.

People were parked all over the roadsides, a parking lot at a nearby boat launch was beyond normal capacity, and I could hear the cheering when the corona jumped out from my place, half a mile away. Obviously some very happy people.

What a wonderful morning.

Solar Eclipse Timing for Peoria, Oregon

Eclipse timing for Peoria, Oregon, but times and alt/az will be close for Corvallis, Albany, Shedd, etc. I was just outside planning how I was going to lay out a white surface to try to photograph any shadow bands that might appear. Thought this might be of interest, so I did a quick cut-n-paste from a text file.

Touches down on the Oregon coast between Lincoln City and Newport at 10:15 a.m. PDT.
Following data from:
https://eclipse.gsfc.nasa.gov/SEgoogle/SEgoogle2001/SE2017Aug21Tgoogle.html
I added the PDT (-7) times.
Lat.: 44.4[redacted]° N
Long.: 123.2[redacted]9° W
Total Solar Eclipse
Duration of Totality: 1m11.4s
Magnitude: 1.003
Obscuration: 100.00%
Event                         Date        Time (UT) PDT       Alt Azi
Start of partial eclipse (C1)  2017/08/21 16:04:50.  09:04:50  6 27.6° 100.8°
Start of total eclipse (C2) :   2017/08/21 17:17:10.  10:17:10  3 39.9° 116.3°
Maximum eclipse :       2017/08/21 17:17:45.  10:17:45  9 40.0° 116.5°
End of total eclipse (C3) :     2017/08/21 17:18:21.  10:18:21  7 40.0° 116.6°
End of partial eclipse (C4) :  2017/08/21 18:37:29.  11:37:29  7 51.2° 139.5°

Crab Nebula with 5 telescopes

The composite is linked above above, from this starting point for all this here on hubblesite.org. There are links to the individual images from the five telescopes involved, covering five spectral regions. Highly recommended.

• Very Large Array (radio)
• Spitzer Space Telescope (infrared)
• Hubble Space Telescope (visual)
• XMM-Newton (ultraviolet)
• Chandra X-ray Observatory (X-ray)

It turns out that there were almost six telescopes involved. Pulling from the science paper (Drubner et. al.):

"We have also used the Atacama Large Millimeter/submillimeter Array (ALMA) to produce the first detailed radio continuum image of the center of the nebula at 100 GHz, although the ALMA observations were not contemporaneous with the others ..."

The features near the center of the nebula were moving at ~ 20% light-speed, and over ~6 months of observations of a  supernova remnant that is 'only' 6500 light years away, had moved ~2 seconds of arc. Which led to a corrupted image.

Too bad, but of course there is a lot going on in the immediate vicinity of a pulsar.

Before leaving that page, I have to point out this short linked video https://media.stsci.edu/uploads/video_file/video_attachment/4408/STScI-H-v1721a-1280x720.mp4. It morphs the images through the spectrum, and it's annotated by spectral band. There's also a version without the annotations.

If you'd like to see changes in the Crab Nebula, similar to what I mentioned mentioned, here you go. This is from Chandra as well, but the page won't load. So here it is on YouTube. Seven months of changes, very near the pulsar, in a few seconds.

So, what else goes on in a nearby supernova remnant?

A lot. As one might expect, there is a lot of very energetic physics packed into a small area, only 6500 light-years away. When the supernova which left the Crab Nebula behind was recorded by Chinese, Japanese, and middle eastern astronomers in 1054, it would have been easily brighter than any star or planet in the sky. 

What began as a star on the order of 10 times the mass of the sun, left a neutron star only about 20 miles across, but weighing ~4.5 billion pounds per cubic inch. It's rotating 30 times per second, because angular momentum was conserved during the explosion. The energy it radiates doesn't stop in the X-ray regime that was observed by Chandra, but extends all the way up to ~10 TeV. That is similar to the LHC, whose initial energy was 7 TeV, but is now running at 13 TeV, but at a vastly higher power level. The LHC beam, after all, is about the diameter of a sewing needle.

The magnetic fields involved are enormous. That doesn't happen due to a conservation law, as in rotation; in fact exact mechanism is still unclear.  What is clear is that their reach is large. Here's an image from NASA's Hubble Captures the Beating Heart of the Crab Nebula (recommended). That pearly blue glow is synchrotron radiation, produced by electrons spiraling in a magnetic field.

Everything near a pulsar that can be ionized is ionized due to those high-energy emissions. What's left are carbon and silicate dusts, and even those seem likely to be mostly charged, though electrostatically). So we have positively charged ions, and those now-free electrons. Those free electrons are pervasive, hence the blue glow.

That ionization, the pulsars intense gravitational and magnetic field, and rotation combine to give rise to those famous pulses that gave pulsars their name. The basic mechanism is 

1. some of the material ejected in the supernova is pulled back into the star
2. the in-fall is caught in the magnetic fields, and pulled toward the poles
3. it's beamed back out at the poles, rather like lighthouse beams
4. the beam sweeps across us, in this case 30 times per second

4. is possible because the axis of rotation is not aligned with that of the magnetic field. That seems a bit counter-intuitive, but it's also the case with the sun, and even here on earth (and something that has to be considered when navigating by compass). That's too much to go into here; books have been written about the Crab. Search on 'dynamo effect' or 'dynamo theory'.

We have been learning about this for a millennium now. While the pace of discovery is accelerating, the story of the Crab Nebula is far from told.

Birding data: there is never just one thing going on

Anomalous high counts of White-crowned Sparrow and Steller's Jay have been had here (Peoria, Linn Co., Willamette riverside) during March. WCSP counts have doubled previous records, and STJA are up by 50%. OTOH, I have yet to find my FOY Belted Kingfisher.

This probably means *nothing at all*. See https://en.wikipedia.org/wiki/Complex_adaptive_system.

First, let's toss the lack of Belted Kingfisher. There's a correlation with river levels: higher water levels than seen this year bring them in to some bottomland where they (also long-legged waders, which are also absent this year) are easily detected. That threshold hasn't been reached. The same thing might explain a complete lack of expected Great Horned and Western Screech Owls: water levels may have been high too high to support the record populations seen over the past couple of years, due to unfavorable conditions for prey. In short, the river levels seem likely to have been almost perfectly wrong.

How might those river levels effect WCSP and STJA? No idea. The data only run to 7 years, and support no conclusion.I suspect an increasing trend in passage of migrating STJA, but can't prove it. The uptick in counts is only 3 years old. In the greater scheme of things, that cannot be distinguished from noise in the data.

Might increased WCSP counts provide some evidence that something has actually changed? Sadly, no. My recording has changed, in a couple of respects.

1- Serious lack of time, for professional reasons

2- Variance in sunflower feeder habits

I think we can all understand #1. #2 is about racoons, of all things;

I used to keep a sunflower feeder hanging, and just top it off in the morning, or whenever it seemed low (working from home). A few months ago, one or more raccoons found it, and as there was no good means of safeguarding it, it was brought in at night. And (crucially) re-hung whenever I got around to it in the morning.

Whenever I got around to it tended to be at a later time than "breakfast is always served". It's likely that I saw fewer WCSP before because they could visit any time, and more later when they had to adapt to a more limited opportunity, hence concentrating their numbers.

I'm probably going to stop feeding birds now. Well, perhaps hang a hummingbird feeder (FOY *female* Rufous Hummingbird today). It would be interesting to see how the data might skew, and it's safest to do it now, while the hardships of winter are largely over, but before breeding territories are established.

I expect the yearly species count (119, two years running) to crash, but that number won't matter either--just more short-term statistical noise. Food for thought, at best. At worst, it's an entirely fallacious racoon-induced population crash. 

Total Solar Eclipse 2017

I live in a pretty cool place, from a certain nerdy viewpoint. 119 species of birds, two years running, might indicate a certain predictability, but a closer look at the data destroys that notion.

So how cool is this place, really? My subjective measures include things like species counts, whether I can get reasonable photos, etc. Subjective in this case means entirely subjective. So what might tip this place into Coolest Place I Have Ever Lived?

Yeah, that might do it. Though no photo did it justice. Film doesn't have the dynamic range, and digital cameras are worse in that respect. There's a wide pearlescant glow from the solar corona seen IRL which is entirely missing from short exposure times. This image predates optimizing over a set of stacked images.

It's a photo of a photo that has been hanging on some wall of pretty much every place I have ever lived since 1979. Yes, I am an old fart: deal with it. No, it's not related to Sauron in any way, save perhaps as being inspirational to film-makers for major production houses. Possibly. The mechanics of of how films are actually made (and taxes, payments to the Tolkien estate avoided, etc.) entirely escape me.

You may want to visit https://en.wikipedia.org/w/index.php?title=Solar_eclipse_of_February_26,_1979&oldid=761573206

That's the link as of this date: I've been burned by not specifying specifying dates. Pull quote:

Many visitors traveled to the Pacific Northwest to view the eclipse,^[1] since it would be the last chance to view a total solar eclipse in the United States for almost four decades. The next over the United States will be the total solar eclipse of August 21, 2017.
Although the path of totality passed through Portland, Oregon in early morning, it was not directly observable from the Portland area due to overcast skies.^[2]

That last line matters. In 1979 I was driving up the Columbia Gorge, seeing small holes of blue sky in wide overcast, and trying to judge where said blue holes might line up with the sun, during that brief period of totality. A fast car and a certain disrespect for law and order won the day. Everything lined up, and I skidded to a stop at Horsethief Lake State Park in time to see the whole event.

• Shadow racing through the gorge at a thousand miles per hour
• Weird greenish light, entirely unexpected, before totality
• Shadow bands rippling across the ground.

It was awesome, in the original sense of the word. This is the Pacific NW. I saw Mount St. Helens erupt a year later, so I'm not a stranger to drama.

So here we are, that long 40 years later, as mentioned in the above pull quote. I'm now a certifiable Old Fart who never expected to live this long. But that narrow path of totality will sweep directly over my place  on August 21. In place of vile winter weather,  I have the best weather of the year, and all I have to do, essentially, is walk outside. How cool is that?

Being a complete nerd, I'll go bit further. I'll spread my parachute canopy across the yard below a second-story deck, and hope for a shadow band photo opportunity, etc. But mostly, I just want to experience the event. I lack the words to describe a total solar eclipse. Perhaps that is the true meaning of 'awesome': you just can't really express it.

Of one thing I am certain: on 2017-08-21, this weird little place in small-town Oregon will become The Coolest Place That I Have Ever Lived.

Exploring Data From the Linux Command Line

A few days ago, we saw the first signs that perhaps the worst of an unusually cold and wet winter might be ending: a temperature over 60°F!

A neighbor commented was made that it had been a long time since the last one, and I was curious as to exactly how long. For reasons of my own, I keep data files on what's recorded at the nearest weather station with what I consider to be fairly reliable data. So it only took a couple of minutes exploratory hacking around at a shell prompt to get my answer. Here’s what I did, and the result I got.

grep ^161[0-2] 1606010000-1612311600 | awk '{print $1" "$5}' | grep -E 6.{3} | tail -n1

1611191600 61.0

It seems longer, but the last day of ≥ 60.0°F temperature was 2016-11-19, and as a side-effect we also get the last time of the last day: 1600 (4PM for those of you who don't use 24-hour time). We could get rid of that side-effect; they are usually a Bad Thing in code. But in this case the source is obvious (as will be shown below), and entirely beneficial. It extracts another piece of information from our data at zero computation cost. Exploratory code for the win.

Before I get into what our pipeline is doing, a note about the file. These are raw data - fields are separated only by whitespace. Lines begin with time and date encoded as YYMMDDTTTT. Hence the first field meaning of the result seen above, and the file name 1606010000-1612311600. It reflects the start-stop dates and times of the file. That can be a useful convention: in this case it immediately reveals that the data are incomplete. The station failed to record after 1600 on New Year's Eve.

Additionally, we can use the wordcount program in linecount mode to see that we are starting with a file containing 5541 lines (records, though there is a 3-line header, which I won't bother to filter out).

wc -l 1606010000-1612311600
5541 1606010000-1612311600

1- grep ^161[0-2] 1606010000-1612311600, in which grep (a pattern-matching tool) is supplying all lines (records) from our file that begin (specified via ^) with 161, if the next digit is 0-2. I was only interested in months 10-12 of 2016 (and 2016 data are all that is in this file), because I knew the last date of ≥ 60.0°F would be in there somewhere. We now have only records from our period of interest. If we ended here, our output would be 

1610010000  24.10    3.0  163.0   53.0   51.0   80.0   12.9  204.0    6.0    0.0

...

1612311600  48.70    2.0   99.0   35.0   35.0   87.0   13.3  173.0    8.0   47.0

I'm using the ellipses in place of 2608 lines of output. wc -l shows 2610. We've filtered out nearly half of our data. Now we pipe (the | character) those lines to awk.

2- awk '{print $1" "$5}', where we instruct awk (a pattern scanning and processing language, of which more later) to print only that first datetime field, a space, then field 5, which contains the temperature, of each line of input it receives. Now we're down to only the fields of interest within our period of interest.  Had we stopped here, our output would still be 2610 lines, but only 2 fields out of 11, formatted as
YYMMDDTTTT NN.N.

This 2nd stage of our filter removed about 2/3 of its incoming data. I'm just guesstimating by looking at line lengths here, but you can get accurate numbers using wc again, before and after this stage. Specify -b instead of -l to count bytes instead of lines. I'll skip the demonstration. Now we send that on to grep again, but specifying different options.

3- grep -E 6.{3} contains the -E (Extended) option, which enables the {} syntax so that we can specify how many instances of a character we want to match. The preceding dot can be read as 'any one character', so a multi-character string would not match.  The trailing '$' matches the end of line -- the opposite of the '^' we used the first time we used when we piped to grep. The net effect is that only content that matches a '6' followed by any 3 single characters, followed by end-of-line, will survive. Given our NN.N format for the field field, we filter out anything except 6N.N and wc -l now shows only 222 of those short lines left, of 2610. Having filtered out all but 1/7 or so of the data coming into this stage, we now we filter down to one line - our answer.

4- tail -n1, which returns only the last n lines, and specify n=1. Because the data are in increasing time/date order (as can be seen in the output of our first filter) this gives us our last datetime, and answers our question, with greater accuracy than we had thought to ask.

If we needed the date and nothing but the date, we could modify our usage of awk, which is a pattern scanning and processing language. GNU awk has some very interesting capabilities, such as floating point math, true multidimensional arrays, etc. This entire task could have been done in awk, but I wanted to show more of the shell tools, and pipelines, not just Cool Things We Can Do With GNU awk'. [1]

The Shell Will Probably Always Belong in Your Toolbox

I often use far more sophisticated tools when I want to take a long hard look at data. But, file formats vary, data may be missing, etc. As a rule of thumb, you can expect to spend half of the total time spent analyzing data just seeing what's there, and cleaning it up. For much of that work, the shell is a great tool, and it's actually very common to spend a bit of time using the command line to explore. In a broad view, command-line tools can help you determine,  quickly, whether a particular data source contains anything of interest at all, and if so, how much, how it's formatted, etc. And finally, the commands can be saved as part of a shell script, and used over an arbitrary number of similar data files. 

To a point, anyway. Shells are slow (particularly bash). Though of course there are tools to quantify that as well, and timing work on a subset of the data can give you an idea of when you are going to have to use something else. 'time' is available as a built-in if you are using the bash shell, and any Unix or Linux will also have a 'time' binary somewhere on your search path if the appropriate package is installed. On this machine it's /usr/bin/time, packaged as 'time'. Everything else, except the shell itself, is in the 'coreutils' package. Which says something about how useful these tools are. If you aren't using them, you quite literally are not using the core of the Linux/Unix tools. 

That is probably a mistake. There is a lot of data out there, stored as textual files of moderate size.

My Ulterior Motive for This

I wanted a post such that:

1. I could advocate the command line, to people who seem to inappropriately default to spreadsheets, which are nothing more than another tool in the box. That box should contain several tools. Consider unstructured data. Or consider binary data formats, which are an intractable problem for both shells and spreadsheets.
2. Had absolutely nothing to do with security work. Because people are going to be justifiably sensitive about exactly whose security data I might be using as an example. But everybody talks about the weather.

If anyone wants to play with the data, it's available at:

https://drive.google.com/open?id=0B0XLFi22OXDpR3h0UUQ1cmNWbkk
Note to self: find another home for this sort of thing. Google Drive can't even preview a text file.
Note to all: this is not a promise to keep it there for any significant period of time. If I need the space for other things (like client-related things), that file is very, very gone. I recently VVG'ed most of what was in /pub.

[1] I do have one idea for something I'll do with awk one of these days. Because who doesn't like univariate summary statistics combined with 4000 year old Babylonian math, and using NIST-certified results to validate (or invalidate, as the case may be) our code?