Monday, August 6, 2012

Keep Calm and Carry On: From MER to MSL

Watching Paint Dry

I always liked the “watching grass grow” analogy to boredom better. Seemed a little more tangible, a little more organic. Plus, paint smells. A team I used to work with back in Colorado actually started growing grass in the hours upon hours of boredom that the spacecraft was affording us. 

1 day, 9 hours, 27 minutes, and 30 seconds.

Press just came into the CMSA. I’m inundated with questions like, “Why the peanuts?” Because of reasons, is why.

1 day, 8 hours, 48 minutes, and 45 seconds.

Just wrapped up a DDOR*. Telemetry coming back in whatever round trip light time is.

Oh yeah, what’s cool is that I bought one of those AeroPress things. Stuff makes quality ‘joe.

Well, wait, that’s not the coolest part. I guess the coolest part is that I’m sitting at a console in the Cruise Mission Support Area (CMSA) for the Mars Science Lab. I'm watching telemetry flow as we approach Newton’s pull from the body of Mars. I’ve been in this room roughly once a week since I started doing uplink operations – and subsequently rolled off of the MER project – and it’s not a whole lot different today than it was on those days. The downlink rate of the MSL spacecraft is just low enough to be annoying. It’s mostly an insanely unfair function of Earth-to-spacecraft range, and that range is [insert big frickin’ number]. With low data rates, we don’t get snapshots of the spacecraft’s state of health as often as we like, so we deal with the reality of watching paint dry.

Watching paint dry, by the way, isn’t all that fascinating. People say they want my job, but they don’t really know what it can be like. It can suck the life out of you. Maybe a telemetry channel will update here, maybe we’ll get a message from the spacecraft there. Not a whole lot. She flies herself, really. 

But in about 30 hours this room will be full of people like me, all of them brimming with anticipation as we watch a machine attached to another machine that just ejected from another machine… land itself on Mars.

1 day, 8 hours, 28 minutes, and 35 seconds.

I still don’t believe that this is actually happening. The cool factor is a little subdued at the moment.

“(Let’s Stop) Reinventing the Wheel”

With the paint drying ever so slowly, there’s time to reflect.

Spacecraft operations teams function mostly on reality. Engineering is done with numbers, and analysis without numbers is only an opinion. The doses of reality come in sobering and inconsistent waves, juxtaposed by the tunnel vision that is, itself, the spawn of equal parts innovation, paranoia, and adrenaline. 

When you start diving deep into the caverns of a spacecraft system, you start to see heritage: this thing or that process or this guideline or that scenario is a particular way because somebody did it on a previous mission the same way and it worked well. There was no need to reinvent a wheel because the operational regime of said thing hasn’t changed in years, if not decades. Time – and its evil brother, Money – is saved when an engineer takes these perfectly acceptable shortcuts in design and process. Efficiency and incremental improvements are nice-to-haves; effectiveness is indispensable.

The need to invent or create is insatiable for an engineer. It gives you a sense of reality and a connection to what you do. It makes it… real. But the desire to create is in exact opposition to reality that there is probably already something or someone out there that can do the same job.

Went to bed – came back again in the morning.

12 hours, 42 minutes, and 42 seconds.

Our flight director just instructed our ACE – the person who clicks the buttons to send the actual ones and zeroes to the spacecraft – to “turn off command modulation.” This is, nominally, the last thing we’ll send to the spacecraft until we’re on the surface.

When a new, big flight project comes along at JPL, this desire to invent and create is magnified. We delude ourselves into the notion that since we’re new and big and bad, everything we make needs to be newer and bigger and badder. It’s a serious problem when we put those blinders on and act like we’re the first in the business to do this business.

On one hand, the inheritances of many pieces that make MSL happen are clear and obvious. But on the other hand, there are many new inventions. The contrast in these two categories is stark.

The trouble comes in the apparent middle ground, where it seems that you can’t place a particular problem that needs to be solved into a well-defined spot in the spectrum from “inheritance” to “invention.” You subconsciously activate your tunnel vision, removing any ability to see what else is around you, and start reinventing things. You work literal days and literal weeks on this new problem to be solved, and then at the end of it someone more astute or removed from the process intelligently points out that your problem had already been solved. If you had simply asked the right question to the right people, you would have been done with this problem weeks ago. 

The truth hurts. Your need to feel connected, to feel reality, was too much, and you cracked.

What’s worse than the subconscious blinders are the blinders of outright denial, the blinders that you artificially place in your view because you refuse to accept that there’s no room (or time, or money) for you to invent something. 

We at JPL – and everyone at every abode to engineering and science in the world – have a problem with these blinders. We must collectively reset ourselves and remove our blinders. Turns out, we’re pretty good at pressing our own reset buttons, which brings me to my point: We learned how to press that reset button because of the MER experience.

10 hours, 48 minutes, and 38 seconds.

My Mission Manager mentioned my glorious “good luck” muttonchops (remember these?) at the 9:30am press conference. They are officially famous, but not Bobak Ferdowsi kind of famous.

The CMSA is quiet. The EDL Manager said that we’re “rationally confident and emotionally terrified.” I only agree with the first half of that. I do have stake in the success of the Sky Crane – that stake being a job – but I’m not so emotionally attached to it. Maybe it’s in my head, maybe I’m just going through that stage of denial. Maybe the acceptance will occur before this countdown clock hits zero.

One thing, though, is clear: MER paved this path. In between every thought about MSL, this one squeezes itself in. You’d be hard-pressed to find someone in this room that would disagree – but only if you pressured them into saying it. It’s easy, though, to find people in this room that don’t dwell on this notion as much as I do.

Then I walk across the street back to Building 264. I hop in the elevator and ride it up to floor 5 – wait, no, sorry, floor 6, old habits. The old Callas Palace has been spliced into two rooms and is now outfitted with sleeker, shinier machines running sleeker, shinier software. The shades are drawn. And yet, with all the differences, it still screams MER. Come Monday morning, we’re going to pick up where we still haven’t left off with Opportunity: tactical daily planning so that we may explore Mars. It's a monument to renewal and reinvention to inject ourselves with a sense of reality.

We hit Curiosity’s Sol 0 at 13:50:00 UTC (06:50:00 PDT) this morning. Breezed by without a blink from the team, most of who are sleeping anyways. Until Opportunity conks out, MSL will be perpetually behind the 3000 sols that she has under her belt already. It’s an astounding wealth of experience that we’re about to start all over again.

So, to my team I silently say, “Let’s please stop inventing the wheel.” We know how to do this. The game has not changed in any fundamental way. We’re the only people in the world that can drive cars on Mars, and we’ve done it all before. This one’s newer, bigger, badder, but it’s still just a machine on six wheels.

Unreserved and unabashed, we have to remind ourselves: “Keep calm and carry on exploring.”

Literally Unbelievable

10 hours, 23 minutes, and 28 seconds.

Most of the team is out to lunch. We’re spying on the media from afar, trying to enjoy our last lunch before what may be the biggest moment JPL has ever experienced.

The word “unbelievable” is, to me, the epitome of hyperbole**. And yet, I feel the need to actually apply it to my experience at JPL.

Interestingly, the enthusiasm and desire to start this blog was intensely opposed by this mental block of mine that causes me to refuse to accept the reality of what it is that I do. Occasionally I’ll reconcile the reality: “Holy crap, I make 1’s and 0’s that get turned into a series of ups and downs in a radio signal that’s leaping across space to be interpreted by a computer on another solid body millions of miles away.” But it's ephemeral, and I forget it immediately. The light that clicks on and reveals the truth is now off again.

It’s just weird. Plain weird.

So now I’m involved with the newer, bigger, badder machine – our wonderful MSL, tucked tight behind an impenetrable heatshield, wrapped from the topside by a metal cage with rockets – and I just want to sit there and invent. I want to create. I want to have a level of involvement, something that can validate the reality that is staring me right in the face. Instead, I feel an extraordinarily benign — if there can be such a descriptor — disconnect from this reality. 

I literally don’t believe that there is a car on Mars, exploring the rim of a crater we call Endeavour. I literally don’t believe that this car gave inspiration and birth to another, bigger, badder, more complex car that is hours away from landing an astronomer’s stone’s throw from this first car. My brain isn’t processing this data.

This feels like a problem to me.

6 hours, 0 minutes, and 0 seconds.

Just finished up the last Cruise Ops Status Tagup meeting. Our Flight Director dropped the last marble into the "days until landing" jar. There was applause, lots and lots of applause. Also just did the last "Playcall" for EDL Parameter Update #4 — everyone agreed we were no-go. She would fly out that 500-meter error.

It hasn't gotten any more real. I can't just sit here and watch. I can't just sit here and enjoy the moment.

1 hour, 10 minutes, and 13 seconds.

The Odyssey spacecraft is rolling to its off-nadir position to point its UHF antenna where the MSL landing site will be in an hour or so. She is out of contact with the DSN, and this is worrisome because it recently had reaction wheel issues. Hopefully she comes back for us when we expect her.

The room I'm in and the rooms surrounding it have filled up to capacity. There are celebrities, big-shot JPL'ers, family, friends. And they all believe. Why can't I? What switch do I need to flip?

50 minutes, 20 seconds.

Odyssey is back in contact with the DSN. The roll maneuver went off without a hitch. She is ready to support the bent-pipe relay for MSL EDL.

This disbelief makes me nervous. My hands are getting cold — maybe it's the air conditioning vent right above me — and I'm pacing. 

13 minutes, 51 seconds.

MSL is either dead or alive. Nothing in between.

The room is growing silent; it's about to happen. The make or break moment. I start recording video with my iPhone.

7 minutes, 0 seconds.

Atmospheric entry. The donut-shaped Cruise stage caused a couple brief blackouts in the radio signal as it passed in between the descent stage's low-gain antenna and Earth. She'll burn up in the atmosphere. She was good to us.

I'm shaking. 

0 minutes, 5 seconds.

Tango-delta nominal. Bridle cut. GNC done. Pyro fire. Closed-loop flyaway controller active. 

It's dead in here.

0 seconds.

We're on the surface of Mars. "Flight, spacecraft mode is SURFACE-NOMINAL"


I believe. That just happened. It really just actually happened. 

While people are debating whether or not you're a bigot based on your desire to consume a chicken sandwich, we got busy landing a robot on Mars. 

I believe. That just happened. It really just actually happened.

While people are going through places of entertainment or worship and opening fire with semi-automatic weapons, we got busy celebrating a multi-country achievement dedicated to all of humanity.

I believe. That just happened. It really just actually happened. 

While people were complaining about $2.5 billion dollars wasted on thinking metal, we got busy looking at pictures of actual wheels of an actual car on an actual other planet.

Yup. There are those pictures, but on the big screen in the CMSA. It hit me like a concrete wall: I believe. I see it, and I believe it. It's there.

And now I can also believe with an intense satisfaction that what I — we —  did and still do with MER is real, too. It may have taken an overpowering experience of seeing a real achievement happen in real time to get me to believe this, but at least something did it. They say that you'll always remember the day you got married, the day your kid was born, and your first Mars landing. I'm not married, I don't have kids, but I now have a Mars landing under my belt. One day later, the nerves still aren't gone. I will sure as hell remember this day. 

Those pictures we look at every day — and boy, do I still peruse the MER pictures every day — are real. The dust-covered, low-resolution images from MSL's landing make this all the more apparent. Those images are really not even on this planet. That's really a car with wheels and solar panels and a crippled arm. It's real.

Now, I must keep calm, and I must carry on exploring. 

We have a new robot that's also doing real things, for real, on another real planet. A real robot with a real older cousin with 3000 real sols under its belt, with no desire or need or, might I say, ability to quit on its own. 

Here's to us. Here's to the people that believe in us.


*DDOR = Delta Differential One-Way Ranging. Gives us velocity of the spacecraft in a particular direction with incredible precision.
**The irony of this statement is not lost on me.

Sunday, August 5, 2012

Stunning and Comprehensive MSL Summary

Hey folks!

If you’re not aware that MSL is landing tonight, then here’s your announcement:


A colleague sent a very impressive summary of the landing system and sequence of events.

Check it out!

I will be watching right from the mission control center at JPL. I know you’re jealous.

Saturday, June 30, 2012

New and updates — and stirring the pot of rumors

How is Opportunity doing? Well… good. Prett-ay good.

Daily Solar Power Is Off the Hizzay

While I idly watch, MER-B's power is shooting through the roof: 600 Watt-hours* predicted. As a reference, we were soaking in about 250 W-hours at the dead of Martian winter (middle of March here on Earth). Tau — a measure of atmospheric opacity — is at an all-time mission low (which means clear skies), and we've had several significant dust cleaning events (which means clean solar arrays). This increase in power is all in spite of the now low northerly tilts that Opportunity is at — you know, the thing that kept her alive in the depths of winter.

If I were so inclined to get in trouble, I'd post the watt-hours plot our Power team puts together. But I'm not so inclined! For that, we'll have to follow our glorious mission manager updates.

Opportunity's Location, Recent Activities, and Future Plans

Opportunity has been skirting the rim of the Cape York geological feature since she departed from her winter parking in the northern havens:

There's been a discovery of new Gypsum (errr, presumably Gypsum) veins:

And we've found a nice juicy spot for a RAT hole — a target named "Grasberg":

Unfortunately, the Mars Odyssey (ODY) spacecraft went into "Safe Mode" a few weeks ago. MER has always been incredibly reliant on ODY for data downlink via the relay chain, and when ODY goes down, MER has to stand down for a few sols. MRO also provides good relay support, but much more rarely than ODY. We have to schedule "passes" with these spacecraft strategically, i.e. weeks ahead of time, so that their teams can piece together their multi-week set of commands appropriately to handle the relay supports. Changing them tactically, i.e., the day of the relay pass or otherwise shortly before then, is a difficult process and things like geometry and spacecraft sequence engineering constraints can bite us.

There is a lot of so-called unsent data on board Opportunity that prevents the team from scheduling data-intense activities (like hi-res Pancams, MIs, or mobility data). As such the work on Grasberg and the surrounding territory is slow.

For the future: We plan to cross the skirt of Cape York to the north to look at the transitional layers (do I sound like a geologist yet?) thereon. Then, southward back from whence we came — to the first entrance into the Endeavour Crater on the south side of Cape York, and then further sound to Cape Tribulation. Betcha we get there before next Martian winter — those RPs are itching to drive the heck out of Oppy, and they're damned good at it, too.

A Mission Milestone -- Sol 3000 
According to my highly technical (read: I cheated and used SPICE again) analysis, Opportunity will hit Sol 3000 at 2012 JUL 02 01:51:32.572 UTC, or 2012 JUL 01 18:51:32.572 PDT.

Holy… what? Seems about right: I started work on the MER project nearabouts sol 2350, which was nearabouts July 2010. 

At the moment I'm not grasping what this really means… I haven't had the time to let it soak in yet. It's just unreal

Rest assured, the MER team is having a big get-together brunch this weekend to celebrate.

Allow me to stir the rumor mill!

I've received several emails asking me about my rumored move away from MER to the Mars Science Lab (MSL) project at JPL. Let me settle all the rumors now: Confirmed. In fact, I left the MER project about two months ago.

Much as I loved working on the MER program, opportunity (pun not intended) awaited me on MSL. It was not a decision that I made without a lot of careful thought — I sat around for a few months working up to actually making the decision. 

There are several good bits to come out of this move, however:

1) I have no plans to discontinue this blog. I never did! MER impacted me in a way that I'll never forget. I'm always going to be plugged into Oppy's updates. She is quickly approaching the off-earth traverse distance record and I want to be blogging about it. No question about it.

2) A large portion of the MSL tactical uplink team comes from MER. ("Gee, you know, it would be prudent of us to hire people who have done all this before…") I see a lot of familiar names and I'm finally meeting some past "MER Legends." Granted, MSL is a new kind of beast, very very unlike MER, but the overarching process of getting 1's and 0's to her is the same. Experience counts.

3) I don't have to move offices! MSL owns floors 4 and 6 of building 264 at JPL, and MER (along with Odyssey and MRO) takes up floor 5. So I still get the chance to swing down to say hello to the MER folks every now and then. In fact I still use the coffee machine in their break room, because if I'm ever going to drink anybody's office-coffee-club coffee, I'm going to drink MER's. **

And several unfortunate ones:

1) I have less time to stay focused on generating consistent, reliable posts here. I'm working crazy hours and will soon be transitioning to "Mars time" shift schedules when we land on Sunday August 5th. POWER THROUGH!

2) I can't blog about MSL, at least not in the capacity that I'm granted here by the PR folks at JPL. 


I've received some good, insightful comments about the Radio Science posts (here and here). I responded with a little too much brevity; and the comments pointed out that I lacked a sense of clarity in the posts! Good catches all a round.

I plan on some supplemental pieces on Radio Science whenever the results go fully public.


*Watt-hours = Energy. Watts are energy (specifically, Joules) per unit time; multiply by a time unit and you get Joules back. You may be asking, "Why not just say 'joules', then?" If we were to express it as Joules — Watt-seconds — we would have these huge numbers that are hard to grasp. Watt-hours are a convenient (though inconsistent!) way of bringing a numb back down to something the human brain to contextualize. It's a colloquial engineering term.
**They've got this old school diner coffee machine that is meant to stay on 24 hours a day; oh boy have I been using it. Late nights on MSL have caused me to leave these silly notes next to the MER coffee machine — something to the effect of "LEAVE ON FOR MATT, 06/29/12 evening" — so nobody turns off my caffeine supply. Sometimes, on top of my note someone from MER will leave me donuts. I love these people…

Sunday, June 10, 2012

Happy 3000th, Spirit

About 3 hours and 32 (Mars) minutes ago, it was zero hundred of Sol 3000 for the Spirit rover. (This means Opportunity's 3000th sol is only a few weeks away.)

Wonder how many sols more it will be until a human stops by and designates the final resting place of Spirit as a historic monument of humanity?

Saturday, June 9, 2012

The Radio Science Campaign, Part 2/2 — The Results


Really, this took much longer to put together than I had imagined. There were a few technical snags — most of which are explained by the fact that I'm a bad programmer — but we finally have something worth discussing.

In Part 1, I talked about the basic science behind the Radio Science campaign that Opportunity undertook from January through the middle of May. Now, we have the results from that. Where in the sky was the Earth when Oppy was doing the experiments? Why does it matter? How did that change over time? How do the various variables relate? How can we easily represent all of this?

I've boiled it down to three plots. First, a few disclaimers and notes:

1) Since I'm not allowed to use operations data or products to create these plots, I have to play stupid and guess at which times of the Martian day to look at the data. As always, I used the SPICE toolkit for the geometric computational heavy lifting. Based on this data and what I remember from all those tactical shifts, I could piece together the relative time of Martian of day at which these happened. I looked at the orientation data of the High Gain Antenna (HGA) to see when it was moving and then — by hand! by hand! — picked out the ones that were "obviously" RS-DTE experiments rather than just the regular DTE/DFE's (see Part 1) when the HGA is also moving. The RS-DTE's are easy to pick out simply by time of day.
2) No durations shown here, just middle of the guessed window. Close enough! Most windows were about half an hour in length.
3) All Martian times of day (TOD) are Hybrid Local Solar Time (HSLT; we also just say "LST"), which is consistent with the rover's flight software.**
4) The plots of position of Earth in the sky (azimuth, elevation) are just showing where the Earth is, not where the antenna was actually pointing. I've done this for simplicity, and because backing out the real position of the HGA difficult. There are ~5 degrees of error here, which, for the purposes of this blog, is well under my "do I care?" radar.

The Evolution of Earth Elevation

We like to schedule DFE/DTE communications with our rover at the same time of Martian day. This gives us predictability, consistency, and flexibility in the plans from sol to sol. However, even if we could get the time on the DSN to do that (nobody can, it's a shared resource), geometry plays a big role. Both planetary geometry and the relationship of Earth and Mars time (i.e., the length of their day) have effects on when we can schedule communications passes — and remember, RS-DTEs are simply communications passes without much data going between the rover and Earth. 

First, as you may know, the Mars day is not equivalent to the Earth day — though it is very close. It's roughly 24 hours and 40 minutes, depending on what you mean by "day". Are we talking solar day? Mean solar day? Hybrid solar day? Then we get different values. The variance of this couples with the fact that, simply due to the relationship of Earth's position to Mars' position over time as they move through the solar system, the Earth will be at different parts of the sky at different times of the year. 

Put it all in a big confusing pot, and baby you got a stew goin'.

Right. So. How we best show this? The plot below shows this basic relationship: I've plotted elevation of the Earth from the local-level horizon at Opportunity's position on Mars at the same time LST (hybrid! hybrid!) from Sol 2800 to Sol 2971, which corresponds roughly to the times of the Radio Science experiments.

I could have picked any arbitrary time of day. The point is to show how the same time of day doesn't give you the same Earth position. We can see that sometime in the middle of February the elevation peaked. This pattern would repeat over and over if I had simply extended the time back. 

The lesson here is this: Earth elevation and Martian time of day only approximate proxies for one another.

Right, so, we see that elevation drifts. How did the RS-DTEs take account for this, or take advantage of this?

The RS-DTE Observations

One of the key characteristics of good RS-DTEs is a low Earth elevation angle. This exaggerates the Doppler-shifted signal, reducing the noise and clarifying the meaning of the data. We know that the Earth will probably be at low elevation angles in the morning and late afternoon, so it would be prudent to schedule our RS-DTEs there. When you slap on operational constraints — can we get that time with the DSN? do we have the solar energy to do it at that time of day? are we doing other things on the rover that preclude HGA movement? etc. — you get large variation from sol to sol in the time of day, and consequently the elevation angle of Earth, for each observation. 

The plot below shows all of the measurements that I gleaned from the SPICE kernels (see above). Details:
1) Left axis (blue): Elevation angle of Earth in degrees during the observation
2) Right axis (green): HLST Time of Day of the observation
3) Bottom axis: Time (duh)

Here are my observations:
> We tried for a lot of elevations under 30 degrees. Notice how we were getting those early on — then the power got too low, and we had to sacrifice the quality of the data for the survival of the rover, so we scheduled the RS-DTEs at an earlier time of day when there was more sun on the arrays.
> Another driver for the earlier and earlier times of day of the observation is found in the first plot from above: The Earth was lower and lower in the sky at the same time of day, and we wanted to hit particular windows of elevation angles, we ended up moving observation times to get the elevation angles right.
> The earlier the measurements got, the more we (presumably) doubled these tracking passes with the DSN as uplink passes to load the next sol's set of plans on board. Win win!

Finally, where was the Earth in the azimuthal direction relative to North? Here we have a plot of azimuth angle clockwise from North and elevation in concentric circles,

North is up (0 degrees azimuth); East is to the right (90 degrees azimuth). For each of the observations, the azimuth was roughly the same; this has mostly to do with the orbits of Earth and Mars, less to do with local solar time. Of course, you can't tell the direction of time in this plot, but that's not the point!

The End of RS-DTEs

There is an incredible amount of input going into these observations; a hundred constraints and a hundred desires. Now we're driving and hunting down veins, leaving radio science to next Martian winter — about an Earth year and a half from now. 

I hope this was educational!


*No, I don't have a Twitter account. No, I don't want one. hashtag PleaseLeaveMeAloneAboutIt
**Fun fact: MSL will be using "mean" solar time — "LMST". It matters.

Sunday, May 13, 2012

[insert song about getting on the road again]

This actually came sooner than I expected: Opportunity drove for the first time since Sol 2795. From Phil Stooke's post:

Paolo the RP wandered into the UMSF forums the other week to announce things like planned (and now executed) "drive direction imaging" and a "turn for comm." The words couldn't have fallen more welcome on our ears.

For now, it'll be step-by-step kind of drives. We still need those recharge sols, and I'm not exactly clear* on the possible continuation of Radio Science. (Speaking of: Part 2 of 2 of this in the works!)

Winter was a challenge — but obviously one we could manage. Next up? Sniff out Morris Hill, maybe do some drive-by IDD'ing of the other potential juicy bits of Cape York, and then it's off to Cape Tribulation. Funny to think that Cape York was this big fat mystery to us when we arrived; it marked, and has since represented in spirit, the arrival of the Opportunity rover at the massive Endeavour Crater. Now, we've done our business and are ready to move on to climb some mountains. It's another new adventure, and this time we don't have to trek tens of kilometers to get there in the first place.


*Not exactly around a lot in MER town these days. Sad.

Friday, May 4, 2012

First Paper about Endeavour is out! (paywall'd)

Here it is! Unfortunately, paywall-blocked. I guess it got published a month ago, but just now showed up on Slashdot.

But hey, it's out there… and it's been in the works for quite a while now.