Log for tohban anomalies

From time to time, an anomaly comes up that the tohban (or someone else on the team) needs to deal with. This page is meant to serve as a record for spacecraft operations, parameters, etc that were out of the ordinary in case that same event happens again. Anomalies or issues with the detector health should be recorded on the dedicated Detector Updates page and not this one.

The list is not meant to be exhaustive, but may aid a tohban in determining if a problem has happened before and what to do about it. The list starts in February of 2013; earlier events are not included. (Unless some intrepid soul wants to go through and add some history...!)

• 2013 February 10. The ADP froze and required a reset, which was performed the next day. Symptoms of the problem were a red error on the ADP program checksum on the spacecraft page as well as zero triggered limbs despite the spacecraft's being in sunlight. Martin was contacted and the ADP was reset. An ADP freeze happens once every year or two and can manifest itself in many ways (not just this one).

• 2013 May 3-4: There was a spin rate snafu. Occasionally the magnetic field configuration is such that the ACS system does not sense any change in magnetic field as it spins, thinks something is wrong, and after a timeout period (5 min) goes into IDLE mode. This problem happens (usually a few occurrences close together) about once a year or so. The ACS system was commanded back to precession mode on the next available pass. No ill effects remained from this issue. Here is the sequence of events:

 2013-123-16:11 - ACS system went into IDLE mode due to spin rate miscalculation.
 2013-124-02:08 - Ground command to precession mode
 2013-124-03:42 - Autonomously transitioned back to Normal mode.

• 2015 Aug 21: Late on the 20th (early on the 21st UTC), detector 9 started spouting out ULD events and quickly filled up the SSR. Once this problem was identified on Friday, detector 9 events (both front and rear) were disabled, and the write pointer motion slowed to a reasonable value. The read pointer was skipped in order to empty the SSR of these bad events (though of course some good data was thrown out with the bath water; see summary below). Since the D9 resets were high (~30k), the HV was turned far down to 900V, and then to 800V, in case there was arcing, in order to prevent permanent damage. At this voltage, resets were ~23k. It was seen later that the ULDs stopped going crazy at the time of this change. (The hyperactive ULD rate never showed in the monitor rates that we see during the passes, which let to delays in its diagnosis.) See attached for a plot of D9 events in the event list, and see this link for the monitor rate plots, at the time D9 started its ULD tantrum.

[See http://sprg.ssl.berkeley.edu/~tohban/browser/?show=monru&date=20150820&time=201739&bar=1]

D9 events were left off and the detector was left at 800V over the weekend, so that there would be no danger of a repeat occurrence during a time when tohban/ops reaction time is likely to be slower. The following Monday we turned on D9 events for a partial orbit to assess the state. While the write pointer speed did increase during the pass, we believe this was due to other changes we made around the same time (fast threshold raises, with corresponding increases in slow rates due to higher livetime). Examination of the VC1 plots shows that the ULDs remained at the “normal” value.