february 08-10, 2010 falda del carmen, córdoba, argentina

Mission Operations Review

February 08-10, 2010Falda del Carmen, Córdoba, ARGENTINA

DAY 2SECTION 08

Aquarius Data Processing System (ADPS)

Gene Carl FeldmanFred Patt

2 of 71February 8-10, 2010

Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review

MOR Agenda – of 3

3 of 71February 8-10, 2010


• Overview and Status• Key Requirements• Previous Reviews• System Interfaces• Key Processes

– Data Acquisition and Ingest– Science Data Processing– Algorithm Integration, Testing and Evaluation– Data Archive and Distribution– Data Quality Control– Validation Support– Periodic Reprocessing

• Facilities/System Configuration/Backup Strategy/ IT Security• Integration and Testing• Staffing, Roles and Responsibilities• Documentation• Risks and Issues

Agenda

4 of 71February 8-10, 2010


Aquarius Data Processing System Overview

• The Aquarius Ground System at NASA/GSFC consists of the Aquarius Data Processing System (ADPS) and the Aquarius Command and Control System (ACCS - next presentation)

• The ADPS will support:– Acquisition and ingest of Aquarius/SAC-D data from CONAE.– Acquisition of dynamic ancillary required to support the science processing.– Processing of Aquarius data to standard product levels using algorithms provided

by the Science Team.– Quality control of data products using methods specified by the Science Team.– Match-up of Aquarius data with in situ salinity measurements.– Archive and distribution of all data products during the active life of the mission to

the science team and general science community.– Delivery of source data (Level-1a) and “validated” Level-2 and Level-3 derived

data products to the PO.DAAC.

• The ADPS will be supported within the existing infrastructure of the Ocean Biology Processing Group (OBPG).

5 of 71February 8-10, 2010


Ocean Biology Processing Group

http://oceancolor.gsfc.nasa.gov

• Located: Goddard Space Flight Center

• Current Archive size: > 900 TBytes

• Distribution (> 2/04): 53 million files

● All missions are supported within the framework and facilities of the current NASA Ocean Data Processing System (ODPS) which has been successfully supporting operational, satellite-based remote-sensing missions since 1996, and its capabilities continue to evolve and expand to meet the demands and challenges of future missions

MISSIONS Currently SUPPORTED

• SeaWiFS : 1997 - present• MODIS (Terra and Aqua) : 2000 -

present• MERIS (US Coastal Waters) Feb 2009 -

present • CZCS / Nimbus-7 : 1978 - 1986• OCTS / ADEOS-I : 1996 - 1997• Glory data system prototype : 2010

launch• Aquarius / SAC-D : 2010 launch• VIIRS / NPP : 2011 launch• Community Processing & Analysis

Software SeaDAS (1991- present)

6 of 71February 8-10, 2010


Status - Science Code only

• If Aquarius were launched today and downlink files in the format expected showed up on the CONAE data server, we could Operationally :– Detect, transfer and ingest the files

– Reformat the files eliminating duplicate records

– Create temporary Level-1A data files and perform telemetry monitoring

– When subsequent downlinks are received, complete Level-1A orbit files produced

– Ingest and preprocess current set of required ancillary data

– Process to Level-2 salinity products and make products (Level-1A & 2) available via the existing web browse, download and order tool

– Process Level-2 files to Level-3 maps and make products available via web browse/download tool.

However

7 of 71February 8-10, 2010


• From now until launch we need to:

– Obtain and incorporate sample SAC-D telemetry files from CONAE into the Level-1A file structure (TBD)

– Receive updated Level-1b and Level-2 processing code from Frank Wentz (November 2009)

– Receive Scatterometer processing code from JPL (November 2009)

– Review and verify delivered code and merge simulated data sets (December 2009)

– Integrate Radiometer and Scatterometer code into data processing system (Jan/Feb 2010)

– Develop ingest procedures for additional ancillary data required by new code (Jan/Feb 2010)

– Verify output products with algorithm providers (February 2010)

– Process one year simulated data set and make available to science community (March 2010)

– Begin real-time mission simulation (March/April 2010 --> Launch)

– Verify Downlink file and SAC-D Housekeeping telemetry format/content (TBD)

– Participate in mission rehearsals as scheduled

Status - Science Code TBD

8 of 71February 8-10, 2010





Agenda

9 of 71February 8-10, 2010


ADPS Key Requirements

• The ADPS shall receive, log, process and archive:– Level 0 raw science data– Aquarius HKT– Service platform HKT (including attitude data)– Ephemeris data

• The ADPS shall acquire and utilize ancillary data products to process Aquarius science data.

• The ADPS shall process science data to Level 1, 2 and 3 products.• The ADPS shall deliver Aquarius data products to the Science Team and

other users.• The ADPS shall deliver data products to the JPL PO.DAAC.• The ADPS shall support Aquarius data product validation by the Aquarius

Validation Data System (AVDS).• The Aquarius Ground System shall utilize and expand upon the GSFC

OBPG facilities as required.• The ADPS shall support reprocessing of the full Aquarius mission in six

months or less.• The ADPS shall support the archive of the full Aquarius mission data set.

10 of 71February 8-10, 2010


Previous Reviews

• Aquarius Preliminary Design Review – June 2005• Aquarius Science/Ground System Pre-CDR – July 2006• Aquarius 1st Science Algorithm Workshop – March 2007• Aquarius 2nd Science Algorithm Workshop – March 2008• ACCS/ADPS Peer Review – March 2008• Aquarius/SAC-D Ground Segment CDR – July 2008• Aquarius Data Processing System Readiness Review – July 2009• Joint CONAE/NASA Mission Operations Working Group – Sept,

Dec 2009• ACCS/ADPS Peer Review – January 2010

All RFAs from previous reviews have been closed.

11 of 71February 8-10, 2010





Agenda

12 of 71February 8-10, 2010


Ground System Interfaces

AQUARIUS Ground System

13 of 71February 8-10, 2010



ADPS ACCSAquarius AquariusData Command &Processing ControlSystem System

14 of 71February 8-10, 2010




CONAEData Mission RFArchive Operations Services

CommandCenterAquarius

Data, SAC-D Telemetry and OrbitProducts

CUSS CODS

1- SAC-D Ground System to Aquarius Ground System ICD

15 of 71February 8-10, 2010





CommandCenter

Aquarius Command SchedulesAnd Reports

Schedule ConfirmationAnd Reports

1- SAC-D Ground System to Aquarius Ground System ICD

Aquarius Data, SAC-D Telemetry and OrbitProducts

CUSS CODS

16 of 71February 8-10, 2010





CommandCenter


S-Band

X-Band

Wallops Svalbard Alaska McMurdo

NASA Ground Network

S-Band

CommandsReal-TimeTelemetry &Coordination


X-Band

ASI Matera

Cordoba

ASI Malindi

1- SAC-D Ground System to Aquarius Ground System ICD2- SAC-D / NASA PSLA, Network Requirements Document & Scheduling Interface3- SD - Observatory Ground IF Specification Document


CUSS CODS

17 of 71February 8-10, 2010





CommandCenter


S-Band

X-Band


NASA Ground Network

S-Band



X-Band

ASI Matera

Cordoba

Principal Investigator andScience Planning Team

Calibration &Instrument

ConfigurationChange Requests

InstrumentEngineering TeamAquarius Instrument Testbed

Flight Software Updates

ASI Malindi

1- SAC-D Ground System to Aquarius Ground System ICD2- SAC-D / NASA PSLA, Network Requirements Document & Scheduling Interface3- SD - Observatory Ground IF Specification Document4- Aquarius Science Team to Aquarius Ground System ICD5- AQ - Aquarius Ground System – Aquarius Instrument Team Anomaly ICD and Aquarius Test bed User Manual


CUSS CODS

18 of 71February 8-10, 2010





CommandCenter


S-Band

X-Band


NASA Ground Network

Science Team andAlgorithm Providers

S-Band

Algorithms &Science Code

Evaluation Products

Ancillary Data Sources



X-Band

ASI Matera

Cordoba






ASI Malindi

1- SAC-D Ground System to Aquarius Ground System ICD2- SAC-D / NASA PSLA, Network Requirements Document & Scheduling Interface3- SD - Observatory Ground IF Specification Document4- Aquarius Science Team to Aquarius Ground System ICD5- AQ - Aquarius Ground System – Aquarius Instrument Team Anomaly ICD and Aquarius Test bed User Manual6- Aquarius Data Product User Guide (PO.DAAC) & Format Specification (ADPS)7- Ocean Data Processing System Data and Software Management Plan, ATBD’s and Software User’s Guides


CUSS CODS

19 of 71February 8-10, 2010





CommandCenter


S-Band

X-Band


NASA Ground Network


S-Band


Evaluation Products




AVDS Aquarius ValidationData System

Insitu DataBias Adjustments

Aquarius Match-up Data

X-Band

ASI Matera

Cordoba






ASI Malindi

1- SAC-D Ground System to Aquarius Ground System ICD2- SAC-D / NASA PSLA, Network Requirements Document & Scheduling Interface3- SD - Observatory Ground IF Specification Document4- Aquarius Science Team to Aquarius Ground System ICD5- AQ - Aquarius Ground System – Aquarius Instrument Team Anomaly ICD and Aquarius Test bed User Manual6- Aquarius Data Product User Guide (PO.DAAC) & Format Specification (ADPS)7- Ocean Data Processing System Data and Software Management Plan, ATBD’s and Software User’s Guides8- A V D S to A D P S ICD


CUSS CODS

20 of 71February 8-10, 2010





CommandCenter


S-Band

X-Band


NASA Ground Network


S-Band


Evaluation Products




AVDS Aquarius ValidationData System PO.DAAC

Level 1, 2, 3Aquarius Data

Data Users andScience Team

Aquarius L1, 2, 3 DataAncillary DataFeedback

SeaDASSoftware

X-Band

ASI Matera

Cordoba






ASI Malindi

1- SAC-D Ground System to Aquarius Ground System ICD2- SAC-D / NASA PSLA, Network Requirements Document & Scheduling Interface3- SD - Observatory Ground IF Specification Document4- Aquarius Science Team to Aquarius Ground System ICD5- AQ - Aquarius Ground System – Aquarius Instrument Team Anomaly ICD and Aquarius Test bed User Manual6- Aquarius Data Product User Guide (PO.DAAC) & Format Specification (ADPS)7- Ocean Data Processing System Data and Software Management Plan, ATBD’s and Software User’s Guides8- A V D S to A D P S ICD9- Aquarius Ground System to PO.DAAC ICD

Insitu DataBias Adjustments

Aquarius Match-up Data


CUSS CODS

21 of 71February 8-10, 2010





Agenda

22 of 71February 8-10, 2010


Data Acquisition and Ingest

• Aquarius/SAC-D data downlink strategy comprises multiple, overlapping downlinks each day of Aquarius science high-rate data (HRD) and Aquarius and SAC-D housekeeping telemetry (HKT).– 3 to 5 passes per 24 hours over the Cordoba ground station.– 1 pass per week over Matera ground station, which is transferred

to Cordoba after pass.– 6 - 11 minutes per pass.– Downlink full science memory of 110 MB at 4Mbps (just under 4

minutes), repeating throughout pass.

• ADPS automatically acquires downlinks from the CUSS (separate files for Science, SAC-D/HKT) after each pass.

• Orbit Products including Ephemeris files are generated daily by the CODS and acquired (Two-line Elements, Predicted Ephemeris and Orbit Ephemeris, Ground Station Contacts, Ground Station Contacts Details, SAA Passes, Eclipses, Events,

Maneuvers List)• All files are ingested and cataloged automatically after

acquisition, and made available for processing.

23 of 71February 8-10, 2010





Agenda

24 of 71February 8-10, 2010


Aquarius Data Processing Development

Approach:1- Use of rapid prototypes evolving into ever more realistic full-mission simulations2- Verify that all requirements are being met3- All components of the system are being thoroughly exercised4- use of "real-world" testing of the end-to-end system on a routine basis - don’t just test for success

Philosophy:

1- Maintain Flexibility - things WILL change2- Strive for maintainability of code and documentation3- Testing, Testing and more Testing:

Get the mechanics right so that when data starts to flow, you can focus on the science

Implementation:1- Start with a full end-to-end data flow description2- Break into the component parts3- Use as close to real data and operational software as possible4- Continuously evolve the simulation to add more credible data, software and interfaces as they

become available

Result:1- All the supporting infrastructure, tools and procedures can be developed, tested and refined as the process continues2- Through this modular, flexible approach, the entire system can gradually evolve to a higher and higher level of fidelity until you have as close to a fully tested and functional launch-ready

system as possible prior to launch

25 of 71February 8-10, 2010


Algorithm Integration, Testing and Evaluation

working code (C, Fortran)

published algorithms

in-house development

written specifications (memos, emails)

prototype code (IDL, MATLAB?)

Science Software Development and

Algorithm Integration

Aquarius Teamalgorithms

issues and test results

Software Repository

Version Controlled Code and Tables

Including ACCS

software updates

ADPS Team

SeaDAS

Software Package

26 of 71February 8-10, 2010


Science Data Processing

The Aquarius science data processing consists of the following steps:

• Level-0 Preprocessor – process each HRD downlink file to produce a single time-ordered set of Aquarius science blocks and removes duplicates– developed by Joel Gales - ADPS.

• Level-0-to-1A – separate Level-0 files into orbits, unpack science data, incorporate overlapping ephemeris and SAC-D HKT and convert Aquarius HKT – developed by Joel Gales - ADPS.

• Telemetry Analysis – HKT data from Level-1A files analyzed for web display on the Aquarius web site – developed by Susie Kennison, Norman Kuring and Liang Hong - ADPS/ACCS.

• Interactive Telemetry Analysis - HKT files analyze using CONAE-provided telemetry viewer.

• Level-1A Merge – consolidate Level-1A files from overlapping downlinks into a single, best-quality full orbit product – Joel Gales - ADPS.

• Level-1A-to-2 – perform calibration, atmospheric correction and salinity retrieval for Aquarius science data – radiometer and scatterometer code delivered by RSS and JPL, respectively; integration and input/output code developed by Joel Gales and Liang Hong - ADPS.

• Level-2-to-3 Binning – Level-2 salinity retrievals for one day geographically projected and collected into equal-area bins – existing OBPG software modified by Joel Gales- ADPS.

• Level-2-to-3 Smoothing – Level-2 salinity retrievals for one day optimally interpolated into equal-area bins – algorithm by Jonathan Lilly/ESR implemented by Joel Gales - ADPS.

• Level-3 Binning and Mapping – Level-3 binned files aggregated to longer time periods (weekly, monthly, etc.) and projected onto map grid – existing OBPG software.

All processing steps are run automatically when all input data sets are available.All software exists in at least preliminary prelaunch form at this time.

27 of 71February 8-10, 2010


Aquarius Data Processing Overview

Level-0 Preprocessor

Level-0 to 1A

Level-1A to 2

Level-1A Merge

Raw Level-0Downlink

File

Level-1Aorbit

Level-2

Level-3 Map

SAC-DHKT Data

Level-2 to 3Binning

Level-2 to 3Smoothing

Level-3 Binning

Level-3 Mapping

DynamicAncillary

Data

StaticAncillary

Data

Level-1Atemp

Level-0

Level-3 Bin

Level-0Report

Level-1aReport

Automated TelemetryAnalysis Tool

Standard Data ProcessingWeb-Based Telemetry Monitoring Tools

TelemetryReport

PredictedDownlink

Times

In-Situ Match-upProcess

Level-2Match-upAVDSBias Adjustment

Web-Based Browse, Download and Order Tools

CONAE-providedInteractive Telemetry

Analysis Tool

SAC-DHKT Data

28 of 71February 8-10, 2010





Agenda

29 of 71February 8-10, 2010


Data Archive and Distribution

• Aquarius data product archiving at ADPS will be performed using the existing ODPS archive catalog database and RAID-based data storage system (> 1 PetaByte of current storage for SeaWiFS, MODIS, MERIS, OCTS, CZCS, etc.)

• Aquarius data distribution during the active life of the mission will be performed using the existing ODPS browse and order system (examples to follow).

• All source data and “validated” data products will be delivered to the JPL PO.DAAC for permanent archive, value-added services and long-term distribution (next slides).

30 of 71February 8-10, 2010


PO.DAAC Support for Aquarius

PO.DAAC (Physical Oceanography Distributed Active Archive Center) responsibilities:

• Long-term (Life of Mission and Beyond) data management, distribution & archive of the Aquarius mission data.

• Ensure data maintenance, usability, access, provenance and security.• Ensure integrity of Aquarius data through quality, quantity, continuity and latency

audits.• Ensure all Aquarius products conform to NASA’s file and metadata standards.• Ensure all Aquarius metadata are delivered to the NASA metadata registry (ECHO).• Implement and maintain PO.DAAC Aquarius ftp site and web pages.• Maintain data Aquarius data access tools including product/granule search,

OpenDAP, L2 and L3 subsetters.• Develop and maintain Aquarius product user guides and read software.• Answer user questions concerning Aquarius data.• Configure and implement the PO.DAAC Data Management & Archive System

(DMAS) to conform to the interface specification and product ingest requirements.• Report data access metrics.

31 of 71February 8-10, 2010


PO.DAAC Delivery Status

• Draft Interface Control Document (Completed September 19, 2008)• Draft Data Management Plan (Completed October 18, 2008)• Final Interface Control Document (February 2010)• Draft User Guide (February 2010)• Final Data Management Plan (May, 2010 2 months prior to

Operational Readiness Review)• Final User Guide (May, 2010)• DMAS adaptation (Ingest, Archive and Distribution tools) (March

2010 for Operations Readiness Test)• Aquarius FTP and web site (including read software) (June 2010 for

Operations Readiness Test)• Data Access tools (July 2010)

32 of 71February 8-10, 2010


Data Archive and Distribution

• Overview of the currently operational web-based browse, order and

distribution tool that is available to support the distribution of

Aquarius data within the Aquarius Data Processing System that also

serves as a data quality monitoring tool for both internal and

external use.

33 of 71February 8-10, 2010


Aquarius Level-1-2 Data Browserhttp://oceancolor.gsfc.nasa.gov/cgi/aquarius

34 of 71February 8-10, 2010



35 of 71February 8-10, 2010



36 of 71February 8-10, 2010


Aquarius Level-3 Data Browser

http://oceancolor.gsfc.nasa.gov/cgi/l3

37 of 71February 8-10, 2010


Beam 1 - daily



38 of 71February 8-10, 2010


All Beams - daily



39 of 71February 8-10, 2010


all beams-weekly

all beams-monthly



40 of 71February 8-10, 2010





Agenda

41 of 71February 8-10, 2010


Data Quality Control

• Automated telemetry monitoring, alarm notification and web tools updated on every downlink

• Interactive telemetry monitoring tools

• Automated format and science data quality checks performed on each file as defined by science team

• Visual inspection via web tools for Level-2 and Level-3 data products

• Level-3 Long-term statistics monitoring tool

• AVDS data analysis by Science Team

42 of 71February 8-10, 2010





Agenda

43 of 71February 8-10, 2010


Validation Support

• The Aquarius Validation Data System (AVDS) will statistically evaluate the Aquarius SSS data product using a global set of in situ salinity measurements (e.g., Argos buoys)

• The ADPS will support the validation activity by acquiring the in situ data from AVDS, performing matchups with the Aquarius Level-2 data products, and uploading the matchup files to AVDS.

• The rationale for the data exchange is to streamline this activity by using preexisting ADPS hardware, software, and satellite data processing facilities, thereby eliminating the redundant effort of developing parallel facilities.

• A successful data exchange simulation was accomplished using in situ water temperature measurements and the MODIS-Aqua 11-micron SST data product.

• The AVDS support testing will be included in the mission simulation described under Integration and Testing.

44 of 71February 8-10, 2010


AVDS-ADPS Data Exchange

in situ data uploaded (daily)

ftp://ftp.esr.org/pub/datasets/avds

ftp://samoa.gsfc.nasa.gov/aquarius

FTP SITESin situ data downloaded daily

limited to new files

in situ data ingested into database

satellite files located & extracted

level-2 hdf files generated

browse images & maps generated

seabass:/data3/adps

DATA STORAGEhttp://seabass.gsfc.nasa.gov/eval/adps.cgi

WEB SITE

satellite data uploaded & posted online

satellite data removed after 1-week

AVDS RESPONSIBILITY

45 of 71February 8-10, 2010





Agenda

46 of 71February 8-10, 2010


Periodic Reprocessing

• Level-1 Requirement: No later than twelve (12) months after the end of the IOC period, the Aquarius Project shall record, process, archive, and deliver the first release of Level 1A and Level 3 data products to a NASA Distributed Active Archive Center (DAAC),

• Level-1 Requirement: After the first release of Aquarius data, the Aquarius Level 1A and Level 3 data products shall be delivered to the DAAC no later than six months after measurements are taken in orbit.

• Level-1 Requirement: During the mission life, all products shall be available to the science community from the Aquarius Data Processing Segment (ADPS) and the final data products produced by the Aquarius Project shall be delivered to the DAAC within six (6) months after the end of the prime mission

47 of 71February 8-10, 2010


Periodic Reprocessing

• Development and test environments allow for multiple algorithm versions to be run on the entire mission data set to test the impacts of calibration and algorithm improvements.

• The results of these “reprocessings” are made immediately available to the science team members via web and ftp mechanisms

• Using SeaWiFS as a benchmark, the most recent reprocessing of the entire mission-long (ten year) data set required 10 hours or 1 year/hour. Aquarius is ½ the data volume of SeaWiFS so data storage and system throughput are NOT a concern. A better estimate of the actual processing throughput will be derived once we process the year-long Aquarius simulated data set later this month

• Once the calibration/algorithm changes have been validated and approved by the Science Team, a mission-long reprocessing will be initiated and the new products and documentation will be made available to the science community and PO.DAAC.

48 of 71February 8-10, 2010





Agenda

49 of 71February 8-10, 2010


OBPG Computing Facility

NASA Goddard Space Flight CenterGreenbelt, Maryland

Server location: Building 28 Room W220

- 900 square feet dedicated to the project

- Raised computer floor with under-floor A/C

- Primary and backup A/C units

- 80 KVA Powerware UPS unit with dual 50 KVA PDUs, additional 5 KVA rack-mounted UPS units

- Connections to GSFC high speed networks: SEN

50 of 71February 8-10, 2010


OBPG System Hardware

Processing Cluster46 processing nodes 15 TB

4 storage nodes 10.9 TB

Ingest Servers2 SeaSpace workstations 78

GB1 storage node 5.1 TB

Distribution Servers (ftp)

8 storage nodes 49.3 TB1 server 250GB

Distribution and Storage Servers (web)

Production: 3 processing nodes75 storage nodes 1.03 PB

Testing Cluster

Network Support Servers

Backup Servers2 storage nodes 19.2 TB

Extreme NetworksBlack Diamond 8810

Gigabit Ethernet switch480 ports

Cal/Val & QC Systems

Mission Operations Systems

Development Servers2 superstore nodes 10 TB

User Desktops

Subversion CM Servers2 superstore nodes 586 GB

Database Servers2 superstore nodes 16.4 TB

AQOPS Servers (web)Production: 2 processing nodes

51 of 71February 8-10, 2010


Aquarius Backup Strategy

• Data• On-site

• Downlinks, L0, L1a data files• primary copy on RAID6 storage node• mirror copy on separate RAID6 storage node, at file creation

• Science Products• primary copy on RAID6 storage node• mirror copy if desired, or regenerated from L1a

• Off-site• Downlinks: stored at CONAE• L1A archived at PO.DAAC

• Source Code• On-site

• primary copy on subversion server, RAID6• mirror copy on backup Subversion server, RAID6, nightly mirror

• Off-site• Subversion data copied to external disk, taken off-site every 1 month• Multiple copies, given to Project Manager, SE, Processing Manager

• Other• On-site

• AQOPS, Forum: database mirrored to backup server nightly• Telemetry Viewer, other: image backups periodically, manual

• Off-site• AQOPS, Forum: databases included in source code off-line backups• Telemetry Viewer, other: image backups stored off-line like source code

52 of 71February 8-10, 2010


IT Security Plan

• ADPS and ACCS: covered under the same plan as other Ocean Biology Processing Group (OBPG) systems.

• NASA IT Security Plan Identification– System name: Sciences and Exploration Directorate (SED) Multi-

Program/Project (MPP) IT Science Systems– System Abbreviation: SED MPP– System Unique Identifier: CD-014-L-GSF-6004– Information System Type: General Support System– Information System Security Categorization: Low (transitioning to

medium within the year)– Information System Operational Status: Operational

• Effective 5/31/2007. Passed Certification & Accreditation in Summer 2007 - in process of being upgraded

• Plan designated Sensitive But Unclassified by NASA

• Implementation questions: addressed off-line

53 of 71February 8-10, 2010





Agenda

54 of 71February 8-10, 2010


Integration and Testing

• ODPS Testing Philosophy:

– As described in the Aquarius Data Processing Development section

• Phased approach to testing: get one thing working and move on to the next step until entire end-to-end stream is operational.

• Testing Aquarius support to be done within operational ODPS environment, with products clearly identified for separation from operational product stream.

• Testing stages:– Interfaces

– Internal functionality (ingest, processing, evaluation, distribution)

– End-to-end (Mission level testing, simulations and rehearsals)

• Test data sources:– Mission simulation

– Aquarius I&T data

55 of 71February 8-10, 2010


Functional Testing

• Functional testing has been completed for most OBPG developed software using simulated Aquarius data:– Level-0 Preprocessor

– Level-0-to-1A*

– Level-1A Merge

– Telemetry Analysis

– Level 2-to-3 Binning**

– Level 2-to-3 Smoothing**

– In situ Matchup**

• Functional testing of Telemetry Analysis software is in progress using I&T files from Bariloche.

• Functional testing for updated Level-1A-to-2 software will begin March 2010

* Need to re-test when final SAC-D HKT format is implemented and made available.

** Need to re-test with new file formats once updated code has been implemented (March 2010)

56 of 71February 8-10, 2010


Interface Testing – Current

• The CONAE MOC interface has been tested using one month of simulated downlink files staged on interim SFTP site, acquired and ingested by ADPS. Recent tests have also been conducted on the CONAE CUSS data server. Anticipate testing CODS interface and products along with actual SAC-D HKT telemetry file in March.

• The Science Team interface has been tested via algorithm code deliveries and distribution of data products generated from simulated data.

• The AVDS interface was tested using SST in situ and MODIS data in 2006 and is awaiting update to processing software to resume.

• The Ancillary Data interfaces for current data sources are exercised routinely as part of ongoing data processing activities.

57 of 71February 8-10, 2010


Interface Testing – Planned

• The CONAE MOC interface using the CUSS and CODS will be tested according to the CONAE I&T schedule:

– Simulated Aquarius downlink files, HKT and ephemeris staged by the Aquarius GS (ongoing).

– Sample SAC-D orbit products and HKT files provided by CONAE (March)

– Aquarius collected during I&T (ongoing) .

– The X-band end-to-end test proposed by NASA (next page).

• This interface, and the Aquarius processing software, will also be exercised extensively during the Mission Simulation (following pages).

• The Ancillary Data interfaces for new sources identified as part of the recent Science Code delivery will be tested in parallel with the functional testing of the integrated software.

• The PO.DAAC interface will be tested starting in March 2010 once final data products with complete formats are generated.

58 of 71February 8-10, 2010


X-band End-to-end Test Configuration

Goal is to verify the downlink file

format containing actual Aquarius

instrument data as produced and

delivered by the CONAE MOC. This

test is absolutely CRITICAL to verify

the actual downlink file format upon

which all of our subsequent

software is based.

Currently scheduled for April 2010

which provides enough time to

resolve any differences

59 of 71February 8-10, 2010


Aquarius Mission Simulation

• The Aquarius Mission Simulation will be a running, day-by-day exercise of the Aquarius/CONAE MOC interface and the ADPS.

• The simulation will follow this daily sequence of events:– The Radiometer (RSS) and Scatterometer (JPL) algorithm teams will

deliver a day’s worth of simulated instrument data to the Aquarius GS.

– The GS will generate simulated downlink (HRD and HKT) files for a typical pass schedule and a simulated daily ephemeris file.

– The simulated files will be staged on the CONAE servers.

– The ADPS will automatically detect, acquire and ingest the files, and process them through all product levels.

– The products will be archived, cataloged, and made available for distribution through the Browse and Order interface and to the PO.DAAC.

– The Aquarius Science Team will use this simulation to test their tools, data quality monitoring processes and anomaly detection.

• The Mission Simulation will commence in early March 2010 and will continue until the Aquarius/SAC-D launch.

60 of 71February 8-10, 2010


Aquarius Simulated Data Set

The Mission Simulation starts with simulated sensor data generated by members of the Aquarius Science Team, using assumed salinity fields and spacecraft orbit data, actual ancillary data and models of the instrument response

61 of 71February 8-10, 2010


Aquarius Simulated Data Set

Once the simulated sensor files are created, these are formatted into Aquarius science blocks and packaged into realistic downlink files based on predicted contact times and durations which are as close to the actual downlink data format as possible given the currently available information and sample telemetry

62 of 71February 8-10, 2010


Integration and Test Schedule

• Most of the interfaces defined in slide #20 have already been tested with first generation data products.

• The Aquarius Data Processing System will be ready to test all interfaces and participate in all Mission Level tests with at-launch data products in March 2010

• Subsequent tests and schedules are presented as part of the Mission Readiness Testing Section 17 presentation

63 of 71February 8-10, 2010





Agenda

64 of 71February 8-10, 2010


Ground System Staffing

NASA Management - Gene Feldman: Oversight of the overall effort, communication/coordination with other NASA organizations and centers and CONAE; oversight of science support, algorithm integration, QC, validation and in situ data support.

Contractor Management - Fred Patt: Oversight of contract support staff, including hiring, tasking, review of deliverables and schedules; ADPS interface definition and product format specification.

Mission Operations - Susie Kennison: Technical lead for instrument operations and commanding support; interface to instrument engineers and CONAE mission operations.

Data Systems Engineering - John Wilding: Technical lead for the data system integration effort; overall system design, implementation and testing.

Software Engineering – Joel Gales: Technical lead for the level conversion / product generation software, including development of the Level 0-to-1 software and integration/packaging of the science (Level 2 and 3) processing software to be provided by the algorithm developers.

Data Systems Support - Dan Knowles: Integrate the product generation software into the existing automated processing system, develop ingest modules, develop database tables for archive products, develop schedules and recipes for data acquisition, production and migration.

Systems Administration - Paul Smith: Specify and acquire new processing and data storage hardware, integrate into existing data processing facility, verify/upgrade network capability for external data transfers.

Quality Control and Validation - Liang Hong: Working with the Science Team, develop methodologies, tools, procedures and reports for operational QC; specify data and develop algorithms and tools for data and science algorithm validation.

In Situ Data Support - Jeremy Werdell: Work closely with the Aquarius Validation Data System (AVDS) to develop methods and tools for data analysis and quality control, building upon the existing SeaWiFS Bio-optical Archive and Storage System (SeaBASS).

Archive and Distribution - Norman Kuring: Incorporate Aquarius product browse, search, order and distribution into existing web-based Ocean Color capability and work with JPL PO.DAAC regarding “institutional” archive support requirements.

65 of 71February 8-10, 2010





Agenda

66 of 71February 8-10, 2010


ADPS Documentation

Name Status Version Date

Aquarius Ground System Level-3 Requirements (AQ-326-0152) Baselined

L3 Aquarius Science Algorithm Requirements (AQ-322-135) Baselined March 30, 2008

Aquarius JPL to GSFC Operations Transition Plan (AQ-211-0261)

Preliminary November 25, 2009

NASA/GSFC Ocean Data Processing System Operations and Data Management Plan

Approved April 2, 2009

Aquarius Ground System Integration Test & Training Plan (AQ-336-0244)

Revision

SAC-D Ground System to Aquarius Ground System ICD (AS-336-0151b)

Review July 14, 2008

AVDS to ADPS ICD (AQ-332-0331) Signed December 18, 2006

Aquarius Science Team to Aquarius Ground System ICD (AQ-336-0243)

Review June 17, 2009

Aquarius Ground System to PO.DAAC ICD (AQ-336-0151d) Review January 4, 2010

Level-1A Product Format Specification (AQ-376-0238) Final September 22, 2009

Level-2 Product Format Specification (AQ-376-0238) Preliminary October 14, 2009

Level-3 Product Format Specification (AQ-376-0238) Review May 26, 2009

67 of 71February 8-10, 2010


Aquarius Website

Consolidated Data access,

Information,

Services and

Community Feedback

68 of 71February 8-10, 2010


Aquarius Wiki

Online Documentation and Software Descriptions available online at:

http://aquarius.nasa.gov/

Under Data and Documents Link labeled “Aquarius Ground System and Mission Operations Wiki Documentation”

Major WIKI documentation update planned in March/April 2010

69 of 71February 8-10, 2010




• Facilities/System Configuration/Backup Strategy/ IT Security• Integration and Testing• Staffing, Roles and Responsibilities• Documentation• Concerns and Summary

Agenda

70 of 71February 8-10, 2010


Key Concerns

• Awaiting delivery of actual CONAE-produced data products (Aquarius science data in downlink format, SAC-D HKT data file and CODS-produced orbit products) to verify formats and contents and resolve any discrepancies or modify our software as required. Anticipated delivery of preliminary versions in March 2010 and key Aquarius data test in April 2010)

• Mission rehearsals are not yet clearly defined in scope, participation and success criteria and overall test coordination needs to be clarified .

71 of 71February 8-10, 2010


Summary

• Aquarius Data Processing System is being built within an operational and already proven and well-tested end to end data system serving multiple missions in a modular, flexible and easily expandable manner.

• Implementation of major components of the science data processing system are in place from ingest to data distribution.

• Smooth and seamless integration of improved algorithms into the development, test and operational environments as they become available.