Importing Mount Wilson filtergrams to the MDI pipeline

A R Jones
1998.11.17

Introduction
Image Conversion
Image Identification Strategy
FITS Header
Additional Information
Automation Scripts
- doproc1
  - dorm
  - dosrfits
  - doftp

Introduction

The background ideas about reprocessing the MWO data for the MDI pipeline are given in the MDI TechNote 98.139 . This document describes the first step in the process; converting the MWO data from sunio format to standard FITS files. This is a two part operation, first actually reading the raw data files from exabyte tape, and then identification of the individual images.

Image Conversion

A modified version of sunio (V 960919 14:10 drs) is the basis of the image conversion utility called srfits. Unlike sunio, srfits is not designed as a general pupose image reading utility, it has only been tested on machines from Sun Microsystems and exabyte tape drives. There is a known problem with sunio and the LINUX kernel, and the same problem will also apply to srfits.

Sunio has been modified in several ways:

rather than just directly streaming from the tape to disk, the data is buffered into images, so that on-the-fly calculations can be made.
the fits headers are written after the image has been buffered and subsraters calculated so that the information can be included in the header
default settings for writing the correct format fits files are set in the code, they can be overridden by normal sunio runtime options

Image Identification Strategy

The current strategy relies on the statistics from 3 image subrasters from a horizontal strip along the center of the CCD (shown in figure 1) and a vertical subraster averaging 2 horizontal pixels. This is not an ideal method, but it works well most of the time. As this will probably be the slowest step in the pipeline, I do not want to make it too much of a bottle neck. The data from the subrasters has been suplememted by calculation of atan(N-S)/(W-E) where N, S, E, W represent simple averages over the entire north, south, east and west hemispheres. The N-S line is used to calculate an FFT to check for ronchi images, as other techniques proved unreliable.

Figure 1 Location of subrasters for image identification

The image statistics calulated are:

allav, allvar: mean and variance pixel value from the entire frame
sr1av, sr1var: mean and variance in subraster 1
sr2av, sr1var: mean and variance in subraster 2
sr3av, sr1var: mean and variance in subraster 3
i-mod: (sr2av - sr1av)/(sr1av + sr2av - 2*sr3av)
north, south, east, west: mean values averaged over the hemispheres
atn2: atan2(north - south),(west - east)

The basic strategy is that an image is first labeled UNKN (unknown), only if it passes a set of tests is it assigned an image type:

FLTR: The filtergrams are the main data product. I have tried various strategies to try and identify `red' and `blue' filtergrams, but none work 100% reliably. The keywords i-mod and atn2 give good clues to the typr of filtergram, but the only reliable way to determine which wing was used is to form Dopplergrams. `if((sr1av + sr2av) > 3 *sr3av){` `strcpy(obstype,filt);` `}`	DARK: The dark frames are taken at the beginning of each observation day, and are used to correct the filtergrams for dark noise. If the 3 equatorial subrasters are the same to within 10 DN the image is classified as dark `if((sr1av - sr3av) < 10 && (sr2av - sr3av) < 10` `&& (sr1av - sr2av) < 10){` `strcpy(obstype,dark);` `}`
Figure 2 Example Filter (`FILT`) Image	Figure 3 Example Dark (`DARK`) Image

RNCI: Ronchi images are recorded at the beginning of each observation. The rulings are aligned with the Earth's E-W axis by drifting the solar image along the rulings. `for(fftindex = 20; fftindex < 30; fftindex ++){` `rnkict = rnkict + fftarray[fftindex];` `}` `for(fftindex = 35; fftindex < 45; fftindex ++){` `rnkiav = rnkiav + fftarray[fftindex];` `}` if((rnkiav/rnkict) > 2.0){ strcpy(obstype,ronchi); }	TEST: The `test' images are written at the begining of an observation to tension the tape. There are several diferent version of the observing code that write different images, and as I find new `test' images they will have to be added. The latest version of the observing code does not write images. `if((tmp1[51700]+tmp1[51701] + tmp1[51702] +` `tmp1[51703]) == 0){` `strcpy(obstype,tension);` `}`
Figure 4 Example Roncki (`RNCI)` Image	Figure 5 Example Test (`TEST`) Image (there are many variants)

FITS Header

SIMPLE =                    T / Written by srfits
BITPIX =                   16 / bits/pixel
NAXIS   =                    2 /
NAXIS1 =                 1024 / Lines
NAXIS2 =                 1024 / Samples/Line
DATE    =         '1997/07/17' / Observation date yyyy/mm/dd
TIME    =           '06:40:31' / Observation time, PDT
KTIME   =           1704483161 / Observation time, 100th sec since...
STATUS =                    0 / Observation status
TYPE    =                 FILT / Type of image Frame
ORIENT =                 SESW / Image orientation
I-MOD   =                   47 / Filtergram modulation
ATN2    =                  113 / 1000*atn2(N-S),(W-E)
END

The value of the ORIENT keyword is based on the date of the observation. Early data was taken with a mirror feeding the CCD.

Additional Information

As well as the header the srfits program generates output about each image which can be piped to a log file, e.g.

19930131:175001.fits file #: 0000 is: TEST imod: 0031 atn2: -1567
19930131:175006.fits file #: 0001 is: TEST imod: 0050 atn2: -1567
19970720:060131.fits file #: 0002 is: RNCI imod: 0013 atn2: -2702
19970720:060136.fits file #: 0003 is: RNCI imod: 0050 atn2: -2979
19970720:060231.fits file #: 0004 is: RNCI imod: 0014 atn2: -2731
19970720:060236.fits file #: 0005 is: RNCI imod: 0050 atn2: -2985
19970720:060331.fits file #: 0006 is: DARK imod: -065 atn2: 00230
19970720:060336.fits file #: 0007 is: DARK imod: -069 atn2: 00232
19970720:060431.fits file #: 0008 is: DARK imod: -073 atn2: 00250
19970720:060436.fits file #: 0009 is: DARK imod: -064 atn2: 00202
19970720:060531.fits file #: 0010 is: FILT imod: 0013 atn2: -2926
19970720:060536.fits file #: 0011 is: FILT imod: 0050 atn2: -3054
19970720:060631.fits file #: 0012 is: FILT imod: 0013 atn2: -2923
19970720:060636.fits file #: 0013 is: FILT imod: 0049 atn2: -3055
19970720:060731.fits file #: 0014 is: FILT imod: 0013 atn2: -2919
19970720:060736.fits file #: 0015 is: FILT imod: 0049 atn2: -3054
19970720:060831.fits file #: 0016 is: FILT imod: 0013 atn2: -2924
19970720:060836.fits file #: 0017 is: FILT imod: 0049 atn2: -3056
19970720:060931.fits file #: 0018 is: FILT imod: 0013 atn2: -2926

...

19970720:173936.fits file #: 1399 is: FILT imod: 0055 atn2: -3017
19970720:174031.fits file #: 1400 is: FILT imod: 0021 atn2: -2885
19970720:174036.fits file #: 1401 is: FILT imod: 0055 atn2: -3016

Automation Scripts

The aim of the project is to allow automatic processing and archiving of data from Mount Wilson by the MDI pipeline at Stanford. The process is driven by shell scripts. The flow of data is shown below:

Raw data tapes will be read using srfits from one ofthe 8mm jukebox drives to a local disk on kala.usc.edu to a dedicated 9GB drive
srfits creates the conformant fits files in a single directory, but also creates script files to move the files to a YYYY/MM/DD/HH directory structure
fits files and shell scripts are ftpd to tarax.stanford.edu
A remote shell job to sort the files into the correct directory structure and flag new data is run
local files on kala.usc.edu are removed
the process is repeated

The processing will be `pushed' from USC due to the limited tapedrive and disk resources. Initial timing of the various processes shows:

srfits reading data on kala.usc.edu	4.5 hrs
FTP data to tarax.stanford.edu	1 hr
move files to correct directory structure on tarax	?
remove files from kala	2 min

This indicates that it is reasonable to process 4days of raw data /day. The 8mm jukebox can be configured to sequentially process 10 tapes before requiring re loading, so the process can be left unatended for 2 days at USC.

doproc1                                       # main routine
   dorm                                       # remove old files
   dogetfree                                  # check free space on kala-03
   dosrfits                                   # read and process data tape
   doftp                                      # ftp data to stanford
              # the next 3 comands would
              # normally execute by rsh
              # at Stanford ( the .sh files
              # are created by srfits and
              # ftpd with the data files )

   chmod 700 *.sh                             # make move shell scripts
                                              #     executable
   dopath.sh                                  # make the directories
   domove.sh                                  # move the data

mail ajones@solar.stanford.edu < proc.log # let me know it finished

   dorm                                                                                                                     # remove old files
      rm -rf tmp
      rm -rf tmp1
      rm -rf tmp2
      rm -rf tmp3
      rm -rf *.sh
      rm -f *.fits
      rm -rf su*

dosrfits # read and process data tape
./srfits -t$TAPE -m

   doftp                                      # ftp data to stanford
      rm -rf 1992*.fits                       # TEST images are
      rm -rf 1993*.fits                       # 1992 or 1993
      ftp -in < testftp

last updated: 17 December 1998
comments and suggestions to: ajones@solar.stanford.edu