Surface¶
Data directory structure¶
The Surface data is organized as xml files every 15 minutes :
> ls -la /g3/tianwh/share/conobs/rec_RSURF_20190*
-rw-r--r-- 1 tianwh obs 11482070 9月 25 00:59 /g3/tianwh/share/conobs/rec_RSURF_2019060100_g_06.dat
-rw-r--r-- 1 tianwh obs 11654320 9月 25 00:59 /g3/tianwh/share/conobs/rec_RSURF_2019060106_g_06.dat
-rw-r--r-- 1 tianwh obs 11637320 9月 25 00:59 /g3/tianwh/share/conobs/rec_RSURF_2019060112_g_06.dat
-rw-r--r-- 1 tianwh obs 11630570 9月 25 00:59 /g3/tianwh/share/conobs/rec_RSURF_2019060118_g_06.dat
-rw-r--r-- 1 tianwh obs 11549820 9月 25 00:59 /g3/tianwh/share/conobs/rec_RSURF_2019060200_g_06.dat
-rw-r--r-- 1 tianwh obs 11672570 9月 25 00:59 /g3/tianwh/share/conobs/rec_RSURF_2019060206_g_06.dat
-rw-r--r-- 1 tianwh obs 11687820 9月 25 00:59 /g3/tianwh/share/conobs/rec_RSURF_2019060212_g_06.dat
-rw-r--r-- 1 tianwh obs 11619320 9月 25 00:59 /g3/tianwh/share/conobs/rec_RSURF_2019060218_g_06.dat
-rw-r--r-- 1 tianwh obs 11566570 9月 25 00:59 /g3/tianwh/share/conobs/rec_RSURF_2019060300_g_06.dat
-rw-r--r-- 1 tianwh obs 11562820 9月 25 00:59 /g3/tianwh/share/conobs/rec_RSURF_2019060306_g_06.dat
-rw-r--r-- 1 tianwh obs 11619320 9月 25 00:59 /g3/tianwh/share/conobs/rec_RSURF_2019060312_g_06.dat
-rw-r--r-- 1 tianwh obs 11468820 9月 25 00:59 /g3/tianwh/share/conobs/rec_RSURF_2019060318_g_06.dat
-rw-r--r-- 1 tianwh obs 11398070 9月 25 00:59 /g3/tianwh/share/conobs/rec_RSURF_2019060400_g_06.dat
-rw-r--r-- 1 tianwh obs 11605820 9月 25 00:59 /g3/tianwh/share/conobs/rec_RSURF_2019060406_g_06.dat
......
Data format¶
CIMISS format data is looks like:
RSURF 2019 6 1 0 45928 6 9 9
00081 81.00 -5.53 -36.87 19.00 1.00 100.00
1013.20 23.05 21.35 347.00 0.00 999999.00 999999.00 999999.00 1011.00 8.00 8.00 8.00 8.00 8.00 8.00 8.00 8.00 8.00
00081 81.00 -5.53 -36.87 19.00 1.00 0.00
1012.80 24.15 21.75 345.00 0.10 999999.00 999999.00 999999.00 1010.60 8.00 8.00 8.00 8.00 8.00 8.00 8.00 8.00 8.00
00081 81.00 -5.53 -36.87 19.00 1.00 200.00
1012.90 22.25 21.25 333.00 0.00 999999.00 999999.00 999999.00 1010.70 8.00 8.00 8.00 8.00 8.00 8.00 8.00 8.00 8.00
00086 86.00 -19.98 -43.96 1214.00 1.00 0.00
1016.00 20.75 14.25 289.00 1.80 999999.00 999999.00 999999.00 882.20 8.00 8.00 8.00 8.00 8.00 8.00 8.00 8.00 8.00
00086 86.00 -19.98 -43.96 1214.00 1.00 100.00
1016.30 20.25 14.35 306.00 1.50 999999.00 999999.00 999999.00 882.30 8.00 8.00 8.00 8.00 8.00 8.00 8.00 8.00 8.00
00086 86.00 -19.98 -43.96 1214.00 31.00 2100.00
1015.10 21.35 13.95 288.00 3.90 999999.00 999999.00 999999.00 881.70 8.00 8.00 8.00 8.00 8.00 8.00 8.00 8.00 8.00
00086 86.00 -19.98 -43.96 1214.00 1.00 200.00
1016.30 20.45 14.35 283.00 2.50 999999.00 999999.00 999999.00 882.40 8.00 8.00 8.00 8.00 8.00 8.00 8.00 8.00 8.00
01001 1001.00 70.94 -8.67 10.00 31.00 2200.00
1015.80 -0.40 -1.80 341.00 10.70 999999.00 999999.00 999999.00 1014.60 0.00 0.00 0.00 0.00 0.00 8.00 8.00 8.00 0.00
01001 1001.00 70.94 -8.67 10.00 1.00 200.00
1014.60 -0.40 -1.97 337.00 10.30 999999.00 999999.00 999999.00 1013.40 0.00 0.00 0.00 0.00 0.00 8.00 8.00 8.00 0.00
01001 1001.00 70.94 -8.67 10.00 1.00 100.00
1015.00 -0.40 -2.06 340.00 11.60 999999.00 999999.00 999999.00 1013.80 0.00 0.00 0.00 0.00 0.00 8.00 8.00 8.00 0.00
01001 1001.00 70.94 -8.67 10.00 31.00 2300.00
1015.20 -0.40 -2.23 341.00 10.40 999999.00 999999.00 999999.00 1014.00 0.00 0.00 0.00 0.00 0.00 8.00 8.00 8.00 0.00
01001 1001.00 70.94 -8.67 10.00 31.00 2100.00
1015.80 -0.20 -1.89 340.00 13.60 999999.00 999999.00 999999.00 1014.60 0.00 0.00 0.00 0.00 0.00 8.00 8.00 8.00 0.00
01001 1001.00 70.94 -8.67 10.00 1.00 0.00
1015.00 -0.20 -1.58 344.00 12.10 999999.00 0.00 999999.00 1013.80 0.00 0.00 0.00 0.00 0.00 8.00 0.00 8.00 0.00
01002 1002.00 80.06 16.25 8.00 1.00 100.00
1020.50 -1.90 -2.89 109.00 6.80 999999.00 999999.00 999999.00 1019.70 0.00 0.00 0.00 0.00 0.00 8.00 8.00 8.00 0.00
01002 1002.00 80.06 16.25 8.00 31.00 2200.00
1020.80 -0.20 -2.30 89.00 6.90 999999.00 999999.00 999999.00 1020.00 0.00 0.00 0.00 0.00 0.00 8.00 8.00 8.00 0.00
01002 1002.00 80.06 16.25 8.00 31.00 2300.00
1020.70 -1.20 -2.59 111.00 6.30 999999.00 999999.00 999999.00 1019.80 0.00 0.00 0.00 0.00 0.00 8.00 8.00 8.00 0.00
01002 1002.00 80.06 16.25 8.00 31.00 2100.00
1020.70 0.00 -1.96 85.00 7.20 999999.00 999999.00 999999.00 1019.90 0.00 0.00 0.00 0.00 0.00 8.00 8.00 8.00 0.00
01002 1002.00 80.06 16.25 8.00 1.00 0.00
1020.70 -1.90 -3.02 119.00 7.30 999999.00 999999.00 999999.00 1019.80 0.00 0.00 0.00 0.00 0.00 8.00 8.00 8.00 0.00
01002 1002.00 80.06 16.25 8.00 1.00 200.00
1020.30 -1.70 -2.96 111.00 8.60 999999.00 999999.00 999999.00 1019.50 0.00 0.00 0.00 0.00 0.00 8.00 8.00 8.00 0.00
01003 1003.00 77.00 15.54 10.00 1.00 100.00
1014.70 -1.30 -2.83 110.00 13.70 999999.00 999999.00 999999.00 1013.30 0.00 0.00 0.00 0.00 0.00 8.00 8.00 8.00 0.00
01003 1003.00 77.00 15.54 10.00 31.00 2300.00
1014.40 1.50 -3.16 62.00 7.60 999999.00 999999.00 999999.00 1013.00 0.00 0.00 0.00 0.00 0.00 8.00 8.00 8.00 0.00
01003 1003.00 77.00 15.54 10.00 31.00 2200.00
1014.00 1.40 -3.26 59.00 12.50 999999.00 999999.00 999999.00 1012.60 0.00 0.00 0.00 0.00 0.00 8.00 8.00 8.00 0.00
01003 1003.00 77.00 15.54 10.00 31.00 2100.00
1013.50 1.40 -3.45 57.00 13.10 999999.00 999999.00 999999.00 1012.10 0.00 1.00 1.00 0.00 0.00 8.00 8.00 8.00 0.00
01003 1003.00 77.00 15.54 10.00 1.00 0.00
1014.40 1.10 -2.99 53.00 9.80 999999.00 0.00 999999.00 1013.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 8.00 0.00
Data extractor¶
A python code is used to extract the desired information from this CIMISS file:
> cd /g5/nwpc_ep8/obsproc/nwprod/decoders
> rm cimiss_data_surface
> ./read_nmc_cimiss_rsurf_rgwst.py -f /g3/tianwh/share/conobs/rec_RSURF_2019060406_g_06.dat
If you want to batch process number of CIMISS files, you can use following command:
> # This command will find all CIMISS files and prcessing the file one by one
> rm cimiss_data_surface
> decod_nmc_cimiss_rsing.py -d /g3/tianwh/share/conobs
The information we want to extract from CIMISS is:
import fortranformat as ff
output_line = ff.FortranRecordWriter('(I8, 3f10.2, 5A8, 2f10.2, 2I8, f10.2)')
output.write(output_line.write([id, lat, lon, elv, obstime.strftime("%Y"), obstime.strftime("%m"), \
obstime.strftime("%d"), obstime.strftime("%H"), obstime.strftime("%M"), \
p, T, round(rh), round(dd), ff]) + "\n")
the content of cimiss_data_surface is:
> less cimiss_data_surface
56076 33.85 102.6333 3429 2018 9 1 16 0 674.7 10.8 97 149 3.4
674002 40.6644 109.6661 1052 2018 9 1 16 0 890.2 15 95 2 1.3
735904 31.0231 118.4314 9 2018 9 1 16 0 999999 26.4 999999 158 1.7
868552 34.55 109.2975 357 2018 9 1 16 0 965.8 24.9 74 356 0.9
718262 22.8133 112.4264 49 2018 9 1 16 0 999999 25.4 999999 225 0
708128 26.8458 116.7783 343 2018 9 1 16 0 970.6 22.9 96 176 1.1
737152 30.7728 117.3278 28.3 2018 9 1 16 0 1005.9 26.3 97 135 0.6
756060 29.1036 119.8056 58 2018 9 1 16 0 999999 26.5 999999 210 1.1
804130 27.4056 110.4781 200 2018 9 1 16 0 999999 23.1 999999 999999 999999
672002 43.2972 116.1158 1284 2018 9 1 16 0 860 13.9 94 217 7.6
663689 40.2278 119.22 300 2018 9 1 16 0 999999 19.2 999999 999999 999999
752628 30.1492 121.1961 5 2018 9 1 16 0 999999 28.1 999999 237 1.6
860410 34.9769 108.5269 1193 2018 9 1 16 0 999999 19.8 999999 999999 999999
868283 34.2217 108.5681 394.5 2018 9 1 16 0 959.8 26.9 64 302 1
754019 29.9078 120.4844 59 2018 9 1 16 0 999999 24.6 999999 324 0
878120 36.0133 108.1183 1129 2018 9 1 16 0 876.3 17.5 99 329 1.4
751675 29.6344 119.5692 95 2018 9 1 16 0 999999 24.7 999999 333 0
Decoder source code¶
- Source code directory:
> cd /g5/nwpc_ep8/obsproc/nwprod/decoders/decod_dccimisssurf/sorc
Subroutines to decode surface data
lsdcod.f
注解
- The cimiss.tbl and
decod_WMO.Res40.headersare not used, although they are required as arguments and read in. - The path and file name of
cimiss_data_surfacefile are hard coded in the subroutines.
- Compile the code
> make
Decode and convert to BUFR format¶
- enter into the exec directory
> cd /g5/nwpc_ep8/obsproc/nwprod/decoders/decod_dccimisssurf/exec
> ls -la
total 2028
drwxr-xr-x. 3 zwtd zwtd 211 Dec 20 22:45 .
drwxr-xr-x. 6 zwtd zwtd 82 Dec 20 22:42 ..
lrwxrwxrwx. 1 zwtd zwtd 34 Dec 9 23:48 bufrtab.000 -> ../../decod_shared/fix/bufrtab.000
-rw-r--r--. 1 zwtd zwtd 1121683 Dec 9 23:48 cimiss.tbl
lrwxrwxrwx. 1 zwtd zwtd 31 Dec 9 23:48 decod_WMO.Res40.headers -> ../parm/decod_WMO.Res40.headers
-rwxr-xr-x. 1 zwtd zwtd 940696 Dec 20 00:24 decod_dccimiss
-rw-r--r--. 1 zwtd zwtd 462 Dec 20 22:45 decod_dccimiss.log
-rwxr-xr-x. 1 zwtd zwtd 484 Dec 20 00:30 run.ksh
-rwxr-xr-x. 1 zwtd zwtd 776 Dec 20 22:44 run_dccimisssurf.py
drwxr-xr-x. 2 zwtd zwtd 58 Dec 20 22:45 tmp
- we provide a script to run the decoder in batch mode:
> ./run_dccimisssurf.py -s 2018121600 -e 2018121700 -i 1
注解
- given the starting datetime and ending datetime, it iterates all cycles (every 1 hours)
- the units of interval is hour (-i)
- this script call run.ksh
- run the decoder script
> run.ksh
> cat run.ksh
!/bin/bash
export DBNBUFRT=120
export TRANJB=/g5/nwpc_ep8/obsproc/nwprod/ush/tranjb
export tank_dir=/g5/nwpc_ep8/obsproc/nwprod/dcom/us007003
export DBNROOT=`pwd`
rm tmp/*
rm decod_dccimiss.log
./decod_dccimiss -d decod_dccimiss.log -b 240 -c $1 bufrtab.000 cimiss.tbl decod_WMO.Res40.headers
ls -la tmp/*
BUFR_FILES=$(echo tmp/BUFR*)
echo ${BUFR_FILES}
for file in ${BUFR_FILES}
do
${TRANJB} ${tank_dir} ${file}
done
注解
- -c $1 : Set the current time (201809011600) used to calculate the time departures of the obs. data.
- -b 240 : Number of hours to decode prior to “current” time (default)
- The observations with date/time between current time - 240 hours and current time + 3 are kept.
- The generated BUFR format file will be saved at
> ls -la tmp
-rw-r--r-- 1 xinzhang staff 4199744 Sep 21 18:45 tmp/BUFR.0.cimiss.1.6436.1536097072.8
Transfer bufr data to BUFR Tanks¶
- put data in BUFR tanks:
> /g5/nwpc_ep8/obsproc/nwprod/ush/tranjb /g5/nwpc_ep8/obsproc/nwprod/dcom/us007003 tmp/BUFR.0.cimiss.1.6436.1536097072.8
> ls -al /g5/nwpc_ep8/obsproc/nwprod/dcom/us007003/20180901/b000/xx01
-rw-r--r-- 1 vagrant vagrant 4235328 Sep 27 04:42 /g5/nwpc_ep8/obsproc/nwprod/dcom/us007003/20180901/b000/xx001
注解
- if environmental variable SCREEN=ON :
- Define Run Time is the system time when the tranjb is running.
- Only observations with date/time between Run Time - 10 days and Run Time + 12 hours are kept.
for retrospective run, set SCREEN=OFF
/g5/nwpc_ep8/obsproc/nwprod/dcom/us007003/yyyymmdd/bmmm/xxsss(where mmm is WMO BUFR message type and xxx is local BUFR message subtype)000.001 (in dump group mnemonic adpsfc): Surface synoptic fixed land reports
BUFR format
Arranged by UTC day and continuously grow throughout the day, if you run decoders many time, the content of the file will grow
No QC (other than rudimentary checks inside decoders)
No duplicate checking
Interested users can use utility
debufrto check the content of the bufr file:> /g5/nwpc_ep8/obsproc/nwprod/util/exec/debufr /g5/nwpc_ep8/obsproc/nwprod/dcom/us007003/20180901/b000/xx001the output is in
debufr.out.