How to read and display Nexrad on AWS using Python

Amazon Web Services (AWS) is making NEXRAD data freely available on Amazon S3 as part of the NOAA Big Data Project. In this Python notebook, I will step you through being able to read and display this data from your Python programs. I will assume that you know Python, how to install Python modules, and can access AWS.  (Follow along by downloading and running this iPython notebook).

What you need

You probably have ipython and matplotlib already. In addition, you may need to install the following Python modules:

  1. boto which I installed using conda:
    conda install boto
  2. pyart which I installed using conda:
    conda install -c https://conda.anaconda.org/jjhelmus pyart

You may also need to configure your AWS credentials to access S3.

Import modules

import matplotlib.pyplot as plt
import numpy.ma as ma
import numpy as np
import pyart.graph
import tempfile
import pyart.io
import boto

Find volume scan

Volume scans from NEXRAD are stored on S3 such that the bucket name is noaa-nexrad-level2 and the key name is something like 2014/04/03/KLSX/KLSX20140403_211350_V06.gz i.e. YYYY/MM/DD/KRAD/KRADYYYYMMDD_HHmmSS_V0?.gz. You can use the boto library to browse and select the keys that you want.

Here, I’ll directly navigate to a NEXRAD file that I know exists.

# read a volume scan file on S3. I happen to know this file exists.
s3conn = boto.connect_s3()
bucket = s3conn.get_bucket('noaa-nexrad-level2')
s3key = bucket.get_key('2015/05/15/KVWX/KVWX20150515_080737_V06.gz')
print s3key

Download volume scan from S3

For further processing, it is helpful to have the file locally on disk, so let’s download it:

# download to a local file, and read it
localfile = tempfile.NamedTemporaryFile()
s3key.get_contents_to_filename(localfile.name)
radar = pyart.io.read_nexrad_archive(localfile.name)

Display the data

Then, let’s use PyArt to display the lowest elevation scan data of all the moments.

# display the lowest elevation scan data
display = pyart.graph.RadarDisplay(radar)
fig = plt.figure(figsize=(9, 12))

plots = [
    # variable-name in pyart, display-name that we want, sweep-number of radar (0=lowest ref, 1=lowest velocity)
    ['reflectivity', 'Reflectivity (dBZ)', 0],
    ['differential_reflectivity', 'Zdr (dB)', 0],
    ['differential_phase', 'Phi_DP (deg)', 0],
    ['cross_correlation_ratio', 'Rho_HV', 0],
    ['velocity', 'Velocity (m/s)', 1],
    ['spectrum_width', 'Spectrum Width', 1]
]

def plot_radar_images(plots):
    ncols = 2
    nrows = len(plots)/2
    for plotno, plot in enumerate(plots, start=1):
        ax = fig.add_subplot(nrows, ncols, plotno)
        display.plot(plot[0], plot[2], ax=ax, title=plot[1],
             colorbar_label='',
             axislabels=('East-West distance from radar (km)' if plotno == 6 else '', 
                         'North-South distance from radar (km)' if plotno == 1 else ''))
        display.set_limits((-300, 300), (-300, 300), ax=ax)
        display.set_aspect_ratio('equal', ax=ax)
        display.plot_range_rings(range(100, 350, 100), lw=0.5, col='black', ax=ax)
    plt.show()

plot_radar_images(plots)

The result will look like this:

Plots of NEXRAD data

Processing data

Not all the reflectivity above is because of weather echoes. Most of the data are actually bioscatter (insects and birds). Let’s mask the reflectivity data using the polarimetric variables to retain only approximately spherical echoes.

refl_grid = radar.get_field(0, 'reflectivity')
print refl_grid[0]
rhohv_grid = radar.get_field(0, 'cross_correlation_ratio')
zdr_grid = radar.get_field(0, 'differential_reflectivity')

# apply rudimentary quality control
reflow = np.less(refl_grid, 20)
zdrhigh = np.greater(np.abs(zdr_grid), 2.3)
rhohvlow = np.less(rhohv_grid, 0.95)
notweather = np.logical_or(reflow, np.logical_or(zdrhigh, rhohvlow))
print notweather[0]

qcrefl_grid = ma.masked_where(notweather, refl_grid)
print qcrefl_grid[0]

# let's create a new object containing only sweep=0 so we can add the QC'ed ref to it for plotting
qced = radar.extract_sweeps([0])
qced.add_field_like('reflectivity', 'reflectivityqc', qcrefl_grid)
display = pyart.graph.RadarDisplay(qced)
fig = plt.figure(figsize=(11, 5))

plots = [
    # variable-name in pyart, display-name that we want, sweep-number of radar (0=lowest ref, 1=lowest velocity)
    ['reflectivity', 'Reflectivity (dBZ)', 0],
    ['reflectivityqc', 'QCed Reflectivity (dBZ)', 0],
]
plot_radar_images(plots)

 

qced

Our simple quality control removes much of the bioscatter and retains the meteorological signal in the north-northeast. However, it is restricted to radial distances below 300 km where polarimetric radar variables are available, and some noise remains. See http://journals.ametsoc.org/doi/abs/10.1175/JTECH-D-13-00073.1 for a better quality-control technique.

Acknowledgment

Thanks to NOAA and Amazon for making the NEXRAD data available.

Tagged with: , ,
Posted in Engineering
8 comments on “How to read and display Nexrad on AWS using Python
  1. […] CartoDB have contributed tutorials to help you get started using NEXRAD on AWS. For example, this tutorial from The Climate Corporation shows you how to read and display the NEXRAD Level II archive data […]

  2. […] CartoDB have contributed tutorials to help you get started using NEXRAD on AWS. For example, this tutorial from The Climate Corporation shows you how to read and display the NEXRAD Level II archive data […]

  3. Scott Collis says:

    Great to see you guys using Py-ART!
    There is so much more you can do too including KDP calculations and a who suite of texture based calculations as well as gridding.. This is all still in development but you can see some examples here: http://nbviewer.ipython.org/github/ARM-DOE/openvap/blob/master/development/cmac2/notebooks/full%20suite.ipynb

  4. Timothy Lang says:

    If, like me, you hate dealing with Amazon credentials (which boto appears to require), hate dealing with the NEXRAD split-cut sweeps and want to use Radx to merge them under the table, plus also want to avoid the Py-ART L2 Message 1 problem for old data, here is a simple function to get a Py-ART object. Should work in Mac/Linux.

    def read_nexrad_aws(filename):
    import tempfile
    import urllib
    import os
    import sys
    import pyart
    baseurl = ‘https://noaa-nexrad-level2.s3.amazonaws.com/’
    localfile = tempfile.NamedTemporaryFile()
    code = filename[0:4]
    yyyy = filename[4:8]
    mm = filename[8:10]
    dd = filename[10:12]
    url = baseurl + yyyy + ‘/’ + mm + ‘/’ + dd + ‘/’ + code + ‘/’ + filename
    print(url)
    # Web module protocols are different depending on Python 2 vs. Python 3
    if sys.version_info < (3, 0, 0):
    handle = urllib.urlretrieve(url, localfile.name)
    else:
    import urllib.request
    import shutil
    # Download the file from 'url' and save it locally under 'file_name':
    with urllib.request.urlopen(url) as response, open(localfile.name, 'wb') as out_file:
    shutil.copyfileobj(response, out_file)
    os.system('mv '+localfile.name+' '+localfile.name+'.gz')
    os.system('gzip -d '+localfile.name+'.gz')
    radar = pyart.aux_io.radx.read_radx(localfile.name)
    return radar

    Then to read a file you just need to know the name, e.g.,
    radar = read_nexrad_aws('KCYS19990614_211854.gz')

  5. […] com tutoriais para ajudá-lo a começar a usar o NEXRAD no AWS. Por exemplo, este tutorial da The Climate Corporation mostra como ler e exibir os dados do arquivo Nível II do NEXRAD a […]

  6. […] NOAA Big Data cooperative research and development agreement with Amazon Web Services (we wrote a blog post about this CRADA in […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: