Hackpads are smart collaborative documents. .
366 days ago
Unfiled. Edited by Aloha Aina 366 days ago
  • Other software
The Community intercomparison suite (http://www.cistools.net/) is a command line tool for the atmospheric sciences that uses a plugin framework to deal with loading and regridding data from a range of sources - simulations, flight campaigns, satellites, ground stations etc.. It can also be used as a python library, but this hasn't been developed particularly far yet.
Pachyderm is an abstraction for pipelines that reduces it the pipeline to io and compute while keeping track of data a-la-git. PyFilesystem is file system abstraction. 
374 days ago
Unfiled. Edited by Guillaume Serazin 374 days ago
Signal processing design
Guillaume S
  • Introduction
Analyzing and characterizing the temporal and spatial scales from the outputs of climate model components (atmosphere, ocean, hydropshere, cryosphere) require the use of signal processing techniques. This python package aims at gathering such tools in the framework of pangeo-data using xarray and dask.
  •  Relation to existing packages
  • pyClim: filtering and timeseries analysis
  • ...
  • Suggested package name
Here is the section for barnstorming about the package name.
For the moment, I am thinking about xscale  (for "across scales") because the idea of this package is to analyze and separate the different time and space scales in a climate signal. I don't really know if something sounding similar to xarray is a good or a bad idea. Any other ideas ?
  • Requirements 
For the moment, we may classify the different techniques required into the different sections:
  1. General statistics
  • Correlation, covariance, autocorrelation, regression, etc
  • Structure functions (turbulence)
  • Significance tests associated to those statistics
  1. Filtering
  • Linear filters based on convolution products
  • Non-linear filters based on regression methods (polynomial, LOESS ...)
  • Spectrally-based filtering
  1. Multi-dimensional spectra
  • Pre-processing (detrending, tapering, zero-padding, multi-tapering)
  • Post-processing (normalization, visualization)
  • Time-frequency spectrum
  • Wavelets
  • Derivative in Fourier space
  • Cross-spectrum analysis
  • Spectrum from structure functions (turbulence)
  1. Principal Component Analysis (PCA)
  • Empirical Orthogonal Functions (EOF) (Standard, Complex, Rotated)
  • Canonical Correlation Analysis (CCA)
  • Single and Multi Tapper Singular Spectrum Analysis (SSA M-SSA) 
  1. Noise reduction techniques
  1. Information theory 
julien l
  • Design approach
Guillaume S First, there are two solutions for the structure of the signal processing package:
  1. One big package that includes all the different signal processing techniques in an unique repository.
  1. One  package that is a collection of several sub-packages, which separate  the different signal processing techniques by field (e.g., spectral,  filtering, eof). Such sub-packages would have their own repository.
Which one should we use ?
  • Open questions
  • Should this "big" package be splitted into several packages ?
  • Should we contribute to the eofs package or implement a new version ?
  • Has anyone approached Andrew (author of eofs)? I reckon he'd be keen to be involved.
  • Not that I'm aware of, but we should.  Would be good in general to loop in in the UK Met Office/Iris/etc. crowd, which he is a part of.
  • I could get in touch with him. Maybe we should wait to have a clear visibility with the website.
  • I am thinking about turbulence diagnostics. This may involve both the vector calculus and the signal processing packages. It could result in another project based on top of those packages and dedicated to compute spectral slopes, cospectrum for energy transfer, structure functions, linear instability etc.
378 days ago
Unfiled. Edited by Spencer Hill 378 days ago
  • windspharm : Compute vorticity, divergence, and others using spherical harmonics.  This is a slightly different use-case than the main mosaic goal, because the calculations are spectral and agnostic to the grid information/finite differencing methods used by the original model.
  • Dependencies and relation to other pangeo packages. 
379 days ago
Unfiled. Edited by Vincent Noel 379 days ago
  • Provide a remap-to interface (`da.remap.remap_to([('lat', 0.5), ('lon', 0.75)], how='nearest')`)
382 days ago
Unfiled. Edited by Joe Hamman 382 days ago
Organization Design
Github organization is here: https://github.com/pangeo-data
Note: if you didn't get a github invite to be a member of the organization, just ask me, I probably couldn't find you username.
  • Challenge / Motivation
There are several building crises facing the Atmosphere / Ocean / Land / Climate (AOC) science community:
383 days ago
Unfiled. Edited by Jeremy McGibbon , Brian Rose , Ryan Abernathey 383 days ago
Jeremy M Thermo Design Document
People need to use equations to calculate thermodynamic quantities all the time. There should be a nice interface to do this.
  • Easy to use interface that minimizes how much you need to memorizes
  • Works on xarray dataarrays
Brian R
  • Takes full advantage of labeled data and automated broadcasting (e.g. for latitude-dependent calculation of absolute salinity)
  • Handles units -- where present in metadata or passed in by user as keyword
Jeremy M
  • Can work for atmosphere or ocean (or arbitrary set of equations)
  • Scales to big data
  • Can take advantage of metadata stored in DataArrays
  • Clear in documentation which form of equations are being solved
Ryan A
  • Support for custom equation of state (particularly for ocean)
Brian R
  • What does it calculate?
  • Atmosphere
  • Potential temperature
  • saturation vapor pressure
  • Slope of adiabats in T-P space etc
  • etc
  • Ocean
  • Density
  • Slope of isopycnals in T-S space
  • etc
  • All functions of thermodynamic state variables
Jeremy M Design
  • Isolate solving code into a new package, that gets used by the atmos package
  • Someone can make an ocean package that does something similar to the atmos package, containing equations for ocean thermodynamics
  • Have saturation vapor pressure assumption treated specially as keyword argument
  • Chain numba calls when doing ocean eqns, they're huge!
Brian R
  • Aim for pure Python / numba implementation
 What are people doing now?
  • Ocean
  • Or using routines extraced from models
  • No one rolls their own (formulas too clunky)
  • Atmosphere
  • aoslib (python wrappers of fortran-based)
  • etc
  • Formulas are simple, so most people just roll their own
  • Don't have a package that can use labeled xarray data effectively
383 days ago
Unfiled. Edited by Spencer Hill 383 days ago
Spencer H
  • 1. presentation of concept behind oocgcm. 
  •      core grids / practical creation of grid objects
  •      need to avoid aditional data structures 
  •      (eg : vector fields, pycomodo container)
  •      change the name. directory structure. 
  •  2. discussion on relevance for atmospheric models 
Members (33)
Rachel Prudden Joy merwin monteiro Guillaume Serazin Vincent Noel Aloha Aina Edward Doddridge Joe Hamman Fabien Maussion Kevin Paul julien lesommer Denis Sergeev Liam Romain Pennel Mary Haley Andre Perkins Todd Ringler Aurelien P Brian Rose james@jamespenn.co.uk Joy Monteiro

Create a New Collection


Move XXX to XXX

XXX will be invited to the XXX on XXX.


Contact Support

Please check out our How-to Guide and FAQ first to see if your question is already answered! :)

If you have a feature request, please add it to this pad. Thanks!

Log in