DS Processing

In this tutorial, we demostrate how to do DS processing with the decorrelation CLI interface. The input data for DS processing is prepared in Load Data.

!ls ./load_data

e.zarr    lat.zarr  meta.toml  phi.zarr    rslc_pyramid  scratch
hgt.zarr  lon.zarr  n.zarr     range.zarr  rslc.zarr     theta.zarr

SHP selection

Here we use two sample Kolmogorov-Smirnov (KS) test for Spatially Homogenious Pixel (SHP) identification.

The output pvalue is the P-value for this test. The smaller the p-value, the more likely the two sample are from same distribution, i.e., the more likely the two pixels are SHP to each other.

import numpy as np
import moraine.cli as mc
import moraine as mr

rslc = './load_data/rslc.zarr'
pvalue = './ds_processing/gix/ras_pvalue.zarr'
az_half_win = 5
r_half_win = 5
method = 'ks'

logger = mc.get_logger()

mc.shp_test(rslc,pvalue,
            az_half_win=az_half_win,r_half_win=r_half_win)

2024-08-09 11:59:19 - log_args - INFO - running function: shp_test
2024-08-09 11:59:19 - log_args - INFO - fetching args:
2024-08-09 11:59:19 - log_args - INFO - rslc = './load_data/rslc.zarr'
2024-08-09 11:59:19 - log_args - INFO - pvalue = './ds_processing/gix/ras_pvalue.zarr'
2024-08-09 11:59:19 - log_args - INFO - az_half_win = 5
2024-08-09 11:59:19 - log_args - INFO - r_half_win = 5
2024-08-09 11:59:19 - log_args - INFO - method = None
2024-08-09 11:59:19 - log_args - INFO - chunks = None
2024-08-09 11:59:19 - log_args - INFO - cuda = False
2024-08-09 11:59:19 - log_args - INFO - processes = None
2024-08-09 11:59:19 - log_args - INFO - n_workers = None
2024-08-09 11:59:19 - log_args - INFO - threads_per_worker = None
2024-08-09 11:59:19 - log_args - INFO - rmm_pool_size = 0.9
2024-08-09 11:59:19 - log_args - INFO - dask_cluster_arg = {}
2024-08-09 11:59:19 - log_args - INFO - fetching args done.
2024-08-09 11:59:19 - shp_test - INFO - hypothetic test method: ks
2024-08-09 11:59:19 - zarr_info - INFO - ./load_data/rslc.zarr zarray shape, chunks, dtype: (2500, 1834, 17), (1000, 1000, 1), complex64
2024-08-09 11:59:19 - shp_test - INFO - starting dask local cluster.
2024-08-09 11:59:20 - shp_test - INFO - dask local cluster started.
2024-08-09 11:59:20 - dask_cluster_info - INFO - dask cluster: LocalCluster(dashboard_link='http://10.211.48.20:8787/status', workers=1, threads=1, memory=1.46 TiB)
2024-08-09 11:59:20 - shp_test - INFO - azimuth half window size: 5; azimuth window size: 11
2024-08-09 11:59:20 - shp_test - INFO - range half window size: 5; range window size: 11
2024-08-09 11:59:20 - darr_info - INFO - rslc with overlap dask array shape, chunksize, dtype: (2520, 1844, 17), (1010, 1005, 17), complex64
2024-08-09 11:59:20 - shp_test - INFO - applying test on rmli stack.
2024-08-09 11:59:20 - shp_test - INFO - trim shared boundaries between p value chunks
2024-08-09 11:59:20 - darr_info - INFO - p value dask array shape, chunksize, dtype: (2500, 1834, 11, 11), (1000, 1000, 11, 11), float32
2024-08-09 11:59:20 - shp_test - INFO - saving p value.
2024-08-09 11:59:21 - zarr_info - INFO - ./ds_processing/gix/ras_pvalue.zarr zarray shape, chunks, dtype: (2500, 1834, 11, 11), (1000, 1000, 1, 1), float32
2024-08-09 11:59:21 - shp_test - INFO - computing graph setted. doing all the computing.
2024-08-09 11:59:28 - shp_test - INFO - computing finished. |  6.9s
2024-08-09 11:59:28 - shp_test - INFO - dask cluster closed.

Then we select SHP by setting a threshold on this Pvalue:

import zarr
from matplotlib import pyplot as plt
import colorcet

is_shp = './ds_processing/gix/ras_is_shp.zarr'
shp_num = './ds_processing/gix/ras_shp_num.zarr'
p_max = 0.05

mc.select_shp(pvalue,is_shp,shp_num,p_max=p_max)

2024-08-09 11:59:28 - log_args - INFO - running function: select_shp
2024-08-09 11:59:28 - log_args - INFO - fetching args:
2024-08-09 11:59:28 - log_args - INFO - pvalue = './ds_processing/gix/ras_pvalue.zarr'
2024-08-09 11:59:28 - log_args - INFO - is_shp = './ds_processing/gix/ras_is_shp.zarr'
2024-08-09 11:59:28 - log_args - INFO - shp_num = './ds_processing/gix/ras_shp_num.zarr'
2024-08-09 11:59:28 - log_args - INFO - p_max = 0.05
2024-08-09 11:59:28 - log_args - INFO - chunks = None
2024-08-09 11:59:28 - log_args - INFO - processes = False
2024-08-09 11:59:28 - log_args - INFO - n_workers = 1
2024-08-09 11:59:28 - log_args - INFO - threads_per_worker = 1
2024-08-09 11:59:28 - log_args - INFO - dask_cluster_arg = {}
2024-08-09 11:59:28 - log_args - INFO - fetching args done.
2024-08-09 11:59:28 - zarr_info - INFO - ./ds_processing/gix/ras_pvalue.zarr zarray shape, chunks, dtype: (2500, 1834, 11, 11), (1000, 1000, 1, 1), float32
2024-08-09 11:59:28 - select_shp - INFO - starting dask cluster.
2024-08-09 11:59:28 - select_shp - INFO - dask cluster started.
2024-08-09 11:59:28 - dask_cluster_info - INFO - dask cluster: LocalCluster(dashboard_link='http://10.211.48.20:8787/status', workers=1, threads=1, memory=1.46 TiB)
2024-08-09 11:59:28 - darr_info - INFO - pvalue dask array shape, chunksize, dtype: (2500, 1834, 11, 11), (1000, 1000, 11, 11), float32
2024-08-09 11:59:28 - select_shp - INFO - selecting SHPs based on pvalue threshold: 0.05
2024-08-09 11:59:28 - darr_info - INFO - is_shp dask array shape, chunksize, dtype: (2500, 1834, 11, 11), (1000, 1000, 11, 11), bool
2024-08-09 11:59:28 - select_shp - INFO - calculate shp_num.
2024-08-09 11:59:28 - darr_info - INFO - shp_num dask array shape, chunksize, dtype: (2500, 1834), (1000, 1000), int32
2024-08-09 11:59:28 - select_shp - INFO - saving is_shp.
2024-08-09 11:59:29 - zarr_info - INFO - ./ds_processing/gix/ras_is_shp.zarr zarray shape, chunks, dtype: (2500, 1834, 11, 11), (1000, 1000, 1, 1), bool
2024-08-09 11:59:29 - select_shp - INFO - saving shp_num.
2024-08-09 11:59:29 - select_shp - INFO - computing graph setted. doing all the computing.
2024-08-09 11:59:34 - select_shp - INFO - computing finished.  4.9s
2024-08-09 11:59:34 - select_shp - INFO - dask cluster closed.

The output is_shp indicate wheather a pixel in the window is a SHP to its center pixel.

shp_num_zarr = zarr.open(shp_num,'r')
shp_num_data = shp_num_zarr[:]
shp_num_plot = mr.plot.ras_plot(shp_num_data)

import holoviews as hv
hv.extension('bokeh')
from bokeh.models import WheelZoomTool
from holoviews import opts

shp_num_plot = shp_num_plot.redim(x=hv.Dimension('r', label='Range'), y=hv.Dimension('az',label='Azimuth'), z=hv.Dimension('n_shp',))

shp_num_plot.opts(opts.Image(cmap='viridis',width=600, height=600, colorbar=True,
                          invert_yaxis=True,
                          default_tools=['pan',WheelZoomTool(zoom_on_axis=False),'save','reset','hover'],active_tools=['wheel_zoom']))

DS candidates selection

ds_can_gix = './ds_processing/gix/ds_can_gix.zarr'
is_ds_can = './ds_processing/gix/ras_is_ds_can.zarr'

mc.pc_logic_ras(shp_num,ds_can_gix,'ras>=50',chunks=200000)
mc.gix2bool(ds_can_gix, is_ds_can, shape= shp_num_zarr.shape)

2024-08-09 11:59:34 - log_args - INFO - running function: pc_logic_ras
2024-08-09 11:59:34 - log_args - INFO - fetching args:
2024-08-09 11:59:34 - log_args - INFO - ras = './ds_processing/gix/ras_shp_num.zarr'
2024-08-09 11:59:34 - log_args - INFO - gix = './ds_processing/gix/ds_can_gix.zarr'
2024-08-09 11:59:34 - log_args - INFO - operation = 'ras>=50'
2024-08-09 11:59:34 - log_args - INFO - chunks = 200000
2024-08-09 11:59:34 - log_args - INFO - fetching args done.
2024-08-09 11:59:34 - zarr_info - INFO - ./ds_processing/gix/ras_shp_num.zarr zarray shape, chunks, dtype: (2500, 1834), (1000, 1000), int32
2024-08-09 11:59:34 - pc_logic_ras - INFO - loading ras into memory.
2024-08-09 11:59:34 - pc_logic_ras - INFO - select pc based on operation: ras>=50
2024-08-09 11:59:34 - pc_logic_ras - INFO - number of selected pixels: 732727.
2024-08-09 11:59:35 - zarr_info - INFO - ./ds_processing/gix/ds_can_gix.zarr zarray shape, chunks, dtype: (732727, 2), (200000, 1), int32
2024-08-09 11:59:35 - pc_logic_ras - INFO - writing gix.
2024-08-09 11:59:35 - pc_logic_ras - INFO - write done.
2024-08-09 11:59:35 - log_args - INFO - running function: gix2bool
2024-08-09 11:59:35 - log_args - INFO - fetching args:
2024-08-09 11:59:35 - log_args - INFO - gix = './ds_processing/gix/ds_can_gix.zarr'
2024-08-09 11:59:35 - log_args - INFO - is_pc = './ds_processing/gix/ras_is_ds_can.zarr'
2024-08-09 11:59:35 - log_args - INFO - shape = (2500, 1834)
2024-08-09 11:59:35 - log_args - INFO - chunks = (1000, 1000)
2024-08-09 11:59:35 - log_args - INFO - fetching args done.
2024-08-09 11:59:35 - zarr_info - INFO - gix zarray shape, chunks, dtype: (732727, 2), (200000, 1), int32
2024-08-09 11:59:35 - gix2bool - INFO - loading gix into memory.
2024-08-09 11:59:35 - gix2bool - INFO - calculate the bool array
2024-08-09 11:59:35 - zarr_info - INFO - is_pc zarray shape, chunks, dtype: (2500, 1834), (1000, 1000), bool
2024-08-09 11:59:35 - gix2bool - INFO - write the bool array.
2024-08-09 11:59:35 - gix2bool - INFO - write done.

ds_can_idx is a int array with shape of (num_of_DS_can, 2). It indicate the position of DS candidate in radar coordinate.

is_ds_can_zarr = zarr.open(is_ds_can,'r')
is_ds_can_data = is_ds_can_zarr[:]
is_ds_can_plot = mr.ras_plot(is_ds_can_data)

is_ds_can_plot = is_ds_can_plot.redim(x=hv.Dimension('r', label='Range'), y=hv.Dimension('az',label='Azimuth'), z=hv.Dimension('is_ds_can',))
is_ds_can_plot.opts(opts.Image(cmap='viridis',width=600, height=600, colorbar=True,
                          invert_yaxis=True,
                          default_tools=['pan',WheelZoomTool(zoom_on_axis=False),'save','reset','hover'],active_tools=['wheel_zoom']))

Coherence matrix estimation

The coherence matrix estimation is kind of complicated as it involves both raster and point cloud data. We also want to save the output point cloud data in the hilbert order.

First, we generate DS candidate point cloud is_shp by ras chunks (which is a directory):

ds_can_is_shp_dir = './ds_processing/gix/ds_can_is_shp'
ds_can_key = './ds_processing/gix/ds_can_key.zarr'

mc.ras2pc_ras_chunk(ds_can_gix,is_shp,ds_can_is_shp_dir,ds_can_key,chunks=(1000,1000))

2024-08-09 11:59:35 - log_args - INFO - running function: ras2pc_ras_chunk
2024-08-09 11:59:35 - log_args - INFO - fetching args:
2024-08-09 11:59:35 - log_args - INFO - gix = './ds_processing/gix/ds_can_gix.zarr'
2024-08-09 11:59:35 - log_args - INFO - ras = './ds_processing/gix/ras_is_shp.zarr'
2024-08-09 11:59:35 - log_args - INFO - pc = './ds_processing/gix/ds_can_is_shp'
2024-08-09 11:59:35 - log_args - INFO - key = './ds_processing/gix/ds_can_key.zarr'
2024-08-09 11:59:35 - log_args - INFO - chunks = (1000, 1000)
2024-08-09 11:59:35 - log_args - INFO - processes = False
2024-08-09 11:59:35 - log_args - INFO - n_workers = 1
2024-08-09 11:59:35 - log_args - INFO - threads_per_worker = 1
2024-08-09 11:59:35 - log_args - INFO - dask_cluster_arg = {}
2024-08-09 11:59:35 - log_args - INFO - fetching args done.
2024-08-09 11:59:35 - zarr_info - INFO - ./ds_processing/gix/ds_can_gix.zarr zarray shape, chunks, dtype: (732727, 2), (200000, 1), int32
2024-08-09 11:59:35 - ras2pc_ras_chunk - INFO - loading gix into memory.
2024-08-09 11:59:35 - ras2pc_ras_chunk - INFO - convert gix to the order of ras chunk
2024-08-09 11:59:41 - ras2pc_ras_chunk - INFO - save key
2024-08-09 11:59:41 - ras2pc_ras_chunk - INFO - starting dask local cluster.
2024-08-09 11:59:42 - ras2pc_ras_chunk - INFO - dask local cluster started.
2024-08-09 11:59:42 - dask_cluster_info - INFO - dask cluster: LocalCluster(dashboard_link='http://10.211.48.20:8787/status', workers=1, threads=1, memory=1.46 TiB)
2024-08-09 11:59:42 - ras2pc_ras_chunk - INFO - start to slice on ./ds_processing/gix/ras_is_shp.zarr
2024-08-09 11:59:42 - zarr_info - INFO - ./ds_processing/gix/ras_is_shp.zarr zarray shape, chunks, dtype: (2500, 1834, 11, 11), (1000, 1000, 1, 1), bool
2024-08-09 11:59:42 - darr_info - INFO - ras dask array shape, chunksize, dtype: (2500, 1834, 11, 11), (1000, 1000, 11, 11), bool
2024-08-09 11:59:42 - darr_info - INFO - pc dask array shape, chunksize, dtype: (732727, 11, 11), (201403, 11, 11), bool
2024-08-09 11:59:42 - ras2pc_ras_chunk - INFO - saving to ds_processing/gix/ds_can_is_shp.
2024-08-09 11:59:42 - ras2pc_ras_chunk - INFO - computing graph setted. doing all the computing.
2024-08-09 11:59:45 - ras2pc_ras_chunk - INFO - computing finished.
2024-08-09 11:59:45 - ras2pc_ras_chunk - INFO - dask cluster closed.

Then we estimate the coherence matrix:

ds_can_coh_dir = './ds_processing/gix/ds_can_coh'

mc.emperical_co_pc(rslc,ds_can_is_shp_dir,ds_can_gix,ds_can_coh_dir,cuda=True)

2024-08-09 11:59:45 - log_args - INFO - running function: emperical_co_pc
2024-08-09 11:59:45 - log_args - INFO - fetching args:
2024-08-09 11:59:45 - log_args - INFO - rslc = './load_data/rslc.zarr'
2024-08-09 11:59:45 - log_args - INFO - is_shp_dir = './ds_processing/gix/ds_can_is_shp'
2024-08-09 11:59:45 - log_args - INFO - gix = './ds_processing/gix/ds_can_gix.zarr'
2024-08-09 11:59:45 - log_args - INFO - coh_dir = './ds_processing/gix/ds_can_coh'
2024-08-09 11:59:45 - log_args - INFO - tnet = None
2024-08-09 11:59:45 - log_args - INFO - chunks = None
2024-08-09 11:59:45 - log_args - INFO - cuda = True
2024-08-09 11:59:45 - log_args - INFO - processes = None
2024-08-09 11:59:45 - log_args - INFO - n_workers = None
2024-08-09 11:59:45 - log_args - INFO - threads_per_worker = None
2024-08-09 11:59:45 - log_args - INFO - rmm_pool_size = 0.9
2024-08-09 11:59:45 - log_args - INFO - dask_cluster_arg = {}
2024-08-09 11:59:45 - log_args - INFO - fetching args done.
2024-08-09 11:59:46 - zarr_info - INFO - ./load_data/rslc.zarr zarray shape, chunks, dtype: (2500, 1834, 17), (1000, 1000, 1), complex64
2024-08-09 11:59:46 - emperical_co_pc - INFO - azimuth window size and half azimuth window size: 11, 5
2024-08-09 11:59:46 - emperical_co_pc - INFO - range window size and half range window size: 11, 5
2024-08-09 11:59:46 - emperical_co_pc - INFO - parallel processing azimuth chunk size: 1000
2024-08-09 11:59:46 - emperical_co_pc - INFO - parallel processing range chunk size: 1000
2024-08-09 11:59:46 - zarr_info - INFO - ./ds_processing/gix/ds_can_gix.zarr zarray shape, chunks, dtype: (732727, 2), (200000, 1), int32
2024-08-09 11:59:46 - emperical_co_pc - INFO - loading gix into memory.
2024-08-09 11:59:46 - emperical_co_pc - INFO - convert gix to the order of ras chunk
2024-08-09 11:59:46 - emperical_co_pc - INFO - starting dask cluster.
2024-08-09 11:59:52 - emperical_co_pc - INFO - dask cluster started.
2024-08-09 11:59:52 - dask_cluster_info - INFO - dask cluster: LocalCUDACluster(dashboard_link='http://127.0.0.1:8787/status', workers=8, threads=8, memory=1.46 TiB)
2024-08-09 11:59:52 - darr_info - INFO - rslc_overlap dask array shape, chunksize, dtype: (2520, 1844, 17), (1010, 1005, 17), complex64
2024-08-09 11:59:52 - darr_info - INFO - gix in ras chunk order dask array shape, chunksize, dtype: (732727, 2), (201403, 2), int32
2024-08-09 11:59:53 - emperical_co_pc - INFO - estimating coherence matrix chunk by chunk.
2024-08-09 11:59:53 - darr_info - INFO - is_shp for chunk 0 dask array shape, chunksize, dtype: (201097, 11, 11), (201097, 11, 11), bool
2024-08-09 11:59:53 - darr_info - INFO - coh for chunk 0 dask array shape, chunksize, dtype: (201097, 136), (201097, 136), complex64
2024-08-09 11:59:53 - emperical_co_pc - INFO - saving coh for chunk 0
2024-08-09 11:59:53 - zarr_info - INFO - ds_processing/gix/ds_can_coh/0.zarr zarray shape, chunks, dtype: (201097, 136), (201097, 1), complex64
2024-08-09 11:59:53 - emperical_co_pc - INFO - computing graph setted. doing all the computing.
2024-08-09 11:59:55 - emperical_co_pc - INFO - computing finished.s
2024-08-09 11:59:57 - emperical_co_pc - INFO - dask cluster closed.

The output of emperical_co_pc is point cloud data in the raster chunk order. So we prepare the key to sort the point cloud data to hilbert order (the hillbert index of DS candidate can be generated by the way):

ds_can_hix_unsorted = './ds_processing/gix/ds_can_hix.zarr'
ds_can_hix = './ds_processing/hix/ds_can/ds_can_hix.zarr'
gix2hix_key = './ds_processing/gix/gix2hix.zarr'

mc.pc_hix(ds_can_gix, ds_can_hix_unsorted, shape=zarr.open(rslc,'r').shape[:2])
mc.pc_sort(
    ds_can_hix_unsorted, ds_can_hix,
    shape=zarr.open(rslc,'r').shape[:2],
    pc_in = './ds_processing/gix/ds_can_gix.zarr',
    pc = './ds_processing/hix/ds_can/ds_can_gix.zarr',
    key=gix2hix_key,
)

2024-08-09 11:59:57 - log_args - INFO - running function: pc_hix
2024-08-09 11:59:57 - log_args - INFO - fetching args:
2024-08-09 11:59:57 - log_args - INFO - gix = './ds_processing/gix/ds_can_gix.zarr'
2024-08-09 11:59:57 - log_args - INFO - hix = './ds_processing/gix/ds_can_hix.zarr'
2024-08-09 11:59:57 - log_args - INFO - shape = (2500, 1834)
2024-08-09 11:59:57 - log_args - INFO - fetching args done.
2024-08-09 11:59:57 - zarr_info - INFO - ./ds_processing/gix/ds_can_gix.zarr zarray shape, chunks, dtype: (732727, 2), (200000, 1), int32
2024-08-09 11:59:57 - zarr_info - INFO - ./ds_processing/gix/ds_can_hix.zarr zarray shape, chunks, dtype: (732727,), (200000,), int64
2024-08-09 11:59:57 - pc_hix - INFO - calculating the hillbert index based on grid index
2024-08-09 11:59:58 - pc_hix - INFO - writing the hillbert index
2024-08-09 11:59:58 - pc_hix - INFO - done.
2024-08-09 11:59:58 - log_args - INFO - running function: pc_sort
2024-08-09 11:59:58 - log_args - INFO - fetching args:
2024-08-09 11:59:58 - log_args - INFO - idx_in = './ds_processing/gix/ds_can_hix.zarr'
2024-08-09 11:59:58 - log_args - INFO - idx = './ds_processing/hix/ds_can/ds_can_hix.zarr'
2024-08-09 11:59:58 - log_args - INFO - pc_in = './ds_processing/gix/ds_can_gix.zarr'
2024-08-09 11:59:58 - log_args - INFO - pc = './ds_processing/hix/ds_can/ds_can_gix.zarr'
2024-08-09 11:59:58 - log_args - INFO - shape = (2500, 1834)
2024-08-09 11:59:58 - log_args - INFO - chunks = None
2024-08-09 11:59:58 - log_args - INFO - key = './ds_processing/gix/gix2hix.zarr'
2024-08-09 11:59:58 - log_args - INFO - processes = False
2024-08-09 11:59:58 - log_args - INFO - n_workers = 1
2024-08-09 11:59:58 - log_args - INFO - threads_per_worker = 1
2024-08-09 11:59:58 - log_args - INFO - dask_cluster_arg = {}
2024-08-09 11:59:58 - log_args - INFO - fetching args done.
2024-08-09 11:59:58 - zarr_info - INFO - ./ds_processing/gix/ds_can_hix.zarr zarray shape, chunks, dtype: (732727,), (200000,), int64
2024-08-09 11:59:58 - pc_sort - INFO - loading idx_in and calculate the sorting indices.
2024-08-09 11:59:58 - pc_sort - INFO - output pc chunk size is 200000
2024-08-09 11:59:58 - pc_sort - INFO - write idx
2024-08-09 11:59:58 - zarr_info - INFO - idx zarray shape, chunks, dtype: (732727,), (200000,), int64
2024-08-09 11:59:58 - pc_sort - INFO - saving key for this sorting
2024-08-09 11:59:59 - pc_sort - INFO - starting dask local cluster.
2024-08-09 11:59:59 - pc_sort - INFO - dask local cluster started.
2024-08-09 11:59:59 - dask_cluster_info - INFO - dask cluster: LocalCluster(dashboard_link='http://10.211.48.20:8787/status', workers=1, threads=1, memory=1.46 TiB)
2024-08-09 11:59:59 - zarr_info - INFO - ./ds_processing/gix/ds_can_gix.zarr zarray shape, chunks, dtype: (732727, 2), (200000, 1), int32
2024-08-09 11:59:59 - darr_info - INFO - pc_in dask array shape, chunksize, dtype: (732727, 2), (732727, 1), int32
2024-08-09 11:59:59 - pc_sort - INFO - set up sorted pc data dask array.
2024-08-09 11:59:59 - darr_info - INFO - pc dask array shape, chunksize, dtype: (732727, 2), (732727, 1), int32
2024-08-09 11:59:59 - pc_sort - INFO - write pc to ./ds_processing/hix/ds_can/ds_can_gix.zarr
2024-08-09 11:59:59 - zarr_info - INFO - ./ds_processing/hix/ds_can/ds_can_gix.zarr zarray shape, chunks, dtype: (732727, 2), (200000, 1), int32
2024-08-09 11:59:59 - pc_sort - INFO - computing graph setted. doing all the computing.
2024-08-09 11:59:59 - pc_sort - INFO - computing finished.d |  0.1s
2024-08-09 11:59:59 - pc_sort - INFO - dask cluster closed.

Then concat and sort ds_can_coh to hilbert order:

ds_can_coh = './ds_processing/hix/ds_can/ds_can_coh.zarr'
ds_can_coh_ave = './ds_processing/hix/ds_can/ds_can_coh_ave.zarr'

chunks = zarr.open(ds_can_gix,'r').chunks[0]
mc.pc_concat(ds_can_coh_dir,ds_can_coh,key=[ds_can_key,gix2hix_key],chunks=chunks)

2024-08-09 11:59:59 - log_args - INFO - running function: pc_concat
2024-08-09 11:59:59 - log_args - INFO - fetching args:
2024-08-09 11:59:59 - log_args - INFO - pcs = './ds_processing/gix/ds_can_coh'
2024-08-09 11:59:59 - log_args - INFO - pc = './ds_processing/hix/ds_can/ds_can_coh.zarr'
2024-08-09 11:59:59 - log_args - INFO - key = ['./ds_processing/gix/ds_can_key.zarr', './ds_processing/gix/gix2hix.zarr']
2024-08-09 11:59:59 - log_args - INFO - chunks = 200000
2024-08-09 11:59:59 - log_args - INFO - processes = False
2024-08-09 11:59:59 - log_args - INFO - n_workers = 1
2024-08-09 11:59:59 - log_args - INFO - threads_per_worker = 1
2024-08-09 11:59:59 - log_args - INFO - dask_cluster_arg = {}
2024-08-09 11:59:59 - log_args - INFO - fetching args done.
2024-08-09 11:59:59 - pc_concat - INFO - input pcs: [[PosixPath('ds_processing/gix/ds_can_coh/0.zarr'), PosixPath('ds_processing/gix/ds_can_coh/1.zarr'), PosixPath('ds_processing/gix/ds_can_coh/2.zarr'), PosixPath('ds_processing/gix/ds_can_coh/3.zarr'), PosixPath('ds_processing/gix/ds_can_coh/4.zarr'), PosixPath('ds_processing/gix/ds_can_coh/5.zarr')]]
2024-08-09 11:59:59 - pc_concat - INFO - output pc: ['./ds_processing/hix/ds_can/ds_can_coh.zarr']
2024-08-09 11:59:59 - pc_concat - INFO - load key
2024-08-09 11:59:59 - zarr_info - INFO - ./ds_processing/gix/ds_can_key.zarr zarray shape, chunks, dtype: (732727,), (200000,), int64
2024-08-09 11:59:59 - zarr_info - INFO - ./ds_processing/gix/gix2hix.zarr zarray shape, chunks, dtype: (732727,), (200000,), int64
2024-08-09 11:59:59 - pc_concat - INFO - starting dask local cluster.
2024-08-09 11:59:59 - pc_concat - INFO - dask local cluster started.
2024-08-09 11:59:59 - dask_cluster_info - INFO - dask cluster: LocalCluster(dashboard_link='http://10.211.48.20:8787/status', workers=1, threads=1, memory=1.46 TiB)
2024-08-09 11:59:59 - pc_concat - INFO - read pc from [PosixPath('ds_processing/gix/ds_can_coh/0.zarr'), PosixPath('ds_processing/gix/ds_can_coh/1.zarr'), PosixPath('ds_processing/gix/ds_can_coh/2.zarr'), PosixPath('ds_processing/gix/ds_can_coh/3.zarr'), PosixPath('ds_processing/gix/ds_can_coh/4.zarr'), PosixPath('ds_processing/gix/ds_can_coh/5.zarr')]
2024-08-09 11:59:59 - darr_info - INFO - concatenated pc dask array shape, chunksize, dtype: (732727, 136), (732727, 1), complex64
2024-08-09 11:59:59 - pc_concat - INFO - sort pc according to key
2024-08-09 11:59:59 - darr_info - INFO - sorted pc dask array shape, chunksize, dtype: (732727, 136), (732727, 1), complex64
2024-08-09 11:59:59 - pc_concat - INFO - save pc to ./ds_processing/hix/ds_can/ds_can_coh.zarr
2024-08-09 11:59:59 - zarr_info - INFO - ./ds_processing/hix/ds_can/ds_can_coh.zarr zarray shape, chunks, dtype: (732727, 136), (200000, 1), complex64
2024-08-09 11:59:59 - pc_concat - INFO - computing graph setted. doing all the computing.
2024-08-09 12:00:05 - pc_concat - INFO - computing finished.|  5.3s
2024-08-09 12:00:05 - pc_concat - INFO - dask cluster closed.

n_points = zarr.open(ds_can_coh,'r').shape[0]
mc.data_reduce(ds_can_coh,ds_can_coh_ave,map_func=np.abs,reduce_func=np.sum,post_map_func=lambda x: x/n_points)

2024-08-09 12:00:05 - log_args - INFO - running function: data_reduce
2024-08-09 12:00:05 - log_args - INFO - fetching args:
2024-08-09 12:00:05 - log_args - INFO - data_in = './ds_processing/hix/ds_can/ds_can_coh.zarr'
2024-08-09 12:00:05 - log_args - INFO - out = './ds_processing/hix/ds_can/ds_can_coh_ave.zarr'
2024-08-09 12:00:05 - log_args - INFO - map_func = <ufunc 'absolute'>
2024-08-09 12:00:05 - log_args - INFO - reduce_func = <function sum>
2024-08-09 12:00:05 - log_args - INFO - axis = 0
2024-08-09 12:00:05 - log_args - INFO - post_map_func = <function <lambda>>
2024-08-09 12:00:05 - log_args - INFO - processes = False
2024-08-09 12:00:05 - log_args - INFO - n_workers = 1
2024-08-09 12:00:05 - log_args - INFO - threads_per_worker = 1
2024-08-09 12:00:05 - log_args - INFO - dask_cluster_arg = {}
2024-08-09 12:00:05 - log_args - INFO - fetching args done.
2024-08-09 12:00:05 - zarr_info - INFO - ./ds_processing/hix/ds_can/ds_can_coh.zarr zarray shape, chunks, dtype: (732727, 136), (200000, 1), complex64
2024-08-09 12:00:05 - data_reduce - INFO - starting dask local cluster.
2024-08-09 12:00:05 - data_reduce - INFO - dask local cluster started.
2024-08-09 12:00:05 - dask_cluster_info - INFO - dask cluster: LocalCluster(dashboard_link='http://10.211.48.20:8787/status', workers=1, threads=1, memory=1.46 TiB)
2024-08-09 12:00:05 - darr_info - INFO - data_in dask array shape, chunksize, dtype: (732727, 136), (200000, 1), complex64
2024-08-09 12:00:05 - darr_info - INFO - maped_data_in dask array shape, chunksize, dtype: (732727, 136), (200000, 1), float32
2024-08-09 12:00:05 - darr_info - INFO - reduced data in every chunk dask array shape, chunksize, dtype: (4, 136), (1, 1), float32
2024-08-09 12:00:05 - data_reduce - INFO - computing graph setted. doing all the computing.
2024-08-09 12:00:07 - data_reduce - INFO - computing finished. 1.9s
2024-08-09 12:00:07 - data_reduce - INFO - dask cluster closed.
2024-08-09 12:00:07 - data_reduce - INFO - continue the reduction on reduced data over every chunk
2024-08-09 12:00:07 - data_reduce - INFO - post mapping
2024-08-09 12:00:07 - data_reduce - INFO - writing output.
2024-08-09 12:00:07 - data_reduce - INFO - done.

ds_can_coh_ave_zarr = zarr.open(ds_can_coh_ave,'r')
ds_can_coh_ave_plot = mr.plot.ras_plot(mr.uncompress_coh(ds_can_coh_ave_zarr[:]))
ds_can_coh_ave_plot = ds_can_coh_ave_plot.redim(x=hv.Dimension('sec_image', label='Secondary image'), y=hv.Dimension('ref_image',label='Reference image'), z=hv.Dimension('ds_can_coh_ave',))
ds_can_coh_ave_plot.opts(opts.Image(cmap='viridis',frame_width=600, frame_height=600, colorbar=True,
                          invert_yaxis=True,
                          default_tools=['pan',WheelZoomTool(zoom_on_axis=False),'save','reset','hover'],active_tools=['wheel_zoom']))

Visualize adpative multi-looked interferograms

Visualize interferograms after adaptive multi-looking and compare with the raw 1 by 1 look interferograms:

mc.pc_pyramid(
    './ds_processing/hix/ds_can/ds_can_coh.zarr',
    './ds_processing/hix/ds_can/ds_can_coh_pyramid',
    yx = './ds_processing/hix/ds_can/ds_can_gix.zarr',
    ras_resolution=2,
    threads_per_worker=8,
)

2024-08-09 12:00:07 - log_args - INFO - running function: pc_pyramid
2024-08-09 12:00:07 - log_args - INFO - fetching args:
2024-08-09 12:00:07 - log_args - INFO - pc = './ds_processing/hix/ds_can/ds_can_coh.zarr'
2024-08-09 12:00:07 - log_args - INFO - out_dir = './ds_processing/hix/ds_can/ds_can_coh_pyramid'
2024-08-09 12:00:07 - log_args - INFO - x = None
2024-08-09 12:00:07 - log_args - INFO - y = None
2024-08-09 12:00:07 - log_args - INFO - yx = './ds_processing/hix/ds_can/ds_can_gix.zarr'
2024-08-09 12:00:07 - log_args - INFO - ras_resolution = 2
2024-08-09 12:00:07 - log_args - INFO - ras_chunks = (256, 256)
2024-08-09 12:00:07 - log_args - INFO - pc_chunks = 65536
2024-08-09 12:00:07 - log_args - INFO - processes = False
2024-08-09 12:00:07 - log_args - INFO - n_workers = 1
2024-08-09 12:00:07 - log_args - INFO - threads_per_worker = 8
2024-08-09 12:00:07 - log_args - INFO - dask_cluster_arg = {}
2024-08-09 12:00:07 - log_args - INFO - fetching args done.
2024-08-09 12:00:10 - zarr_info - INFO - ./ds_processing/hix/ds_can/ds_can_coh.zarr zarray shape, chunks, dtype: (732727, 136), (200000, 1), complex64
2024-08-09 12:00:10 - pc_pyramid - INFO - rendering point cloud data coordinates:
2024-08-09 12:00:10 - pc_pyramid - INFO - rasterizing point cloud data to grid with bounds: [0, 0, 1832, 2498].
2024-08-09 12:00:10 - pc_pyramid - INFO - dask local cluster started.
2024-08-09 12:00:10 - dask_cluster_info - INFO - dask cluster: LocalCluster(dashboard_link='http://10.211.48.20:8787/status', workers=1, threads=8, memory=1.46 TiB)
2024-08-09 12:00:10 - pc_pyramid - INFO - pc data coordinates rendering ends.
2024-08-09 12:00:10 - zarr_info - INFO - ds_processing/hix/ds_can/ds_can_coh_pyramid/pc.zarr zarray shape, chunks, dtype: (732727, 136), (65536, 1), complex64
2024-08-09 12:00:10 - pc_pyramid - INFO - pc data rendering ends.
2024-08-09 12:00:10 - darr_info - INFO - rasterized pc data at level 0 dask array shape, chunksize, dtype: (1250, 917, 136), (1250, 917, 1), complex64
2024-08-09 12:00:10 - darr_info - INFO - rasterized pc index at level 0 dask array shape, chunksize, dtype: (1250, 917), (1250, 917), int64
2024-08-09 12:00:10 - zarr_info - INFO - ds_processing/hix/ds_can/ds_can_coh_pyramid/0.zarr zarray shape, chunks, dtype: (1250, 917, 136), (256, 256, 1), complex64
2024-08-09 12:00:10 - zarr_info - INFO - ds_processing/hix/ds_can/ds_can_coh_pyramid/idx_0.zarr zarray shape, chunks, dtype: (1250, 917), (256, 256), int64
2024-08-09 12:00:10 - darr_info - INFO - rasterized pc data at level 1 dask array shape, chunksize, dtype: (625, 459, 136), (625, 459, 1), complex64
2024-08-09 12:00:10 - darr_info - INFO - rasterized pc index at level 1 dask array shape, chunksize, dtype: (625, 459), (625, 459), int64
2024-08-09 12:00:10 - zarr_info - INFO - ds_processing/hix/ds_can/ds_can_coh_pyramid/1.zarr zarray shape, chunks, dtype: (625, 459, 136), (256, 256, 1), complex64
2024-08-09 12:00:11 - zarr_info - INFO - ds_processing/hix/ds_can/ds_can_coh_pyramid/idx_1.zarr zarray shape, chunks, dtype: (625, 459), (256, 256), int64
2024-08-09 12:00:11 - darr_info - INFO - rasterized pc data at level 2 dask array shape, chunksize, dtype: (313, 230, 136), (313, 230, 1), complex64
2024-08-09 12:00:11 - darr_info - INFO - rasterized pc index at level 2 dask array shape, chunksize, dtype: (313, 230), (313, 230), int64
2024-08-09 12:00:11 - zarr_info - INFO - ds_processing/hix/ds_can/ds_can_coh_pyramid/2.zarr zarray shape, chunks, dtype: (313, 230, 136), (256, 256, 1), complex64
2024-08-09 12:00:11 - zarr_info - INFO - ds_processing/hix/ds_can/ds_can_coh_pyramid/idx_2.zarr zarray shape, chunks, dtype: (313, 230), (256, 256), int64
2024-08-09 12:00:11 - darr_info - INFO - rasterized pc data at level 3 dask array shape, chunksize, dtype: (157, 115, 136), (157, 115, 1), complex64
2024-08-09 12:00:11 - darr_info - INFO - rasterized pc index at level 3 dask array shape, chunksize, dtype: (157, 115), (157, 115), int64
2024-08-09 12:00:11 - zarr_info - INFO - ds_processing/hix/ds_can/ds_can_coh_pyramid/3.zarr zarray shape, chunks, dtype: (157, 115, 136), (256, 256, 1), complex64
2024-08-09 12:00:11 - zarr_info - INFO - ds_processing/hix/ds_can/ds_can_coh_pyramid/idx_3.zarr zarray shape, chunks, dtype: (157, 115), (256, 256), int64
2024-08-09 12:00:11 - darr_info - INFO - rasterized pc data at level 4 dask array shape, chunksize, dtype: (79, 58, 136), (79, 58, 1), complex64
2024-08-09 12:00:11 - darr_info - INFO - rasterized pc index at level 4 dask array shape, chunksize, dtype: (79, 58), (79, 58), int64
2024-08-09 12:00:11 - zarr_info - INFO - ds_processing/hix/ds_can/ds_can_coh_pyramid/4.zarr zarray shape, chunks, dtype: (79, 58, 136), (256, 256, 1), complex64
2024-08-09 12:00:11 - zarr_info - INFO - ds_processing/hix/ds_can/ds_can_coh_pyramid/idx_4.zarr zarray shape, chunks, dtype: (79, 58), (256, 256), int64
2024-08-09 12:00:11 - darr_info - INFO - rasterized pc data at level 5 dask array shape, chunksize, dtype: (40, 29, 136), (40, 29, 1), complex64
2024-08-09 12:00:11 - darr_info - INFO - rasterized pc index at level 5 dask array shape, chunksize, dtype: (40, 29), (40, 29), int64
2024-08-09 12:00:11 - zarr_info - INFO - ds_processing/hix/ds_can/ds_can_coh_pyramid/5.zarr zarray shape, chunks, dtype: (40, 29, 136), (256, 256, 1), complex64
2024-08-09 12:00:11 - zarr_info - INFO - ds_processing/hix/ds_can/ds_can_coh_pyramid/idx_5.zarr zarray shape, chunks, dtype: (40, 29), (256, 256), int64
2024-08-09 12:00:11 - darr_info - INFO - rasterized pc data at level 6 dask array shape, chunksize, dtype: (20, 15, 136), (20, 15, 1), complex64
2024-08-09 12:00:11 - darr_info - INFO - rasterized pc index at level 6 dask array shape, chunksize, dtype: (20, 15), (20, 15), int64
2024-08-09 12:00:11 - zarr_info - INFO - ds_processing/hix/ds_can/ds_can_coh_pyramid/6.zarr zarray shape, chunks, dtype: (20, 15, 136), (256, 256, 1), complex64
2024-08-09 12:00:11 - zarr_info - INFO - ds_processing/hix/ds_can/ds_can_coh_pyramid/idx_6.zarr zarray shape, chunks, dtype: (20, 15), (256, 256), int64
2024-08-09 12:00:11 - darr_info - INFO - rasterized pc data at level 7 dask array shape, chunksize, dtype: (10, 8, 136), (10, 8, 1), complex64
2024-08-09 12:00:11 - darr_info - INFO - rasterized pc index at level 7 dask array shape, chunksize, dtype: (10, 8), (10, 8), int64
2024-08-09 12:00:11 - zarr_info - INFO - ds_processing/hix/ds_can/ds_can_coh_pyramid/7.zarr zarray shape, chunks, dtype: (10, 8, 136), (256, 256, 1), complex64
2024-08-09 12:00:11 - zarr_info - INFO - ds_processing/hix/ds_can/ds_can_coh_pyramid/idx_7.zarr zarray shape, chunks, dtype: (10, 8), (256, 256), int64
2024-08-09 12:00:11 - darr_info - INFO - rasterized pc data at level 8 dask array shape, chunksize, dtype: (5, 4, 136), (5, 4, 1), complex64
2024-08-09 12:00:11 - darr_info - INFO - rasterized pc index at level 8 dask array shape, chunksize, dtype: (5, 4), (5, 4), int64
2024-08-09 12:00:11 - zarr_info - INFO - ds_processing/hix/ds_can/ds_can_coh_pyramid/8.zarr zarray shape, chunks, dtype: (5, 4, 136), (256, 256, 1), complex64
2024-08-09 12:00:11 - zarr_info - INFO - ds_processing/hix/ds_can/ds_can_coh_pyramid/idx_8.zarr zarray shape, chunks, dtype: (5, 4), (256, 256), int64
2024-08-09 12:00:11 - darr_info - INFO - rasterized pc data at level 9 dask array shape, chunksize, dtype: (3, 2, 136), (3, 2, 1), complex64
2024-08-09 12:00:11 - darr_info - INFO - rasterized pc index at level 9 dask array shape, chunksize, dtype: (3, 2), (3, 2), int64
2024-08-09 12:00:11 - zarr_info - INFO - ds_processing/hix/ds_can/ds_can_coh_pyramid/9.zarr zarray shape, chunks, dtype: (3, 2, 136), (256, 256, 1), complex64
2024-08-09 12:00:11 - zarr_info - INFO - ds_processing/hix/ds_can/ds_can_coh_pyramid/idx_9.zarr zarray shape, chunks, dtype: (3, 2), (256, 256), int64
2024-08-09 12:00:11 - pc_pyramid - INFO - computing graph setted. doing all the computing.

/users/kangl/miniforge3/envs/work/lib/python3.10/site-packages/distributed/client.py:3162: UserWarning: Sending large graph of size 20.48 MiB.
This may cause some slowdown.
Consider scattering data ahead of time and using futures.
  warnings.warn(

2024-08-09 12:00:24 - pc_pyramid - INFO - computing finished. 13.0s
2024-08-09 12:00:25 - pc_pyramid - INFO - dask cluster closed.

rslc_zarr = zarr.open(rslc,'r')
tnet = mr.TempNet.from_bandwidth(rslc_zarr.shape[2])

def co_phase_post_proc_ras(data_zarr,xslice,yslice,i,j):
    if i == j:
        nx = xslice.stop-xslice.start
        ny = yslice.stop-yslice.start
        mask = np.isnan(data_zarr[yslice,xslice,0])
        data = np.zeros((ny,nx),dtype=np.float32)
        data[mask] = np.nan
        return data
    elif i > j:
        ref, sec = j, i
        conj = True
    else:
        ref, sec = i, j
        conj = False
    image_pair_idx = tnet.image_pairs_idx(ref=ref,sec=sec)
    if image_pair_idx == -1:
        nx = xslice.stop-xslice.start
        ny = yslice.stop-yslice.start
        return np.full((ny,nx),fill_value=np.nan,dtype=np.float32)
    data = data_zarr[yslice,xslice,image_pair_idx]
    if conj:
        return -np.angle(data)
    else:
        return np.angle(data)

def co_phase_post_proc_pc(data_zarr,idx_array,i,j):
    if i == j:
        data = np.zeros_like(idx_array,dtype=np.float32)
        return data
    elif i > j:
        ref, sec = j, i
        conj = True
    else:
        ref, sec = i, j
        conj = False
    image_pair_idx = tnet.image_pairs_idx(ref=ref,sec=sec)
    if image_pair_idx == -1:
        return np.full_like(idx_array,fill_value=np.nan,dtype=np.float32)
    data = data_zarr[idx_array,image_pair_idx]
    if conj:
        return -np.angle(data)
    else:
        return np.angle(data)

ds_can_intf_plot = mc.pc_plot(
    './ds_processing/hix/ds_can/ds_can_coh_pyramid',
    post_proc_ras=co_phase_post_proc_ras,
    post_proc_pc=co_phase_post_proc_pc,
    n_kdim=2,level_increase=1)
raw_intf_plot = mc.ras_plot('./load_data/rslc_pyramid',post_proc='intf_all',n_kdim=2,level_increase=1)
intf_plots = raw_intf_plot.relabel('Raw Interferograms') + ds_can_intf_plot.relabel('Adaptively multilooked Interferograms')

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[17], line 3
      1 ds_can_intf_plot = mc.pc_plot(
      2     './ds_processing/hix/ds_can/ds_can_coh_pyramid',
----> 3     post_proc_ras=co_phase_post_proc_ras,
      4     post_proc_pc=co_phase_post_proc_pc,
      5     n_kdim=2,level_increase=1)
      6 raw_intf_plot = mc.ras_plot('./load_data/rslc_pyramid',post_proc='intf_all',n_kdim=2,level_increase=1)
      7 intf_plots = raw_intf_plot.relabel('Raw Interferograms') + ds_can_intf_plot.relabel('Adaptively multilooked Interferograms')

NameError: name 'co_phase_post_proc_ras' is not defined

import holoviews as hv
from bokeh.models import WheelZoomTool
hv.extension('bokeh')
import toml
with open('load_data/meta.toml','r') as f:
    dates = toml.load(f)['dates']
intf_plots = intf_plots.redim(
    i=hv.Dimension('i', label='Reference Image', range=(0,16), value_format=(lambda i: dates[i])),
    j=hv.Dimension('j', label='Secondary Image', range=(0,16), value_format=(lambda i: dates[i])),
    x=hv.Dimension('r', label='Range'),
    y=hv.Dimension('az',label='Azimuth'),
    z=hv.Dimension('Phase',range=(-np.pi,np.pi))
)

hv.output(widget_location='bottom')
intf_plots.opts(
    hv.opts.Image(
        cmap='colorwheel',frame_width=500, frame_height=600, colorbar=True,
        default_tools=['pan',WheelZoomTool(zoom_on_axis=False),'save','reset','hover'],
        active_tools=['wheel_zoom'],
        invert_yaxis=True,
    ),
    hv.opts.Points(
        color='Phase', cmap='colorwheel',frame_width=500, frame_height=600, colorbar=True,
        default_tools=['pan',WheelZoomTool(zoom_on_axis=False),'save','reset','hover'],
        active_tools=['wheel_zoom'],
        invert_yaxis=True,
    ),
)

Phase linking

To get rid of the closure phase, phase linking is applied here:

ds_can_coh = './ds_processing/hix/ds_can/ds_can_coh.zarr'
ds_can_ph = './ds_processing/hix/ds_can/ds_can_ph.zarr'
ds_can_emi_quality = './ds_processing/hix/ds_can/ds_can_emi_quality.zarr'
emi_quality = './ds_processing/hix/ds_can/ras_ds_can_emi_quality.zarr'

mc.emi(ds_can_coh,ds_can_ph,ds_can_emi_quality)

2024-08-09 12:00:28 - log_args - INFO - running function: emi
2024-08-09 12:00:28 - log_args - INFO - fetching args:
2024-08-09 12:00:28 - log_args - INFO - coh = './ds_processing/hix/ds_can/ds_can_coh.zarr'
2024-08-09 12:00:28 - log_args - INFO - ph = './ds_processing/hix/ds_can/ds_can_ph.zarr'
2024-08-09 12:00:28 - log_args - INFO - emi_quality = './ds_processing/hix/ds_can/ds_can_emi_quality.zarr'
2024-08-09 12:00:28 - log_args - INFO - ref = 0
2024-08-09 12:00:28 - log_args - INFO - chunks = None
2024-08-09 12:00:28 - log_args - INFO - cuda = False
2024-08-09 12:00:28 - log_args - INFO - processes = None
2024-08-09 12:00:28 - log_args - INFO - n_workers = None
2024-08-09 12:00:28 - log_args - INFO - threads_per_worker = None
2024-08-09 12:00:28 - log_args - INFO - rmm_pool_size = 0.9
2024-08-09 12:00:28 - log_args - INFO - dask_cluster_arg = {}
2024-08-09 12:00:28 - log_args - INFO - fetching args done.
2024-08-09 12:00:28 - zarr_info - INFO - ./ds_processing/hix/ds_can/ds_can_coh.zarr zarray shape, chunks, dtype: (732727, 136), (200000, 1), complex64
2024-08-09 12:00:28 - emi - INFO - starting dask cluster.
2024-08-09 12:00:28 - emi - INFO - dask cluster started.
2024-08-09 12:00:28 - dask_cluster_info - INFO - dask cluster: LocalCluster(dashboard_link='http://10.211.48.20:8787/status', workers=1, threads=2, memory=1.46 TiB)
2024-08-09 12:00:28 - darr_info - INFO - coh dask array shape, chunksize, dtype: (732727, 136), (200000, 136), complex64
2024-08-09 12:00:28 - emi - INFO - phase linking with EMI.
2024-08-09 12:00:28 - emi - INFO - got ph and emi_quality.
2024-08-09 12:00:28 - darr_info - INFO - ph dask array shape, chunksize, dtype: (732727, 17), (200000, 17), complex64
2024-08-09 12:00:28 - darr_info - INFO - emi_quality dask array shape, chunksize, dtype: (732727,), (200000,), float32
2024-08-09 12:00:28 - emi - INFO - saving ph and emi_quality.
2024-08-09 12:00:28 - zarr_info - INFO - ./ds_processing/hix/ds_can/ds_can_ph.zarr zarray shape, chunks, dtype: (732727, 17), (200000, 1), complex64
2024-08-09 12:00:28 - zarr_info - INFO - ./ds_processing/hix/ds_can/ds_can_emi_quality.zarr zarray shape, chunks, dtype: (732727,), (200000,), float32
2024-08-09 12:00:28 - emi - INFO - computing graph setted. doing all the computing.
2024-08-09 12:00:34 - emi - INFO - computing finished.leted |  5.6s
2024-08-09 12:00:34 - emi - INFO - dask cluster closed.

rslc_zarr = zarr.open(rslc,'r')
mc.pc2ras(ds_can_hix, ds_can_emi_quality,emi_quality,rslc_zarr.shape[:2])

2024-08-09 12:00:34 - log_args - INFO - running function: pc2ras
2024-08-09 12:00:34 - log_args - INFO - fetching args:
2024-08-09 12:00:34 - log_args - INFO - idx = './ds_processing/hix/ds_can/ds_can_hix.zarr'
2024-08-09 12:00:34 - log_args - INFO - pc = './ds_processing/hix/ds_can/ds_can_emi_quality.zarr'
2024-08-09 12:00:34 - log_args - INFO - ras = './ds_processing/hix/ds_can/ras_ds_can_emi_quality.zarr'
2024-08-09 12:00:34 - log_args - INFO - shape = (2500, 1834)
2024-08-09 12:00:34 - log_args - INFO - chunks = (1000, 1000)
2024-08-09 12:00:34 - log_args - INFO - processes = False
2024-08-09 12:00:34 - log_args - INFO - n_workers = 1
2024-08-09 12:00:34 - log_args - INFO - threads_per_worker = 1
2024-08-09 12:00:34 - log_args - INFO - dask_cluster_arg = {}
2024-08-09 12:00:34 - log_args - INFO - fetching args done.
2024-08-09 12:00:34 - zarr_info - INFO - ./ds_processing/hix/ds_can/ds_can_hix.zarr zarray shape, chunks, dtype: (732727,), (200000,), int64
2024-08-09 12:00:34 - pc2ras - INFO - loading hix into memory and convert to gix
2024-08-09 12:00:35 - pc2ras - INFO - starting dask local cluster.
2024-08-09 12:00:35 - pc2ras - INFO - dask local cluster started.
2024-08-09 12:00:35 - dask_cluster_info - INFO - dask cluster: LocalCluster(dashboard_link='http://10.211.48.20:8787/status', workers=1, threads=1, memory=1.46 TiB)
2024-08-09 12:00:35 - pc2ras - INFO - start to work on ./ds_processing/hix/ds_can/ds_can_emi_quality.zarr
2024-08-09 12:00:35 - zarr_info - INFO - ./ds_processing/hix/ds_can/ds_can_emi_quality.zarr zarray shape, chunks, dtype: (732727,), (200000,), float32
2024-08-09 12:00:35 - darr_info - INFO - pc dask array shape, chunksize, dtype: (732727,), (732727,), float32
2024-08-09 12:00:35 - pc2ras - INFO - create ras dask array
2024-08-09 12:00:35 - darr_info - INFO - ras dask array shape, chunksize, dtype: (2500, 1834), (2500, 1834), float32
2024-08-09 12:00:35 - pc2ras - INFO - save ras to ./ds_processing/hix/ds_can/ras_ds_can_emi_quality.zarr
2024-08-09 12:00:35 - zarr_info - INFO - ./ds_processing/hix/ds_can/ras_ds_can_emi_quality.zarr zarray shape, chunks, dtype: (2500, 1834), (1000, 1000), float32
2024-08-09 12:00:35 - pc2ras - INFO - computing graph setted. doing all the computing.
2024-08-09 12:00:35 - pc2ras - INFO - computing finished.ed |  0.1s
2024-08-09 12:00:35 - pc2ras - INFO - dask cluster closed.

emi_quality_zarr = zarr.open(emi_quality,'r')
emi_quality_plot = mr.plot.ras_plot(emi_quality_zarr[:])
emi_quality_plot = emi_quality_plot.redim(x=hv.Dimension('r', label='Range'), y=hv.Dimension('az',label='Azimuth'), z=hv.Dimension('emi_quality',range=(0.5,1.3)))
emi_quality_plot.opts(opts.Image(cmap='viridis',frame_width=500, frame_height=600, colorbar=True,
                          invert_yaxis=True,
                          default_tools=['pan',WheelZoomTool(zoom_on_axis=False),'save','reset','hover'],active_tools=['wheel_zoom']))

We can also estimate a temporal coherence for DS:

ds_can_t_coh = './ds_processing/hix/ds_can/ds_can_t_coh.zarr'

mc.ds_temp_coh(ds_can_coh,ds_can_ph, ds_can_t_coh)

2024-08-09 12:00:35 - log_args - INFO - running function: ds_temp_coh
2024-08-09 12:00:35 - log_args - INFO - fetching args:
2024-08-09 12:00:35 - log_args - INFO - coh = './ds_processing/hix/ds_can/ds_can_coh.zarr'
2024-08-09 12:00:35 - log_args - INFO - ph = './ds_processing/hix/ds_can/ds_can_ph.zarr'
2024-08-09 12:00:35 - log_args - INFO - t_coh = './ds_processing/hix/ds_can/ds_can_t_coh.zarr'
2024-08-09 12:00:35 - log_args - INFO - tnet = None
2024-08-09 12:00:35 - log_args - INFO - chunks = None
2024-08-09 12:00:35 - log_args - INFO - cuda = False
2024-08-09 12:00:35 - log_args - INFO - processes = None
2024-08-09 12:00:35 - log_args - INFO - n_workers = None
2024-08-09 12:00:35 - log_args - INFO - threads_per_worker = None
2024-08-09 12:00:35 - log_args - INFO - rmm_pool_size = 0.9
2024-08-09 12:00:35 - log_args - INFO - dask_cluster_arg = {}
2024-08-09 12:00:35 - log_args - INFO - fetching args done.
2024-08-09 12:00:35 - zarr_info - INFO - ./ds_processing/hix/ds_can/ds_can_coh.zarr zarray shape, chunks, dtype: (732727, 136), (200000, 1), complex64
2024-08-09 12:00:35 - zarr_info - INFO - ./ds_processing/hix/ds_can/ds_can_ph.zarr zarray shape, chunks, dtype: (732727, 17), (200000, 1), complex64
2024-08-09 12:00:35 - ds_temp_coh - INFO - starting dask local cluster.
2024-08-09 12:00:35 - ds_temp_coh - INFO - dask local cluster started.
2024-08-09 12:00:35 - dask_cluster_info - INFO - dask cluster: LocalCluster(dashboard_link='http://10.211.48.20:8787/status', workers=1, threads=2, memory=1.46 TiB)
2024-08-09 12:00:35 - darr_info - INFO - coh dask array shape, chunksize, dtype: (732727, 136), (200000, 136), complex64
2024-08-09 12:00:35 - darr_info - INFO - ph dask array shape, chunksize, dtype: (732727, 17), (200000, 17), complex64
2024-08-09 12:00:35 - ds_temp_coh - INFO - Estimate temporal coherence for DS.
2024-08-09 12:00:35 - ds_temp_coh - INFO - got temporal coherence t_coh.
2024-08-09 12:00:35 - darr_info - INFO - t_coh dask array shape, chunksize, dtype: (732727,), (200000,), float32
2024-08-09 12:00:35 - ds_temp_coh - INFO - saving t_coh.
2024-08-09 12:00:35 - ds_temp_coh - INFO - computing graph setted. doing all the computing.
2024-08-09 12:00:36 - ds_temp_coh - INFO - computing finished. 0.9s
2024-08-09 12:00:36 - ds_temp_coh - INFO - dask cluster closed.

Plot it:

t_coh = './ds_processing/hix/ds_can/ras_ds_can_t_coh.zarr'
rslc_zarr = zarr.open(rslc,'r')
mc.pc2ras(ds_can_hix, ds_can_t_coh,t_coh,rslc_zarr.shape[:2])

2024-08-09 12:00:37 - log_args - INFO - running function: pc2ras
2024-08-09 12:00:37 - log_args - INFO - fetching args:
2024-08-09 12:00:37 - log_args - INFO - idx = './ds_processing/hix/ds_can/ds_can_hix.zarr'
2024-08-09 12:00:37 - log_args - INFO - pc = './ds_processing/hix/ds_can/ds_can_t_coh.zarr'
2024-08-09 12:00:37 - log_args - INFO - ras = './ds_processing/hix/ds_can/ras_ds_can_t_coh.zarr'
2024-08-09 12:00:37 - log_args - INFO - shape = (2500, 1834)
2024-08-09 12:00:37 - log_args - INFO - chunks = (1000, 1000)
2024-08-09 12:00:37 - log_args - INFO - processes = False
2024-08-09 12:00:37 - log_args - INFO - n_workers = 1
2024-08-09 12:00:37 - log_args - INFO - threads_per_worker = 1
2024-08-09 12:00:37 - log_args - INFO - dask_cluster_arg = {}
2024-08-09 12:00:37 - log_args - INFO - fetching args done.
2024-08-09 12:00:37 - zarr_info - INFO - ./ds_processing/hix/ds_can/ds_can_hix.zarr zarray shape, chunks, dtype: (732727,), (200000,), int64
2024-08-09 12:00:37 - pc2ras - INFO - loading hix into memory and convert to gix
2024-08-09 12:00:37 - pc2ras - INFO - starting dask local cluster.
2024-08-09 12:00:37 - pc2ras - INFO - dask local cluster started.
2024-08-09 12:00:37 - dask_cluster_info - INFO - dask cluster: LocalCluster(dashboard_link='http://10.211.48.20:8787/status', workers=1, threads=1, memory=1.46 TiB)
2024-08-09 12:00:37 - pc2ras - INFO - start to work on ./ds_processing/hix/ds_can/ds_can_t_coh.zarr
2024-08-09 12:00:37 - zarr_info - INFO - ./ds_processing/hix/ds_can/ds_can_t_coh.zarr zarray shape, chunks, dtype: (732727,), (200000,), float32
2024-08-09 12:00:37 - darr_info - INFO - pc dask array shape, chunksize, dtype: (732727,), (732727,), float32
2024-08-09 12:00:37 - pc2ras - INFO - create ras dask array
2024-08-09 12:00:37 - darr_info - INFO - ras dask array shape, chunksize, dtype: (2500, 1834), (2500, 1834), float32
2024-08-09 12:00:37 - pc2ras - INFO - save ras to ./ds_processing/hix/ds_can/ras_ds_can_t_coh.zarr
2024-08-09 12:00:37 - zarr_info - INFO - ./ds_processing/hix/ds_can/ras_ds_can_t_coh.zarr zarray shape, chunks, dtype: (2500, 1834), (1000, 1000), float32
2024-08-09 12:00:37 - pc2ras - INFO - computing graph setted. doing all the computing.
2024-08-09 12:00:37 - pc2ras - INFO - computing finished.ed |  0.1s
2024-08-09 12:00:37 - pc2ras - INFO - dask cluster closed.

t_coh_zarr = zarr.open(t_coh,'r')
t_coh_plot = mr.plot.ras_plot(t_coh_zarr[:])
t_coh_plot = t_coh_plot.redim(x=hv.Dimension('r', label='Range'), y=hv.Dimension('az',label='Azimuth'), z=hv.Dimension('t_coh',range=(0,1)))
t_coh_plot.opts(opts.Image(cmap='viridis',frame_width=500, frame_height=600, colorbar=True,
                          invert_yaxis=True,
                          default_tools=['pan',WheelZoomTool(zoom_on_axis=False),'save','reset','hover'],active_tools=['wheel_zoom']))

The full coherence matrix may be too big to be saved. It is also possible to do the coherence matrix estimation, phase linking together:

ds_can_ph_dir = './ds_processing/gix/ds_can_ph'
ds_can_emi_quality_dir = './ds_processing/gix/ds_can_emi_quality'
ds_can_t_coh_dir = './ds_processing/gix/ds_can_t_coh'

mc.emperical_co_emi_temp_coh_pc(rslc,ds_can_is_shp_dir,ds_can_gix,ds_can_ph_dir,ds_can_emi_quality_dir,ds_can_t_coh_dir,chunks=(1000,1000),cuda=True)
mc.pc_concat(
    [ds_can_ph_dir,ds_can_emi_quality_dir,ds_can_t_coh_dir],
    [ds_can_ph,ds_can_emi_quality,ds_can_t_coh],
    key=[ds_can_key,gix2hix_key],
    chunks=200000)

2024-08-09 12:00:37 - log_args - INFO - running function: emperical_co_emi_temp_coh_pc
2024-08-09 12:00:37 - log_args - INFO - fetching args:
2024-08-09 12:00:37 - log_args - INFO - rslc = './load_data/rslc.zarr'
2024-08-09 12:00:37 - log_args - INFO - is_shp_dir = './ds_processing/gix/ds_can_is_shp'
2024-08-09 12:00:37 - log_args - INFO - gix = './ds_processing/gix/ds_can_gix.zarr'
2024-08-09 12:00:37 - log_args - INFO - ph_dir = './ds_processing/gix/ds_can_ph'
2024-08-09 12:00:37 - log_args - INFO - emi_quality_dir = './ds_processing/gix/ds_can_emi_quality'
2024-08-09 12:00:37 - log_args - INFO - t_coh_dir = './ds_processing/gix/ds_can_t_coh'
2024-08-09 12:00:37 - log_args - INFO - tnet = None
2024-08-09 12:00:37 - log_args - INFO - chunks = (1000, 1000)
2024-08-09 12:00:37 - log_args - INFO - cuda = True
2024-08-09 12:00:37 - log_args - INFO - processes = None
2024-08-09 12:00:37 - log_args - INFO - n_workers = None
2024-08-09 12:00:37 - log_args - INFO - threads_per_worker = None
2024-08-09 12:00:37 - log_args - INFO - rmm_pool_size = 0.9
2024-08-09 12:00:37 - log_args - INFO - dask_cluster_arg = {}
2024-08-09 12:00:37 - log_args - INFO - fetching args done.
2024-08-09 12:00:37 - zarr_info - INFO - ./load_data/rslc.zarr zarray shape, chunks, dtype: (2500, 1834, 17), (1000, 1000, 1), complex64
2024-08-09 12:00:38 - emperical_co_emi_temp_coh_pc - INFO - azimuth window size and half azimuth window size: 11, 5
2024-08-09 12:00:38 - emperical_co_emi_temp_coh_pc - INFO - range window size and half range window size: 11, 5
2024-08-09 12:00:38 - emperical_co_emi_temp_coh_pc - INFO - parallel processing azimuth chunk size: 1000
2024-08-09 12:00:38 - emperical_co_emi_temp_coh_pc - INFO - parallel processing range chunk size: 1000
2024-08-09 12:00:38 - zarr_info - INFO - ./ds_processing/gix/ds_can_gix.zarr zarray shape, chunks, dtype: (732727, 2), (200000, 1), int32
2024-08-09 12:00:38 - emperical_co_emi_temp_coh_pc - INFO - loading gix into memory.
2024-08-09 12:00:38 - emperical_co_emi_temp_coh_pc - INFO - convert gix to the order of ras chunk
2024-08-09 12:00:38 - emperical_co_emi_temp_coh_pc - INFO - starting dask cluster.
2024-08-09 12:00:44 - emperical_co_emi_temp_coh_pc - INFO - dask cluster started.
2024-08-09 12:00:44 - dask_cluster_info - INFO - dask cluster: LocalCUDACluster(dashboard_link='http://127.0.0.1:8787/status', workers=8, threads=8, memory=1.46 TiB)
2024-08-09 12:00:44 - darr_info - INFO - rslc_overlap dask array shape, chunksize, dtype: (2520, 1844, 17), (1010, 1005, 17), complex64
2024-08-09 12:00:44 - darr_info - INFO - gix in ras chunk order dask array shape, chunksize, dtype: (732727, 2), (201403, 2), int32
2024-08-09 12:00:44 - emperical_co_emi_temp_coh_pc - INFO - estimating coherence matrix chunk by chunk.
2024-08-09 12:00:44 - darr_info - INFO - is_shp for chunk 0 dask array shape, chunksize, dtype: (201097, 11, 11), (201097, 11, 11), bool
2024-08-09 12:00:44 - darr_info - INFO - ph for chunk 0 dask array shape, chunksize, dtype: (201097, 17), (201097, 17), complex64
2024-08-09 12:00:44 - darr_info - INFO - emi_quality for chunk 0 dask array shape, chunksize, dtype: (201097,), (201097,), float32
2024-08-09 12:00:44 - darr_info - INFO - t_coh for chunk 0 dask array shape, chunksize, dtype: (201097,), (201097,), float32
2024-08-09 12:00:44 - emperical_co_emi_temp_coh_pc - INFO - saving ph, emi_quality, t_coh for chunk 0
2024-08-09 12:00:44 - zarr_info - INFO - ds_processing/gix/ds_can_ph/0.zarr zarray shape, chunks, dtype: (201097, 17), (201097, 1), complex64
2024-08-09 12:00:44 - zarr_info - INFO - ds_processing/gix/ds_can_emi_quality/0.zarr zarray shape, chunks, dtype: (201097,), (201097,), float32
2024-08-09 12:00:44 - zarr_info - INFO - ds_processing/gix/ds_can_t_coh/0.zarr zarray shape, chunks, dtype: (201097,), (201097,), float32
2024-08-09 12:00:44 - emperical_co_emi_temp_coh_pc - INFO - computing graph setted. doing all the computing.
2024-08-09 12:00:49 - emperical_co_emi_temp_coh_pc - INFO - computing finished.
2024-08-09 12:00:51 - emperical_co_emi_temp_coh_pc - INFO - dask cluster closed.
2024-08-09 12:00:51 - log_args - INFO - running function: pc_concat
2024-08-09 12:00:51 - log_args - INFO - fetching args:
2024-08-09 12:00:51 - log_args - INFO - pcs = ['./ds_processing/gix/ds_can_ph', './ds_processing/gix/ds_can_emi_quality', './ds_processing/gix/ds_can_t_coh']
2024-08-09 12:00:51 - log_args - INFO - pc = ['./ds_processing/hix/ds_can/ds_can_ph.zarr', './ds_processing/hix/ds_can/ds_can_emi_quality.zarr', './ds_processing/hix/ds_can/ds_can_t_coh.zarr']
2024-08-09 12:00:51 - log_args - INFO - key = ['./ds_processing/gix/ds_can_key.zarr', './ds_processing/gix/gix2hix.zarr']
2024-08-09 12:00:51 - log_args - INFO - chunks = 200000
2024-08-09 12:00:51 - log_args - INFO - processes = False
2024-08-09 12:00:51 - log_args - INFO - n_workers = 1
2024-08-09 12:00:51 - log_args - INFO - threads_per_worker = 1
2024-08-09 12:00:51 - log_args - INFO - dask_cluster_arg = {}
2024-08-09 12:00:51 - log_args - INFO - fetching args done.
2024-08-09 12:00:51 - pc_concat - INFO - input pcs: [[PosixPath('ds_processing/gix/ds_can_ph/0.zarr'), PosixPath('ds_processing/gix/ds_can_ph/1.zarr'), PosixPath('ds_processing/gix/ds_can_ph/2.zarr'), PosixPath('ds_processing/gix/ds_can_ph/3.zarr'), PosixPath('ds_processing/gix/ds_can_ph/4.zarr'), PosixPath('ds_processing/gix/ds_can_ph/5.zarr')], [PosixPath('ds_processing/gix/ds_can_emi_quality/0.zarr'), PosixPath('ds_processing/gix/ds_can_emi_quality/1.zarr'), PosixPath('ds_processing/gix/ds_can_emi_quality/2.zarr'), PosixPath('ds_processing/gix/ds_can_emi_quality/3.zarr'), PosixPath('ds_processing/gix/ds_can_emi_quality/4.zarr'), PosixPath('ds_processing/gix/ds_can_emi_quality/5.zarr')], [PosixPath('ds_processing/gix/ds_can_t_coh/0.zarr'), PosixPath('ds_processing/gix/ds_can_t_coh/1.zarr'), PosixPath('ds_processing/gix/ds_can_t_coh/2.zarr'), PosixPath('ds_processing/gix/ds_can_t_coh/3.zarr'), PosixPath('ds_processing/gix/ds_can_t_coh/4.zarr'), PosixPath('ds_processing/gix/ds_can_t_coh/5.zarr')]]
2024-08-09 12:00:51 - pc_concat - INFO - output pc: ['./ds_processing/hix/ds_can/ds_can_ph.zarr', './ds_processing/hix/ds_can/ds_can_emi_quality.zarr', './ds_processing/hix/ds_can/ds_can_t_coh.zarr']
2024-08-09 12:00:51 - pc_concat - INFO - load key
2024-08-09 12:00:51 - zarr_info - INFO - ./ds_processing/gix/ds_can_key.zarr zarray shape, chunks, dtype: (732727,), (200000,), int64
2024-08-09 12:00:51 - zarr_info - INFO - ./ds_processing/gix/gix2hix.zarr zarray shape, chunks, dtype: (732727,), (200000,), int64
2024-08-09 12:00:51 - pc_concat - INFO - starting dask local cluster.
2024-08-09 12:00:52 - pc_concat - INFO - dask local cluster started.
2024-08-09 12:00:52 - dask_cluster_info - INFO - dask cluster: LocalCluster(dashboard_link='http://10.211.48.20:8787/status', workers=1, threads=1, memory=1.46 TiB)
2024-08-09 12:00:52 - pc_concat - INFO - read pc from [PosixPath('ds_processing/gix/ds_can_ph/0.zarr'), PosixPath('ds_processing/gix/ds_can_ph/1.zarr'), PosixPath('ds_processing/gix/ds_can_ph/2.zarr'), PosixPath('ds_processing/gix/ds_can_ph/3.zarr'), PosixPath('ds_processing/gix/ds_can_ph/4.zarr'), PosixPath('ds_processing/gix/ds_can_ph/5.zarr')]
2024-08-09 12:00:52 - darr_info - INFO - concatenated pc dask array shape, chunksize, dtype: (732727, 17), (732727, 1), complex64
2024-08-09 12:00:52 - pc_concat - INFO - sort pc according to key
2024-08-09 12:00:52 - darr_info - INFO - sorted pc dask array shape, chunksize, dtype: (732727, 17), (732727, 1), complex64
2024-08-09 12:00:52 - pc_concat - INFO - save pc to ./ds_processing/hix/ds_can/ds_can_ph.zarr
2024-08-09 12:00:52 - zarr_info - INFO - ./ds_processing/hix/ds_can/ds_can_ph.zarr zarray shape, chunks, dtype: (732727, 17), (200000, 1), complex64
2024-08-09 12:00:52 - pc_concat - INFO - read pc from [PosixPath('ds_processing/gix/ds_can_emi_quality/0.zarr'), PosixPath('ds_processing/gix/ds_can_emi_quality/1.zarr'), PosixPath('ds_processing/gix/ds_can_emi_quality/2.zarr'), PosixPath('ds_processing/gix/ds_can_emi_quality/3.zarr'), PosixPath('ds_processing/gix/ds_can_emi_quality/4.zarr'), PosixPath('ds_processing/gix/ds_can_emi_quality/5.zarr')]
2024-08-09 12:00:52 - darr_info - INFO - concatenated pc dask array shape, chunksize, dtype: (732727,), (732727,), float32
2024-08-09 12:00:52 - pc_concat - INFO - sort pc according to key
2024-08-09 12:00:52 - darr_info - INFO - sorted pc dask array shape, chunksize, dtype: (732727,), (732727,), float32
2024-08-09 12:00:52 - pc_concat - INFO - save pc to ./ds_processing/hix/ds_can/ds_can_emi_quality.zarr
2024-08-09 12:00:52 - zarr_info - INFO - ./ds_processing/hix/ds_can/ds_can_emi_quality.zarr zarray shape, chunks, dtype: (732727,), (200000,), float32
2024-08-09 12:00:52 - pc_concat - INFO - read pc from [PosixPath('ds_processing/gix/ds_can_t_coh/0.zarr'), PosixPath('ds_processing/gix/ds_can_t_coh/1.zarr'), PosixPath('ds_processing/gix/ds_can_t_coh/2.zarr'), PosixPath('ds_processing/gix/ds_can_t_coh/3.zarr'), PosixPath('ds_processing/gix/ds_can_t_coh/4.zarr'), PosixPath('ds_processing/gix/ds_can_t_coh/5.zarr')]
2024-08-09 12:00:52 - darr_info - INFO - concatenated pc dask array shape, chunksize, dtype: (732727,), (732727,), float32
2024-08-09 12:00:52 - pc_concat - INFO - sort pc according to key
2024-08-09 12:00:52 - darr_info - INFO - sorted pc dask array shape, chunksize, dtype: (732727,), (732727,), float32
2024-08-09 12:00:52 - pc_concat - INFO - save pc to ./ds_processing/hix/ds_can/ds_can_t_coh.zarr
2024-08-09 12:00:52 - zarr_info - INFO - ./ds_processing/hix/ds_can/ds_can_t_coh.zarr zarray shape, chunks, dtype: (732727,), (200000,), float32
2024-08-09 12:00:52 - pc_concat - INFO - computing graph setted. doing all the computing.
2024-08-09 12:00:53 - pc_concat - INFO - computing finished.|  0.8s
2024-08-09 12:00:53 - pc_concat - INFO - dask cluster closed.

DS candidiates refinement

Then, we refine the ds candidate using EMI quality factor and the temporal coherence:

ds_can_r1_hix = './ds_processing/hix/ds_can/ds_can_r1_hix.zarr'
mc.pc_logic_pc(ds_can_hix, ds_can_emi_quality,ds_can_r1_hix,'(pc_in>=1.0)&(pc_in<=1.05)')

2024-08-09 12:00:53 - log_args - INFO - running function: pc_logic_pc
2024-08-09 12:00:53 - log_args - INFO - fetching args:
2024-08-09 12:00:53 - log_args - INFO - idx_in = './ds_processing/hix/ds_can/ds_can_hix.zarr'
2024-08-09 12:00:53 - log_args - INFO - pc_in = './ds_processing/hix/ds_can/ds_can_emi_quality.zarr'
2024-08-09 12:00:53 - log_args - INFO - idx = './ds_processing/hix/ds_can/ds_can_r1_hix.zarr'
2024-08-09 12:00:53 - log_args - INFO - operation = '(pc_in>=1.0)&(pc_in<=1.05)'
2024-08-09 12:00:53 - log_args - INFO - chunks = None
2024-08-09 12:00:53 - log_args - INFO - fetching args done.
2024-08-09 12:00:53 - zarr_info - INFO - ./ds_processing/hix/ds_can/ds_can_hix.zarr zarray shape, chunks, dtype: (732727,), (200000,), int64
2024-08-09 12:00:53 - zarr_info - INFO - ./ds_processing/hix/ds_can/ds_can_emi_quality.zarr zarray shape, chunks, dtype: (732727,), (200000,), float32
2024-08-09 12:00:53 - pc_logic_pc - INFO - loading idx_in into memory.
2024-08-09 12:00:53 - pc_logic_pc - INFO - loading pc_in into memory.
2024-08-09 12:00:53 - pc_logic_pc - INFO - select pc based on operation: (pc_in>=1.0)&(pc_in<=1.05)
2024-08-09 12:00:53 - pc_logic_pc - INFO - number of selected pixels: 404009.
2024-08-09 12:00:53 - zarr_info - INFO - idx zarray shape, chunks, dtype: (404009,), (200000,), int64
2024-08-09 12:00:53 - pc_logic_pc - INFO - writing idx.
2024-08-09 12:00:53 - pc_logic_pc - INFO - write done.

ds_can_r2_hix = './ds_processing/hix/ds_can/ds_can_r2_hix.zarr'
mc.pc_logic_pc(ds_can_hix, ds_can_t_coh,ds_can_r2_hix,'(pc_in>=0.8)&(pc_in<=1.0)')

2024-08-09 12:00:53 - log_args - INFO - running function: pc_logic_pc
2024-08-09 12:00:53 - log_args - INFO - fetching args:
2024-08-09 12:00:53 - log_args - INFO - idx_in = './ds_processing/hix/ds_can/ds_can_hix.zarr'
2024-08-09 12:00:53 - log_args - INFO - pc_in = './ds_processing/hix/ds_can/ds_can_t_coh.zarr'
2024-08-09 12:00:53 - log_args - INFO - idx = './ds_processing/hix/ds_can/ds_can_r2_hix.zarr'
2024-08-09 12:00:53 - log_args - INFO - operation = '(pc_in>=0.8)&(pc_in<=1.0)'
2024-08-09 12:00:53 - log_args - INFO - chunks = None
2024-08-09 12:00:53 - log_args - INFO - fetching args done.
2024-08-09 12:00:53 - zarr_info - INFO - ./ds_processing/hix/ds_can/ds_can_hix.zarr zarray shape, chunks, dtype: (732727,), (200000,), int64
2024-08-09 12:00:53 - zarr_info - INFO - ./ds_processing/hix/ds_can/ds_can_t_coh.zarr zarray shape, chunks, dtype: (732727,), (200000,), float32
2024-08-09 12:00:53 - pc_logic_pc - INFO - loading idx_in into memory.
2024-08-09 12:00:53 - pc_logic_pc - INFO - loading pc_in into memory.
2024-08-09 12:00:53 - pc_logic_pc - INFO - select pc based on operation: (pc_in>=0.8)&(pc_in<=1.0)
2024-08-09 12:00:53 - pc_logic_pc - INFO - number of selected pixels: 379467.
2024-08-09 12:00:53 - zarr_info - INFO - idx zarray shape, chunks, dtype: (379467,), (200000,), int64
2024-08-09 12:00:53 - pc_logic_pc - INFO - writing idx.
2024-08-09 12:00:53 - pc_logic_pc - INFO - write done.

Get their intersection:

Note

Note that pc_union can be applied if the coherence is not good and no much pixels selected.

ds_hix = './ds_processing/hix/ds/ds_hix.zarr'

mc.pc_intersect(ds_can_r1_hix,ds_can_r2_hix,ds_hix)

2024-08-09 12:00:53 - log_args - INFO - running function: pc_intersect
2024-08-09 12:00:53 - log_args - INFO - fetching args:
2024-08-09 12:00:53 - log_args - INFO - idx1 = './ds_processing/hix/ds_can/ds_can_r1_hix.zarr'
2024-08-09 12:00:53 - log_args - INFO - idx2 = './ds_processing/hix/ds_can/ds_can_r2_hix.zarr'
2024-08-09 12:00:53 - log_args - INFO - idx = './ds_processing/hix/ds/ds_hix.zarr'
2024-08-09 12:00:53 - log_args - INFO - pc1 = None
2024-08-09 12:00:53 - log_args - INFO - pc2 = None
2024-08-09 12:00:53 - log_args - INFO - pc = None
2024-08-09 12:00:53 - log_args - INFO - shape = None
2024-08-09 12:00:53 - log_args - INFO - chunks = None
2024-08-09 12:00:53 - log_args - INFO - prefer_1 = True
2024-08-09 12:00:53 - log_args - INFO - processes = False
2024-08-09 12:00:53 - log_args - INFO - n_workers = 1
2024-08-09 12:00:53 - log_args - INFO - threads_per_worker = 1
2024-08-09 12:00:53 - log_args - INFO - dask_cluster_arg = {}
2024-08-09 12:00:53 - log_args - INFO - fetching args done.
2024-08-09 12:00:53 - zarr_info - INFO - ./ds_processing/hix/ds_can/ds_can_r1_hix.zarr zarray shape, chunks, dtype: (404009,), (200000,), int64
2024-08-09 12:00:53 - zarr_info - INFO - ./ds_processing/hix/ds_can/ds_can_r2_hix.zarr zarray shape, chunks, dtype: (379467,), (200000,), int64
2024-08-09 12:00:53 - pc_intersect - INFO - loading idx1 and idx2 into memory.
2024-08-09 12:00:53 - pc_intersect - INFO - calculate the intersection
2024-08-09 12:00:53 - pc_intersect - INFO - number of points in the intersection: 292623
2024-08-09 12:00:53 - pc_intersect - INFO - write intersect idx
2024-08-09 12:00:53 - pc_intersect - INFO - write done
2024-08-09 12:00:53 - zarr_info - INFO - ./ds_processing/hix/ds/ds_hix.zarr zarray shape, chunks, dtype: (292623,), (200000,), int64
2024-08-09 12:00:53 - pc_intersect - INFO - no point cloud data provided, exit.

Visualize phase linked interferograms at refined ds points

ds_coh = './ds_processing/hix/ds/ds_coh.zarr'
ds_ph = './ds_processing/hix/ds/ds_ph.zarr'
ds_gix = './ds_processing/hix/ds/ds_gix.zarr'

mc.pc_select_data(
    ds_can_hix,ds_hix,
    pc_in=[ds_can_coh, ds_can_ph, './ds_processing/hix/ds_can/ds_can_gix.zarr/' ],
    pc=[ds_coh,ds_ph, ds_gix],threads_per_worker=32)

2024-08-09 12:00:53 - log_args - INFO - running function: pc_select_data
2024-08-09 12:00:53 - log_args - INFO - fetching args:
2024-08-09 12:00:53 - log_args - INFO - idx_in = './ds_processing/hix/ds_can/ds_can_hix.zarr'
2024-08-09 12:00:53 - log_args - INFO - idx = './ds_processing/hix/ds/ds_hix.zarr'
2024-08-09 12:00:53 - log_args - INFO - pc_in = ['./ds_processing/hix/ds_can/ds_can_coh.zarr', './ds_processing/hix/ds_can/ds_can_ph.zarr', './ds_processing/hix/ds_can/ds_can_gix.zarr/']
2024-08-09 12:00:53 - log_args - INFO - pc = ['./ds_processing/hix/ds/ds_coh.zarr', './ds_processing/hix/ds/ds_ph.zarr', './ds_processing/hix/ds/ds_gix.zarr']
2024-08-09 12:00:53 - log_args - INFO - shape = None
2024-08-09 12:00:53 - log_args - INFO - chunks = None
2024-08-09 12:00:53 - log_args - INFO - processes = False
2024-08-09 12:00:53 - log_args - INFO - n_workers = 1
2024-08-09 12:00:53 - log_args - INFO - threads_per_worker = 32
2024-08-09 12:00:53 - log_args - INFO - dask_cluster_arg = {}
2024-08-09 12:00:53 - log_args - INFO - fetching args done.
2024-08-09 12:00:53 - zarr_info - INFO - ./ds_processing/hix/ds_can/ds_can_hix.zarr zarray shape, chunks, dtype: (732727,), (200000,), int64
2024-08-09 12:00:53 - zarr_info - INFO - ./ds_processing/hix/ds/ds_hix.zarr zarray shape, chunks, dtype: (292623,), (200000,), int64
2024-08-09 12:00:53 - pc_select_data - INFO - loading idx_in and idx into memory.
2024-08-09 12:00:53 - pc_select_data - INFO - starting dask local cluster.
2024-08-09 12:00:53 - pc_select_data - INFO - dask local cluster started.
2024-08-09 12:00:53 - dask_cluster_info - INFO - dask cluster: LocalCluster(dashboard_link='http://10.211.48.20:8787/status', workers=1, threads=32, memory=1.46 TiB)
2024-08-09 12:00:53 - zarr_info - INFO - ./ds_processing/hix/ds_can/ds_can_coh.zarr zarray shape, chunks, dtype: (732727, 136), (200000, 1), complex64
2024-08-09 12:00:53 - darr_info - INFO - pc_in dask array shape, chunksize, dtype: (732727, 136), (732727, 1), complex64
2024-08-09 12:00:53 - pc_select_data - INFO - set up selected pc data dask array.
2024-08-09 12:00:53 - darr_info - INFO - pc dask array shape, chunksize, dtype: (292623, 136), (292623, 1), complex64
2024-08-09 12:00:53 - pc_select_data - INFO - write pc to ./ds_processing/hix/ds/ds_coh.zarr
2024-08-09 12:00:53 - zarr_info - INFO - ./ds_processing/hix/ds/ds_coh.zarr zarray shape, chunks, dtype: (292623, 136), (200000, 1), complex64
2024-08-09 12:00:53 - zarr_info - INFO - ./ds_processing/hix/ds_can/ds_can_ph.zarr zarray shape, chunks, dtype: (732727, 17), (200000, 1), complex64
2024-08-09 12:00:53 - darr_info - INFO - pc_in dask array shape, chunksize, dtype: (732727, 17), (732727, 1), complex64
2024-08-09 12:00:53 - pc_select_data - INFO - set up selected pc data dask array.
2024-08-09 12:00:53 - darr_info - INFO - pc dask array shape, chunksize, dtype: (292623, 17), (292623, 1), complex64
2024-08-09 12:00:53 - pc_select_data - INFO - write pc to ./ds_processing/hix/ds/ds_ph.zarr
2024-08-09 12:00:53 - zarr_info - INFO - ./ds_processing/hix/ds/ds_ph.zarr zarray shape, chunks, dtype: (292623, 17), (200000, 1), complex64
2024-08-09 12:00:53 - zarr_info - INFO - ./ds_processing/hix/ds_can/ds_can_gix.zarr/ zarray shape, chunks, dtype: (732727, 2), (200000, 1), int32
2024-08-09 12:00:53 - darr_info - INFO - pc_in dask array shape, chunksize, dtype: (732727, 2), (732727, 1), int32
2024-08-09 12:00:53 - pc_select_data - INFO - set up selected pc data dask array.
2024-08-09 12:00:53 - darr_info - INFO - pc dask array shape, chunksize, dtype: (292623, 2), (292623, 1), int32
2024-08-09 12:00:53 - pc_select_data - INFO - write pc to ./ds_processing/hix/ds/ds_gix.zarr
2024-08-09 12:00:53 - zarr_info - INFO - ./ds_processing/hix/ds/ds_gix.zarr zarray shape, chunks, dtype: (292623, 2), (200000, 1), int32
2024-08-09 12:00:53 - pc_select_data - INFO - computing graph setted. doing all the computing.
2024-08-09 12:00:55 - pc_select_data - INFO - computing finished.5s
2024-08-09 12:00:55 - pc_select_data - INFO - dask cluster closed.

Then plot them:

mc.pc_pyramid(
    './ds_processing/hix/ds/ds_coh.zarr',
    './ds_processing/hix/ds/ds_coh_pyramid',
    yx = './ds_processing/hix/ds/ds_gix.zarr',
    ras_resolution=2,
)

2024-08-09 12:00:55 - log_args - INFO - running function: pc_pyramid
2024-08-09 12:00:55 - log_args - INFO - fetching args:
2024-08-09 12:00:55 - log_args - INFO - pc = './ds_processing/hix/ds/ds_coh.zarr'
2024-08-09 12:00:55 - log_args - INFO - out_dir = './ds_processing/hix/ds/ds_coh_pyramid'
2024-08-09 12:00:55 - log_args - INFO - x = None
2024-08-09 12:00:55 - log_args - INFO - y = None
2024-08-09 12:00:55 - log_args - INFO - yx = './ds_processing/hix/ds/ds_gix.zarr'
2024-08-09 12:00:55 - log_args - INFO - ras_resolution = 2
2024-08-09 12:00:55 - log_args - INFO - ras_chunks = (256, 256)
2024-08-09 12:00:55 - log_args - INFO - pc_chunks = 65536
2024-08-09 12:00:55 - log_args - INFO - processes = False
2024-08-09 12:00:55 - log_args - INFO - n_workers = 1
2024-08-09 12:00:55 - log_args - INFO - threads_per_worker = 2
2024-08-09 12:00:55 - log_args - INFO - dask_cluster_arg = {}
2024-08-09 12:00:55 - log_args - INFO - fetching args done.
2024-08-09 12:00:57 - zarr_info - INFO - ./ds_processing/hix/ds/ds_coh.zarr zarray shape, chunks, dtype: (292623, 136), (200000, 1), complex64
2024-08-09 12:00:57 - pc_pyramid - INFO - rendering point cloud data coordinates:
2024-08-09 12:00:57 - pc_pyramid - INFO - rasterizing point cloud data to grid with bounds: [0, 0, 1832, 2498].
2024-08-09 12:00:58 - pc_pyramid - INFO - dask local cluster started.
2024-08-09 12:00:58 - dask_cluster_info - INFO - dask cluster: LocalCluster(dashboard_link='http://10.211.48.20:8787/status', workers=1, threads=2, memory=1.46 TiB)
2024-08-09 12:00:58 - pc_pyramid - INFO - pc data coordinates rendering ends.
2024-08-09 12:00:58 - zarr_info - INFO - ds_processing/hix/ds/ds_coh_pyramid/pc.zarr zarray shape, chunks, dtype: (292623, 136), (65536, 1), complex64
2024-08-09 12:00:58 - pc_pyramid - INFO - pc data rendering ends.
2024-08-09 12:00:58 - darr_info - INFO - rasterized pc data at level 0 dask array shape, chunksize, dtype: (1250, 917, 136), (1250, 917, 1), complex64
2024-08-09 12:00:58 - darr_info - INFO - rasterized pc index at level 0 dask array shape, chunksize, dtype: (1250, 917), (1250, 917), int64
2024-08-09 12:00:58 - zarr_info - INFO - ds_processing/hix/ds/ds_coh_pyramid/0.zarr zarray shape, chunks, dtype: (1250, 917, 136), (256, 256, 1), complex64
2024-08-09 12:00:58 - zarr_info - INFO - ds_processing/hix/ds/ds_coh_pyramid/idx_0.zarr zarray shape, chunks, dtype: (1250, 917), (256, 256), int64
2024-08-09 12:00:58 - darr_info - INFO - rasterized pc data at level 1 dask array shape, chunksize, dtype: (625, 459, 136), (625, 459, 1), complex64
2024-08-09 12:00:58 - darr_info - INFO - rasterized pc index at level 1 dask array shape, chunksize, dtype: (625, 459), (625, 459), int64
2024-08-09 12:00:58 - zarr_info - INFO - ds_processing/hix/ds/ds_coh_pyramid/1.zarr zarray shape, chunks, dtype: (625, 459, 136), (256, 256, 1), complex64
2024-08-09 12:00:58 - zarr_info - INFO - ds_processing/hix/ds/ds_coh_pyramid/idx_1.zarr zarray shape, chunks, dtype: (625, 459), (256, 256), int64
2024-08-09 12:00:58 - darr_info - INFO - rasterized pc data at level 2 dask array shape, chunksize, dtype: (313, 230, 136), (313, 230, 1), complex64
2024-08-09 12:00:58 - darr_info - INFO - rasterized pc index at level 2 dask array shape, chunksize, dtype: (313, 230), (313, 230), int64
2024-08-09 12:00:58 - zarr_info - INFO - ds_processing/hix/ds/ds_coh_pyramid/2.zarr zarray shape, chunks, dtype: (313, 230, 136), (256, 256, 1), complex64
2024-08-09 12:00:58 - zarr_info - INFO - ds_processing/hix/ds/ds_coh_pyramid/idx_2.zarr zarray shape, chunks, dtype: (313, 230), (256, 256), int64
2024-08-09 12:00:58 - darr_info - INFO - rasterized pc data at level 3 dask array shape, chunksize, dtype: (157, 115, 136), (157, 115, 1), complex64
2024-08-09 12:00:58 - darr_info - INFO - rasterized pc index at level 3 dask array shape, chunksize, dtype: (157, 115), (157, 115), int64
2024-08-09 12:00:58 - zarr_info - INFO - ds_processing/hix/ds/ds_coh_pyramid/3.zarr zarray shape, chunks, dtype: (157, 115, 136), (256, 256, 1), complex64
2024-08-09 12:00:58 - zarr_info - INFO - ds_processing/hix/ds/ds_coh_pyramid/idx_3.zarr zarray shape, chunks, dtype: (157, 115), (256, 256), int64
2024-08-09 12:00:58 - darr_info - INFO - rasterized pc data at level 4 dask array shape, chunksize, dtype: (79, 58, 136), (79, 58, 1), complex64
2024-08-09 12:00:58 - darr_info - INFO - rasterized pc index at level 4 dask array shape, chunksize, dtype: (79, 58), (79, 58), int64
2024-08-09 12:00:58 - zarr_info - INFO - ds_processing/hix/ds/ds_coh_pyramid/4.zarr zarray shape, chunks, dtype: (79, 58, 136), (256, 256, 1), complex64
2024-08-09 12:00:58 - zarr_info - INFO - ds_processing/hix/ds/ds_coh_pyramid/idx_4.zarr zarray shape, chunks, dtype: (79, 58), (256, 256), int64
2024-08-09 12:00:58 - darr_info - INFO - rasterized pc data at level 5 dask array shape, chunksize, dtype: (40, 29, 136), (40, 29, 1), complex64
2024-08-09 12:00:58 - darr_info - INFO - rasterized pc index at level 5 dask array shape, chunksize, dtype: (40, 29), (40, 29), int64
2024-08-09 12:00:58 - zarr_info - INFO - ds_processing/hix/ds/ds_coh_pyramid/5.zarr zarray shape, chunks, dtype: (40, 29, 136), (256, 256, 1), complex64
2024-08-09 12:00:58 - zarr_info - INFO - ds_processing/hix/ds/ds_coh_pyramid/idx_5.zarr zarray shape, chunks, dtype: (40, 29), (256, 256), int64
2024-08-09 12:00:58 - darr_info - INFO - rasterized pc data at level 6 dask array shape, chunksize, dtype: (20, 15, 136), (20, 15, 1), complex64
2024-08-09 12:00:58 - darr_info - INFO - rasterized pc index at level 6 dask array shape, chunksize, dtype: (20, 15), (20, 15), int64
2024-08-09 12:00:58 - zarr_info - INFO - ds_processing/hix/ds/ds_coh_pyramid/6.zarr zarray shape, chunks, dtype: (20, 15, 136), (256, 256, 1), complex64
2024-08-09 12:00:58 - zarr_info - INFO - ds_processing/hix/ds/ds_coh_pyramid/idx_6.zarr zarray shape, chunks, dtype: (20, 15), (256, 256), int64
2024-08-09 12:00:58 - darr_info - INFO - rasterized pc data at level 7 dask array shape, chunksize, dtype: (10, 8, 136), (10, 8, 1), complex64
2024-08-09 12:00:58 - darr_info - INFO - rasterized pc index at level 7 dask array shape, chunksize, dtype: (10, 8), (10, 8), int64
2024-08-09 12:00:58 - zarr_info - INFO - ds_processing/hix/ds/ds_coh_pyramid/7.zarr zarray shape, chunks, dtype: (10, 8, 136), (256, 256, 1), complex64
2024-08-09 12:00:58 - zarr_info - INFO - ds_processing/hix/ds/ds_coh_pyramid/idx_7.zarr zarray shape, chunks, dtype: (10, 8), (256, 256), int64
2024-08-09 12:00:58 - darr_info - INFO - rasterized pc data at level 8 dask array shape, chunksize, dtype: (5, 4, 136), (5, 4, 1), complex64
2024-08-09 12:00:58 - darr_info - INFO - rasterized pc index at level 8 dask array shape, chunksize, dtype: (5, 4), (5, 4), int64
2024-08-09 12:00:58 - zarr_info - INFO - ds_processing/hix/ds/ds_coh_pyramid/8.zarr zarray shape, chunks, dtype: (5, 4, 136), (256, 256, 1), complex64
2024-08-09 12:00:58 - zarr_info - INFO - ds_processing/hix/ds/ds_coh_pyramid/idx_8.zarr zarray shape, chunks, dtype: (5, 4), (256, 256), int64
2024-08-09 12:00:58 - darr_info - INFO - rasterized pc data at level 9 dask array shape, chunksize, dtype: (3, 2, 136), (3, 2, 1), complex64
2024-08-09 12:00:58 - darr_info - INFO - rasterized pc index at level 9 dask array shape, chunksize, dtype: (3, 2), (3, 2), int64
2024-08-09 12:00:58 - zarr_info - INFO - ds_processing/hix/ds/ds_coh_pyramid/9.zarr zarray shape, chunks, dtype: (3, 2, 136), (256, 256, 1), complex64
2024-08-09 12:00:59 - zarr_info - INFO - ds_processing/hix/ds/ds_coh_pyramid/idx_9.zarr zarray shape, chunks, dtype: (3, 2), (256, 256), int64
2024-08-09 12:00:59 - pc_pyramid - INFO - computing graph setted. doing all the computing.

/users/kangl/miniforge3/envs/work/lib/python3.10/site-packages/distributed/client.py:3162: UserWarning: Sending large graph of size 13.76 MiB.
This may cause some slowdown.
Consider scattering data ahead of time and using futures.
  warnings.warn(

2024-08-09 12:01:18 - pc_pyramid - INFO - computing finished. 19.0s
2024-08-09 12:01:18 - pc_pyramid - INFO - dask cluster closed.

mc.pc_pyramid(
    './ds_processing/hix/ds/ds_ph.zarr',
    './ds_processing/hix/ds/ds_ph_pyramid',
    yx = './ds_processing/hix/ds/ds_gix.zarr',
    ras_resolution=2,
    threads_per_worker=32,
)

2024-08-09 12:01:18 - log_args - INFO - running function: pc_pyramid
2024-08-09 12:01:18 - log_args - INFO - fetching args:
2024-08-09 12:01:18 - log_args - INFO - pc = './ds_processing/hix/ds/ds_ph.zarr'
2024-08-09 12:01:18 - log_args - INFO - out_dir = './ds_processing/hix/ds/ds_ph_pyramid'
2024-08-09 12:01:18 - log_args - INFO - x = None
2024-08-09 12:01:18 - log_args - INFO - y = None
2024-08-09 12:01:18 - log_args - INFO - yx = './ds_processing/hix/ds/ds_gix.zarr'
2024-08-09 12:01:18 - log_args - INFO - ras_resolution = 2
2024-08-09 12:01:18 - log_args - INFO - ras_chunks = (256, 256)
2024-08-09 12:01:18 - log_args - INFO - pc_chunks = 65536
2024-08-09 12:01:18 - log_args - INFO - processes = False
2024-08-09 12:01:18 - log_args - INFO - n_workers = 1
2024-08-09 12:01:18 - log_args - INFO - threads_per_worker = 32
2024-08-09 12:01:18 - log_args - INFO - dask_cluster_arg = {}
2024-08-09 12:01:18 - log_args - INFO - fetching args done.
2024-08-09 12:01:19 - zarr_info - INFO - ./ds_processing/hix/ds/ds_ph.zarr zarray shape, chunks, dtype: (292623, 17), (200000, 1), complex64
2024-08-09 12:01:19 - pc_pyramid - INFO - rendering point cloud data coordinates:
2024-08-09 12:01:19 - pc_pyramid - INFO - rasterizing point cloud data to grid with bounds: [0, 0, 1832, 2498].
2024-08-09 12:01:19 - pc_pyramid - INFO - dask local cluster started.
2024-08-09 12:01:19 - dask_cluster_info - INFO - dask cluster: LocalCluster(dashboard_link='http://10.211.48.20:8787/status', workers=1, threads=32, memory=1.46 TiB)
2024-08-09 12:01:19 - pc_pyramid - INFO - pc data coordinates rendering ends.
2024-08-09 12:01:19 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_pyramid/pc.zarr zarray shape, chunks, dtype: (292623, 17), (65536, 1), complex64
2024-08-09 12:01:19 - pc_pyramid - INFO - pc data rendering ends.
2024-08-09 12:01:19 - darr_info - INFO - rasterized pc data at level 0 dask array shape, chunksize, dtype: (1250, 917, 17), (1250, 917, 1), complex64
2024-08-09 12:01:19 - darr_info - INFO - rasterized pc index at level 0 dask array shape, chunksize, dtype: (1250, 917), (1250, 917), int64
2024-08-09 12:01:19 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_pyramid/0.zarr zarray shape, chunks, dtype: (1250, 917, 17), (256, 256, 1), complex64
2024-08-09 12:01:19 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_pyramid/idx_0.zarr zarray shape, chunks, dtype: (1250, 917), (256, 256), int64
2024-08-09 12:01:19 - darr_info - INFO - rasterized pc data at level 1 dask array shape, chunksize, dtype: (625, 459, 17), (625, 459, 1), complex64
2024-08-09 12:01:19 - darr_info - INFO - rasterized pc index at level 1 dask array shape, chunksize, dtype: (625, 459), (625, 459), int64
2024-08-09 12:01:19 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_pyramid/1.zarr zarray shape, chunks, dtype: (625, 459, 17), (256, 256, 1), complex64
2024-08-09 12:01:19 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_pyramid/idx_1.zarr zarray shape, chunks, dtype: (625, 459), (256, 256), int64
2024-08-09 12:01:19 - darr_info - INFO - rasterized pc data at level 2 dask array shape, chunksize, dtype: (313, 230, 17), (313, 230, 1), complex64
2024-08-09 12:01:19 - darr_info - INFO - rasterized pc index at level 2 dask array shape, chunksize, dtype: (313, 230), (313, 230), int64
2024-08-09 12:01:19 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_pyramid/2.zarr zarray shape, chunks, dtype: (313, 230, 17), (256, 256, 1), complex64
2024-08-09 12:01:19 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_pyramid/idx_2.zarr zarray shape, chunks, dtype: (313, 230), (256, 256), int64
2024-08-09 12:01:19 - darr_info - INFO - rasterized pc data at level 3 dask array shape, chunksize, dtype: (157, 115, 17), (157, 115, 1), complex64
2024-08-09 12:01:19 - darr_info - INFO - rasterized pc index at level 3 dask array shape, chunksize, dtype: (157, 115), (157, 115), int64
2024-08-09 12:01:19 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_pyramid/3.zarr zarray shape, chunks, dtype: (157, 115, 17), (256, 256, 1), complex64
2024-08-09 12:01:19 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_pyramid/idx_3.zarr zarray shape, chunks, dtype: (157, 115), (256, 256), int64
2024-08-09 12:01:19 - darr_info - INFO - rasterized pc data at level 4 dask array shape, chunksize, dtype: (79, 58, 17), (79, 58, 1), complex64
2024-08-09 12:01:19 - darr_info - INFO - rasterized pc index at level 4 dask array shape, chunksize, dtype: (79, 58), (79, 58), int64
2024-08-09 12:01:19 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_pyramid/4.zarr zarray shape, chunks, dtype: (79, 58, 17), (256, 256, 1), complex64
2024-08-09 12:01:19 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_pyramid/idx_4.zarr zarray shape, chunks, dtype: (79, 58), (256, 256), int64
2024-08-09 12:01:19 - darr_info - INFO - rasterized pc data at level 5 dask array shape, chunksize, dtype: (40, 29, 17), (40, 29, 1), complex64
2024-08-09 12:01:19 - darr_info - INFO - rasterized pc index at level 5 dask array shape, chunksize, dtype: (40, 29), (40, 29), int64
2024-08-09 12:01:19 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_pyramid/5.zarr zarray shape, chunks, dtype: (40, 29, 17), (256, 256, 1), complex64
2024-08-09 12:01:19 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_pyramid/idx_5.zarr zarray shape, chunks, dtype: (40, 29), (256, 256), int64
2024-08-09 12:01:19 - darr_info - INFO - rasterized pc data at level 6 dask array shape, chunksize, dtype: (20, 15, 17), (20, 15, 1), complex64
2024-08-09 12:01:19 - darr_info - INFO - rasterized pc index at level 6 dask array shape, chunksize, dtype: (20, 15), (20, 15), int64
2024-08-09 12:01:19 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_pyramid/6.zarr zarray shape, chunks, dtype: (20, 15, 17), (256, 256, 1), complex64
2024-08-09 12:01:19 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_pyramid/idx_6.zarr zarray shape, chunks, dtype: (20, 15), (256, 256), int64
2024-08-09 12:01:19 - darr_info - INFO - rasterized pc data at level 7 dask array shape, chunksize, dtype: (10, 8, 17), (10, 8, 1), complex64
2024-08-09 12:01:19 - darr_info - INFO - rasterized pc index at level 7 dask array shape, chunksize, dtype: (10, 8), (10, 8), int64
2024-08-09 12:01:19 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_pyramid/7.zarr zarray shape, chunks, dtype: (10, 8, 17), (256, 256, 1), complex64
2024-08-09 12:01:19 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_pyramid/idx_7.zarr zarray shape, chunks, dtype: (10, 8), (256, 256), int64
2024-08-09 12:01:19 - darr_info - INFO - rasterized pc data at level 8 dask array shape, chunksize, dtype: (5, 4, 17), (5, 4, 1), complex64
2024-08-09 12:01:19 - darr_info - INFO - rasterized pc index at level 8 dask array shape, chunksize, dtype: (5, 4), (5, 4), int64
2024-08-09 12:01:19 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_pyramid/8.zarr zarray shape, chunks, dtype: (5, 4, 17), (256, 256, 1), complex64
2024-08-09 12:01:19 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_pyramid/idx_8.zarr zarray shape, chunks, dtype: (5, 4), (256, 256), int64
2024-08-09 12:01:19 - darr_info - INFO - rasterized pc data at level 9 dask array shape, chunksize, dtype: (3, 2, 17), (3, 2, 1), complex64
2024-08-09 12:01:19 - darr_info - INFO - rasterized pc index at level 9 dask array shape, chunksize, dtype: (3, 2), (3, 2), int64
2024-08-09 12:01:19 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_pyramid/9.zarr zarray shape, chunks, dtype: (3, 2, 17), (256, 256, 1), complex64
2024-08-09 12:01:19 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_pyramid/idx_9.zarr zarray shape, chunks, dtype: (3, 2), (256, 256), int64
2024-08-09 12:01:19 - pc_pyramid - INFO - computing graph setted. doing all the computing.
[#                                       ] | 3% Completed |  0.1s

/users/kangl/miniforge3/envs/work/lib/python3.10/site-packages/distributed/client.py:3162: UserWarning: Sending large graph of size 13.31 MiB.
This may cause some slowdown.
Consider scattering data ahead of time and using futures.
  warnings.warn(

2024-08-09 12:01:21 - pc_pyramid - INFO - computing finished.  2.1s
2024-08-09 12:01:21 - pc_pyramid - INFO - dask cluster closed.

ds_intf_plot = mc.pc_plot(
    './ds_processing/hix/ds/ds_coh_pyramid',
    post_proc_ras=co_phase_post_proc_ras,
    post_proc_pc=co_phase_post_proc_pc,
    n_kdim=2,
    level_increase=0)
raw_intf_plot = mc.ras_plot('./load_data/rslc_pyramid',post_proc='intf_all',n_kdim=2,level_increase=0)
ds_pl_intf_plot = mc.pc_plot('./ds_processing/hix/ds/ds_ph_pyramid',post_proc_ras='intf_all', post_proc_pc='intf_all',level_increase=0)
ds_intf_plots = raw_intf_plot.relabel('Raw Interferograms') + \
ds_intf_plot.relabel('Adaptively multilooked Interferograms') + \
ds_pl_intf_plot.relabel('Phase linked Interferograms')

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[20], line 3
      1 ds_intf_plot = mc.pc_plot(
      2     './ds_processing/hix/ds/ds_coh_pyramid',
----> 3     post_proc_ras=co_phase_post_proc_ras,
      4     post_proc_pc=co_phase_post_proc_pc,
      5     n_kdim=2,
      6     level_increase=0)
      7 raw_intf_plot = mc.ras_plot('./load_data/rslc_pyramid',post_proc='intf_all',n_kdim=2,level_increase=0)
      8 ds_pl_intf_plot = mc.pc_plot('./ds_processing/hix/ds/ds_ph_pyramid',post_proc_ras='intf_all', post_proc_pc='intf_all',level_increase=0)

NameError: name 'co_phase_post_proc_ras' is not defined

ds_intf_plots = ds_intf_plots.redim(
    i=hv.Dimension('i', label='Reference Image', range=(0,16), value_format=(lambda i: dates[i])),
    j=hv.Dimension('j', label='Secondary Image', range=(0,16), value_format=(lambda i: dates[i])),
    x=hv.Dimension('r', label='Range'),
    y=hv.Dimension('az',label='Azimuth'),
    z=hv.Dimension('Phase',range=(-np.pi,np.pi))
)

ds_intf_plots.opts(
    hv.opts.Image(
        cmap='colorwheel',frame_width=400, frame_height=500, colorbar=True,
        default_tools=['pan',WheelZoomTool(zoom_on_axis=False),'save','reset','hover'],
        active_tools=['wheel_zoom'],
        invert_yaxis=True,
    ),
    hv.opts.Points(
        color='Phase', cmap='colorwheel',frame_width=400, frame_height=500, colorbar=True,
        default_tools=['pan',WheelZoomTool(zoom_on_axis=False),'save','reset','hover'],
        active_tools=['wheel_zoom'],
        invert_yaxis=True,
    ),
)

Visualize phase linked interferograms at refined ds points in earth coordinates

Get the needed data for ds:

ds_ph = './ds_processing/hix/ds/ds_ph.zarr'
ds_e = './ds_processing/hix/ds/ds_e.zarr'
ds_n = './ds_processing/hix/ds/ds_n.zarr'

mc.ras2pc(ds_hix,
          ras=['./load_data/e.zarr/','./load_data/n.zarr/'],
          pc=[ds_e, ds_n],
         threads_per_worker=32)

2024-08-18 15:01:10 - log_args - INFO - running function: ras2pc
2024-08-18 15:01:10 - log_args - INFO - fetching args:
2024-08-18 15:01:10 - log_args - INFO - idx = './ds_processing/hix/ds/ds_hix.zarr'
2024-08-18 15:01:10 - log_args - INFO - ras = ['./load_data/e.zarr/', './load_data/n.zarr/']
2024-08-18 15:01:10 - log_args - INFO - pc = ['./ds_processing/hix/ds/ds_e.zarr', './ds_processing/hix/ds/ds_n.zarr']
2024-08-18 15:01:10 - log_args - INFO - chunks = None
2024-08-18 15:01:10 - log_args - INFO - processes = False
2024-08-18 15:01:10 - log_args - INFO - n_workers = 1
2024-08-18 15:01:10 - log_args - INFO - threads_per_worker = 32
2024-08-18 15:01:10 - log_args - INFO - dask_cluster_arg = {}
2024-08-18 15:01:10 - log_args - INFO - fetching args done.
2024-08-18 15:01:10 - zarr_info - INFO - ./ds_processing/hix/ds/ds_hix.zarr zarray shape, chunks, dtype: (292623,), (200000,), int64
2024-08-18 15:01:10 - ras2pc - INFO - loading hix into memory and convert to gix
2024-08-18 15:01:12 - ras2pc - INFO - starting dask local cluster.
2024-08-18 15:01:14 - ras2pc - INFO - dask local cluster started.
2024-08-18 15:01:14 - dask_cluster_info - INFO - dask cluster: LocalCluster(dashboard_link='http://10.211.48.9:8787/status', workers=1, threads=32, memory=1.46 TiB)
2024-08-18 15:01:14 - ras2pc - INFO - start to slice on ./load_data/e.zarr/
2024-08-18 15:01:14 - zarr_info - INFO - ./load_data/e.zarr/ zarray shape, chunks, dtype: (2500, 1834), (1000, 1000), float64
2024-08-18 15:01:14 - darr_info - INFO - ras dask array shape, chunksize, dtype: (2500, 1834), (2500, 1834), float64
2024-08-18 15:01:14 - darr_info - INFO - pc dask array shape, chunksize, dtype: (292623,), (292623,), float64
2024-08-18 15:01:14 - ras2pc - INFO - saving to ./ds_processing/hix/ds/ds_e.zarr.
2024-08-18 15:01:14 - zarr_info - INFO - ./ds_processing/hix/ds/ds_e.zarr zarray shape, chunks, dtype: (292623,), (200000,), float64
2024-08-18 15:01:14 - ras2pc - INFO - start to slice on ./load_data/n.zarr/
2024-08-18 15:01:14 - zarr_info - INFO - ./load_data/n.zarr/ zarray shape, chunks, dtype: (2500, 1834), (1000, 1000), float64
2024-08-18 15:01:14 - darr_info - INFO - ras dask array shape, chunksize, dtype: (2500, 1834), (2500, 1834), float64
2024-08-18 15:01:14 - darr_info - INFO - pc dask array shape, chunksize, dtype: (292623,), (292623,), float64
2024-08-18 15:01:14 - ras2pc - INFO - saving to ./ds_processing/hix/ds/ds_n.zarr.
2024-08-18 15:01:14 - zarr_info - INFO - ./ds_processing/hix/ds/ds_n.zarr zarray shape, chunks, dtype: (292623,), (200000,), float64
2024-08-18 15:01:14 - ras2pc - INFO - computing graph setted. doing all the computing.
2024-08-18 15:01:15 - ras2pc - INFO - computing finished.ed |  0.2s
2024-08-18 15:01:15 - ras2pc - INFO - dask cluster closed.

mc.pc_pyramid(
    './ds_processing/hix/ds/ds_ph.zarr',
    './ds_processing/hix/ds/ds_ph_geo_pyramid',
    x = ds_e,
    y = ds_n,
    ras_resolution=20,
    threads_per_worker=32,
)

2024-08-09 12:01:23 - log_args - INFO - running function: pc_pyramid
2024-08-09 12:01:23 - log_args - INFO - fetching args:
2024-08-09 12:01:23 - log_args - INFO - pc = './ds_processing/hix/ds/ds_ph.zarr'
2024-08-09 12:01:23 - log_args - INFO - out_dir = './ds_processing/hix/ds/ds_ph_geo_pyramid'
2024-08-09 12:01:23 - log_args - INFO - x = './ds_processing/hix/ds/ds_e.zarr'
2024-08-09 12:01:23 - log_args - INFO - y = './ds_processing/hix/ds/ds_n.zarr'
2024-08-09 12:01:23 - log_args - INFO - yx = None
2024-08-09 12:01:23 - log_args - INFO - ras_resolution = 20
2024-08-09 12:01:23 - log_args - INFO - ras_chunks = (256, 256)
2024-08-09 12:01:23 - log_args - INFO - pc_chunks = 65536
2024-08-09 12:01:23 - log_args - INFO - processes = False
2024-08-09 12:01:23 - log_args - INFO - n_workers = 1
2024-08-09 12:01:23 - log_args - INFO - threads_per_worker = 32
2024-08-09 12:01:23 - log_args - INFO - dask_cluster_arg = {}
2024-08-09 12:01:23 - log_args - INFO - fetching args done.
2024-08-09 12:01:23 - zarr_info - INFO - ./ds_processing/hix/ds/ds_ph.zarr zarray shape, chunks, dtype: (292623, 17), (200000, 1), complex64
2024-08-09 12:01:23 - pc_pyramid - INFO - rendering point cloud data coordinates:
2024-08-09 12:01:23 - pc_pyramid - INFO - rasterizing point cloud data to grid with bounds: [-16498471.798140068, 8649651.33657227, -16470131.798140068, 8674811.33657227].
2024-08-09 12:01:24 - pc_pyramid - INFO - dask local cluster started.
2024-08-09 12:01:24 - dask_cluster_info - INFO - dask cluster: LocalCluster(dashboard_link='http://10.211.48.20:8787/status', workers=1, threads=32, memory=1.46 TiB)
2024-08-09 12:01:24 - pc_pyramid - INFO - pc data coordinates rendering ends.
2024-08-09 12:01:24 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_geo_pyramid/pc.zarr zarray shape, chunks, dtype: (292623, 17), (65536, 1), complex64
2024-08-09 12:01:24 - pc_pyramid - INFO - pc data rendering ends.
2024-08-09 12:01:24 - darr_info - INFO - rasterized pc data at level 0 dask array shape, chunksize, dtype: (1259, 1418, 17), (1259, 1418, 1), complex64
2024-08-09 12:01:24 - darr_info - INFO - rasterized pc index at level 0 dask array shape, chunksize, dtype: (1259, 1418), (1259, 1418), int64
2024-08-09 12:01:24 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_geo_pyramid/0.zarr zarray shape, chunks, dtype: (1259, 1418, 17), (256, 256, 1), complex64
2024-08-09 12:01:24 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_geo_pyramid/idx_0.zarr zarray shape, chunks, dtype: (1259, 1418), (256, 256), int64
2024-08-09 12:01:24 - darr_info - INFO - rasterized pc data at level 1 dask array shape, chunksize, dtype: (630, 709, 17), (630, 709, 1), complex64
2024-08-09 12:01:24 - darr_info - INFO - rasterized pc index at level 1 dask array shape, chunksize, dtype: (630, 709), (630, 709), int64
2024-08-09 12:01:24 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_geo_pyramid/1.zarr zarray shape, chunks, dtype: (630, 709, 17), (256, 256, 1), complex64
2024-08-09 12:01:24 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_geo_pyramid/idx_1.zarr zarray shape, chunks, dtype: (630, 709), (256, 256), int64
2024-08-09 12:01:24 - darr_info - INFO - rasterized pc data at level 2 dask array shape, chunksize, dtype: (315, 355, 17), (315, 355, 1), complex64
2024-08-09 12:01:24 - darr_info - INFO - rasterized pc index at level 2 dask array shape, chunksize, dtype: (315, 355), (315, 355), int64
2024-08-09 12:01:24 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_geo_pyramid/2.zarr zarray shape, chunks, dtype: (315, 355, 17), (256, 256, 1), complex64
2024-08-09 12:01:24 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_geo_pyramid/idx_2.zarr zarray shape, chunks, dtype: (315, 355), (256, 256), int64
2024-08-09 12:01:24 - darr_info - INFO - rasterized pc data at level 3 dask array shape, chunksize, dtype: (158, 178, 17), (158, 178, 1), complex64
2024-08-09 12:01:24 - darr_info - INFO - rasterized pc index at level 3 dask array shape, chunksize, dtype: (158, 178), (158, 178), int64
2024-08-09 12:01:24 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_geo_pyramid/3.zarr zarray shape, chunks, dtype: (158, 178, 17), (256, 256, 1), complex64
2024-08-09 12:01:24 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_geo_pyramid/idx_3.zarr zarray shape, chunks, dtype: (158, 178), (256, 256), int64
2024-08-09 12:01:24 - darr_info - INFO - rasterized pc data at level 4 dask array shape, chunksize, dtype: (79, 89, 17), (79, 89, 1), complex64
2024-08-09 12:01:24 - darr_info - INFO - rasterized pc index at level 4 dask array shape, chunksize, dtype: (79, 89), (79, 89), int64
2024-08-09 12:01:24 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_geo_pyramid/4.zarr zarray shape, chunks, dtype: (79, 89, 17), (256, 256, 1), complex64
2024-08-09 12:01:24 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_geo_pyramid/idx_4.zarr zarray shape, chunks, dtype: (79, 89), (256, 256), int64
2024-08-09 12:01:24 - darr_info - INFO - rasterized pc data at level 5 dask array shape, chunksize, dtype: (40, 45, 17), (40, 45, 1), complex64
2024-08-09 12:01:24 - darr_info - INFO - rasterized pc index at level 5 dask array shape, chunksize, dtype: (40, 45), (40, 45), int64
2024-08-09 12:01:24 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_geo_pyramid/5.zarr zarray shape, chunks, dtype: (40, 45, 17), (256, 256, 1), complex64
2024-08-09 12:01:24 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_geo_pyramid/idx_5.zarr zarray shape, chunks, dtype: (40, 45), (256, 256), int64
2024-08-09 12:01:24 - darr_info - INFO - rasterized pc data at level 6 dask array shape, chunksize, dtype: (20, 23, 17), (20, 23, 1), complex64
2024-08-09 12:01:24 - darr_info - INFO - rasterized pc index at level 6 dask array shape, chunksize, dtype: (20, 23), (20, 23), int64
2024-08-09 12:01:24 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_geo_pyramid/6.zarr zarray shape, chunks, dtype: (20, 23, 17), (256, 256, 1), complex64
2024-08-09 12:01:24 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_geo_pyramid/idx_6.zarr zarray shape, chunks, dtype: (20, 23), (256, 256), int64
2024-08-09 12:01:24 - darr_info - INFO - rasterized pc data at level 7 dask array shape, chunksize, dtype: (10, 12, 17), (10, 12, 1), complex64
2024-08-09 12:01:24 - darr_info - INFO - rasterized pc index at level 7 dask array shape, chunksize, dtype: (10, 12), (10, 12), int64
2024-08-09 12:01:24 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_geo_pyramid/7.zarr zarray shape, chunks, dtype: (10, 12, 17), (256, 256, 1), complex64
2024-08-09 12:01:24 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_geo_pyramid/idx_7.zarr zarray shape, chunks, dtype: (10, 12), (256, 256), int64
2024-08-09 12:01:24 - darr_info - INFO - rasterized pc data at level 8 dask array shape, chunksize, dtype: (5, 6, 17), (5, 6, 1), complex64
2024-08-09 12:01:24 - darr_info - INFO - rasterized pc index at level 8 dask array shape, chunksize, dtype: (5, 6), (5, 6), int64
2024-08-09 12:01:24 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_geo_pyramid/8.zarr zarray shape, chunks, dtype: (5, 6, 17), (256, 256, 1), complex64
2024-08-09 12:01:24 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_geo_pyramid/idx_8.zarr zarray shape, chunks, dtype: (5, 6), (256, 256), int64
2024-08-09 12:01:24 - darr_info - INFO - rasterized pc data at level 9 dask array shape, chunksize, dtype: (3, 3, 17), (3, 3, 1), complex64
2024-08-09 12:01:24 - darr_info - INFO - rasterized pc index at level 9 dask array shape, chunksize, dtype: (3, 3), (3, 3), int64
2024-08-09 12:01:24 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_geo_pyramid/9.zarr zarray shape, chunks, dtype: (3, 3, 17), (256, 256, 1), complex64
2024-08-09 12:01:24 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_geo_pyramid/idx_9.zarr zarray shape, chunks, dtype: (3, 3), (256, 256), int64
2024-08-09 12:01:24 - darr_info - INFO - rasterized pc data at level 10 dask array shape, chunksize, dtype: (2, 2, 17), (2, 2, 1), complex64
2024-08-09 12:01:24 - darr_info - INFO - rasterized pc index at level 10 dask array shape, chunksize, dtype: (2, 2), (2, 2), int64
2024-08-09 12:01:24 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_geo_pyramid/10.zarr zarray shape, chunks, dtype: (2, 2, 17), (256, 256, 1), complex64
2024-08-09 12:01:24 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_geo_pyramid/idx_10.zarr zarray shape, chunks, dtype: (2, 2), (256, 256), int64
2024-08-09 12:01:24 - pc_pyramid - INFO - computing graph setted. doing all the computing.
[#                                       ] | 3% Completed |  0.1s

/users/kangl/miniforge3/envs/work/lib/python3.10/site-packages/distributed/client.py:3162: UserWarning: Sending large graph of size 20.43 MiB.
This may cause some slowdown.
Consider scattering data ahead of time and using futures.
  warnings.warn(

2024-08-09 12:01:26 - pc_pyramid - INFO - computing finished.  2.0s
2024-08-09 12:01:26 - pc_pyramid - INFO - dask cluster closed.

import toml
with open('load_data/meta.toml','r') as f:
    dates = toml.load(f)['dates']
ds_geo_intf_plot = mc.pc_plot('./ds_processing/hix/ds/ds_ph_geo_pyramid',post_proc_ras='intf_all', post_proc_pc='intf_all',level_increase=1)
ds_geo_intf_plot = ds_geo_intf_plot.redim(
    i=hv.Dimension('i', label='Reference Image', range=(0,16), value_format=(lambda i: dates[i])),
    j=hv.Dimension('j', label='Secondary Image', range=(0,16), value_format=(lambda i: dates[i])),
    x=hv.Dimension('lon', label='Longitude'),
    y=hv.Dimension('lat',label='Latitude'),
    z=hv.Dimension('Phase',range=(-np.pi,np.pi))
)

hv.element.tiles.EsriImagery()*ds_geo_intf_plot.opts(
    hv.opts.Image(
        cmap='colorwheel',frame_width=500, frame_height=400, colorbar=True,
        default_tools=['pan',WheelZoomTool(zoom_on_axis=False),'save','reset','hover'],
        active_tools=['wheel_zoom']
    ),
    hv.opts.Points(
        color='Phase', cmap='colorwheel',frame_width=500, frame_height=400, colorbar=True,
        default_tools=['pan',WheelZoomTool(zoom_on_axis=False),'save','reset','hover'],
        active_tools=['wheel_zoom']
    ),
)