DS Processing

In this tutorial, we demostrate how to do DS processing with the decorrelation CLI interface. The input data for DS processing is prepared in Load Data.

!ls ./load_data

e.zarr    lat.zarr  meta.toml  phi.zarr    rslc_pyramid  scratch
hgt.zarr  lon.zarr  n.zarr     range.zarr  rslc.zarr     theta.zarr

SHP selection

Here we use two sample Kolmogorov-Smirnov (KS) test for Spatially Homogenious Pixel (SHP) identification.

The output pvalue is the P-value for this test. The smaller the p-value, the more likely the two sample are from same distribution, i.e., the more likely the two pixels are SHP to each other.

import numpy as np
import moraine.cli as mc
import moraine as mr

rslc = './load_data/rslc.zarr'
pvalue = './ds_processing/gix/ras_pvalue.zarr'
az_half_win = 5
r_half_win = 5
method = 'ks'

logger = mc.get_logger()

mc.shp_test(rslc,pvalue,
            az_half_win=az_half_win,r_half_win=r_half_win)

2025-06-24 22:18:13 - log_args - INFO - running function: shp_test
2025-06-24 22:18:13 - log_args - INFO - fetching args:
2025-06-24 22:18:13 - log_args - INFO - rslc = './load_data/rslc.zarr'
2025-06-24 22:18:13 - log_args - INFO - pvalue = './ds_processing/gix/ras_pvalue.zarr'
2025-06-24 22:18:13 - log_args - INFO - az_half_win = 5
2025-06-24 22:18:13 - log_args - INFO - r_half_win = 5
2025-06-24 22:18:13 - log_args - INFO - method = None
2025-06-24 22:18:13 - log_args - INFO - chunks = None
2025-06-24 22:18:13 - log_args - INFO - cuda = False
2025-06-24 22:18:13 - log_args - INFO - processes = None
2025-06-24 22:18:13 - log_args - INFO - n_workers = None
2025-06-24 22:18:13 - log_args - INFO - threads_per_worker = None
2025-06-24 22:18:13 - log_args - INFO - rmm_pool_size = 0.9
2025-06-24 22:18:13 - log_args - INFO - dask_cluster_arg = {}
2025-06-24 22:18:13 - log_args - INFO - fetching args done.
2025-06-24 22:18:13 - shp_test - INFO - hypothetic test method: ks
2025-06-24 22:18:13 - zarr_info - INFO - ./load_data/rslc.zarr zarray shape, chunks, dtype: (2500, 1834, 17), (1000, 1000, 1), complex64
2025-06-24 22:18:13 - shp_test - INFO - starting dask local cluster.
2025-06-24 22:18:15 - shp_test - INFO - dask local cluster started.
2025-06-24 22:18:15 - dask_cluster_info - INFO - dask cluster: LocalCluster(dashboard_link='http://10.211.48.18:8787/status', workers=1, threads=1, memory=1.46 TiB)
2025-06-24 22:18:15 - shp_test - INFO - azimuth half window size: 5; azimuth window size: 11
2025-06-24 22:18:15 - shp_test - INFO - range half window size: 5; range window size: 11
2025-06-24 22:18:15 - darr_info - INFO - rslc with overlap dask array shape, chunksize, dtype: (2520, 1844, 17), (1010, 1005, 17), complex64
2025-06-24 22:18:16 - shp_test - INFO - applying test on rmli stack.
2025-06-24 22:18:16 - shp_test - INFO - trim shared boundaries between p value chunks
2025-06-24 22:18:16 - darr_info - INFO - p value dask array shape, chunksize, dtype: (2500, 1834, 11, 11), (1000, 1000, 11, 11), float32
2025-06-24 22:18:16 - shp_test - INFO - saving p value.
2025-06-24 22:18:16 - zarr_info - INFO - ./ds_processing/gix/ras_pvalue.zarr zarray shape, chunks, dtype: (2500, 1834, 11, 11), (1000, 1000, 1, 1), float32
2025-06-24 22:18:16 - shp_test - INFO - computing graph setted. doing all the computing.
2025-06-24 22:18:23 - shp_test - INFO - computing finished. |  6.9s
2025-06-24 22:18:23 - shp_test - INFO - dask cluster closed.

Then we select SHP by setting a threshold on this Pvalue:

import zarr
from matplotlib import pyplot as plt
import colorcet

is_shp = './ds_processing/gix/ras_is_shp.zarr'
shp_num = './ds_processing/gix/ras_shp_num.zarr'
p_max = 0.05

mc.select_shp(pvalue,is_shp,shp_num,p_max=p_max)

2025-06-24 22:18:24 - log_args - INFO - running function: select_shp
2025-06-24 22:18:24 - log_args - INFO - fetching args:
2025-06-24 22:18:24 - log_args - INFO - pvalue = './ds_processing/gix/ras_pvalue.zarr'
2025-06-24 22:18:24 - log_args - INFO - is_shp = './ds_processing/gix/ras_is_shp.zarr'
2025-06-24 22:18:24 - log_args - INFO - shp_num = './ds_processing/gix/ras_shp_num.zarr'
2025-06-24 22:18:24 - log_args - INFO - p_max = 0.05
2025-06-24 22:18:24 - log_args - INFO - chunks = None
2025-06-24 22:18:24 - log_args - INFO - processes = False
2025-06-24 22:18:24 - log_args - INFO - n_workers = 1
2025-06-24 22:18:24 - log_args - INFO - threads_per_worker = 1
2025-06-24 22:18:24 - log_args - INFO - dask_cluster_arg = {}
2025-06-24 22:18:24 - log_args - INFO - fetching args done.
2025-06-24 22:18:24 - zarr_info - INFO - ./ds_processing/gix/ras_pvalue.zarr zarray shape, chunks, dtype: (2500, 1834, 11, 11), (1000, 1000, 1, 1), float32
2025-06-24 22:18:24 - select_shp - INFO - starting dask cluster.
2025-06-24 22:18:24 - select_shp - INFO - dask cluster started.
2025-06-24 22:18:24 - dask_cluster_info - INFO - dask cluster: LocalCluster(dashboard_link='http://10.211.48.18:8787/status', workers=1, threads=1, memory=1.46 TiB)
2025-06-24 22:18:24 - darr_info - INFO - pvalue dask array shape, chunksize, dtype: (2500, 1834, 11, 11), (1000, 1000, 11, 11), float32
2025-06-24 22:18:24 - select_shp - INFO - selecting SHPs based on pvalue threshold: 0.05
2025-06-24 22:18:24 - darr_info - INFO - is_shp dask array shape, chunksize, dtype: (2500, 1834, 11, 11), (1000, 1000, 11, 11), bool
2025-06-24 22:18:24 - select_shp - INFO - calculate shp_num.
2025-06-24 22:18:24 - darr_info - INFO - shp_num dask array shape, chunksize, dtype: (2500, 1834), (1000, 1000), int32
2025-06-24 22:18:24 - select_shp - INFO - saving is_shp.
2025-06-24 22:18:24 - zarr_info - INFO - ./ds_processing/gix/ras_is_shp.zarr zarray shape, chunks, dtype: (2500, 1834, 11, 11), (1000, 1000, 1, 1), bool
2025-06-24 22:18:24 - select_shp - INFO - saving shp_num.
2025-06-24 22:18:24 - zarr_info - INFO - ./ds_processing/gix/ras_shp_num.zarr zarray shape, chunks, dtype: (2500, 1834), (1000, 1000), int32
2025-06-24 22:18:24 - select_shp - INFO - computing graph setted. doing all the computing.
2025-06-24 22:18:29 - select_shp - INFO - computing finished.  4.4s
2025-06-24 22:18:29 - select_shp - INFO - dask cluster closed.

The output is_shp indicate wheather a pixel in the window is a SHP to its center pixel.

shp_num_zarr = zarr.open(shp_num,'r')
shp_num_data = shp_num_zarr[:]
shp_num_plot = mr.plot.ras_plot(shp_num_data)

import holoviews as hv
hv.extension('bokeh')
from bokeh.models import WheelZoomTool
from holoviews import opts

shp_num_plot = shp_num_plot.redim(x=hv.Dimension('r', label='Range'), y=hv.Dimension('az',label='Azimuth'), z=hv.Dimension('n_shp',))

shp_num_plot.opts(opts.Image(cmap='viridis',width=600, height=600, colorbar=True,
                          invert_yaxis=True,
                          default_tools=['pan',WheelZoomTool(zoom_on_axis=False),'save','reset','hover'],active_tools=['wheel_zoom']))

DS candidates selection

ds_can_gix = './ds_processing/gix/ds_can_gix.zarr'
is_ds_can = './ds_processing/gix/ras_is_ds_can.zarr'

mc.pc_logic_ras(shp_num,ds_can_gix,'ras>=50',chunks=200000)
mc.gix2bool(ds_can_gix, is_ds_can, shape= shp_num_zarr.shape)

2025-06-24 22:18:30 - log_args - INFO - running function: pc_logic_ras
2025-06-24 22:18:30 - log_args - INFO - fetching args:
2025-06-24 22:18:30 - log_args - INFO - ras = './ds_processing/gix/ras_shp_num.zarr'
2025-06-24 22:18:30 - log_args - INFO - gix = './ds_processing/gix/ds_can_gix.zarr'
2025-06-24 22:18:30 - log_args - INFO - operation = 'ras>=50'
2025-06-24 22:18:30 - log_args - INFO - chunks = 200000
2025-06-24 22:18:30 - log_args - INFO - fetching args done.
2025-06-24 22:18:30 - zarr_info - INFO - ./ds_processing/gix/ras_shp_num.zarr zarray shape, chunks, dtype: (2500, 1834), (1000, 1000), int32
2025-06-24 22:18:30 - pc_logic_ras - INFO - loading ras into memory.
2025-06-24 22:18:30 - pc_logic_ras - INFO - select pc based on operation: ras>=50
2025-06-24 22:18:30 - pc_logic_ras - INFO - number of selected pixels: 732727.
2025-06-24 22:18:30 - zarr_info - INFO - ./ds_processing/gix/ds_can_gix.zarr zarray shape, chunks, dtype: (732727, 2), (200000, 1), int32
2025-06-24 22:18:30 - pc_logic_ras - INFO - writing gix.
2025-06-24 22:18:30 - pc_logic_ras - INFO - write done.
2025-06-24 22:18:30 - log_args - INFO - running function: gix2bool
2025-06-24 22:18:30 - log_args - INFO - fetching args:
2025-06-24 22:18:30 - log_args - INFO - gix = './ds_processing/gix/ds_can_gix.zarr'
2025-06-24 22:18:30 - log_args - INFO - is_pc = './ds_processing/gix/ras_is_ds_can.zarr'
2025-06-24 22:18:30 - log_args - INFO - shape = (2500, 1834)
2025-06-24 22:18:30 - log_args - INFO - chunks = (1000, 1000)
2025-06-24 22:18:30 - log_args - INFO - fetching args done.
2025-06-24 22:18:30 - zarr_info - INFO - gix zarray shape, chunks, dtype: (732727, 2), (200000, 1), int32
2025-06-24 22:18:30 - gix2bool - INFO - loading gix into memory.
2025-06-24 22:18:30 - gix2bool - INFO - calculate the bool array
2025-06-24 22:18:30 - zarr_info - INFO - is_pc zarray shape, chunks, dtype: (2500, 1834), (1000, 1000), bool
2025-06-24 22:18:30 - gix2bool - INFO - write the bool array.
2025-06-24 22:18:30 - gix2bool - INFO - write done.

ds_can_idx is a int array with shape of (num_of_DS_can, 2). It indicate the position of DS candidate in radar coordinate.

is_ds_can_zarr = zarr.open(is_ds_can,'r')
is_ds_can_data = is_ds_can_zarr[:]
is_ds_can_plot = mr.ras_plot(is_ds_can_data)

is_ds_can_plot = is_ds_can_plot.redim(x=hv.Dimension('r', label='Range'), y=hv.Dimension('az',label='Azimuth'), z=hv.Dimension('is_ds_can',))
is_ds_can_plot.opts(opts.Image(cmap='viridis',width=600, height=600, colorbar=True,
                          invert_yaxis=True,
                          default_tools=['pan',WheelZoomTool(zoom_on_axis=False),'save','reset','hover'],active_tools=['wheel_zoom']))

Coherence matrix estimation

The coherence matrix estimation is kind of complicated as it involves both raster and point cloud data. We also want to save the output point cloud data in the hilbert order.

First, we generate DS candidate point cloud is_shp by ras chunks (which is a directory):

ds_can_is_shp_dir = './ds_processing/gix/ds_can_is_shp'
ds_can_key = './ds_processing/gix/ds_can_key.zarr'

mc.ras2pc_ras_chunk(ds_can_gix,is_shp,ds_can_is_shp_dir,ds_can_key,chunks=(1000,1000))

2025-06-24 22:18:30 - log_args - INFO - running function: ras2pc_ras_chunk
2025-06-24 22:18:30 - log_args - INFO - fetching args:
2025-06-24 22:18:30 - log_args - INFO - gix = './ds_processing/gix/ds_can_gix.zarr'
2025-06-24 22:18:30 - log_args - INFO - ras = './ds_processing/gix/ras_is_shp.zarr'
2025-06-24 22:18:30 - log_args - INFO - pc = './ds_processing/gix/ds_can_is_shp'
2025-06-24 22:18:30 - log_args - INFO - key = './ds_processing/gix/ds_can_key.zarr'
2025-06-24 22:18:30 - log_args - INFO - chunks = (1000, 1000)
2025-06-24 22:18:30 - log_args - INFO - processes = False
2025-06-24 22:18:30 - log_args - INFO - n_workers = 1
2025-06-24 22:18:30 - log_args - INFO - threads_per_worker = 1
2025-06-24 22:18:30 - log_args - INFO - dask_cluster_arg = {}
2025-06-24 22:18:30 - log_args - INFO - fetching args done.
2025-06-24 22:18:30 - zarr_info - INFO - ./ds_processing/gix/ds_can_gix.zarr zarray shape, chunks, dtype: (732727, 2), (200000, 1), int32
2025-06-24 22:18:30 - ras2pc_ras_chunk - INFO - loading gix into memory.
2025-06-24 22:18:30 - ras2pc_ras_chunk - INFO - convert gix to the order of ras chunk
2025-06-24 22:18:36 - ras2pc_ras_chunk - INFO - save key
2025-06-24 22:18:37 - ras2pc_ras_chunk - INFO - starting dask local cluster.
2025-06-24 22:18:37 - ras2pc_ras_chunk - INFO - dask local cluster started.
2025-06-24 22:18:37 - dask_cluster_info - INFO - dask cluster: LocalCluster(dashboard_link='http://10.211.48.18:8787/status', workers=1, threads=1, memory=1.46 TiB)
2025-06-24 22:18:37 - ras2pc_ras_chunk - INFO - start to slice on ./ds_processing/gix/ras_is_shp.zarr
2025-06-24 22:18:37 - zarr_info - INFO - ./ds_processing/gix/ras_is_shp.zarr zarray shape, chunks, dtype: (2500, 1834, 11, 11), (1000, 1000, 1, 1), bool
2025-06-24 22:18:37 - darr_info - INFO - ras dask array shape, chunksize, dtype: (2500, 1834, 11, 11), (1000, 1000, 11, 11), bool
2025-06-24 22:18:37 - darr_info - INFO - pc dask array shape, chunksize, dtype: (732727, 11, 11), (201403, 11, 11), bool
2025-06-24 22:18:37 - ras2pc_ras_chunk - INFO - saving to ds_processing/gix/ds_can_is_shp.
2025-06-24 22:18:37 - ras2pc_ras_chunk - INFO - computing graph setted. doing all the computing.
2025-06-24 22:18:41 - ras2pc_ras_chunk - INFO - computing finished.
2025-06-24 22:18:41 - ras2pc_ras_chunk - INFO - dask cluster closed.

Then we estimate the coherence matrix:

ds_can_coh_dir = './ds_processing/gix/ds_can_coh'

mc.emperical_co_pc(rslc,ds_can_is_shp_dir,ds_can_gix,ds_can_coh_dir,cuda=True)

2025-06-24 22:18:41 - log_args - INFO - running function: emperical_co_pc
2025-06-24 22:18:41 - log_args - INFO - fetching args:
2025-06-24 22:18:41 - log_args - INFO - rslc = './load_data/rslc.zarr'
2025-06-24 22:18:41 - log_args - INFO - is_shp_dir = './ds_processing/gix/ds_can_is_shp'
2025-06-24 22:18:41 - log_args - INFO - gix = './ds_processing/gix/ds_can_gix.zarr'
2025-06-24 22:18:41 - log_args - INFO - coh_dir = './ds_processing/gix/ds_can_coh'
2025-06-24 22:18:41 - log_args - INFO - image_pairs = None
2025-06-24 22:18:41 - log_args - INFO - chunks = None
2025-06-24 22:18:41 - log_args - INFO - cuda = True
2025-06-24 22:18:41 - log_args - INFO - processes = None
2025-06-24 22:18:41 - log_args - INFO - n_workers = None
2025-06-24 22:18:41 - log_args - INFO - threads_per_worker = None
2025-06-24 22:18:41 - log_args - INFO - rmm_pool_size = 0.9
2025-06-24 22:18:41 - log_args - INFO - dask_cluster_arg = {}
2025-06-24 22:18:41 - log_args - INFO - fetching args done.
2025-06-24 22:18:41 - zarr_info - INFO - ./load_data/rslc.zarr zarray shape, chunks, dtype: (2500, 1834, 17), (1000, 1000, 1), complex64
2025-06-24 22:18:41 - emperical_co_pc - INFO - azimuth window size and half azimuth window size: 11, 5
2025-06-24 22:18:41 - emperical_co_pc - INFO - range window size and half range window size: 11, 5
2025-06-24 22:18:41 - emperical_co_pc - INFO - parallel processing azimuth chunk size: 1000
2025-06-24 22:18:41 - emperical_co_pc - INFO - parallel processing range chunk size: 1000
2025-06-24 22:18:41 - zarr_info - INFO - ./ds_processing/gix/ds_can_gix.zarr zarray shape, chunks, dtype: (732727, 2), (200000, 1), int32
2025-06-24 22:18:41 - emperical_co_pc - INFO - loading gix into memory.
2025-06-24 22:18:41 - emperical_co_pc - INFO - convert gix to the order of ras chunk
2025-06-24 22:18:42 - emperical_co_pc - INFO - starting dask cluster.
2025-06-24 22:18:49 - emperical_co_pc - INFO - dask cluster started.
2025-06-24 22:18:49 - dask_cluster_info - INFO - dask cluster: LocalCUDACluster(dashboard_link='http://127.0.0.1:8787/status', workers=8, threads=8, memory=1.46 TiB)
2025-06-24 22:18:49 - darr_info - INFO - rslc_overlap dask array shape, chunksize, dtype: (2520, 1844, 17), (1010, 1005, 17), complex64
2025-06-24 22:18:49 - darr_info - INFO - gix in ras chunk order dask array shape, chunksize, dtype: (732727, 2), (201403, 2), int32
2025-06-24 22:18:49 - emperical_co_pc - INFO - estimating coherence matrix chunk by chunk.
2025-06-24 22:18:49 - darr_info - INFO - is_shp for chunk 0 dask array shape, chunksize, dtype: (201097, 11, 11), (201097, 11, 11), bool
2025-06-24 22:18:49 - darr_info - INFO - coh for chunk 0 dask array shape, chunksize, dtype: (201097, 136), (201097, 136), complex64
2025-06-24 22:18:49 - emperical_co_pc - INFO - saving coh for chunk 0
2025-06-24 22:18:49 - zarr_info - INFO - ds_processing/gix/ds_can_coh/0.zarr zarray shape, chunks, dtype: (201097, 136), (201097, 1), complex64
2025-06-24 22:18:49 - darr_info - INFO - is_shp for chunk 1 dask array shape, chunksize, dtype: (137562, 11, 11), (137562, 11, 11), bool
2025-06-24 22:18:49 - darr_info - INFO - coh for chunk 1 dask array shape, chunksize, dtype: (137562, 136), (137562, 136), complex64
2025-06-24 22:18:49 - emperical_co_pc - INFO - saving coh for chunk 1
2025-06-24 22:18:49 - zarr_info - INFO - ds_processing/gix/ds_can_coh/1.zarr zarray shape, chunks, dtype: (137562, 136), (137562, 1), complex64
2025-06-24 22:18:49 - darr_info - INFO - is_shp for chunk 2 dask array shape, chunksize, dtype: (201403, 11, 11), (201403, 11, 11), bool
2025-06-24 22:18:49 - darr_info - INFO - coh for chunk 2 dask array shape, chunksize, dtype: (201403, 136), (201403, 136), complex64
2025-06-24 22:18:49 - emperical_co_pc - INFO - saving coh for chunk 2
2025-06-24 22:18:49 - zarr_info - INFO - ds_processing/gix/ds_can_coh/2.zarr zarray shape, chunks, dtype: (201403, 136), (201403, 1), complex64
2025-06-24 22:18:49 - darr_info - INFO - is_shp for chunk 3 dask array shape, chunksize, dtype: (73518, 11, 11), (73518, 11, 11), bool
2025-06-24 22:18:49 - darr_info - INFO - coh for chunk 3 dask array shape, chunksize, dtype: (73518, 136), (73518, 136), complex64
2025-06-24 22:18:49 - emperical_co_pc - INFO - saving coh for chunk 3
2025-06-24 22:18:49 - zarr_info - INFO - ds_processing/gix/ds_can_coh/3.zarr zarray shape, chunks, dtype: (73518, 136), (73518, 1), complex64
2025-06-24 22:18:49 - darr_info - INFO - is_shp for chunk 4 dask array shape, chunksize, dtype: (78445, 11, 11), (78445, 11, 11), bool
2025-06-24 22:18:49 - darr_info - INFO - coh for chunk 4 dask array shape, chunksize, dtype: (78445, 136), (78445, 136), complex64
2025-06-24 22:18:49 - emperical_co_pc - INFO - saving coh for chunk 4
2025-06-24 22:18:49 - zarr_info - INFO - ds_processing/gix/ds_can_coh/4.zarr zarray shape, chunks, dtype: (78445, 136), (78445, 1), complex64
2025-06-24 22:18:49 - darr_info - INFO - is_shp for chunk 5 dask array shape, chunksize, dtype: (40702, 11, 11), (40702, 11, 11), bool
2025-06-24 22:18:49 - darr_info - INFO - coh for chunk 5 dask array shape, chunksize, dtype: (40702, 136), (40702, 136), complex64
2025-06-24 22:18:49 - emperical_co_pc - INFO - saving coh for chunk 5
2025-06-24 22:18:49 - zarr_info - INFO - ds_processing/gix/ds_can_coh/5.zarr zarray shape, chunks, dtype: (40702, 136), (40702, 1), complex64
2025-06-24 22:18:49 - emperical_co_pc - INFO - computing graph setted. doing all the computing.
2025-06-24 22:18:52 - emperical_co_pc - INFO - computing finished.s
2025-06-24 22:18:53 - emperical_co_pc - INFO - dask cluster closed.

The output of emperical_co_pc is point cloud data in the raster chunk order. So we prepare the key to sort the point cloud data to hilbert order (the hillbert index of DS candidate can be generated by the way):

ds_can_hix_unsorted = './ds_processing/gix/ds_can_hix.zarr'
ds_can_hix = './ds_processing/hix/ds_can/ds_can_hix.zarr'
gix2hix_key = './ds_processing/gix/gix2hix.zarr'

mc.pc_hix(ds_can_gix, ds_can_hix_unsorted, shape=zarr.open(rslc,'r').shape[:2])
mc.pc_sort(
    ds_can_hix_unsorted, ds_can_hix,
    shape=zarr.open(rslc,'r').shape[:2],
    pc_in = './ds_processing/gix/ds_can_gix.zarr',
    pc = './ds_processing/hix/ds_can/ds_can_gix.zarr',
    key=gix2hix_key,
)

2025-06-24 22:18:53 - log_args - INFO - running function: pc_hix
2025-06-24 22:18:53 - log_args - INFO - fetching args:
2025-06-24 22:18:53 - log_args - INFO - gix = './ds_processing/gix/ds_can_gix.zarr'
2025-06-24 22:18:53 - log_args - INFO - hix = './ds_processing/gix/ds_can_hix.zarr'
2025-06-24 22:18:53 - log_args - INFO - shape = (2500, 1834)
2025-06-24 22:18:53 - log_args - INFO - fetching args done.
2025-06-24 22:18:53 - zarr_info - INFO - ./ds_processing/gix/ds_can_gix.zarr zarray shape, chunks, dtype: (732727, 2), (200000, 1), int32
2025-06-24 22:18:53 - zarr_info - INFO - ./ds_processing/gix/ds_can_hix.zarr zarray shape, chunks, dtype: (732727,), (200000,), int64
2025-06-24 22:18:53 - pc_hix - INFO - calculating the hillbert index based on grid index
2025-06-24 22:18:55 - pc_hix - INFO - writing the hillbert index
2025-06-24 22:18:55 - pc_hix - INFO - done.
2025-06-24 22:18:55 - log_args - INFO - running function: pc_sort
2025-06-24 22:18:55 - log_args - INFO - fetching args:
2025-06-24 22:18:55 - log_args - INFO - idx_in = './ds_processing/gix/ds_can_hix.zarr'
2025-06-24 22:18:55 - log_args - INFO - idx = './ds_processing/hix/ds_can/ds_can_hix.zarr'
2025-06-24 22:18:55 - log_args - INFO - pc_in = './ds_processing/gix/ds_can_gix.zarr'
2025-06-24 22:18:55 - log_args - INFO - pc = './ds_processing/hix/ds_can/ds_can_gix.zarr'
2025-06-24 22:18:55 - log_args - INFO - shape = (2500, 1834)
2025-06-24 22:18:55 - log_args - INFO - chunks = None
2025-06-24 22:18:55 - log_args - INFO - key = './ds_processing/gix/gix2hix.zarr'
2025-06-24 22:18:55 - log_args - INFO - processes = False
2025-06-24 22:18:55 - log_args - INFO - n_workers = 1
2025-06-24 22:18:55 - log_args - INFO - threads_per_worker = 1
2025-06-24 22:18:55 - log_args - INFO - dask_cluster_arg = {}
2025-06-24 22:18:55 - log_args - INFO - fetching args done.
2025-06-24 22:18:55 - zarr_info - INFO - ./ds_processing/gix/ds_can_hix.zarr zarray shape, chunks, dtype: (732727,), (200000,), int64
2025-06-24 22:18:55 - pc_sort - INFO - loading idx_in and calculate the sorting indices.
2025-06-24 22:18:55 - pc_sort - INFO - output pc chunk size is 200000
2025-06-24 22:18:55 - pc_sort - INFO - write idx
2025-06-24 22:18:55 - zarr_info - INFO - idx zarray shape, chunks, dtype: (732727,), (200000,), int64
2025-06-24 22:18:55 - pc_sort - INFO - saving key for this sorting
2025-06-24 22:18:55 - pc_sort - INFO - starting dask local cluster.
2025-06-24 22:18:55 - pc_sort - INFO - dask local cluster started.
2025-06-24 22:18:55 - dask_cluster_info - INFO - dask cluster: LocalCluster(dashboard_link='http://10.211.48.18:8787/status', workers=1, threads=1, memory=1.46 TiB)
2025-06-24 22:18:55 - zarr_info - INFO - ./ds_processing/gix/ds_can_gix.zarr zarray shape, chunks, dtype: (732727, 2), (200000, 1), int32
2025-06-24 22:18:55 - darr_info - INFO - pc_in dask array shape, chunksize, dtype: (732727, 2), (732727, 1), int32
2025-06-24 22:18:55 - pc_sort - INFO - set up sorted pc data dask array.
2025-06-24 22:18:55 - darr_info - INFO - pc dask array shape, chunksize, dtype: (732727, 2), (732727, 1), int32
2025-06-24 22:18:55 - pc_sort - INFO - write pc to ./ds_processing/hix/ds_can/ds_can_gix.zarr
2025-06-24 22:18:55 - zarr_info - INFO - ./ds_processing/hix/ds_can/ds_can_gix.zarr zarray shape, chunks, dtype: (732727, 2), (200000, 1), int32
2025-06-24 22:18:55 - pc_sort - INFO - computing graph setted. doing all the computing.
2025-06-24 22:18:55 - pc_sort - INFO - computing finished.d |  0.1s
2025-06-24 22:18:55 - pc_sort - INFO - dask cluster closed.

Then concat and sort ds_can_coh to hilbert order:

ds_can_coh = './ds_processing/hix/ds_can/ds_can_coh.zarr'
ds_can_coh_ave = './ds_processing/hix/ds_can/ds_can_coh_ave.zarr'

chunks = zarr.open(ds_can_gix,'r').chunks[0]
mc.pc_concat(ds_can_coh_dir,ds_can_coh,key=[ds_can_key,gix2hix_key],chunks=chunks)

2025-06-24 22:18:55 - log_args - INFO - running function: pc_concat
2025-06-24 22:18:55 - log_args - INFO - fetching args:
2025-06-24 22:18:55 - log_args - INFO - pcs = './ds_processing/gix/ds_can_coh'
2025-06-24 22:18:55 - log_args - INFO - pc = './ds_processing/hix/ds_can/ds_can_coh.zarr'
2025-06-24 22:18:55 - log_args - INFO - key = ['./ds_processing/gix/ds_can_key.zarr', './ds_processing/gix/gix2hix.zarr']
2025-06-24 22:18:55 - log_args - INFO - chunks = 200000
2025-06-24 22:18:55 - log_args - INFO - processes = False
2025-06-24 22:18:55 - log_args - INFO - n_workers = 1
2025-06-24 22:18:55 - log_args - INFO - threads_per_worker = 1
2025-06-24 22:18:55 - log_args - INFO - dask_cluster_arg = {}
2025-06-24 22:18:55 - log_args - INFO - fetching args done.
2025-06-24 22:18:55 - pc_concat - INFO - input pcs: [[PosixPath('ds_processing/gix/ds_can_coh/0.zarr'), PosixPath('ds_processing/gix/ds_can_coh/1.zarr'), PosixPath('ds_processing/gix/ds_can_coh/2.zarr'), PosixPath('ds_processing/gix/ds_can_coh/3.zarr'), PosixPath('ds_processing/gix/ds_can_coh/4.zarr'), PosixPath('ds_processing/gix/ds_can_coh/5.zarr')]]
2025-06-24 22:18:55 - pc_concat - INFO - output pc: ['./ds_processing/hix/ds_can/ds_can_coh.zarr']
2025-06-24 22:18:55 - pc_concat - INFO - load key
2025-06-24 22:18:55 - zarr_info - INFO - ./ds_processing/gix/ds_can_key.zarr zarray shape, chunks, dtype: (732727,), (200000,), int64
2025-06-24 22:18:55 - zarr_info - INFO - ./ds_processing/gix/gix2hix.zarr zarray shape, chunks, dtype: (732727,), (200000,), int64
2025-06-24 22:18:55 - pc_concat - INFO - starting dask local cluster.
2025-06-24 22:18:55 - pc_concat - INFO - dask local cluster started.
2025-06-24 22:18:55 - dask_cluster_info - INFO - dask cluster: LocalCluster(dashboard_link='http://10.211.48.18:8787/status', workers=1, threads=1, memory=1.46 TiB)
2025-06-24 22:18:55 - pc_concat - INFO - read pc from [PosixPath('ds_processing/gix/ds_can_coh/0.zarr'), PosixPath('ds_processing/gix/ds_can_coh/1.zarr'), PosixPath('ds_processing/gix/ds_can_coh/2.zarr'), PosixPath('ds_processing/gix/ds_can_coh/3.zarr'), PosixPath('ds_processing/gix/ds_can_coh/4.zarr'), PosixPath('ds_processing/gix/ds_can_coh/5.zarr')]
2025-06-24 22:18:55 - darr_info - INFO - concatenated pc dask array shape, chunksize, dtype: (732727, 136), (732727, 1), complex64
2025-06-24 22:18:55 - pc_concat - INFO - sort pc according to key
2025-06-24 22:18:55 - darr_info - INFO - sorted pc dask array shape, chunksize, dtype: (732727, 136), (732727, 1), complex64
2025-06-24 22:18:55 - pc_concat - INFO - save pc to ./ds_processing/hix/ds_can/ds_can_coh.zarr
2025-06-24 22:18:55 - zarr_info - INFO - ./ds_processing/hix/ds_can/ds_can_coh.zarr zarray shape, chunks, dtype: (732727, 136), (200000, 1), complex64
2025-06-24 22:18:55 - pc_concat - INFO - computing graph setted. doing all the computing.
2025-06-24 22:19:00 - pc_concat - INFO - computing finished.|  4.2s
2025-06-24 22:19:00 - pc_concat - INFO - dask cluster closed.

n_points = zarr.open(ds_can_coh,'r').shape[0]
mc.data_reduce(ds_can_coh,ds_can_coh_ave,map_func=np.abs,reduce_func=np.sum,post_map_func=lambda x: x/n_points)

2025-06-24 22:19:00 - log_args - INFO - running function: data_reduce
2025-06-24 22:19:00 - log_args - INFO - fetching args:
2025-06-24 22:19:00 - log_args - INFO - data_in = './ds_processing/hix/ds_can/ds_can_coh.zarr'
2025-06-24 22:19:00 - log_args - INFO - out = './ds_processing/hix/ds_can/ds_can_coh_ave.zarr'
2025-06-24 22:19:00 - log_args - INFO - map_func = <ufunc 'absolute'>
2025-06-24 22:19:00 - log_args - INFO - reduce_func = <function sum>
2025-06-24 22:19:00 - log_args - INFO - axis = 0
2025-06-24 22:19:00 - log_args - INFO - post_map_func = <function <lambda>>
2025-06-24 22:19:00 - log_args - INFO - processes = False
2025-06-24 22:19:00 - log_args - INFO - n_workers = 1
2025-06-24 22:19:00 - log_args - INFO - threads_per_worker = 1
2025-06-24 22:19:00 - log_args - INFO - dask_cluster_arg = {}
2025-06-24 22:19:00 - log_args - INFO - fetching args done.
2025-06-24 22:19:00 - zarr_info - INFO - ./ds_processing/hix/ds_can/ds_can_coh.zarr zarray shape, chunks, dtype: (732727, 136), (200000, 1), complex64
2025-06-24 22:19:00 - data_reduce - INFO - starting dask local cluster.
2025-06-24 22:19:00 - data_reduce - INFO - dask local cluster started.
2025-06-24 22:19:00 - dask_cluster_info - INFO - dask cluster: LocalCluster(dashboard_link='http://10.211.48.18:8787/status', workers=1, threads=1, memory=1.46 TiB)
2025-06-24 22:19:00 - darr_info - INFO - data_in dask array shape, chunksize, dtype: (732727, 136), (200000, 1), complex64
2025-06-24 22:19:00 - darr_info - INFO - maped_data_in dask array shape, chunksize, dtype: (732727, 136), (200000, 1), float32
2025-06-24 22:19:00 - darr_info - INFO - reduced data in every chunk dask array shape, chunksize, dtype: (4, 136), (1, 1), float32
2025-06-24 22:19:00 - data_reduce - INFO - computing graph setted. doing all the computing.
2025-06-24 22:19:02 - data_reduce - INFO - computing finished. 2.0s
2025-06-24 22:19:02 - data_reduce - INFO - dask cluster closed.
2025-06-24 22:19:02 - data_reduce - INFO - continue the reduction on reduced data over every chunk
2025-06-24 22:19:02 - data_reduce - INFO - post mapping
2025-06-24 22:19:02 - data_reduce - INFO - writing output.
2025-06-24 22:19:02 - data_reduce - INFO - done.

ds_can_coh_ave_zarr = zarr.open(ds_can_coh_ave,'r')
ds_can_coh_ave_plot = mr.plot.ras_plot(mr.uncompress_coh(ds_can_coh_ave_zarr[:]))
ds_can_coh_ave_plot = ds_can_coh_ave_plot.redim(x=hv.Dimension('sec_image', label='Secondary image'), y=hv.Dimension('ref_image',label='Reference image'), z=hv.Dimension('ds_can_coh_ave',))
ds_can_coh_ave_plot.opts(opts.Image(cmap='viridis',frame_width=600, frame_height=600, colorbar=True,
                          invert_yaxis=True,
                          default_tools=['pan',WheelZoomTool(zoom_on_axis=False),'save','reset','hover'],active_tools=['wheel_zoom']))

Visualize adpative multi-looked interferograms

Visualize interferograms after adaptive multi-looking and compare with the raw 1 by 1 look interferograms:

mc.pc_pyramid(
    './ds_processing/hix/ds_can/ds_can_coh.zarr',
    './ds_processing/hix/ds_can/ds_can_coh_pyramid',
    yx = './ds_processing/hix/ds_can/ds_can_gix.zarr',
    ras_resolution=2,
)

2025-06-24 22:19:03 - log_args - INFO - running function: pc_pyramid
2025-06-24 22:19:03 - log_args - INFO - fetching args:
2025-06-24 22:19:03 - log_args - INFO - pc = './ds_processing/hix/ds_can/ds_can_coh.zarr'
2025-06-24 22:19:03 - log_args - INFO - out_dir = './ds_processing/hix/ds_can/ds_can_coh_pyramid'
2025-06-24 22:19:03 - log_args - INFO - x = None
2025-06-24 22:19:03 - log_args - INFO - y = None
2025-06-24 22:19:03 - log_args - INFO - yx = './ds_processing/hix/ds_can/ds_can_gix.zarr'
2025-06-24 22:19:03 - log_args - INFO - ras_resolution = 2
2025-06-24 22:19:03 - log_args - INFO - ras_chunks = (256, 256)
2025-06-24 22:19:03 - log_args - INFO - pc_chunks = 65536
2025-06-24 22:19:03 - log_args - INFO - processes = False
2025-06-24 22:19:03 - log_args - INFO - n_workers = 1
2025-06-24 22:19:03 - log_args - INFO - threads_per_worker = 2
2025-06-24 22:19:03 - log_args - INFO - dask_cluster_arg = {}
2025-06-24 22:19:03 - log_args - INFO - fetching args done.
2025-06-24 22:19:03 - pc_pyramid - INFO - clean out dir
2025-06-24 22:19:04 - zarr_info - INFO - ./ds_processing/hix/ds_can/ds_can_coh.zarr zarray shape, chunks, dtype: (732727, 136), (200000, 1), complex64
2025-06-24 22:19:04 - pc_pyramid - INFO - rendering point cloud data coordinates:
2025-06-24 22:19:04 - pc_pyramid - INFO - rasterizing point cloud data to grid with bounds: [0, 0, 1832, 2498].
2025-06-24 22:19:04 - zarr_info - INFO - ds_processing/hix/ds_can/ds_can_coh_pyramid/x.zarr zarray shape, chunks, dtype: (732727,), (65536,), int32
2025-06-24 22:19:04 - zarr_info - INFO - ds_processing/hix/ds_can/ds_can_coh_pyramid/y.zarr zarray shape, chunks, dtype: (732727,), (65536,), int32
2025-06-24 22:19:04 - pc_pyramid - INFO - pc data coordinates rendering ends.
2025-06-24 22:19:04 - zarr_info - INFO - ds_processing/hix/ds_can/ds_can_coh_pyramid/idx_0.zarr zarray shape, chunks, dtype: (1250, 917), (256, 256), int64
2025-06-24 22:19:06 - zarr_info - INFO - ds_processing/hix/ds_can/ds_can_coh_pyramid/idx_1.zarr zarray shape, chunks, dtype: (625, 459), (256, 256), int64
2025-06-24 22:19:06 - zarr_info - INFO - ds_processing/hix/ds_can/ds_can_coh_pyramid/idx_2.zarr zarray shape, chunks, dtype: (313, 230), (256, 256), int64
2025-06-24 22:19:06 - zarr_info - INFO - ds_processing/hix/ds_can/ds_can_coh_pyramid/idx_3.zarr zarray shape, chunks, dtype: (157, 115), (256, 256), int64
2025-06-24 22:19:06 - zarr_info - INFO - ds_processing/hix/ds_can/ds_can_coh_pyramid/idx_4.zarr zarray shape, chunks, dtype: (79, 58), (256, 256), int64
2025-06-24 22:19:06 - zarr_info - INFO - ds_processing/hix/ds_can/ds_can_coh_pyramid/idx_5.zarr zarray shape, chunks, dtype: (40, 29), (256, 256), int64
2025-06-24 22:19:06 - zarr_info - INFO - ds_processing/hix/ds_can/ds_can_coh_pyramid/idx_6.zarr zarray shape, chunks, dtype: (20, 15), (256, 256), int64
2025-06-24 22:19:06 - zarr_info - INFO - ds_processing/hix/ds_can/ds_can_coh_pyramid/idx_7.zarr zarray shape, chunks, dtype: (10, 8), (256, 256), int64
2025-06-24 22:19:06 - zarr_info - INFO - ds_processing/hix/ds_can/ds_can_coh_pyramid/idx_8.zarr zarray shape, chunks, dtype: (5, 4), (256, 256), int64
2025-06-24 22:19:06 - zarr_info - INFO - ds_processing/hix/ds_can/ds_can_coh_pyramid/idx_9.zarr zarray shape, chunks, dtype: (3, 2), (256, 256), int64
2025-06-24 22:19:06 - pc_pyramid - INFO - rasterized idx rendering ends
2025-06-24 22:19:06 - pc_pyramid - INFO - dask local cluster started to render pc data.
2025-06-24 22:19:06 - dask_cluster_info - INFO - dask cluster: LocalCluster(dashboard_link='http://10.211.48.18:8787/status', workers=1, threads=2, memory=1.46 TiB)
2025-06-24 22:19:06 - zarr_info - INFO - ds_processing/hix/ds_can/ds_can_coh_pyramid/pc.zarr zarray shape, chunks, dtype: (732727, 136), (65536, 1), complex64
2025-06-24 22:19:06 - zarr_info - INFO - ds_processing/hix/ds_can/ds_can_coh_pyramid/0.zarr zarray shape, chunks, dtype: (1250, 917, 136), (256, 256, 1), complex64
2025-06-24 22:19:06 - zarr_info - INFO - ds_processing/hix/ds_can/ds_can_coh_pyramid/1.zarr zarray shape, chunks, dtype: (625, 459, 136), (256, 256, 1), complex64
2025-06-24 22:19:06 - zarr_info - INFO - ds_processing/hix/ds_can/ds_can_coh_pyramid/2.zarr zarray shape, chunks, dtype: (313, 230, 136), (256, 256, 1), complex64
2025-06-24 22:19:06 - zarr_info - INFO - ds_processing/hix/ds_can/ds_can_coh_pyramid/3.zarr zarray shape, chunks, dtype: (157, 115, 136), (256, 256, 1), complex64
2025-06-24 22:19:06 - zarr_info - INFO - ds_processing/hix/ds_can/ds_can_coh_pyramid/4.zarr zarray shape, chunks, dtype: (79, 58, 136), (256, 256, 1), complex64
2025-06-24 22:19:06 - zarr_info - INFO - ds_processing/hix/ds_can/ds_can_coh_pyramid/5.zarr zarray shape, chunks, dtype: (40, 29, 136), (256, 256, 1), complex64
2025-06-24 22:19:06 - zarr_info - INFO - ds_processing/hix/ds_can/ds_can_coh_pyramid/6.zarr zarray shape, chunks, dtype: (20, 15, 136), (256, 256, 1), complex64
2025-06-24 22:19:06 - zarr_info - INFO - ds_processing/hix/ds_can/ds_can_coh_pyramid/7.zarr zarray shape, chunks, dtype: (10, 8, 136), (256, 256, 1), complex64
2025-06-24 22:19:06 - zarr_info - INFO - ds_processing/hix/ds_can/ds_can_coh_pyramid/8.zarr zarray shape, chunks, dtype: (5, 4, 136), (256, 256, 1), complex64
2025-06-24 22:19:06 - zarr_info - INFO - ds_processing/hix/ds_can/ds_can_coh_pyramid/9.zarr zarray shape, chunks, dtype: (3, 2, 136), (256, 256, 1), complex64
2025-06-24 22:19:07 - pc_pyramid - INFO - computing graph setted. doing all the computing.
2025-06-24 22:19:16 - pc_pyramid - INFO - computing finished.  9.1s
2025-06-24 22:19:16 - pc_pyramid - INFO - dask cluster closed.

rslc_zarr = zarr.open(rslc,'r')
tnet = mr.TempNet.from_bandwidth(rslc_zarr.shape[2])

def co_phase_post_proc_ras(data_zarr,xslice,yslice,i,j):
    if i == j:
        nx = xslice.stop-xslice.start
        ny = yslice.stop-yslice.start
        mask = np.isnan(data_zarr[yslice,xslice,0])
        data = np.zeros((ny,nx),dtype=np.float32)
        data[mask] = np.nan
        return data
    elif i > j:
        ref, sec = j, i
        conj = True
    else:
        ref, sec = i, j
        conj = False
    image_pair_idx = tnet.image_pairs_idx(ref=ref,sec=sec)
    if image_pair_idx == -1:
        nx = xslice.stop-xslice.start
        ny = yslice.stop-yslice.start
        return np.full((ny,nx),fill_value=np.nan,dtype=np.float32)
    data = data_zarr[yslice,xslice,image_pair_idx]
    if conj:
        return -np.angle(data)
    else:
        return np.angle(data)

def co_phase_post_proc_pc(data_zarr,idx_array,i,j):
    if i == j:
        data = np.zeros_like(idx_array,dtype=np.float32)
        return data
    elif i > j:
        ref, sec = j, i
        conj = True
    else:
        ref, sec = i, j
        conj = False
    image_pair_idx = tnet.image_pairs_idx(ref=ref,sec=sec)
    if image_pair_idx == -1:
        return np.full_like(idx_array,fill_value=np.nan,dtype=np.float32)
    data = data_zarr[idx_array,image_pair_idx]
    if conj:
        return -np.angle(data)
    else:
        return np.angle(data)

ds_can_intf_plot = mc.pc_plot(
    './ds_processing/hix/ds_can/ds_can_coh_pyramid',
    post_proc_ras=co_phase_post_proc_ras,
    post_proc_pc=co_phase_post_proc_pc,
    n_kdim=2,level_increase=1)
raw_intf_plot = mc.ras_plot('./load_data/rslc_pyramid',post_proc='intf_all',n_kdim=2,level_increase=1)
intf_plots = raw_intf_plot.relabel('Raw Interferograms') + ds_can_intf_plot.relabel('Adaptively multilooked Interferograms')

import holoviews as hv
from bokeh.models import WheelZoomTool
hv.extension('bokeh')
import toml
with open('load_data/meta.toml','r') as f:
    dates = toml.load(f)['dates']
intf_plots = intf_plots.redim(
    i=hv.Dimension('i', label='Reference Image', range=(0,16), value_format=(lambda i: dates[i])),
    j=hv.Dimension('j', label='Secondary Image', range=(0,16), value_format=(lambda i: dates[i])),
    x=hv.Dimension('r', label='Range'),
    y=hv.Dimension('az',label='Azimuth'),
    z=hv.Dimension('Phase',range=(-np.pi,np.pi))
)

hv.output(widget_location='bottom')
intf_plots.opts(
    hv.opts.Image(
        cmap='colorwheel',frame_width=500, frame_height=600, colorbar=True,
        default_tools=['pan',WheelZoomTool(zoom_on_axis=False),'save','reset','hover'],
        active_tools=['wheel_zoom'],
        invert_yaxis=True,
    ),
    hv.opts.Points(
        color='Phase', cmap='colorwheel',frame_width=500, frame_height=600, colorbar=True,
        default_tools=['pan',WheelZoomTool(zoom_on_axis=False),'save','reset','hover'],
        active_tools=['wheel_zoom'],
        invert_yaxis=True,
    ),
)

Phase linking

To get rid of the closure phase, phase linking is applied here:

ds_can_coh = './ds_processing/hix/ds_can/ds_can_coh.zarr'
ds_can_ph = './ds_processing/hix/ds_can/ds_can_ph.zarr'
ds_can_emi_quality = './ds_processing/hix/ds_can/ds_can_emi_quality.zarr'
emi_quality = './ds_processing/hix/ds_can/ras_ds_can_emi_quality.zarr'

mc.emi(ds_can_coh,ds_can_ph,ds_can_emi_quality)

2025-06-24 22:19:19 - log_args - INFO - running function: emi
2025-06-24 22:19:19 - log_args - INFO - fetching args:
2025-06-24 22:19:19 - log_args - INFO - coh = './ds_processing/hix/ds_can/ds_can_coh.zarr'
2025-06-24 22:19:19 - log_args - INFO - ph = './ds_processing/hix/ds_can/ds_can_ph.zarr'
2025-06-24 22:19:19 - log_args - INFO - emi_quality = './ds_processing/hix/ds_can/ds_can_emi_quality.zarr'
2025-06-24 22:19:19 - log_args - INFO - ref = 0
2025-06-24 22:19:19 - log_args - INFO - chunks = None
2025-06-24 22:19:19 - log_args - INFO - cuda = False
2025-06-24 22:19:19 - log_args - INFO - processes = None
2025-06-24 22:19:19 - log_args - INFO - n_workers = None
2025-06-24 22:19:19 - log_args - INFO - threads_per_worker = None
2025-06-24 22:19:19 - log_args - INFO - rmm_pool_size = 0.9
2025-06-24 22:19:19 - log_args - INFO - dask_cluster_arg = {}
2025-06-24 22:19:19 - log_args - INFO - fetching args done.
2025-06-24 22:19:19 - zarr_info - INFO - ./ds_processing/hix/ds_can/ds_can_coh.zarr zarray shape, chunks, dtype: (732727, 136), (200000, 1), complex64
2025-06-24 22:19:19 - emi - INFO - starting dask cluster.
2025-06-24 22:19:19 - emi - INFO - dask cluster started.
2025-06-24 22:19:19 - dask_cluster_info - INFO - dask cluster: LocalCluster(dashboard_link='http://10.211.48.18:8787/status', workers=1, threads=2, memory=1.46 TiB)
2025-06-24 22:19:19 - darr_info - INFO - coh dask array shape, chunksize, dtype: (732727, 136), (200000, 136), complex64
2025-06-24 22:19:19 - emi - INFO - phase linking with EMI.
2025-06-24 22:19:19 - emi - INFO - got ph and emi_quality.
2025-06-24 22:19:19 - darr_info - INFO - ph dask array shape, chunksize, dtype: (732727, 17), (200000, 17), complex64
2025-06-24 22:19:19 - darr_info - INFO - emi_quality dask array shape, chunksize, dtype: (732727,), (200000,), float32
2025-06-24 22:19:19 - emi - INFO - saving ph and emi_quality.
2025-06-24 22:19:19 - zarr_info - INFO - ./ds_processing/hix/ds_can/ds_can_ph.zarr zarray shape, chunks, dtype: (732727, 17), (200000, 1), complex64
2025-06-24 22:19:19 - zarr_info - INFO - ./ds_processing/hix/ds_can/ds_can_emi_quality.zarr zarray shape, chunks, dtype: (732727,), (200000,), float32
2025-06-24 22:19:19 - emi - INFO - computing graph setted. doing all the computing.
2025-06-24 22:19:25 - emi - INFO - computing finished.leted |  5.0s
2025-06-24 22:19:25 - emi - INFO - dask cluster closed.

rslc_zarr = zarr.open(rslc,'r')
mc.pc2ras(ds_can_hix, ds_can_emi_quality,emi_quality,rslc_zarr.shape[:2])

2025-06-24 22:19:25 - log_args - INFO - running function: pc2ras
2025-06-24 22:19:25 - log_args - INFO - fetching args:
2025-06-24 22:19:25 - log_args - INFO - idx = './ds_processing/hix/ds_can/ds_can_hix.zarr'
2025-06-24 22:19:25 - log_args - INFO - pc = './ds_processing/hix/ds_can/ds_can_emi_quality.zarr'
2025-06-24 22:19:25 - log_args - INFO - ras = './ds_processing/hix/ds_can/ras_ds_can_emi_quality.zarr'
2025-06-24 22:19:25 - log_args - INFO - shape = (2500, 1834)
2025-06-24 22:19:25 - log_args - INFO - chunks = (1000, 1000)
2025-06-24 22:19:25 - log_args - INFO - processes = False
2025-06-24 22:19:25 - log_args - INFO - n_workers = 1
2025-06-24 22:19:25 - log_args - INFO - threads_per_worker = 1
2025-06-24 22:19:25 - log_args - INFO - dask_cluster_arg = {}
2025-06-24 22:19:25 - log_args - INFO - fetching args done.
2025-06-24 22:19:25 - zarr_info - INFO - ./ds_processing/hix/ds_can/ds_can_hix.zarr zarray shape, chunks, dtype: (732727,), (200000,), int64
2025-06-24 22:19:25 - pc2ras - INFO - loading hix into memory and convert to gix
2025-06-24 22:19:26 - pc2ras - INFO - starting dask local cluster.
2025-06-24 22:19:26 - pc2ras - INFO - dask local cluster started.
2025-06-24 22:19:26 - dask_cluster_info - INFO - dask cluster: LocalCluster(dashboard_link='http://10.211.48.18:8787/status', workers=1, threads=1, memory=1.46 TiB)
2025-06-24 22:19:26 - pc2ras - INFO - start to work on ./ds_processing/hix/ds_can/ds_can_emi_quality.zarr
2025-06-24 22:19:26 - zarr_info - INFO - ./ds_processing/hix/ds_can/ds_can_emi_quality.zarr zarray shape, chunks, dtype: (732727,), (200000,), float32
2025-06-24 22:19:26 - darr_info - INFO - pc dask array shape, chunksize, dtype: (732727,), (732727,), float32
2025-06-24 22:19:26 - pc2ras - INFO - create ras dask array
2025-06-24 22:19:26 - darr_info - INFO - ras dask array shape, chunksize, dtype: (2500, 1834), (2500, 1834), float32
2025-06-24 22:19:26 - pc2ras - INFO - save ras to ./ds_processing/hix/ds_can/ras_ds_can_emi_quality.zarr
2025-06-24 22:19:26 - zarr_info - INFO - ./ds_processing/hix/ds_can/ras_ds_can_emi_quality.zarr zarray shape, chunks, dtype: (2500, 1834), (1000, 1000), float32
2025-06-24 22:19:26 - pc2ras - INFO - computing graph setted. doing all the computing.
2025-06-24 22:19:26 - pc2ras - INFO - computing finished.ed |  0.1s
2025-06-24 22:19:26 - pc2ras - INFO - dask cluster closed.

emi_quality_zarr = zarr.open(emi_quality,'r')
emi_quality_plot = mr.plot.ras_plot(emi_quality_zarr[:])
emi_quality_plot = emi_quality_plot.redim(x=hv.Dimension('r', label='Range'), y=hv.Dimension('az',label='Azimuth'), z=hv.Dimension('emi_quality',range=(0.5,1.3)))
emi_quality_plot.opts(opts.Image(cmap='viridis',frame_width=500, frame_height=600, colorbar=True,
                          invert_yaxis=True,
                          default_tools=['pan',WheelZoomTool(zoom_on_axis=False),'save','reset','hover'],active_tools=['wheel_zoom']))

We can also estimate a temporal coherence for DS:

ds_can_t_coh = './ds_processing/hix/ds_can/ds_can_t_coh.zarr'

mc.ds_temp_coh(ds_can_coh,ds_can_ph, ds_can_t_coh)

2025-06-24 22:19:26 - log_args - INFO - running function: ds_temp_coh
2025-06-24 22:19:26 - log_args - INFO - fetching args:
2025-06-24 22:19:26 - log_args - INFO - coh = './ds_processing/hix/ds_can/ds_can_coh.zarr'
2025-06-24 22:19:26 - log_args - INFO - ph = './ds_processing/hix/ds_can/ds_can_ph.zarr'
2025-06-24 22:19:26 - log_args - INFO - t_coh = './ds_processing/hix/ds_can/ds_can_t_coh.zarr'
2025-06-24 22:19:26 - log_args - INFO - tnet = None
2025-06-24 22:19:26 - log_args - INFO - chunks = None
2025-06-24 22:19:26 - log_args - INFO - cuda = False
2025-06-24 22:19:26 - log_args - INFO - processes = None
2025-06-24 22:19:26 - log_args - INFO - n_workers = None
2025-06-24 22:19:26 - log_args - INFO - threads_per_worker = None
2025-06-24 22:19:26 - log_args - INFO - rmm_pool_size = 0.9
2025-06-24 22:19:26 - log_args - INFO - dask_cluster_arg = {}
2025-06-24 22:19:26 - log_args - INFO - fetching args done.
2025-06-24 22:19:26 - zarr_info - INFO - ./ds_processing/hix/ds_can/ds_can_coh.zarr zarray shape, chunks, dtype: (732727, 136), (200000, 1), complex64
2025-06-24 22:19:26 - zarr_info - INFO - ./ds_processing/hix/ds_can/ds_can_ph.zarr zarray shape, chunks, dtype: (732727, 17), (200000, 1), complex64
2025-06-24 22:19:26 - ds_temp_coh - INFO - starting dask local cluster.
2025-06-24 22:19:26 - ds_temp_coh - INFO - dask local cluster started.
2025-06-24 22:19:26 - dask_cluster_info - INFO - dask cluster: LocalCluster(dashboard_link='http://10.211.48.18:8787/status', workers=1, threads=2, memory=1.46 TiB)
2025-06-24 22:19:26 - darr_info - INFO - coh dask array shape, chunksize, dtype: (732727, 136), (200000, 136), complex64
2025-06-24 22:19:26 - darr_info - INFO - ph dask array shape, chunksize, dtype: (732727, 17), (200000, 17), complex64
2025-06-24 22:19:26 - ds_temp_coh - INFO - Estimate temporal coherence for DS.
2025-06-24 22:19:26 - ds_temp_coh - INFO - got temporal coherence t_coh.
2025-06-24 22:19:26 - darr_info - INFO - t_coh dask array shape, chunksize, dtype: (732727,), (200000,), float32
2025-06-24 22:19:26 - ds_temp_coh - INFO - saving t_coh.
2025-06-24 22:19:26 - ds_temp_coh - INFO - computing graph setted. doing all the computing.
2025-06-24 22:19:27 - ds_temp_coh - INFO - computing finished. 0.8s
2025-06-24 22:19:27 - ds_temp_coh - INFO - dask cluster closed.

Plot it:

t_coh = './ds_processing/hix/ds_can/ras_ds_can_t_coh.zarr'
rslc_zarr = zarr.open(rslc,'r')
mc.pc2ras(ds_can_hix, ds_can_t_coh,t_coh,rslc_zarr.shape[:2])

2025-06-24 22:19:27 - log_args - INFO - running function: pc2ras
2025-06-24 22:19:27 - log_args - INFO - fetching args:
2025-06-24 22:19:27 - log_args - INFO - idx = './ds_processing/hix/ds_can/ds_can_hix.zarr'
2025-06-24 22:19:27 - log_args - INFO - pc = './ds_processing/hix/ds_can/ds_can_t_coh.zarr'
2025-06-24 22:19:27 - log_args - INFO - ras = './ds_processing/hix/ds_can/ras_ds_can_t_coh.zarr'
2025-06-24 22:19:27 - log_args - INFO - shape = (2500, 1834)
2025-06-24 22:19:27 - log_args - INFO - chunks = (1000, 1000)
2025-06-24 22:19:27 - log_args - INFO - processes = False
2025-06-24 22:19:27 - log_args - INFO - n_workers = 1
2025-06-24 22:19:27 - log_args - INFO - threads_per_worker = 1
2025-06-24 22:19:27 - log_args - INFO - dask_cluster_arg = {}
2025-06-24 22:19:27 - log_args - INFO - fetching args done.
2025-06-24 22:19:27 - zarr_info - INFO - ./ds_processing/hix/ds_can/ds_can_hix.zarr zarray shape, chunks, dtype: (732727,), (200000,), int64
2025-06-24 22:19:27 - pc2ras - INFO - loading hix into memory and convert to gix
2025-06-24 22:19:27 - pc2ras - INFO - starting dask local cluster.
2025-06-24 22:19:27 - pc2ras - INFO - dask local cluster started.
2025-06-24 22:19:27 - dask_cluster_info - INFO - dask cluster: LocalCluster(dashboard_link='http://10.211.48.18:8787/status', workers=1, threads=1, memory=1.46 TiB)
2025-06-24 22:19:28 - pc2ras - INFO - start to work on ./ds_processing/hix/ds_can/ds_can_t_coh.zarr
2025-06-24 22:19:28 - zarr_info - INFO - ./ds_processing/hix/ds_can/ds_can_t_coh.zarr zarray shape, chunks, dtype: (732727,), (200000,), float32
2025-06-24 22:19:28 - darr_info - INFO - pc dask array shape, chunksize, dtype: (732727,), (732727,), float32
2025-06-24 22:19:28 - pc2ras - INFO - create ras dask array
2025-06-24 22:19:28 - darr_info - INFO - ras dask array shape, chunksize, dtype: (2500, 1834), (2500, 1834), float32
2025-06-24 22:19:28 - pc2ras - INFO - save ras to ./ds_processing/hix/ds_can/ras_ds_can_t_coh.zarr
2025-06-24 22:19:28 - zarr_info - INFO - ./ds_processing/hix/ds_can/ras_ds_can_t_coh.zarr zarray shape, chunks, dtype: (2500, 1834), (1000, 1000), float32
2025-06-24 22:19:28 - pc2ras - INFO - computing graph setted. doing all the computing.
2025-06-24 22:19:28 - pc2ras - INFO - computing finished.ed |  0.1s
2025-06-24 22:19:28 - pc2ras - INFO - dask cluster closed.

t_coh_zarr = zarr.open(t_coh,'r')
t_coh_plot = mr.plot.ras_plot(t_coh_zarr[:])
t_coh_plot = t_coh_plot.redim(x=hv.Dimension('r', label='Range'), y=hv.Dimension('az',label='Azimuth'), z=hv.Dimension('t_coh',range=(0,1)))
t_coh_plot.opts(opts.Image(cmap='viridis',frame_width=500, frame_height=600, colorbar=True,
                          invert_yaxis=True,
                          default_tools=['pan',WheelZoomTool(zoom_on_axis=False),'save','reset','hover'],active_tools=['wheel_zoom']))

The full coherence matrix may be too big to be saved. It is also possible to do the coherence matrix estimation, phase linking together:

ds_can_ph_dir = './ds_processing/gix/ds_can_ph'
ds_can_emi_quality_dir = './ds_processing/gix/ds_can_emi_quality'
ds_can_t_coh_dir = './ds_processing/gix/ds_can_t_coh'

mc.emperical_co_emi_temp_coh_pc(rslc,ds_can_is_shp_dir,ds_can_gix,ds_can_ph_dir,ds_can_emi_quality_dir,ds_can_t_coh_dir,chunks=(1000,1000),cuda=True)
mc.pc_concat(
    [ds_can_ph_dir,ds_can_emi_quality_dir,ds_can_t_coh_dir],
    [ds_can_ph,ds_can_emi_quality,ds_can_t_coh],
    key=[ds_can_key,gix2hix_key],
    chunks=200000)

2025-06-24 22:19:28 - log_args - INFO - running function: emperical_co_emi_temp_coh_pc
2025-06-24 22:19:28 - log_args - INFO - fetching args:
2025-06-24 22:19:28 - log_args - INFO - rslc = './load_data/rslc.zarr'
2025-06-24 22:19:28 - log_args - INFO - is_shp_dir = './ds_processing/gix/ds_can_is_shp'
2025-06-24 22:19:28 - log_args - INFO - gix = './ds_processing/gix/ds_can_gix.zarr'
2025-06-24 22:19:28 - log_args - INFO - ph_dir = './ds_processing/gix/ds_can_ph'
2025-06-24 22:19:28 - log_args - INFO - emi_quality_dir = './ds_processing/gix/ds_can_emi_quality'
2025-06-24 22:19:28 - log_args - INFO - t_coh_dir = './ds_processing/gix/ds_can_t_coh'
2025-06-24 22:19:28 - log_args - INFO - batch_size = 1000
2025-06-24 22:19:28 - log_args - INFO - chunks = (1000, 1000)
2025-06-24 22:19:28 - log_args - INFO - cuda = True
2025-06-24 22:19:28 - log_args - INFO - processes = None
2025-06-24 22:19:28 - log_args - INFO - n_workers = None
2025-06-24 22:19:28 - log_args - INFO - threads_per_worker = None
2025-06-24 22:19:28 - log_args - INFO - rmm_pool_size = 0.9
2025-06-24 22:19:28 - log_args - INFO - dask_cluster_arg = {}
2025-06-24 22:19:28 - log_args - INFO - fetching args done.
2025-06-24 22:19:28 - zarr_info - INFO - ./load_data/rslc.zarr zarray shape, chunks, dtype: (2500, 1834, 17), (1000, 1000, 1), complex64
2025-06-24 22:19:28 - emperical_co_emi_temp_coh_pc - INFO - azimuth window size and half azimuth window size: 11, 5
2025-06-24 22:19:28 - emperical_co_emi_temp_coh_pc - INFO - range window size and half range window size: 11, 5
2025-06-24 22:19:28 - emperical_co_emi_temp_coh_pc - INFO - parallel processing azimuth chunk size: 1000
2025-06-24 22:19:28 - emperical_co_emi_temp_coh_pc - INFO - parallel processing range chunk size: 1000
2025-06-24 22:19:28 - zarr_info - INFO - ./ds_processing/gix/ds_can_gix.zarr zarray shape, chunks, dtype: (732727, 2), (200000, 1), int32
2025-06-24 22:19:28 - emperical_co_emi_temp_coh_pc - INFO - loading gix into memory.
2025-06-24 22:19:28 - emperical_co_emi_temp_coh_pc - INFO - convert gix to the order of ras chunk
2025-06-24 22:19:28 - emperical_co_emi_temp_coh_pc - INFO - starting dask cluster.
2025-06-24 22:19:35 - emperical_co_emi_temp_coh_pc - INFO - dask cluster started.
2025-06-24 22:19:35 - dask_cluster_info - INFO - dask cluster: LocalCUDACluster(dashboard_link='http://127.0.0.1:8787/status', workers=8, threads=8, memory=1.46 TiB)
2025-06-24 22:19:35 - darr_info - INFO - rslc_overlap dask array shape, chunksize, dtype: (2520, 1844, 17), (1010, 1005, 17), complex64
2025-06-24 22:19:35 - darr_info - INFO - gix in ras chunk order dask array shape, chunksize, dtype: (732727, 2), (201403, 2), int32
2025-06-24 22:19:35 - emperical_co_emi_temp_coh_pc - INFO - estimating coherence matrix chunk by chunk.
2025-06-24 22:19:35 - darr_info - INFO - is_shp for chunk 0 dask array shape, chunksize, dtype: (201097, 11, 11), (201097, 11, 11), bool
2025-06-24 22:19:35 - darr_info - INFO - ph for chunk 0 dask array shape, chunksize, dtype: (201097, 17), (201097, 17), complex64
2025-06-24 22:19:35 - darr_info - INFO - emi_quality for chunk 0 dask array shape, chunksize, dtype: (201097,), (201097,), float32
2025-06-24 22:19:35 - darr_info - INFO - t_coh for chunk 0 dask array shape, chunksize, dtype: (201097,), (201097,), float32
2025-06-24 22:19:35 - emperical_co_emi_temp_coh_pc - INFO - saving ph, emi_quality, t_coh for chunk 0
2025-06-24 22:19:35 - zarr_info - INFO - ds_processing/gix/ds_can_ph/0.zarr zarray shape, chunks, dtype: (201097, 17), (201097, 1), complex64
2025-06-24 22:19:35 - zarr_info - INFO - ds_processing/gix/ds_can_emi_quality/0.zarr zarray shape, chunks, dtype: (201097,), (201097,), float32
2025-06-24 22:19:35 - zarr_info - INFO - ds_processing/gix/ds_can_t_coh/0.zarr zarray shape, chunks, dtype: (201097,), (201097,), float32
2025-06-24 22:19:35 - darr_info - INFO - is_shp for chunk 1 dask array shape, chunksize, dtype: (137562, 11, 11), (137562, 11, 11), bool
2025-06-24 22:19:35 - darr_info - INFO - ph for chunk 1 dask array shape, chunksize, dtype: (137562, 17), (137562, 17), complex64
2025-06-24 22:19:35 - darr_info - INFO - emi_quality for chunk 1 dask array shape, chunksize, dtype: (137562,), (137562,), float32
2025-06-24 22:19:35 - darr_info - INFO - t_coh for chunk 1 dask array shape, chunksize, dtype: (137562,), (137562,), float32
2025-06-24 22:19:35 - emperical_co_emi_temp_coh_pc - INFO - saving ph, emi_quality, t_coh for chunk 1
2025-06-24 22:19:35 - zarr_info - INFO - ds_processing/gix/ds_can_ph/1.zarr zarray shape, chunks, dtype: (137562, 17), (137562, 1), complex64
2025-06-24 22:19:35 - zarr_info - INFO - ds_processing/gix/ds_can_emi_quality/1.zarr zarray shape, chunks, dtype: (137562,), (137562,), float32
2025-06-24 22:19:35 - zarr_info - INFO - ds_processing/gix/ds_can_t_coh/1.zarr zarray shape, chunks, dtype: (137562,), (137562,), float32
2025-06-24 22:19:35 - darr_info - INFO - is_shp for chunk 2 dask array shape, chunksize, dtype: (201403, 11, 11), (201403, 11, 11), bool
2025-06-24 22:19:35 - darr_info - INFO - ph for chunk 2 dask array shape, chunksize, dtype: (201403, 17), (201403, 17), complex64
2025-06-24 22:19:35 - darr_info - INFO - emi_quality for chunk 2 dask array shape, chunksize, dtype: (201403,), (201403,), float32
2025-06-24 22:19:35 - darr_info - INFO - t_coh for chunk 2 dask array shape, chunksize, dtype: (201403,), (201403,), float32
2025-06-24 22:19:35 - emperical_co_emi_temp_coh_pc - INFO - saving ph, emi_quality, t_coh for chunk 2
2025-06-24 22:19:35 - zarr_info - INFO - ds_processing/gix/ds_can_ph/2.zarr zarray shape, chunks, dtype: (201403, 17), (201403, 1), complex64
2025-06-24 22:19:35 - zarr_info - INFO - ds_processing/gix/ds_can_emi_quality/2.zarr zarray shape, chunks, dtype: (201403,), (201403,), float32
2025-06-24 22:19:35 - zarr_info - INFO - ds_processing/gix/ds_can_t_coh/2.zarr zarray shape, chunks, dtype: (201403,), (201403,), float32
2025-06-24 22:19:35 - darr_info - INFO - is_shp for chunk 3 dask array shape, chunksize, dtype: (73518, 11, 11), (73518, 11, 11), bool
2025-06-24 22:19:35 - darr_info - INFO - ph for chunk 3 dask array shape, chunksize, dtype: (73518, 17), (73518, 17), complex64
2025-06-24 22:19:35 - darr_info - INFO - emi_quality for chunk 3 dask array shape, chunksize, dtype: (73518,), (73518,), float32
2025-06-24 22:19:35 - darr_info - INFO - t_coh for chunk 3 dask array shape, chunksize, dtype: (73518,), (73518,), float32
2025-06-24 22:19:35 - emperical_co_emi_temp_coh_pc - INFO - saving ph, emi_quality, t_coh for chunk 3
2025-06-24 22:19:35 - zarr_info - INFO - ds_processing/gix/ds_can_ph/3.zarr zarray shape, chunks, dtype: (73518, 17), (73518, 1), complex64
2025-06-24 22:19:35 - zarr_info - INFO - ds_processing/gix/ds_can_emi_quality/3.zarr zarray shape, chunks, dtype: (73518,), (73518,), float32
2025-06-24 22:19:35 - zarr_info - INFO - ds_processing/gix/ds_can_t_coh/3.zarr zarray shape, chunks, dtype: (73518,), (73518,), float32
2025-06-24 22:19:35 - darr_info - INFO - is_shp for chunk 4 dask array shape, chunksize, dtype: (78445, 11, 11), (78445, 11, 11), bool
2025-06-24 22:19:35 - darr_info - INFO - ph for chunk 4 dask array shape, chunksize, dtype: (78445, 17), (78445, 17), complex64
2025-06-24 22:19:35 - darr_info - INFO - emi_quality for chunk 4 dask array shape, chunksize, dtype: (78445,), (78445,), float32
2025-06-24 22:19:35 - darr_info - INFO - t_coh for chunk 4 dask array shape, chunksize, dtype: (78445,), (78445,), float32
2025-06-24 22:19:35 - emperical_co_emi_temp_coh_pc - INFO - saving ph, emi_quality, t_coh for chunk 4
2025-06-24 22:19:35 - zarr_info - INFO - ds_processing/gix/ds_can_ph/4.zarr zarray shape, chunks, dtype: (78445, 17), (78445, 1), complex64
2025-06-24 22:19:35 - zarr_info - INFO - ds_processing/gix/ds_can_emi_quality/4.zarr zarray shape, chunks, dtype: (78445,), (78445,), float32
2025-06-24 22:19:35 - zarr_info - INFO - ds_processing/gix/ds_can_t_coh/4.zarr zarray shape, chunks, dtype: (78445,), (78445,), float32
2025-06-24 22:19:35 - darr_info - INFO - is_shp for chunk 5 dask array shape, chunksize, dtype: (40702, 11, 11), (40702, 11, 11), bool
2025-06-24 22:19:35 - darr_info - INFO - ph for chunk 5 dask array shape, chunksize, dtype: (40702, 17), (40702, 17), complex64
2025-06-24 22:19:35 - darr_info - INFO - emi_quality for chunk 5 dask array shape, chunksize, dtype: (40702,), (40702,), float32
2025-06-24 22:19:35 - darr_info - INFO - t_coh for chunk 5 dask array shape, chunksize, dtype: (40702,), (40702,), float32
2025-06-24 22:19:35 - emperical_co_emi_temp_coh_pc - INFO - saving ph, emi_quality, t_coh for chunk 5
2025-06-24 22:19:35 - zarr_info - INFO - ds_processing/gix/ds_can_ph/5.zarr zarray shape, chunks, dtype: (40702, 17), (40702, 1), complex64
2025-06-24 22:19:35 - zarr_info - INFO - ds_processing/gix/ds_can_emi_quality/5.zarr zarray shape, chunks, dtype: (40702,), (40702,), float32
2025-06-24 22:19:35 - zarr_info - INFO - ds_processing/gix/ds_can_t_coh/5.zarr zarray shape, chunks, dtype: (40702,), (40702,), float32
2025-06-24 22:19:35 - emperical_co_emi_temp_coh_pc - INFO - computing graph setted. doing all the computing.
2025-06-24 22:19:42 - emperical_co_emi_temp_coh_pc - INFO - computing finished.
2025-06-24 22:19:45 - emperical_co_emi_temp_coh_pc - INFO - dask cluster closed.
2025-06-24 22:19:45 - log_args - INFO - running function: pc_concat
2025-06-24 22:19:45 - log_args - INFO - fetching args:
2025-06-24 22:19:45 - log_args - INFO - pcs = ['./ds_processing/gix/ds_can_ph', './ds_processing/gix/ds_can_emi_quality', './ds_processing/gix/ds_can_t_coh']
2025-06-24 22:19:45 - log_args - INFO - pc = ['./ds_processing/hix/ds_can/ds_can_ph.zarr', './ds_processing/hix/ds_can/ds_can_emi_quality.zarr', './ds_processing/hix/ds_can/ds_can_t_coh.zarr']
2025-06-24 22:19:45 - log_args - INFO - key = ['./ds_processing/gix/ds_can_key.zarr', './ds_processing/gix/gix2hix.zarr']
2025-06-24 22:19:45 - log_args - INFO - chunks = 200000
2025-06-24 22:19:45 - log_args - INFO - processes = False
2025-06-24 22:19:45 - log_args - INFO - n_workers = 1
2025-06-24 22:19:45 - log_args - INFO - threads_per_worker = 1
2025-06-24 22:19:45 - log_args - INFO - dask_cluster_arg = {}
2025-06-24 22:19:45 - log_args - INFO - fetching args done.
2025-06-24 22:19:45 - pc_concat - INFO - input pcs: [[PosixPath('ds_processing/gix/ds_can_ph/0.zarr'), PosixPath('ds_processing/gix/ds_can_ph/1.zarr'), PosixPath('ds_processing/gix/ds_can_ph/2.zarr'), PosixPath('ds_processing/gix/ds_can_ph/3.zarr'), PosixPath('ds_processing/gix/ds_can_ph/4.zarr'), PosixPath('ds_processing/gix/ds_can_ph/5.zarr')], [PosixPath('ds_processing/gix/ds_can_emi_quality/0.zarr'), PosixPath('ds_processing/gix/ds_can_emi_quality/1.zarr'), PosixPath('ds_processing/gix/ds_can_emi_quality/2.zarr'), PosixPath('ds_processing/gix/ds_can_emi_quality/3.zarr'), PosixPath('ds_processing/gix/ds_can_emi_quality/4.zarr'), PosixPath('ds_processing/gix/ds_can_emi_quality/5.zarr')], [PosixPath('ds_processing/gix/ds_can_t_coh/0.zarr'), PosixPath('ds_processing/gix/ds_can_t_coh/1.zarr'), PosixPath('ds_processing/gix/ds_can_t_coh/2.zarr'), PosixPath('ds_processing/gix/ds_can_t_coh/3.zarr'), PosixPath('ds_processing/gix/ds_can_t_coh/4.zarr'), PosixPath('ds_processing/gix/ds_can_t_coh/5.zarr')]]
2025-06-24 22:19:45 - pc_concat - INFO - output pc: ['./ds_processing/hix/ds_can/ds_can_ph.zarr', './ds_processing/hix/ds_can/ds_can_emi_quality.zarr', './ds_processing/hix/ds_can/ds_can_t_coh.zarr']
2025-06-24 22:19:45 - pc_concat - INFO - load key
2025-06-24 22:19:45 - zarr_info - INFO - ./ds_processing/gix/ds_can_key.zarr zarray shape, chunks, dtype: (732727,), (200000,), int64
2025-06-24 22:19:45 - zarr_info - INFO - ./ds_processing/gix/gix2hix.zarr zarray shape, chunks, dtype: (732727,), (200000,), int64
2025-06-24 22:19:45 - pc_concat - INFO - starting dask local cluster.
2025-06-24 22:19:45 - pc_concat - INFO - dask local cluster started.
2025-06-24 22:19:45 - dask_cluster_info - INFO - dask cluster: LocalCluster(dashboard_link='http://10.211.48.18:8787/status', workers=1, threads=1, memory=1.46 TiB)
2025-06-24 22:19:45 - pc_concat - INFO - read pc from [PosixPath('ds_processing/gix/ds_can_ph/0.zarr'), PosixPath('ds_processing/gix/ds_can_ph/1.zarr'), PosixPath('ds_processing/gix/ds_can_ph/2.zarr'), PosixPath('ds_processing/gix/ds_can_ph/3.zarr'), PosixPath('ds_processing/gix/ds_can_ph/4.zarr'), PosixPath('ds_processing/gix/ds_can_ph/5.zarr')]
2025-06-24 22:19:45 - darr_info - INFO - concatenated pc dask array shape, chunksize, dtype: (732727, 17), (732727, 1), complex64
2025-06-24 22:19:45 - pc_concat - INFO - sort pc according to key
2025-06-24 22:19:45 - darr_info - INFO - sorted pc dask array shape, chunksize, dtype: (732727, 17), (732727, 1), complex64
2025-06-24 22:19:45 - pc_concat - INFO - save pc to ./ds_processing/hix/ds_can/ds_can_ph.zarr
2025-06-24 22:19:45 - zarr_info - INFO - ./ds_processing/hix/ds_can/ds_can_ph.zarr zarray shape, chunks, dtype: (732727, 17), (200000, 1), complex64
2025-06-24 22:19:45 - pc_concat - INFO - read pc from [PosixPath('ds_processing/gix/ds_can_emi_quality/0.zarr'), PosixPath('ds_processing/gix/ds_can_emi_quality/1.zarr'), PosixPath('ds_processing/gix/ds_can_emi_quality/2.zarr'), PosixPath('ds_processing/gix/ds_can_emi_quality/3.zarr'), PosixPath('ds_processing/gix/ds_can_emi_quality/4.zarr'), PosixPath('ds_processing/gix/ds_can_emi_quality/5.zarr')]
2025-06-24 22:19:45 - darr_info - INFO - concatenated pc dask array shape, chunksize, dtype: (732727,), (732727,), float32
2025-06-24 22:19:45 - pc_concat - INFO - sort pc according to key
2025-06-24 22:19:45 - darr_info - INFO - sorted pc dask array shape, chunksize, dtype: (732727,), (732727,), float32
2025-06-24 22:19:45 - pc_concat - INFO - save pc to ./ds_processing/hix/ds_can/ds_can_emi_quality.zarr
2025-06-24 22:19:45 - zarr_info - INFO - ./ds_processing/hix/ds_can/ds_can_emi_quality.zarr zarray shape, chunks, dtype: (732727,), (200000,), float32
2025-06-24 22:19:45 - pc_concat - INFO - read pc from [PosixPath('ds_processing/gix/ds_can_t_coh/0.zarr'), PosixPath('ds_processing/gix/ds_can_t_coh/1.zarr'), PosixPath('ds_processing/gix/ds_can_t_coh/2.zarr'), PosixPath('ds_processing/gix/ds_can_t_coh/3.zarr'), PosixPath('ds_processing/gix/ds_can_t_coh/4.zarr'), PosixPath('ds_processing/gix/ds_can_t_coh/5.zarr')]
2025-06-24 22:19:45 - darr_info - INFO - concatenated pc dask array shape, chunksize, dtype: (732727,), (732727,), float32
2025-06-24 22:19:45 - pc_concat - INFO - sort pc according to key
2025-06-24 22:19:45 - darr_info - INFO - sorted pc dask array shape, chunksize, dtype: (732727,), (732727,), float32
2025-06-24 22:19:45 - pc_concat - INFO - save pc to ./ds_processing/hix/ds_can/ds_can_t_coh.zarr
2025-06-24 22:19:45 - zarr_info - INFO - ./ds_processing/hix/ds_can/ds_can_t_coh.zarr zarray shape, chunks, dtype: (732727,), (200000,), float32
2025-06-24 22:19:45 - pc_concat - INFO - computing graph setted. doing all the computing.
2025-06-24 22:19:46 - pc_concat - INFO - computing finished.|  0.7s
2025-06-24 22:19:46 - pc_concat - INFO - dask cluster closed.

DS candidiates refinement

Then, we refine the ds candidate using EMI quality factor and the temporal coherence:

ds_can_r1_hix = './ds_processing/hix/ds_can/ds_can_r1_hix.zarr'
mc.pc_logic_pc(ds_can_hix, ds_can_emi_quality,ds_can_r1_hix,'(pc_in>=1.0)&(pc_in<=1.05)')

2025-06-24 22:19:46 - log_args - INFO - running function: pc_logic_pc
2025-06-24 22:19:46 - log_args - INFO - fetching args:
2025-06-24 22:19:46 - log_args - INFO - idx_in = './ds_processing/hix/ds_can/ds_can_hix.zarr'
2025-06-24 22:19:46 - log_args - INFO - pc_in = './ds_processing/hix/ds_can/ds_can_emi_quality.zarr'
2025-06-24 22:19:46 - log_args - INFO - idx = './ds_processing/hix/ds_can/ds_can_r1_hix.zarr'
2025-06-24 22:19:46 - log_args - INFO - operation = '(pc_in>=1.0)&(pc_in<=1.05)'
2025-06-24 22:19:46 - log_args - INFO - chunks = None
2025-06-24 22:19:46 - log_args - INFO - fetching args done.
2025-06-24 22:19:46 - zarr_info - INFO - ./ds_processing/hix/ds_can/ds_can_hix.zarr zarray shape, chunks, dtype: (732727,), (200000,), int64
2025-06-24 22:19:46 - zarr_info - INFO - ./ds_processing/hix/ds_can/ds_can_emi_quality.zarr zarray shape, chunks, dtype: (732727,), (200000,), float32
2025-06-24 22:19:46 - pc_logic_pc - INFO - loading idx_in into memory.
2025-06-24 22:19:46 - pc_logic_pc - INFO - loading pc_in into memory.
2025-06-24 22:19:46 - pc_logic_pc - INFO - select pc based on operation: (pc_in>=1.0)&(pc_in<=1.05)
2025-06-24 22:19:46 - pc_logic_pc - INFO - number of selected pixels: 404013.
2025-06-24 22:19:46 - zarr_info - INFO - idx zarray shape, chunks, dtype: (404013,), (200000,), int64
2025-06-24 22:19:46 - pc_logic_pc - INFO - writing idx.
2025-06-24 22:19:46 - pc_logic_pc - INFO - write done.

ds_can_r2_hix = './ds_processing/hix/ds_can/ds_can_r2_hix.zarr'
mc.pc_logic_pc(ds_can_hix, ds_can_t_coh,ds_can_r2_hix,'(pc_in>=0.8)&(pc_in<=1.0)')

2025-06-24 22:19:46 - log_args - INFO - running function: pc_logic_pc
2025-06-24 22:19:46 - log_args - INFO - fetching args:
2025-06-24 22:19:46 - log_args - INFO - idx_in = './ds_processing/hix/ds_can/ds_can_hix.zarr'
2025-06-24 22:19:46 - log_args - INFO - pc_in = './ds_processing/hix/ds_can/ds_can_t_coh.zarr'
2025-06-24 22:19:46 - log_args - INFO - idx = './ds_processing/hix/ds_can/ds_can_r2_hix.zarr'
2025-06-24 22:19:46 - log_args - INFO - operation = '(pc_in>=0.8)&(pc_in<=1.0)'
2025-06-24 22:19:46 - log_args - INFO - chunks = None
2025-06-24 22:19:46 - log_args - INFO - fetching args done.
2025-06-24 22:19:46 - zarr_info - INFO - ./ds_processing/hix/ds_can/ds_can_hix.zarr zarray shape, chunks, dtype: (732727,), (200000,), int64
2025-06-24 22:19:46 - zarr_info - INFO - ./ds_processing/hix/ds_can/ds_can_t_coh.zarr zarray shape, chunks, dtype: (732727,), (200000,), float32
2025-06-24 22:19:46 - pc_logic_pc - INFO - loading idx_in into memory.
2025-06-24 22:19:46 - pc_logic_pc - INFO - loading pc_in into memory.
2025-06-24 22:19:46 - pc_logic_pc - INFO - select pc based on operation: (pc_in>=0.8)&(pc_in<=1.0)
2025-06-24 22:19:46 - pc_logic_pc - INFO - number of selected pixels: 381992.
2025-06-24 22:19:46 - zarr_info - INFO - idx zarray shape, chunks, dtype: (381992,), (200000,), int64
2025-06-24 22:19:46 - pc_logic_pc - INFO - writing idx.
2025-06-24 22:19:46 - pc_logic_pc - INFO - write done.

Get their intersection:

Note

Note that pc_union can be applied if the coherence is not good and no much pixels selected.

ds_hix = './ds_processing/hix/ds/ds_hix.zarr'

mc.pc_intersect(ds_can_r1_hix,ds_can_r2_hix,ds_hix)

2025-06-24 22:19:46 - log_args - INFO - running function: pc_intersect
2025-06-24 22:19:46 - log_args - INFO - fetching args:
2025-06-24 22:19:46 - log_args - INFO - idx1 = './ds_processing/hix/ds_can/ds_can_r1_hix.zarr'
2025-06-24 22:19:46 - log_args - INFO - idx2 = './ds_processing/hix/ds_can/ds_can_r2_hix.zarr'
2025-06-24 22:19:46 - log_args - INFO - idx = './ds_processing/hix/ds/ds_hix.zarr'
2025-06-24 22:19:46 - log_args - INFO - pc1 = None
2025-06-24 22:19:46 - log_args - INFO - pc2 = None
2025-06-24 22:19:46 - log_args - INFO - pc = None
2025-06-24 22:19:46 - log_args - INFO - shape = None
2025-06-24 22:19:46 - log_args - INFO - chunks = None
2025-06-24 22:19:46 - log_args - INFO - prefer_1 = True
2025-06-24 22:19:46 - log_args - INFO - processes = False
2025-06-24 22:19:46 - log_args - INFO - n_workers = 1
2025-06-24 22:19:46 - log_args - INFO - threads_per_worker = 1
2025-06-24 22:19:46 - log_args - INFO - dask_cluster_arg = {}
2025-06-24 22:19:46 - log_args - INFO - fetching args done.
2025-06-24 22:19:46 - zarr_info - INFO - ./ds_processing/hix/ds_can/ds_can_r1_hix.zarr zarray shape, chunks, dtype: (404013,), (200000,), int64
2025-06-24 22:19:46 - zarr_info - INFO - ./ds_processing/hix/ds_can/ds_can_r2_hix.zarr zarray shape, chunks, dtype: (381992,), (200000,), int64
2025-06-24 22:19:46 - pc_intersect - INFO - loading idx1 and idx2 into memory.
2025-06-24 22:19:46 - pc_intersect - INFO - calculate the intersection
2025-06-24 22:19:46 - pc_intersect - INFO - number of points in the intersection: 293814
2025-06-24 22:19:46 - pc_intersect - INFO - write intersect idx
2025-06-24 22:19:46 - pc_intersect - INFO - write done
2025-06-24 22:19:46 - zarr_info - INFO - ./ds_processing/hix/ds/ds_hix.zarr zarray shape, chunks, dtype: (293814,), (200000,), int64
2025-06-24 22:19:46 - pc_intersect - INFO - no point cloud data provided, exit.

Visualize phase linked interferograms at refined ds points

ds_coh = './ds_processing/hix/ds/ds_coh.zarr'
ds_ph = './ds_processing/hix/ds/ds_ph.zarr'
ds_gix = './ds_processing/hix/ds/ds_gix.zarr'

mc.pc_select_data(
    ds_can_hix,ds_hix,
    pc_in=[ds_can_coh, ds_can_ph, './ds_processing/hix/ds_can/ds_can_gix.zarr/' ],
    pc=[ds_coh,ds_ph, ds_gix],threads_per_worker=32)

2025-06-24 22:19:46 - log_args - INFO - running function: pc_select_data
2025-06-24 22:19:46 - log_args - INFO - fetching args:
2025-06-24 22:19:46 - log_args - INFO - idx_in = './ds_processing/hix/ds_can/ds_can_hix.zarr'
2025-06-24 22:19:46 - log_args - INFO - idx = './ds_processing/hix/ds/ds_hix.zarr'
2025-06-24 22:19:46 - log_args - INFO - pc_in = ['./ds_processing/hix/ds_can/ds_can_coh.zarr', './ds_processing/hix/ds_can/ds_can_ph.zarr', './ds_processing/hix/ds_can/ds_can_gix.zarr/']
2025-06-24 22:19:46 - log_args - INFO - pc = ['./ds_processing/hix/ds/ds_coh.zarr', './ds_processing/hix/ds/ds_ph.zarr', './ds_processing/hix/ds/ds_gix.zarr']
2025-06-24 22:19:46 - log_args - INFO - shape = None
2025-06-24 22:19:46 - log_args - INFO - chunks = None
2025-06-24 22:19:46 - log_args - INFO - processes = False
2025-06-24 22:19:46 - log_args - INFO - n_workers = 1
2025-06-24 22:19:46 - log_args - INFO - threads_per_worker = 32
2025-06-24 22:19:46 - log_args - INFO - dask_cluster_arg = {}
2025-06-24 22:19:46 - log_args - INFO - fetching args done.
2025-06-24 22:19:46 - zarr_info - INFO - ./ds_processing/hix/ds_can/ds_can_hix.zarr zarray shape, chunks, dtype: (732727,), (200000,), int64
2025-06-24 22:19:46 - zarr_info - INFO - ./ds_processing/hix/ds/ds_hix.zarr zarray shape, chunks, dtype: (293814,), (200000,), int64
2025-06-24 22:19:46 - pc_select_data - INFO - loading idx_in and idx into memory.
2025-06-24 22:19:46 - pc_select_data - INFO - starting dask local cluster.
2025-06-24 22:19:46 - pc_select_data - INFO - dask local cluster started.
2025-06-24 22:19:46 - dask_cluster_info - INFO - dask cluster: LocalCluster(dashboard_link='http://10.211.48.18:8787/status', workers=1, threads=32, memory=1.46 TiB)
2025-06-24 22:19:46 - zarr_info - INFO - ./ds_processing/hix/ds_can/ds_can_coh.zarr zarray shape, chunks, dtype: (732727, 136), (200000, 1), complex64
2025-06-24 22:19:46 - darr_info - INFO - pc_in dask array shape, chunksize, dtype: (732727, 136), (732727, 1), complex64
2025-06-24 22:19:46 - pc_select_data - INFO - set up selected pc data dask array.
2025-06-24 22:19:46 - darr_info - INFO - pc dask array shape, chunksize, dtype: (293814, 136), (293814, 1), complex64
2025-06-24 22:19:46 - pc_select_data - INFO - write pc to ./ds_processing/hix/ds/ds_coh.zarr
2025-06-24 22:19:46 - zarr_info - INFO - ./ds_processing/hix/ds/ds_coh.zarr zarray shape, chunks, dtype: (293814, 136), (200000, 1), complex64
2025-06-24 22:19:46 - zarr_info - INFO - ./ds_processing/hix/ds_can/ds_can_ph.zarr zarray shape, chunks, dtype: (732727, 17), (200000, 1), complex64
2025-06-24 22:19:46 - darr_info - INFO - pc_in dask array shape, chunksize, dtype: (732727, 17), (732727, 1), complex64
2025-06-24 22:19:46 - pc_select_data - INFO - set up selected pc data dask array.
2025-06-24 22:19:46 - darr_info - INFO - pc dask array shape, chunksize, dtype: (293814, 17), (293814, 1), complex64
2025-06-24 22:19:46 - pc_select_data - INFO - write pc to ./ds_processing/hix/ds/ds_ph.zarr
2025-06-24 22:19:46 - zarr_info - INFO - ./ds_processing/hix/ds/ds_ph.zarr zarray shape, chunks, dtype: (293814, 17), (200000, 1), complex64
2025-06-24 22:19:46 - zarr_info - INFO - ./ds_processing/hix/ds_can/ds_can_gix.zarr/ zarray shape, chunks, dtype: (732727, 2), (200000, 1), int32
2025-06-24 22:19:46 - darr_info - INFO - pc_in dask array shape, chunksize, dtype: (732727, 2), (732727, 1), int32
2025-06-24 22:19:46 - pc_select_data - INFO - set up selected pc data dask array.
2025-06-24 22:19:46 - darr_info - INFO - pc dask array shape, chunksize, dtype: (293814, 2), (293814, 1), int32
2025-06-24 22:19:46 - pc_select_data - INFO - write pc to ./ds_processing/hix/ds/ds_gix.zarr
2025-06-24 22:19:46 - zarr_info - INFO - ./ds_processing/hix/ds/ds_gix.zarr zarray shape, chunks, dtype: (293814, 2), (200000, 1), int32
2025-06-24 22:19:46 - pc_select_data - INFO - computing graph setted. doing all the computing.
2025-06-24 22:19:48 - pc_select_data - INFO - computing finished.2s
2025-06-24 22:19:48 - pc_select_data - INFO - dask cluster closed.

Then plot them:

mc.pc_pyramid(
    './ds_processing/hix/ds/ds_coh.zarr',
    './ds_processing/hix/ds/ds_coh_pyramid',
    yx = './ds_processing/hix/ds/ds_gix.zarr',
    ras_resolution=2,
)

2025-06-24 22:19:48 - log_args - INFO - running function: pc_pyramid
2025-06-24 22:19:48 - log_args - INFO - fetching args:
2025-06-24 22:19:48 - log_args - INFO - pc = './ds_processing/hix/ds/ds_coh.zarr'
2025-06-24 22:19:48 - log_args - INFO - out_dir = './ds_processing/hix/ds/ds_coh_pyramid'
2025-06-24 22:19:48 - log_args - INFO - x = None
2025-06-24 22:19:48 - log_args - INFO - y = None
2025-06-24 22:19:48 - log_args - INFO - yx = './ds_processing/hix/ds/ds_gix.zarr'
2025-06-24 22:19:48 - log_args - INFO - ras_resolution = 2
2025-06-24 22:19:48 - log_args - INFO - ras_chunks = (256, 256)
2025-06-24 22:19:48 - log_args - INFO - pc_chunks = 65536
2025-06-24 22:19:48 - log_args - INFO - processes = False
2025-06-24 22:19:48 - log_args - INFO - n_workers = 1
2025-06-24 22:19:48 - log_args - INFO - threads_per_worker = 2
2025-06-24 22:19:48 - log_args - INFO - dask_cluster_arg = {}
2025-06-24 22:19:48 - log_args - INFO - fetching args done.
2025-06-24 22:19:48 - pc_pyramid - INFO - clean out dir
2025-06-24 22:19:48 - zarr_info - INFO - ./ds_processing/hix/ds/ds_coh.zarr zarray shape, chunks, dtype: (293814, 136), (200000, 1), complex64
2025-06-24 22:19:48 - pc_pyramid - INFO - rendering point cloud data coordinates:
2025-06-24 22:19:48 - pc_pyramid - INFO - rasterizing point cloud data to grid with bounds: [0, 0, 1832, 2498].
2025-06-24 22:19:48 - zarr_info - INFO - ds_processing/hix/ds/ds_coh_pyramid/x.zarr zarray shape, chunks, dtype: (293814,), (65536,), int32
2025-06-24 22:19:48 - zarr_info - INFO - ds_processing/hix/ds/ds_coh_pyramid/y.zarr zarray shape, chunks, dtype: (293814,), (65536,), int32
2025-06-24 22:19:48 - pc_pyramid - INFO - pc data coordinates rendering ends.
2025-06-24 22:19:48 - zarr_info - INFO - ds_processing/hix/ds/ds_coh_pyramid/idx_0.zarr zarray shape, chunks, dtype: (1250, 917), (256, 256), int64
2025-06-24 22:19:49 - zarr_info - INFO - ds_processing/hix/ds/ds_coh_pyramid/idx_1.zarr zarray shape, chunks, dtype: (625, 459), (256, 256), int64
2025-06-24 22:19:49 - zarr_info - INFO - ds_processing/hix/ds/ds_coh_pyramid/idx_2.zarr zarray shape, chunks, dtype: (313, 230), (256, 256), int64
2025-06-24 22:19:49 - zarr_info - INFO - ds_processing/hix/ds/ds_coh_pyramid/idx_3.zarr zarray shape, chunks, dtype: (157, 115), (256, 256), int64
2025-06-24 22:19:49 - zarr_info - INFO - ds_processing/hix/ds/ds_coh_pyramid/idx_4.zarr zarray shape, chunks, dtype: (79, 58), (256, 256), int64
2025-06-24 22:19:49 - zarr_info - INFO - ds_processing/hix/ds/ds_coh_pyramid/idx_5.zarr zarray shape, chunks, dtype: (40, 29), (256, 256), int64
2025-06-24 22:19:49 - zarr_info - INFO - ds_processing/hix/ds/ds_coh_pyramid/idx_6.zarr zarray shape, chunks, dtype: (20, 15), (256, 256), int64
2025-06-24 22:19:49 - zarr_info - INFO - ds_processing/hix/ds/ds_coh_pyramid/idx_7.zarr zarray shape, chunks, dtype: (10, 8), (256, 256), int64
2025-06-24 22:19:49 - zarr_info - INFO - ds_processing/hix/ds/ds_coh_pyramid/idx_8.zarr zarray shape, chunks, dtype: (5, 4), (256, 256), int64
2025-06-24 22:19:49 - zarr_info - INFO - ds_processing/hix/ds/ds_coh_pyramid/idx_9.zarr zarray shape, chunks, dtype: (3, 2), (256, 256), int64
2025-06-24 22:19:49 - pc_pyramid - INFO - rasterized idx rendering ends
2025-06-24 22:19:49 - pc_pyramid - INFO - dask local cluster started to render pc data.
2025-06-24 22:19:49 - dask_cluster_info - INFO - dask cluster: LocalCluster(dashboard_link='http://10.211.48.18:8787/status', workers=1, threads=2, memory=1.46 TiB)
2025-06-24 22:19:49 - zarr_info - INFO - ds_processing/hix/ds/ds_coh_pyramid/pc.zarr zarray shape, chunks, dtype: (293814, 136), (65536, 1), complex64
2025-06-24 22:19:49 - zarr_info - INFO - ds_processing/hix/ds/ds_coh_pyramid/0.zarr zarray shape, chunks, dtype: (1250, 917, 136), (256, 256, 1), complex64
2025-06-24 22:19:49 - zarr_info - INFO - ds_processing/hix/ds/ds_coh_pyramid/1.zarr zarray shape, chunks, dtype: (625, 459, 136), (256, 256, 1), complex64
2025-06-24 22:19:49 - zarr_info - INFO - ds_processing/hix/ds/ds_coh_pyramid/2.zarr zarray shape, chunks, dtype: (313, 230, 136), (256, 256, 1), complex64
2025-06-24 22:19:49 - zarr_info - INFO - ds_processing/hix/ds/ds_coh_pyramid/3.zarr zarray shape, chunks, dtype: (157, 115, 136), (256, 256, 1), complex64
2025-06-24 22:19:49 - zarr_info - INFO - ds_processing/hix/ds/ds_coh_pyramid/4.zarr zarray shape, chunks, dtype: (79, 58, 136), (256, 256, 1), complex64
2025-06-24 22:19:49 - zarr_info - INFO - ds_processing/hix/ds/ds_coh_pyramid/5.zarr zarray shape, chunks, dtype: (40, 29, 136), (256, 256, 1), complex64
2025-06-24 22:19:49 - zarr_info - INFO - ds_processing/hix/ds/ds_coh_pyramid/6.zarr zarray shape, chunks, dtype: (20, 15, 136), (256, 256, 1), complex64
2025-06-24 22:19:49 - zarr_info - INFO - ds_processing/hix/ds/ds_coh_pyramid/7.zarr zarray shape, chunks, dtype: (10, 8, 136), (256, 256, 1), complex64
2025-06-24 22:19:49 - zarr_info - INFO - ds_processing/hix/ds/ds_coh_pyramid/8.zarr zarray shape, chunks, dtype: (5, 4, 136), (256, 256, 1), complex64
2025-06-24 22:19:49 - zarr_info - INFO - ds_processing/hix/ds/ds_coh_pyramid/9.zarr zarray shape, chunks, dtype: (3, 2, 136), (256, 256, 1), complex64
2025-06-24 22:19:50 - pc_pyramid - INFO - computing graph setted. doing all the computing.
2025-06-24 22:19:57 - pc_pyramid - INFO - computing finished.  7.0s
2025-06-24 22:19:57 - pc_pyramid - INFO - dask cluster closed.

mc.pc_pyramid(
    './ds_processing/hix/ds/ds_ph.zarr',
    './ds_processing/hix/ds/ds_ph_pyramid',
    yx = './ds_processing/hix/ds/ds_gix.zarr',
    ras_resolution=2,
)

2025-06-24 22:19:57 - log_args - INFO - running function: pc_pyramid
2025-06-24 22:19:57 - log_args - INFO - fetching args:
2025-06-24 22:19:57 - log_args - INFO - pc = './ds_processing/hix/ds/ds_ph.zarr'
2025-06-24 22:19:57 - log_args - INFO - out_dir = './ds_processing/hix/ds/ds_ph_pyramid'
2025-06-24 22:19:57 - log_args - INFO - x = None
2025-06-24 22:19:57 - log_args - INFO - y = None
2025-06-24 22:19:57 - log_args - INFO - yx = './ds_processing/hix/ds/ds_gix.zarr'
2025-06-24 22:19:57 - log_args - INFO - ras_resolution = 2
2025-06-24 22:19:57 - log_args - INFO - ras_chunks = (256, 256)
2025-06-24 22:19:57 - log_args - INFO - pc_chunks = 65536
2025-06-24 22:19:57 - log_args - INFO - processes = False
2025-06-24 22:19:57 - log_args - INFO - n_workers = 1
2025-06-24 22:19:57 - log_args - INFO - threads_per_worker = 2
2025-06-24 22:19:57 - log_args - INFO - dask_cluster_arg = {}
2025-06-24 22:19:57 - log_args - INFO - fetching args done.
2025-06-24 22:19:57 - pc_pyramid - INFO - clean out dir
2025-06-24 22:19:57 - zarr_info - INFO - ./ds_processing/hix/ds/ds_ph.zarr zarray shape, chunks, dtype: (293814, 17), (200000, 1), complex64
2025-06-24 22:19:57 - pc_pyramid - INFO - rendering point cloud data coordinates:
2025-06-24 22:19:57 - pc_pyramid - INFO - rasterizing point cloud data to grid with bounds: [0, 0, 1832, 2498].
2025-06-24 22:19:57 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_pyramid/x.zarr zarray shape, chunks, dtype: (293814,), (65536,), int32
2025-06-24 22:19:57 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_pyramid/y.zarr zarray shape, chunks, dtype: (293814,), (65536,), int32
2025-06-24 22:19:57 - pc_pyramid - INFO - pc data coordinates rendering ends.
2025-06-24 22:19:57 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_pyramid/idx_0.zarr zarray shape, chunks, dtype: (1250, 917), (256, 256), int64
2025-06-24 22:19:57 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_pyramid/idx_1.zarr zarray shape, chunks, dtype: (625, 459), (256, 256), int64
2025-06-24 22:19:57 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_pyramid/idx_2.zarr zarray shape, chunks, dtype: (313, 230), (256, 256), int64
2025-06-24 22:19:57 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_pyramid/idx_3.zarr zarray shape, chunks, dtype: (157, 115), (256, 256), int64
2025-06-24 22:19:57 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_pyramid/idx_4.zarr zarray shape, chunks, dtype: (79, 58), (256, 256), int64
2025-06-24 22:19:57 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_pyramid/idx_5.zarr zarray shape, chunks, dtype: (40, 29), (256, 256), int64
2025-06-24 22:19:57 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_pyramid/idx_6.zarr zarray shape, chunks, dtype: (20, 15), (256, 256), int64
2025-06-24 22:19:57 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_pyramid/idx_7.zarr zarray shape, chunks, dtype: (10, 8), (256, 256), int64
2025-06-24 22:19:57 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_pyramid/idx_8.zarr zarray shape, chunks, dtype: (5, 4), (256, 256), int64
2025-06-24 22:19:57 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_pyramid/idx_9.zarr zarray shape, chunks, dtype: (3, 2), (256, 256), int64
2025-06-24 22:19:57 - pc_pyramid - INFO - rasterized idx rendering ends
2025-06-24 22:19:57 - pc_pyramid - INFO - dask local cluster started to render pc data.
2025-06-24 22:19:57 - dask_cluster_info - INFO - dask cluster: LocalCluster(dashboard_link='http://10.211.48.18:8787/status', workers=1, threads=2, memory=1.46 TiB)
2025-06-24 22:19:57 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_pyramid/pc.zarr zarray shape, chunks, dtype: (293814, 17), (65536, 1), complex64
2025-06-24 22:19:57 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_pyramid/0.zarr zarray shape, chunks, dtype: (1250, 917, 17), (256, 256, 1), complex64
2025-06-24 22:19:57 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_pyramid/1.zarr zarray shape, chunks, dtype: (625, 459, 17), (256, 256, 1), complex64
2025-06-24 22:19:57 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_pyramid/2.zarr zarray shape, chunks, dtype: (313, 230, 17), (256, 256, 1), complex64
2025-06-24 22:19:58 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_pyramid/3.zarr zarray shape, chunks, dtype: (157, 115, 17), (256, 256, 1), complex64
2025-06-24 22:19:58 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_pyramid/4.zarr zarray shape, chunks, dtype: (79, 58, 17), (256, 256, 1), complex64
2025-06-24 22:19:58 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_pyramid/5.zarr zarray shape, chunks, dtype: (40, 29, 17), (256, 256, 1), complex64
2025-06-24 22:19:58 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_pyramid/6.zarr zarray shape, chunks, dtype: (20, 15, 17), (256, 256, 1), complex64
2025-06-24 22:19:58 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_pyramid/7.zarr zarray shape, chunks, dtype: (10, 8, 17), (256, 256, 1), complex64
2025-06-24 22:19:58 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_pyramid/8.zarr zarray shape, chunks, dtype: (5, 4, 17), (256, 256, 1), complex64
2025-06-24 22:19:58 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_pyramid/9.zarr zarray shape, chunks, dtype: (3, 2, 17), (256, 256, 1), complex64
2025-06-24 22:19:58 - pc_pyramid - INFO - computing graph setted. doing all the computing.
2025-06-24 22:19:59 - pc_pyramid - INFO - computing finished.  0.9s
2025-06-24 22:19:59 - pc_pyramid - INFO - dask cluster closed.

ds_intf_plot = mc.pc_plot(
    './ds_processing/hix/ds/ds_coh_pyramid',
    post_proc_ras=co_phase_post_proc_ras,
    post_proc_pc=co_phase_post_proc_pc,
    n_kdim=2,
    level_increase=0)
raw_intf_plot = mc.ras_plot('./load_data/rslc_pyramid',post_proc='intf_all',n_kdim=2,level_increase=0)
ds_pl_intf_plot = mc.pc_plot('./ds_processing/hix/ds/ds_ph_pyramid',post_proc_ras='intf_all', post_proc_pc='intf_all',level_increase=0)
ds_intf_plots = raw_intf_plot.relabel('Raw Interferograms') + \
ds_intf_plot.relabel('Adaptively multilooked Interferograms') + \
ds_pl_intf_plot.relabel('Phase linked Interferograms')

ds_intf_plots = ds_intf_plots.redim(
    i=hv.Dimension('i', label='Reference Image', range=(0,16), value_format=(lambda i: dates[i])),
    j=hv.Dimension('j', label='Secondary Image', range=(0,16), value_format=(lambda i: dates[i])),
    x=hv.Dimension('r', label='Range'),
    y=hv.Dimension('az',label='Azimuth'),
    z=hv.Dimension('Phase',range=(-np.pi,np.pi))
)

ds_intf_plots.opts(
    hv.opts.Image(
        cmap='colorwheel',frame_width=400, frame_height=500, colorbar=True,
        default_tools=['pan',WheelZoomTool(zoom_on_axis=False),'save','reset','hover'],
        active_tools=['wheel_zoom'],
        invert_yaxis=True,
    ),
    hv.opts.Points(
        color='Phase', cmap='colorwheel',frame_width=400, frame_height=500, colorbar=True,
        default_tools=['pan',WheelZoomTool(zoom_on_axis=False),'save','reset','hover'],
        active_tools=['wheel_zoom'],
        invert_yaxis=True,
    ),
)

Visualize phase linked interferograms at refined ds points in earth coordinates

Get the needed data for ds:

ds_ph = './ds_processing/hix/ds/ds_ph.zarr'
ds_e = './ds_processing/hix/ds/ds_e.zarr'
ds_n = './ds_processing/hix/ds/ds_n.zarr'

mc.ras2pc(ds_hix,
          ras=['./load_data/e.zarr/','./load_data/n.zarr/'],
          pc=[ds_e, ds_n],)

2025-06-24 22:20:00 - log_args - INFO - running function: ras2pc
2025-06-24 22:20:00 - log_args - INFO - fetching args:
2025-06-24 22:20:00 - log_args - INFO - idx = './ds_processing/hix/ds/ds_hix.zarr'
2025-06-24 22:20:00 - log_args - INFO - ras = ['./load_data/e.zarr/', './load_data/n.zarr/']
2025-06-24 22:20:00 - log_args - INFO - pc = ['./ds_processing/hix/ds/ds_e.zarr', './ds_processing/hix/ds/ds_n.zarr']
2025-06-24 22:20:00 - log_args - INFO - chunks = None
2025-06-24 22:20:00 - log_args - INFO - processes = False
2025-06-24 22:20:00 - log_args - INFO - n_workers = 1
2025-06-24 22:20:00 - log_args - INFO - threads_per_worker = 1
2025-06-24 22:20:00 - log_args - INFO - dask_cluster_arg = {}
2025-06-24 22:20:00 - log_args - INFO - fetching args done.
2025-06-24 22:20:00 - zarr_info - INFO - ./ds_processing/hix/ds/ds_hix.zarr zarray shape, chunks, dtype: (293814,), (200000,), int64
2025-06-24 22:20:00 - ras2pc - INFO - loading hix into memory and convert to gix
2025-06-24 22:20:00 - ras2pc - INFO - starting dask local cluster.
2025-06-24 22:20:00 - ras2pc - INFO - dask local cluster started.
2025-06-24 22:20:00 - dask_cluster_info - INFO - dask cluster: LocalCluster(dashboard_link='http://10.211.48.18:8787/status', workers=1, threads=1, memory=1.46 TiB)
2025-06-24 22:20:00 - ras2pc - INFO - start to slice on ./load_data/e.zarr/
2025-06-24 22:20:00 - zarr_info - INFO - ./load_data/e.zarr/ zarray shape, chunks, dtype: (2500, 1834), (1000, 1000), float64
2025-06-24 22:20:00 - darr_info - INFO - ras dask array shape, chunksize, dtype: (2500, 1834), (2500, 1834), float64
2025-06-24 22:20:00 - darr_info - INFO - pc dask array shape, chunksize, dtype: (293814,), (293814,), float64
2025-06-24 22:20:00 - ras2pc - INFO - saving to ./ds_processing/hix/ds/ds_e.zarr.
2025-06-24 22:20:00 - zarr_info - INFO - ./ds_processing/hix/ds/ds_e.zarr zarray shape, chunks, dtype: (293814,), (200000,), float64
2025-06-24 22:20:00 - ras2pc - INFO - start to slice on ./load_data/n.zarr/
2025-06-24 22:20:00 - zarr_info - INFO - ./load_data/n.zarr/ zarray shape, chunks, dtype: (2500, 1834), (1000, 1000), float64
2025-06-24 22:20:00 - darr_info - INFO - ras dask array shape, chunksize, dtype: (2500, 1834), (2500, 1834), float64
2025-06-24 22:20:00 - darr_info - INFO - pc dask array shape, chunksize, dtype: (293814,), (293814,), float64
2025-06-24 22:20:00 - ras2pc - INFO - saving to ./ds_processing/hix/ds/ds_n.zarr.
2025-06-24 22:20:00 - zarr_info - INFO - ./ds_processing/hix/ds/ds_n.zarr zarray shape, chunks, dtype: (293814,), (200000,), float64
2025-06-24 22:20:00 - ras2pc - INFO - computing graph setted. doing all the computing.
2025-06-24 22:20:00 - ras2pc - INFO - computing finished.ed |  0.1s
2025-06-24 22:20:00 - ras2pc - INFO - dask cluster closed.

mc.pc_pyramid(
    './ds_processing/hix/ds/ds_ph.zarr',
    './ds_processing/hix/ds/ds_ph_geo_pyramid',
    x = ds_e,
    y = ds_n,
    ras_resolution=20,
)

2025-06-24 22:20:00 - log_args - INFO - running function: pc_pyramid
2025-06-24 22:20:00 - log_args - INFO - fetching args:
2025-06-24 22:20:00 - log_args - INFO - pc = './ds_processing/hix/ds/ds_ph.zarr'
2025-06-24 22:20:00 - log_args - INFO - out_dir = './ds_processing/hix/ds/ds_ph_geo_pyramid'
2025-06-24 22:20:00 - log_args - INFO - x = './ds_processing/hix/ds/ds_e.zarr'
2025-06-24 22:20:00 - log_args - INFO - y = './ds_processing/hix/ds/ds_n.zarr'
2025-06-24 22:20:00 - log_args - INFO - yx = None
2025-06-24 22:20:00 - log_args - INFO - ras_resolution = 20
2025-06-24 22:20:00 - log_args - INFO - ras_chunks = (256, 256)
2025-06-24 22:20:00 - log_args - INFO - pc_chunks = 65536
2025-06-24 22:20:00 - log_args - INFO - processes = False
2025-06-24 22:20:00 - log_args - INFO - n_workers = 1
2025-06-24 22:20:00 - log_args - INFO - threads_per_worker = 2
2025-06-24 22:20:00 - log_args - INFO - dask_cluster_arg = {}
2025-06-24 22:20:00 - log_args - INFO - fetching args done.
2025-06-24 22:20:00 - pc_pyramid - INFO - clean out dir
2025-06-24 22:20:00 - zarr_info - INFO - ./ds_processing/hix/ds/ds_ph.zarr zarray shape, chunks, dtype: (293814, 17), (200000, 1), complex64
2025-06-24 22:20:00 - pc_pyramid - INFO - rendering point cloud data coordinates:
2025-06-24 22:20:00 - pc_pyramid - INFO - rasterizing point cloud data to grid with bounds: [-16498471.798140068, 8649651.33657227, -16470131.798140068, 8674811.33657227].
2025-06-24 22:20:00 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_geo_pyramid/x.zarr zarray shape, chunks, dtype: (293814,), (65536,), float64
2025-06-24 22:20:00 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_geo_pyramid/y.zarr zarray shape, chunks, dtype: (293814,), (65536,), float64
2025-06-24 22:20:00 - pc_pyramid - INFO - pc data coordinates rendering ends.
2025-06-24 22:20:00 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_geo_pyramid/idx_0.zarr zarray shape, chunks, dtype: (1259, 1418), (256, 256), int64
2025-06-24 22:20:00 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_geo_pyramid/idx_1.zarr zarray shape, chunks, dtype: (630, 709), (256, 256), int64
2025-06-24 22:20:01 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_geo_pyramid/idx_2.zarr zarray shape, chunks, dtype: (315, 355), (256, 256), int64
2025-06-24 22:20:01 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_geo_pyramid/idx_3.zarr zarray shape, chunks, dtype: (158, 178), (256, 256), int64
2025-06-24 22:20:01 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_geo_pyramid/idx_4.zarr zarray shape, chunks, dtype: (79, 89), (256, 256), int64
2025-06-24 22:20:01 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_geo_pyramid/idx_5.zarr zarray shape, chunks, dtype: (40, 45), (256, 256), int64
2025-06-24 22:20:01 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_geo_pyramid/idx_6.zarr zarray shape, chunks, dtype: (20, 23), (256, 256), int64
2025-06-24 22:20:01 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_geo_pyramid/idx_7.zarr zarray shape, chunks, dtype: (10, 12), (256, 256), int64
2025-06-24 22:20:01 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_geo_pyramid/idx_8.zarr zarray shape, chunks, dtype: (5, 6), (256, 256), int64
2025-06-24 22:20:01 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_geo_pyramid/idx_9.zarr zarray shape, chunks, dtype: (3, 3), (256, 256), int64
2025-06-24 22:20:01 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_geo_pyramid/idx_10.zarr zarray shape, chunks, dtype: (2, 2), (256, 256), int64
2025-06-24 22:20:01 - pc_pyramid - INFO - rasterized idx rendering ends
2025-06-24 22:20:01 - pc_pyramid - INFO - dask local cluster started to render pc data.
2025-06-24 22:20:01 - dask_cluster_info - INFO - dask cluster: LocalCluster(dashboard_link='http://10.211.48.18:8787/status', workers=1, threads=2, memory=1.46 TiB)
2025-06-24 22:20:01 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_geo_pyramid/pc.zarr zarray shape, chunks, dtype: (293814, 17), (65536, 1), complex64
2025-06-24 22:20:01 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_geo_pyramid/0.zarr zarray shape, chunks, dtype: (1259, 1418, 17), (256, 256, 1), complex64
2025-06-24 22:20:01 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_geo_pyramid/1.zarr zarray shape, chunks, dtype: (630, 709, 17), (256, 256, 1), complex64
2025-06-24 22:20:01 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_geo_pyramid/2.zarr zarray shape, chunks, dtype: (315, 355, 17), (256, 256, 1), complex64
2025-06-24 22:20:01 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_geo_pyramid/3.zarr zarray shape, chunks, dtype: (158, 178, 17), (256, 256, 1), complex64
2025-06-24 22:20:01 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_geo_pyramid/4.zarr zarray shape, chunks, dtype: (79, 89, 17), (256, 256, 1), complex64
2025-06-24 22:20:01 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_geo_pyramid/5.zarr zarray shape, chunks, dtype: (40, 45, 17), (256, 256, 1), complex64
2025-06-24 22:20:01 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_geo_pyramid/6.zarr zarray shape, chunks, dtype: (20, 23, 17), (256, 256, 1), complex64
2025-06-24 22:20:01 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_geo_pyramid/7.zarr zarray shape, chunks, dtype: (10, 12, 17), (256, 256, 1), complex64
2025-06-24 22:20:01 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_geo_pyramid/8.zarr zarray shape, chunks, dtype: (5, 6, 17), (256, 256, 1), complex64
2025-06-24 22:20:01 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_geo_pyramid/9.zarr zarray shape, chunks, dtype: (3, 3, 17), (256, 256, 1), complex64
2025-06-24 22:20:01 - zarr_info - INFO - ds_processing/hix/ds/ds_ph_geo_pyramid/10.zarr zarray shape, chunks, dtype: (2, 2, 17), (256, 256, 1), complex64
2025-06-24 22:20:01 - pc_pyramid - INFO - computing graph setted. doing all the computing.
2025-06-24 22:20:02 - pc_pyramid - INFO - computing finished.  1.1s
2025-06-24 22:20:02 - pc_pyramid - INFO - dask cluster closed.

import toml
with open('load_data/meta.toml','r') as f:
    dates = toml.load(f)['dates']
ds_geo_intf_plot = mc.pc_plot('./ds_processing/hix/ds/ds_ph_geo_pyramid',post_proc_ras='intf_all', post_proc_pc='intf_all',level_increase=1)
ds_geo_intf_plot = ds_geo_intf_plot.redim(
    i=hv.Dimension('i', label='Reference Image', range=(0,16), value_format=(lambda i: dates[i])),
    j=hv.Dimension('j', label='Secondary Image', range=(0,16), value_format=(lambda i: dates[i])),
    x=hv.Dimension('lon', label='Longitude'),
    y=hv.Dimension('lat',label='Latitude'),
    z=hv.Dimension('Phase',range=(-np.pi,np.pi))
)

hv.element.tiles.EsriImagery()*ds_geo_intf_plot.opts(
    hv.opts.Image(
        cmap='colorwheel',frame_width=500, frame_height=400, colorbar=True,
        default_tools=['pan',WheelZoomTool(zoom_on_axis=False),'save','reset','hover'],
        active_tools=['wheel_zoom']
    ),
    hv.opts.Points(
        color='Phase', cmap='colorwheel',frame_width=500, frame_height=400, colorbar=True,
        default_tools=['pan',WheelZoomTool(zoom_on_axis=False),'save','reset','hover'],
        active_tools=['wheel_zoom']
    ),
)