dl: Deep learning operators (CLI)

Commands of moraine.cli.dl (source); the same functions are called in Python as mc.<name>(...) with import moraine.cli as mc. The help below is generated from the docstrings, like moraine COMMAND --help.

command	summary
n2f	Noise2Fringe (n2f) filtering of raster interferograms
n2ft	Noise2Fringe Transformer (n2ft) filtering of point cloud interferograms

n2f

Noise2Fringe (n2f) filtering of raster interferograms. With cuda the network runs in half precision. A task filters one output chunk with all image pairs: it holds the rslc of one processing chunk with its overlap for all images (8 bytes per pixel and image), the filtered interferograms of the output chunk (8 bytes per pixel and image pair) and the model's activations of one interferogram (about 1 kB per pixel of the processing chunk on the GPU, 2 kB on the CPU); with cuda all of it on the GPU. A worker runs threads_per_worker tasks at a time.

moraine n2f --rslc STR --intf STR --image_pairs FILE [--chunks INT INT] [--out_chunks INT INT] [--depths INT INT] [--model STR] [--cuda | --no-cuda] [--processes VALUE] [--n_workers VALUE] [--threads_per_worker VALUE]

argument	type	default	description
--rslc	str	required	input rslc stack, shape (nlines, width, nimages)
--intf	str	required	output filtered interferograms, complex64 with unit amplitude (nan where the input is nan), shape (nlines, width, n_image_pairs)
--image_pairs	np.ndarray	required	input image pairs (reference, secondary), shape (n_image_pairs, 2); make a file with moraine image-pairs
--chunks	tuple[int, int]	None	(azimuth, range) processing chunk size, same as rslc by default; the processing chunks are cut at the output chunks
--out_chunks	tuple[int, int]	None	(azimuth, range) chunk size of the output, same as rslc by default; it sets the memory of a worker
--depths	tuple[int, int], 0)	(0, 0)	(azimuth, range) overlap in pixels between processing chunks, reduces chunk border effects
--model	str	None	path to the model weights (.pth), use the model comes with this package by default
--cuda	bool	False	if use cuda for processing, false by default
--processes		None	use processes (True) or threads (False) for the workers, only for cpu processing. Default: True
--n_workers		None	number of workers. Default: 1 for cpu, one per GPU for cuda
--threads_per_worker		None	output chunks a worker filters at the same time. Default: 1

Global options: --json, --traceback, --log FILE, -q / --quiet (conventions).

In a pipeline (examples/02_ps.toml, step n2f):

[[step]]
name = "n2f"                      # Noise2Fringe filtering of the sequential interferograms
run = "n2f"
rslc = "raw/rslc.zarr"
intf = "ps/ras_n2f_intf.zarr"
image_pairs = "ps/pairs_seq.txt"

In Python:

import moraine.cli as mc
mc.n2f(rslc, intf, image_pairs, chunks=None, out_chunks=None, depths=(0, 0), model=None, cuda=False, processes=None, n_workers=None, threads_per_worker=None)

The array function mr.n2f does the same on numpy / cupy arrays in memory.

n2ft

Noise2Fringe Transformer (n2ft) filtering of point cloud interferograms. Every worker filters one output chunk of points with all image pairs at a time: it holds the rslc of the points of the chunk and of their halos for the images of the pairs (8 bytes per point and image), the filtered interferograms of the chunk (8 bytes per point and image pair) and about 400 bytes per point of the chunk more; with cuda a model call holds about a fifth of the GPU memory. Before, the main process finds the halos of all processing chunks with about 70 bytes per point and 32 threads of 16 x k x chunks bytes (41 MB each by default).

moraine n2ft --x STR --y STR --rslc STR --intf STR --image_pairs FILE [--chunks VALUE] [--out_chunks VALUE] [--k VALUE] [--model STR] [--cuda | --no-cuda] [--compile | --no-compile] [--processes VALUE] [--n_workers VALUE] [--threads_per_worker VALUE]

argument	type	default	description
--x	str	required	input x coordinate of the points (e.g. longitude or easting), shape (n_points,)
--y	str	required	input y coordinate of the points (e.g. latitude or northing), shape (n_points,)
--rslc	str	required	input rslc of the points, shape (n_points, nimages)
--intf	str	required	output filtered interferograms, complex64 with unit amplitude, shape (n_points, n_image_pairs)
--image_pairs	np.ndarray	required	input image pairs (reference, secondary), shape (n_image_pairs, 2); make a file with moraine image-pairs
--chunks	int	None	number of points per processing chunk, same as rslc by default; the processing chunks do not cross the output chunks
--out_chunks	int	None	point chunk size of the output, same as rslc by default; it sets the memory of a worker
--k	int	128	number of nearest neighbours of every point of a processing chunk that are filtered with it (halo)
--model	str	None	path to the model weights (.pth), use the model comes with this package by default
--cuda	bool	False	if use cuda for processing, false by default
--compile	bool	None	compile the model with torch.compile in every worker: the model then runs about 3 times faster on a GPU, but the compilation takes 15-40 s per worker (less when torch has cached it on disk). Default: when points times image pairs is at least 1e8
--processes		None	use processes (True) or threads (False) for the workers, only for cpu processing. Default: True
--n_workers		None	number of workers. Default: 1 for cpu, one per GPU for cuda
--threads_per_worker		None	tasks a worker runs at the same time. Default: 1

Global options: --json, --traceback, --log FILE, -q / --quiet (conventions).

In a pipeline (examples/04_refine.toml, step n2ft):

[[step]]
name = "n2ft"                     # Noise2Fringe Transformer filtering of the sequential interferograms
run = "n2ft"
x = "pc/pc_can_lon.zarr"
y = "pc/pc_can_lat.zarr"
rslc = "pc/pc_can_ph.zarr"
intf = "pc/pc_can_n2ft_intf.zarr"
image_pairs = "pc/pairs_seq.txt"
chunks = 20000

In Python:

import moraine.cli as mc
mc.n2ft(x, y, rslc, intf, image_pairs, chunks=None, out_chunks=None, k=128, model=None, cuda=False, compile=None, processes=None, n_workers=None, threads_per_worker=None)

The array function mr.n2ft does the same on numpy / cupy arrays in memory.