script_tools

library of functions helpful for cli script development and parallel computing particulary with subprocess calls.

clasp.script_tools.try_mkdir(s)

silently ignore exceptions on mkdir

clasp.script_tools.arange(start, stop=None, step=1)

like numpy.arange for integers

clasp.script_tools.int_rng(s)

expand start:end:inc notation into range

clasp.script_tools.rm_dup(seq)

removes duplicates from list while preserving order

clasp.script_tools.warn_match(kwargs, sargs)

clasp.script_tools.kwarg_match(func, kwargs, debug=False)

filters dict for keys used by func

clasp.script_tools.arg_match(func, kwargs, *args)

filters dict for positional arguments used by func

clasp.script_tools.kwarg_arg(func, kwargs, skip=None)

returns ordered list of optional arg values

clasp.script_tools.crossref(l1, l2)

return all possible pairs of 2 lists

clasp.script_tools.crossref_all(l, followers=[])

return all possible combos of list of lists

clasp.script_tools.subpipe(commands)

parses special syntax in pipe expressions

$(some command) executes to a temporary file whose path is inserted in

the command.

$((expression)) evaluates a arithmetic expression in place +-*/()

clasp.script_tools.pipeline(commands, outfile=None, inp=None, close=False, cwd=None, writemode='w', forceinpfile=False, caperr=False)

executes pipeline of shell commands (given as list of strings)

special syntax:

$(some command) executes to a temporary file whose path is inserted in

the command.

$((expression)) evaluates a arithmetic expression in place +-*/()

Parameters:

• commands (list) – list of commands to execute in order

• outfile (writeable file object) – optional destination for stdout

• inp (str or filebuffer) – string to feed to stdin at start of pipeline

• close (bool) – if true closes file object before returning

• cwd (str) – directory to execute pipeline (temp files and Popen cwd)

• writemode (str) – passed to open() for outfile (‘w’, ‘wb’ for write or ‘a’ for append)

• forceinpfile (bool) – always treat inp as a file, if a string, open the path for reading

Returns:

out – returns stdout of pipeline (will be None if outfile is given)

Return type:

str

clasp.script_tools.flat_list(l)

flattens any depth list

clasp.script_tools.pool_call(func, args, kwargs={}, cwd=None, order=True, expand=False, handle=False, test=False)

execute func with concurrent.futures return output

Parameters:

• func (python function) – function to execute

• args (list of tuples) – each set is mapped to function

• kwargs (dict) – constant keyword args for func

• cwd (str) – directory in which to execute function calls

• order (bool) – whether to maintain order of input

• expand (bool) – whether to expand items in args to map to function args

• handle (bool) – whether to return future objects or results

Return type:

list of results unless handle=True then returns iterable of futures

clasp.script_tools.cluster_call(func, args, kwargs={}, timeout=0.1, cwd=None, debug=False)

for backwards compatibility only

clasp.script_tools.read_epw(epw)

read daylight sky data from epw or wea file

Returns:

out – (month, day, hour, dirnorn, difhoriz, globhoriz, skycover)

Return type:

tuple

clasp.script_tools.isnum(s)

test if input can be converted to float

clasp.script_tools.try_float(s)

attempt conversion to float

clasp.script_tools.coerce_data(datastr, i_vals, dataf, coerce=True)

ensure all data points parsed are valid numbers

clasp.script_tools.get_i(i, d_vals)

if (x, y) return y if x == i if y return y

clasp.script_tools.read_data_file(dataf, header=False, xheader=False, comment='#', delim='\t, ', coerce=True)

clasp.script_tools.read_data(dataf, x_vals=[0], y_vals=[-1], rows=False, header=False, weax=None, reverse=False, autox=None, comment='#', xheader=False, delim='\t, ', coerce=True, weatherfile=False, drange=None)

read generic csv/tsv data file

Parameters:

• dataf (str) – file to read data from

• x_vals (list of ints) – column (or row with rows=True) indices for x values

• y_vals (list of ints) – column (or row with rows=True) indices for y values

• rows (Boolean) – if True read data in rows

• header (Boolean) – return first row (or column with rows=True) as series labels

• weax (2 item list of ints) – idx for month and day to use day number as x_vals, if given ignores x_val

• reverse (Boolean) – reverse order of data (use with autox)

• autox (Boolean) – assigns integers (starting at 0) as x_vals

• comment (str) – comment line signifiers (inserted in regex ^[comment].*)

• delim (str) – delimeters for parsing data (inserted in regex [delim]+)

• coerce (Boolean) – raise exception if all values are are not numbers

• weatherfile (str of file path) – handles wea and epw file formates returning daylight parameters

• drange (list of ints) – limit series output to given indices.

Returns:

• datax (list) – list of x_vals for each y_val (pads with last item if necessary) if there are more x_vals than y_vals does not return excess datax

• datay (list) – list for each y_val

• head (list) – if header=True list of labels for each y_val else []

clasp.script_tools.read_all_data(datafs, x_vals=[], y_vals=[], **kwargs)

read multiple data files and pair x and y data call read_data

Parameters:

• datafs (list of str) – files to read data from

• x_vals (list of ints or tuple int pairs) – (fileidx, colidx) or colidx to read from each file

• y_vals (list of ints or tuple int pairs) – (fileidx, colidx) or colidx to read from each file

• kwargs – optional arguments for read_data

Returns:

• datax (list) – list of x_vals for each y_val (pads with last item if necessary)

• datay (list) – list for each y_val

• head (list) – if header=True list of labels for each y_val else []

clasp.script_tools.clean_tmp(ctx)

clasp.script_tools.expandpat(pat, s, mark=0)

expand sglob pattern for each character option

Parameters:

• pat (regex) – regex pattern to split on

• s (str) – string to split

• mark (int) – 0: include splitting mark in output 1: skip splitting mark (assume 1 character in length)

Returns:

allpat – list of strings enumerating all possible combinations of pattern

Return type:

list of strings

clasp.script_tools.sglob(s)

super glob includes [abc] notation + [!abc] exclude notation