python - How to vectorize/tensorize operations in numpy with irregular array shapes -

i perform operation

if had regular shape, use np.einsum, believe syntax

np.einsum('ijp,ipk->ijk',x, alpha) 

unfortunately, data x has non regular structure on 1st (if 0 index) axis.

to give little more context, refers p^th feature of j^th member of i^th group. because groups have different sizes, effectively, list of lists of different lengths, of lists of same length.

has regular structure , can saved standard numpy array (it comes in 1-dimensional , use alpha.reshape(a,b,c) a,b,c problem specific integers)

i avoid storing x list of lists of lists or list of np.arrays of different dimensions , writing like

a = [] in range(num_groups):     temp = np.empty(group_sizes[i], dtype=float)     j in range(group_sizes[i]):         temp[i] = np.einsum('p,pk->k',x[i][j], alpha[i,:,:])     a.append(temp) 

is nice numpy function/data structure doing or going have compromise partially vectorised implementation?

i know sounds obvious, but, if can afford memory, i'd start checking performance padding data have uniform size, is, adding zeros , perform operation. simpler solution faster more supposedly optimal 1 has more python/c roundtrips.

if doesn't work, best bet, tom wyllie suggested, bucketing strategy. assuming x list of lists of lists , alpha array, can start collecting sizes of second index (maybe have this):

x_sizes = np.array([len(x_i) x_i in x]) 

and sort them:

idx_sort = np.argsort(x_sizes) x_sizes_sorted = x_sizes[idx_sort] 

then choose number of buckets, number of divisions of work. let's pick buckets = 4. need divide data more or less each piece same size:

sizes_cumsum = np.cumsum(x_sizes_sorted) total = sizes_cumsum[-1] bucket_idx = [] in range(buckets):     low = np.round(i * total / float(buckets))     high = np.round((i + 1) * total / float(buckets))     m = sizes_cumsum >= low & sizes_cumsum < high     idx = np.where(m),     # make relative x, not idx_sort     idx = idx_sort[idx]     bucket_idx.append(idx) 

and make computation each bucket:

bucket_results = [] idx in bucket_idx:     # last index in bucket biggest     bucket_size = x_sizes[idx[-1]]     # fill bucket array     x_bucket = np.zeros((len(x), bucket_size, len(x[0][0])), dtype=x.dtype)     i, x_i in enumerate(idx):         x_bucket[i, :x_sizes[x_i]] = x[x_i]     # compute     res = np.einsum('ijp,ipk->ijk',x, alpha[:, :bucket_size, :])     bucket_results.append(res) 

filling array x_bucket slow in part. again, if can afford memory, more efficient have x in single padded array , slice x[idx, :bucket_size, :].

finally, can put results list:

result = [none] * len(x) res, idx in zip(bucket_results, bucket_idx):     r, x_i in zip(res, idx):         result[x_i] = res[:x_sizes[x_i]] 

sorry i'm not giving proper function, i'm not sure how input or expected output put pieces , can use them see fit.