My dataset is too big (5M cells) to process in memory.
I know about backed feature and chunked loading, but can I save data back to h5ad in chunks?
Here is more concrete example of me normalizing large data file:
chunk_size =100 ad = sc.read_h5ad(file_name, backed='r+') for chunk, start, end in ad.chunked_X(chunk_size): sc.pp.normalize_total(AnnData(chunk), target_sum=1e6, inplace=True)
but how can I write the processed chunks incrementally to a new file?