Importing graphs from Seurat

I posted a comment on the issue: https://github.com/theislab/scanpy/issues/598#issuecomment-653220911
which I am hoping to get some help with. To read in the Loom file created from Seurat, one must remove the graphs (FindNeighbors) from Seurat (according to the responses of that issue). However, I would like to use this from Seurat to maintain consistency rather than having scanpy recompute. I am able to extract and save the sparse matrix containing the graph information from Seurat and load it into Python (e.g. with pandas). I am wondering how I can reload this object back into the resulting scanpy object from reading the Loom file?