Create my own model database

[oMersa344]

Hi,
As I understand, if I want to use the visualization functions of micom, I need to analyze the data with the micom workflow, and the grow function.
To use the manifest function, I need model db, which I want to build with the existing model I’ve created with CarveMe. However, the link - create your own database <>__ is empty and leads to the MICOM workflow page.
Is it possible to build my own db from the existing models? If yes, How can I do it?
Thank you

[cdiener]

Oh yeah sorry, that is not well documented in the MICOM docs. Without QIIME you would use the build_database function. The q2-micom tutorial has a bit more on that.

Alternatively, you can use build without a model database. For that, your MICOM taxonomy table has to include a column named file that points to a metabolic model in SBML format and that is consistent with the used taxonomic rank. Though, the recommendation would be to assemble a DB first. It makes build much faster, performs a lot of checks and fixes, and is easier to publish.

[oMersa344]

Thank you.
I tried using build_database function with a manifest file as requested to build the database.

However, I get an error while using the function -

sourdoughDB = build_database(manifest=mani,
                             out_path='/Users/kashilab/Documents/Omer_Sabach/impact_rich_media/joined_models',
                             threads=10,
                             progress=True)

Traceback (most recent call last):
  File "/Users/kashilab/Documents/Omer_Sabach/impact_rich_media/draft.py", line 19, in <module>
    sourdoughDB = build_database(manifest=mani,
  File "/Users/kashilab/opt/anaconda3/envs/impact_rich_media/lib/python3.9/site-packages/micom/workflows/build.py", line 179, in build_database
    bad = meta.file.apply(lambda x: not os.path.exists(x))
  File "/Users/kashilab/opt/anaconda3/envs/impact_rich_media/lib/python3.9/site-packages/pandas/core/series.py", line 4357, in apply
    return SeriesApply(self, func, convert_dtype, args, kwargs).apply()
  File "/Users/kashilab/opt/anaconda3/envs/impact_rich_media/lib/python3.9/site-packages/pandas/core/apply.py", line 1043, in apply
    return self.apply_standard()
  File "/Users/kashilab/opt/anaconda3/envs/impact_rich_media/lib/python3.9/site-packages/pandas/core/apply.py", line 1098, in apply_standard
    mapped = lib.map_infer(
  File "pandas/_libs/lib.pyx", line 2859, in pandas._libs.lib.map_infer
  File "/Users/kashilab/opt/anaconda3/envs/impact_rich_media/lib/python3.9/site-packages/micom/workflows/build.py", line 179, in <lambda>
    bad = meta.file.apply(lambda x: not os.path.exists(x))
  File "/Users/kashilab/opt/anaconda3/envs/impact_rich_media/lib/python3.9/genericpath.py", line 19, in exists
    os.stat(path)
TypeError: stat: path should be string, bytes, os.PathLike or integer, not float

It seems that no matter what method I use to insert the file output path, I still get the same error.
I even tried using just '/Users' with additional folders, and it still doesn't work.
Can you please help me?

[nigiord]

Hi @oMersa344, I might be wrong but from the error message I would say the problem is in the manifest dataframe (mani variable) and not the out_path parameters. Apparently the entries of your file column are interpreted as float, so the program is not able to check if the files exist or not.

You might want to double check what's inside the mani variable.

[oMersa344]

Hi @nigiord,
you were right, indeed the problem was in the manifest dataframe.
As I fixed it, I get a new error -

/Users/kashilab/opt/anaconda3/envs/impact_rich_media/lib/python3.9/site-packages/micom/workflows/build.py:188: FutureWarning: The default value of regex will change from True to False in a future version.
  meta.index = meta[rank].str.replace("[^\\w\\_]", "_")
/Users/kashilab/opt/anaconda3/envs/impact_rich_media/lib/python3.9/site-packages/micom/workflows/build.py:188: FutureWarning: The default value of regex will change from True to False in a future version.
  meta.index = meta[rank].str.replace("[^\\w\\_]", "_")
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/kashilab/opt/anaconda3/envs/impact_rich_media/lib/python3.9/multiprocessing/spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "/Users/kashilab/opt/anaconda3/envs/impact_rich_media/lib/python3.9/multiprocessing/spawn.py", line 125, in _main
    prepare(preparation_data)
  File "/Users/kashilab/opt/anaconda3/envs/impact_rich_media/lib/python3.9/multiprocessing/spawn.py", line 236, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "/Users/kashilab/opt/anaconda3/envs/impact_rich_media/lib/python3.9/multiprocessing/spawn.py", line 287, in _fixup_main_from_path
    main_content = runpy.run_path(main_path,
  File "/Users/kashilab/opt/anaconda3/envs/impact_rich_media/lib/python3.9/runpy.py", line 268, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "/Users/kashilab/opt/anaconda3/envs/impact_rich_media/lib/python3.9/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/Users/kashilab/opt/anaconda3/envs/impact_rich_media/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/kashilab/Documents/Omer_Sabach/impact_rich_media/draft.py", line 20, in <module>
    sourdoughDB = build_database(manifest=mani,
  File "/Users/kashilab/opt/anaconda3/envs/impact_rich_media/lib/python3.9/site-packages/micom/workflows/build.py", line 237, in build_database
    workflow(_summarize_models, args, threads)
  File "/Users/kashilab/opt/anaconda3/envs/impact_rich_media/lib/python3.9/site-packages/micom/workflows/core.py", line 37, in workflow
    with Pool(processes=n_jobs, maxtasksperchild=1) as pool:
  File "/Users/kashilab/opt/anaconda3/envs/impact_rich_media/lib/python3.9/multiprocessing/context.py", line 119, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
  File "/Users/kashilab/opt/anaconda3/envs/impact_rich_media/lib/python3.9/multiprocessing/pool.py", line 212, in __init__
    self._repopulate_pool()
  File "/Users/kashilab/opt/anaconda3/envs/impact_rich_media/lib/python3.9/multiprocessing/pool.py", line 303, in _repopulate_pool
    return self._repopulate_pool_static(self._ctx, self.Process,
  File "/Users/kashilab/opt/anaconda3/envs/impact_rich_media/lib/python3.9/multiprocessing/pool.py", line 326, in _repopulate_pool_static
    w.start()
  File "/Users/kashilab/opt/anaconda3/envs/impact_rich_media/lib/python3.9/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/Users/kashilab/opt/anaconda3/envs/impact_rich_media/lib/python3.9/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/Users/kashilab/opt/anaconda3/envs/impact_rich_media/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/Users/kashilab/opt/anaconda3/envs/impact_rich_media/lib/python3.9/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/Users/kashilab/opt/anaconda3/envs/impact_rich_media/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 42, in _launch
    prep_data = spawn.get_preparation_data(process_obj._name)
  File "/Users/kashilab/opt/anaconda3/envs/impact_rich_media/lib/python3.9/multiprocessing/spawn.py", line 154, in get_preparation_data
    _check_not_importing_main()
  File "/Users/kashilab/opt/anaconda3/envs/impact_rich_media/lib/python3.9/multiprocessing/spawn.py", line 134, in _check_not_importing_main
    raise RuntimeError('''
RuntimeError: 
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.

I've copied the error once, but it keeps jumping again and again while running.

[oMersa344]

A similar error occur when I'm trying using build function without a pre-built database:
manifest = build(all_com, out_folder="built_models", model_db=None, cutoff=0.0001, threads=10)

I get this error which keep jumping again and again:

  File "<string>", line 1, in <module>
  File "/Users/kashilab/opt/anaconda3/envs/impact_rich_media/lib/python3.9/multiprocessing/spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "/Users/kashilab/opt/anaconda3/envs/impact_rich_media/lib/python3.9/multiprocessing/spawn.py", line 125, in _main
    prepare(preparation_data)
  File "/Users/kashilab/opt/anaconda3/envs/impact_rich_media/lib/python3.9/multiprocessing/spawn.py", line 236, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "/Users/kashilab/opt/anaconda3/envs/impact_rich_media/lib/python3.9/multiprocessing/spawn.py", line 287, in _fixup_main_from_path
    main_content = runpy.run_path(main_path,
  File "/Users/kashilab/opt/anaconda3/envs/impact_rich_media/lib/python3.9/runpy.py", line 268, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "/Users/kashilab/opt/anaconda3/envs/impact_rich_media/lib/python3.9/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/Users/kashilab/opt/anaconda3/envs/impact_rich_media/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/kashilab/Documents/Omer_Sabach/impact_rich_media/draft2.py", line 46, in <module>
    manifest = build(all_com, out_folder="built_models", model_db=None, cutoff=0.0001, threads=10)
  File "/Users/kashilab/opt/anaconda3/envs/impact_rich_media/lib/python3.9/site-packages/micom/workflows/build.py", line 93, in build
    res = workflow(build_and_save, args, threads)
  File "/Users/kashilab/opt/anaconda3/envs/impact_rich_media/lib/python3.9/site-packages/micom/workflows/core.py", line 37, in workflow
    with Pool(processes=n_jobs, maxtasksperchild=1) as pool:
  File "/Users/kashilab/opt/anaconda3/envs/impact_rich_media/lib/python3.9/multiprocessing/context.py", line 119, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
  File "/Users/kashilab/opt/anaconda3/envs/impact_rich_media/lib/python3.9/multiprocessing/pool.py", line 212, in __init__
    self._repopulate_pool()
  File "/Users/kashilab/opt/anaconda3/envs/impact_rich_media/lib/python3.9/multiprocessing/pool.py", line 303, in _repopulate_pool
    return self._repopulate_pool_static(self._ctx, self.Process,
  File "/Users/kashilab/opt/anaconda3/envs/impact_rich_media/lib/python3.9/multiprocessing/pool.py", line 326, in _repopulate_pool_static
    w.start()
  File "/Users/kashilab/opt/anaconda3/envs/impact_rich_media/lib/python3.9/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/Users/kashilab/opt/anaconda3/envs/impact_rich_media/lib/python3.9/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/Users/kashilab/opt/anaconda3/envs/impact_rich_media/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/Users/kashilab/opt/anaconda3/envs/impact_rich_media/lib/python3.9/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/Users/kashilab/opt/anaconda3/envs/impact_rich_media/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 42, in _launch
    prep_data = spawn.get_preparation_data(process_obj._name)
  File "/Users/kashilab/opt/anaconda3/envs/impact_rich_media/lib/python3.9/multiprocessing/spawn.py", line 154, in get_preparation_data
    _check_not_importing_main()
  File "/Users/kashilab/opt/anaconda3/envs/impact_rich_media/lib/python3.9/multiprocessing/spawn.py", line 134, in _check_not_importing_main
    raise RuntimeError('''
RuntimeError: 
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.

all_com is a dataframe looks like

   sample_id           id                                               file  \
1  community1    l_sanfran  /Users/kashilab/Documents/Omer_Sabach/impact_r...   
2  community1  l_plantarum  /Users/kashilab/Documents/Omer_Sabach/impact_r...   
3  community1     l_brevis  /Users/kashilab/Documents/Omer_Sabach/impact_r...   
0  community1        S288c  /Users/kashilab/Documents/Omer_Sabach/impact_r...   
1  community2    l_sanfran  /Users/kashilab/Documents/Omer_Sabach/impact_r...   

   abundance           species  
1        0.2  sanfranciscensis  
2        0.2         plantarum  
3        0.2            brevis  
0        0.4        cerevisiae  
1        0.2  sanfranciscensis  

[cdiener]

Sorry, thought I already replied. The errors you are seeing are coming from multiprocessing (from the Python standard library) and have something to do how processes are spawned on your machine. Those are always hard to diagnose since they are often specific to the operating system and version and how you are executing your code.
Are you running this in a Python script or in Jupyter? If it is a script it should be called in the the main method and you could try the fix the error suggests. Setting threads=1 will also work but this would be slower.

[cdiener]

I added some recent fixes to make avoid some deadlocks. You could try with a recent version to see if that gets around those issues. 🤞