Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
63 commits
Select commit Hold shift + click to select a range
8743837
update
Roy-Kid Feb 6, 2024
9468b2c
rewrite proj
Roy-Kid Feb 17, 2024
1a0e03a
update
Roy-Kid Feb 28, 2024
b69e7fb
update script
Roy-Kid Feb 28, 2024
f8ce937
add cache
Roy-Kid Feb 28, 2024
bc1c2a9
using pysqa to generate script and submit
Roy-Kid Mar 1, 2024
f6ff0dd
using pysqa to generate script and submit
Roy-Kid Mar 1, 2024
42ff7f0
using Exp name instead of hardcode
Roy-Kid Mar 1, 2024
a2e611d
add real example
Roy-Kid Mar 2, 2024
a376167
fix bug
Roy-Kid Mar 2, 2024
b27815e
update slurm example
Roy-Kid Mar 4, 2024
a99aca9
rewrite submitor
Roy-Kid Mar 21, 2024
c597d4b
Merge branch 'master' of https://github.com/MolCrafts/molexp
Roy-Kid Mar 21, 2024
8eda858
add monitor and project
Roy-Kid Mar 25, 2024
0389ee6
using ExpTracker to seperated mkdir
Roy-Kid Mar 27, 2024
3da026d
update proj
Roy-Kid Mar 27, 2024
663a4ef
use resolve to parallel exec exp
Roy-Kid Apr 2, 2024
147e9eb
remove central monitor
Roy-Kid Apr 2, 2024
e96185c
fix resolve problem
Roy-Kid Apr 12, 2024
2cb9db9
fix conflict
Roy-Kid Apr 12, 2024
b0ee3bc
add custom tracker
Roy-Kid Apr 26, 2024
bbd126c
update map reduce method
Roy-Kid Apr 29, 2024
c8b31c6
customize tracker
Roy-Kid May 10, 2024
ccbcbb7
back to use hamilton.experimenttracker instead customized
Roy-Kid May 11, 2024
fec0a99
Changes how the project module is run
skrawcz May 11, 2024
151588f
Fixes parallelization
skrawcz May 11, 2024
9ae1589
Updates doc string
skrawcz May 11, 2024
8b6eed8
Merge pull request #7 from MolCrafts/attempt_fix_serialization
Roy-Kid May 12, 2024
d1a7742
add resume
Roy-Kid May 14, 2024
23e7dd2
rewrite serial proj
Roy-Kid May 17, 2024
93686fb
fix save_to usage bug
Roy-Kid May 17, 2024
f4749ca
refine query exp
Roy-Kid May 17, 2024
ee12304
[feat] script
Roy-Kid May 17, 2024
13956a8
add tag(cd_run_dir) to control where node is working at
Roy-Kid May 17, 2024
659d34c
[adapter] introduce hamilton ui
Roy-Kid May 31, 2024
8e2fca5
[proj] using parallelizable instead of parameterize
Roy-Kid Jun 2, 2024
f3d4288
[proj] add cache support
Roy-Kid Jun 17, 2024
30c8e75
redesign
Roy-Kid Jul 16, 2024
4c7e2ba
resume from an exist task
Roy-Kid Jul 16, 2024
d1681b4
remove proj.resume_task, add exp.resume_task
Roy-Kid Jul 17, 2024
01cff4a
add tear down
Roy-Kid Jul 17, 2024
85701d0
seperate task module
Roy-Kid Jul 18, 2024
86f7d44
merge resume and def test
Roy-Kid Jul 18, 2024
c428f27
clean up code
Roy-Kid Jul 18, 2024
6d57fa0
bug when serialization
Roy-Kid Jul 18, 2024
44171c9
add asset
Roy-Kid Jul 19, 2024
8eead1e
cases
Roy-Kid Jul 19, 2024
7e69610
rewrite exp, proj, task
Roy-Kid Aug 1, 2024
d87f989
redesign logic
Roy-Kid Aug 1, 2024
46fd3da
update
Roy-Kid Aug 1, 2024
bbfd6af
add start_tasks support
Roy-Kid Aug 5, 2024
cbe3876
decouple all level
Roy-Kid Aug 16, 2024
48e741a
refactor
Roy-Kid Aug 19, 2024
df1d2b9
add map reduce
Roy-Kid Aug 21, 2024
1aec5c5
update deps check
Aug 21, 2024
cf8c2a5
update deps check
Aug 21, 2024
a9879bd
update state and template
Roy-Kid Aug 21, 2024
883c709
fix map_reduce
Aug 21, 2024
cc9d466
Merge branch 'dev' of https://github.com/MolCrafts/molexp into dev
Roy-Kid Aug 21, 2024
2a895d4
update
Roy-Kid Aug 29, 2024
f16feb1
dardel update
Aug 29, 2024
44a477a
fix
Aug 29, 2024
ebe919d
update
Sep 2, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .devcontainer/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
FROM mcr.microsoft.com/devcontainers/miniconda:1-3

RUN sh -c "$(wget -O- https://github.com/deluan/zsh-in-docker/releases/download/v1.2.0/zsh-in-docker.sh)" -- \
-t nanotech
34 changes: 34 additions & 0 deletions .devcontainer/devcontainer.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
// For format details, see https://aka.ms/devcontainer.json. For config options, see the
// README at: https://github.com/devcontainers/templates/tree/main/src/anaconda
{
"name": "MolExp",
"build": {
"context": ".",
"dockerfile": "Dockerfile"
},
// Features to add to the dev container. More info: https://containers.dev/features.
// "features": {
// "ghcr.io/devcontainers/features/conda:1": {}
// },
// Use 'forwardPorts' to make a list of ports inside the container available locally.
// "forwardPorts": [],
// Use 'postCreateCommand' to run commands after the container is created.
"postCreateCommand": "pip install -e \".[dev]\"",
// Configure tool-specific properties.
// "customizations": {},
"customizations": {
"vscode": {
"extensions": [
"ms-python.python",
"ms-python.black-formatter",
"ms-python.isort",
"ms-toolsai.jupyter",
"github.copilot",
"ms-vscode.test-adapter-converter",
"njpwerner.autodocstr"
]
}
},
// Uncomment to connect as root instead. More info: https://aka.ms/dev-containers-non-root.
"remoteUser": "root"
}
3 changes: 3 additions & 0 deletions .vscode/settings.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{
"python.analysis.autoImportCompletions": true
}
33 changes: 31 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,2 +1,31 @@
# molexp
[WIP] Molecular experiment managerment
# molexp: a computational chemistry workflow and experiment management package

`molexp` is a comprehensive computational chemistry workflow and experiment management package, designed to streamline and enhance the efficiency of chemistry-related research and experimentation. At its core, `molexp` leverages the robust capabilities of [Hamilton](https://github.com/DAGWorks-Inc/hamilton), allowing users to define workflows in a manner that is testable, modular, and self-documenting. This ensures that all workflows are easy to understand, maintain, and share, as they are written in pure Python code.

One of the standout features of `molexp` is its advanced experiment management system. This system is meticulously designed to facilitate self-explanatory task organization, making it simple for researchers to manage and document their experiments effectively. Users can perform Create, Read, Update, and Delete (CRUD) operations directly from the command line, using specific variables to tailor the management process to their unique requirements.

## Features

1. **Integration with Hamilton**: By utilizing Hamilton, `molexp` ensures that workflows are not only powerful and flexible but also maintain high standards of code quality and reusability.

2. **Modular Workflow Design**: Workflows in `molexp` are built in a modular fashion, promoting reusability and ease of testing. This modularity allows for individual components to be tested and debugged independently, significantly reducing the time and effort required to ensure the reliability of complex workflows.

3. **Self-Documenting Code**: The use of pure Python code means that all workflows and experiments are inherently self-documenting. This transparency is crucial for collaborative research environments, where clear documentation and easy-to-follow code are essential for effective teamwork.

4. **Comprehensive Experiment Management**: The experiment management system in `molexp` is designed to simplify the organization of tasks. It supports comprehensive CRUD operations via the command line, allowing users to create, view, modify, and delete experiments with ease.

5. **Customizable Variables**: The system’s flexibility is further enhanced by the ability to use specific variables to manage experiments. This feature allows users to tailor their workflow management processes to suit their specific needs and preferences.

6. **User-Friendly Command Line Interface**: The command line interface of `molexp` is intuitive and user-friendly, ensuring that even users with minimal programming experience can effectively manage their computational chemistry workflows and experiments.

7. **Scalability and Flexibility**: `molexp` is designed to handle projects of varying sizes and complexities. Whether you are managing a small-scale experiment or a large, multifaceted research project, `molexp` scales to meet your needs.

8. **Community and Support**: As an open-source package, `molexp` benefits from a growing community of users and contributors. This ensures continuous improvements, updates, and a wealth of shared knowledge and resources.

## Concepts

`project`: a project represents a stand alone research topic, and contains necessary index and doc files

`Experiment`: an experiment is new science trial with new variable

`Task`: several tasks in an experiment, each task is a complete workflow or graph or executation.
141 changes: 141 additions & 0 deletions docs/getting_started.ipynb
Original file line number Diff line number Diff line change
@@ -0,0 +1,141 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Getting Startted with MolExp\n",
"\n",
"let's we start from hierarchy "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Three level "
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"%load_ext autoreload\n",
"%autoreload 2"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"import molexp as me\n",
"\n",
"proj = me.Project(\n",
" name='tutorial_proj',\n",
" description='A tutorial project',\n",
" root='.'\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"exp1 = proj.def_exp(\n",
" name = 'exp_1',\n",
" description = 'test exp',\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"<Asset: global_asset>"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"proj.add_asset(\n",
" 'global_asset'\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"<Task: task_1>"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"exp1.def_task(\n",
" name = 'task_1',\n",
" description = 'test task'\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
"proj.init()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"exp2 = exp1.clone()"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "base",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.2"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
2 changes: 2 additions & 0 deletions docs/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
# Welcome to MolExp

41 changes: 0 additions & 41 deletions example/cmdline/funcs.py

This file was deleted.

52 changes: 0 additions & 52 deletions example/cmdline/run.py

This file was deleted.

43 changes: 0 additions & 43 deletions example/cmdline/slurm_funcs.py

This file was deleted.

19 changes: 19 additions & 0 deletions mkdocs.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
site_name: MolExp

site_url: https://github.com/MolCrafts/molexp
theme:
name: material
features:
- navigation.instant
- navigation.instant.progress
- navigation.tracking
- navigation.tabs

markdown_extensions:
- pymdownx.arithmatex:
generic: true

extra_javascript:
- javascripts/mathjax.js
- https://polyfill.io/v3/polyfill.min.js?features=es6
- https://unpkg.com/mathjax@3/es5/tex-mml-chtml.js
Loading