Skip to content

TODO:

Repo setup

  • TODO:
    • Docs for injection and container
    • Remove tests in pip installation, move to directory base
    • Docs!
    • Set up all loggers
  • Done:
    • Versioneer
    • Action for testing package install & running pytests
    • Tests for injection and container 7/5
    • Environment variables with extensible prefix 7/5
    • License should be passed or something generic. SLAC license shouldn't be in LUME 4/12
    • Add versioneer config. Versioneer install needs to happen on init. 4/12
    • Transition from pytest-mysql to remove pip dependency 4/15
    • Basics of injection and container 4/15
    • Action for building docs
    • Automatically generate diagram from .sql schema? 5/6s
    • Move any services in init files to a service.py
    • Fix prefect context bug during pytest

Template

TODO: - [ ] CLI tool - [ ] Instructions for templating tool and configuration - [ ] Templated README - [ ] Handling of input/output variables at outset - [ ] Drop required defaults for lume-model variables - [ ] Check functionality of all - [ ] Docs - Done: - [x] Action for testing package install & running pytests - [x] Upload docker image to repo artifacts 5/3 - [x] Docker build action for templated repos 4/12

Databases

  • TODO:
    • Test database failure cases
    • Rename base classes for DBService
    • Dashboard image field in mongodb for impact
    • Docs
  • Done:
    • Initial tests for mysql model db
    • Abstraction of database schema and auth config from query execution
    • Add flow_to_deployments into db
    • Change model_versions to deployments in schema
    • Make mongodb multiprocessing safe
    • Add uniqueness to results
    • Fix test connection with mysql in-package plugin. "Connection refused" 4/13

Backlog

  • TODO:
    • Implement/plan for logging infrastructure
    • Synchronous snapshot service (-> SLAC-services)
    • HPC interface
    • Slurm interface
    • Output service (-> LUME-EPICS?)
  • Done:
    • Finish util testing

Scheduler

  • TODO:
    • Test flow of flows composition
    • Add mapped parameters to database?
    • Add requirements table
    • Kubernetes backend tests
    • How do we handle submission of environment variables to the scheduler? For example, how do we communicate the aliasing of services in a docker-compose app?
    • Docs
    • Environment solving for containerized jobs
  • Done:
    • Refactor scheduler 7/13
    • Create prefect flow image 6/26
    • Remove redundant flow storage during build in pytest 7/6
    • Pin docker-py version 7/15
    • Constrain KubernetesBackend image pull policy to existing options
    • Drop prefect subfolder 6/24
    • Create docker backend 6/27
    • Tesing infrastructure for prefect w/ docker compose 6/27
    • Result tasks
    • Add scheduler to context
    • Drop all but apollo from config 8/3
    • Improve service wait_until_responsive checks and move into docker 8/9

Misc

  • TODO:
    • Separate template into own repo and use git submodule
    • Could abstract docker compose versions
    • Use environment variable fixture in tests instead of modifying env
  • Done:
    • Rename file.systems to file.filesystems and all files names service.py
    • Move fixtures from conftest to designated files under tests/fixtures
    • Change LUME-model SurrogateModel to BaseModel for generalizability

Files

  • TODO:
    • File locks
  • Done:
    • Implement local file handler 5/2
    • Implement mounted filesystem handler 5/2

Models

  • TODO:
    • Create model interface with injected services
    • Add utility for loading flow objects
    • Preprocessing structure
    • Registration of preprocessing flows