PyTorch API reference

Every public name of gpulayer.pytorch (import gpulayer.pytorch as glt), with its signature and docstring. This page is generated by scripts/make_api_reference.py from the code; the guides explain how the pieces fit together.

glt.LightningModule and glt.LightningDataModule are the same classes as glt.Module and glt.DataModule, for code moving from PyTorch Lightning.

Trainer

Runs the loop here or launches it on workers. Guide: Trainer and Module.

`Trainer`

python

glt.Trainer(
    *,
    accelerator: str = 'auto',
    strategy: str | DDPStrategy | FSDPStrategy = 'ddp',
    devices: int | str = 1,
    num_nodes: int = 1,
    precision: str | int = '32-true',
    logger: Logger | list[Logger] | bool | None = None,
    callbacks: Callback | list[Callback] | None = None,
    fast_dev_run: bool | int = False,
    max_epochs: int | None = 1,
    min_epochs: int | None = None,
    max_steps: int | None = None,
    min_steps: int | None = None,
    max_time: float | timedelta | dict[str, int] | str | None = None,
    limit_train_batches: int | float | None = None,
    limit_val_batches: int | float | None = None,
    limit_test_batches: int | float | None = None,
    limit_predict_batches: int | float | None = None,
    overfit_batches: int | float = 0.0,
    val_check_interval: int | float | None = None,
    check_val_every_n_epoch: int | None = 1,
    num_sanity_val_steps: int = 2,
    log_every_n_steps: int = 50,
    enable_checkpointing: bool = True,
    enable_progress_bar: bool = False,
    enable_model_summary: bool = False,
    accumulate_grad_batches: int = 1,
    gradient_clip_val: float | None = None,
    gradient_clip_algorithm: str = 'norm',
    deterministic: bool | str = False,
    benchmark: bool | None = None,
    inference_mode: bool = True,
    use_distributed_sampler: bool = True,
    profiler: str | Profiler | None = None,
    detect_anomaly: bool = False,
    barebones: bool = False,
    plugins: Any = None,
    sync_batchnorm: bool = False,
    reload_dataloaders_every_n_epochs: int = 0,
    default_root_dir: str | Path = 'gpulayer_runs',
    target: str | None = None,
    gradient_sync: str = 'auto',
    prefetch: bool = False,
    launch_options: dict[str, Any] | None = None,
)

Train a Module: Trainer(max_epochs=5).fit(model, train_loader, val_loader).

With one device and no target the loop runs in this process. With devices > 1, num_nodes > 1 or a target, fit submits a job and the same script runs on GPUlayer workers, one process per rank; the trained weights come back to model.

Every argument is described in the guide's table: Trainer arguments. Names and meanings follow PyTorch Lightning; the GPUlayer-only ones are target, gradient_sync, prefetch and launch_options.

Method
`fit(model, train_dataloaders=None, val_dataloaders=None, datamodule=None, ckpt_path=None)`	Train `model`. The loaders may be given as functions that return them, or come from a `DataModule` or the model's `train_dataloader`/`val_dataloader`: then they are built only in the processes that train. When the Trainer launches a job, the launching process only waits, so data it built would be thrown away (measured: a 6 s dataset built for nothing made a 2-GPU run look 25% slower per step).
`log_metrics(metrics, step=None)`	Record extra values for the current step (used by monitoring callbacks): they go to metrics.csv and the loggers, from rank 0.
`predict(model=None, dataloaders=None, datamodule=None, return_predictions=None, ckpt_path=None)`	Run `predict_step` over `dataloaders` in this process (no gradients) and return one output per batch, moved to the CPU. With `return_predictions=False` (for example with a `BasePredictionWriter`) nothing is kept in memory.
`save_checkpoint(filepath, weights_only=False, storage_options=None, dirpath=None)`	Write a checkpoint from rank 0 (other ranks return None) and return its path. A plain file name goes to `checkpoints/` of the run (or `dirpath`); a path is used as it is. Call it on every rank: with `strategy="fsdp"` the full state is gathered from all of them.
`test(model=None, dataloaders=None, ckpt_path=None, verbose=True, datamodule=None)`	Run `test_step` over `dataloaders` in this process and return the averaged logs, e.g. `{"test_acc": 0.93}`.
`validate(model=None, dataloaders=None, ckpt_path=None, verbose=True, datamodule=None)`	Run `validation_step` over `dataloaders` in this process and return the averaged logs, e.g. `{"val_loss": 0.21}` (`result[0]` also works, as in Lightning).

Property
`checkpoint_callback`	The first `ModelCheckpoint`, or None.
`checkpoint_callbacks`	Every `ModelCheckpoint` among the callbacks.
`early_stopping_callback`	The first `EarlyStopping`, or None.
`early_stopping_callbacks`	Every `EarlyStopping` among the callbacks.
`estimated_stepping_batches`	Optimizer steps this fit plans (`inf` when the loader's length is unknown).
`global_rank`	This process's rank among all processes (0 in a single process).
`is_global_zero`	True in the process with global rank 0 (and in a single process).
`lightning_module`	The Module being trained or evaluated (Lightning's name).
`local_rank`	This process's rank on its machine.
`log_dir`	The first logger's directory, else the run's output directory.
`logger`	The first logger, or None.
`model`	The Module as the loop calls it: wrapped in DDP when distributed, else the Module.
`node_rank`	This machine's rank among the machines of the job.
`num_devices`	Processes (devices) per node, from `devices=`.
`predicting`	True while `predict` runs.
`sanity_checking`	True during the validation check before training.
`strategy`	The strategy object (`DDPStrategy` or `FSDPStrategy`) from `strategy=`.
`testing`	True while `test` runs.
`training`	True while training batches run.
`validating`	True while validation runs (inside fit or `validate`).
`will_launch`	True when `fit` will run on GPUlayer workers instead of this process.
`world_size`	Number of processes training together.

`LRSchedulerConfig`

python

glt.LRSchedulerConfig(
    scheduler: Any,
    name: str | None = None,
    interval: str = 'epoch',
    frequency: int = 1,
    reduce_on_plateau: bool = False,
    monitor: str | None = None,
    strict: bool = True,
)

A learning-rate scheduler and when to step it, as in Lightning.

Module and DataModule

Your model and data. Guide: Trainer and Module.

`Module`

python

glt.Module(*args: Any, **kwargs: Any)

Subclass and implement training_step and configure_optimizers.

validation_step, test_step and predict_step are optional. Call self.log(name, value) in any step; the Trainer averages logged values per epoch across steps and ranks. Override any of the on_* hooks to run code at that point of the loop.

Method
`all_gather(data, sync_grads=False)`	Gather tensors from every process: each tensor gets a leading world-size dimension (unchanged in a single process).
`backward(loss, args, *kwargs)`	Run the backward pass; override to pass arguments to `loss.backward`.
`clip_gradients(optimizer, gradient_clip_val=None, gradient_clip_algorithm=None)`	Clip the gradients of `optimizer`'s parameters (unscaled first with 16-mixed).
`configure_callbacks()`	Callbacks this model always needs; they are added to the Trainer's.
`configure_gradient_clipping(optimizer, gradient_clip_val=None, gradient_clip_algorithm=None)`	Clip gradients before each optimizer step; by default with the Trainer's values.
`configure_model()`	Create layers here to build them on the training device (called once per fit).
`configure_optimizers()`	Return an optimizer; a list of optimizers; `([optimizers], [schedulers])`; `(optimizer, scheduler)`; or `{"optimizer": ..., "lr_scheduler": ...}`, where the scheduler may be a dict `{"scheduler": ..., "interval": "epoch" \| "step", "frequency": 1, "monitor": "val_loss"}`. Several optimizers need manual optimization (`self.automatic_optimization = False`), as in Lightning.
`freeze()`	No gradients for any parameter, and evaluation mode.
`load_from_checkpoint(checkpoint_path, args, map_location='cpu', strict=True, *kwargs)`	Build the model and load the weights saved by the Trainer. Arguments recorded with `save_hyperparameters` are used unless given again here.
`log(name, value, prog_bar=False, logger=None, on_step=None, on_epoch=None, reduce_fx='mean', enable_graph=False, sync_dist=False, sync_dist_group=None, add_dataloader_idx=True, batch_size=None, metric_attribute=None, rank_zero_only=False)`	Record a scalar for this step. The Trainer aggregates it over the epoch with `reduce_fx` ("mean", "sum", "max" or "min"; a mean is weighted by `batch_size` when given) and averages it across processes. `prog_bar` shows it in the progress bar. `sync_dist` is accepted for compatibility: epoch values are always combined across processes.
`log_dict(dictionary, **kwargs)`	`self.log` for each item of `dictionary`, with the same options.
`lr_scheduler_step(scheduler, metric)`	Step a scheduler; `metric` is the monitored value for ReduceLROnPlateau.
`lr_schedulers()`	The scheduler(s) from `configure_optimizers`: one, a list, or None.
`manual_backward(loss, args, *kwargs)`	`loss.backward()` for manual optimization, with the Trainer's gradient scaling.
`optimizer_step(epoch, batch_idx, optimizer, closure=None)`	Run the optimizer step (override to change it, e.g. learning-rate warm-up).
`optimizer_zero_grad(epoch, batch_idx, optimizer)`	Reset gradients after an optimizer step (`set_to_none=True`); override to change it.
`optimizers(use_pl_optimizer=True)`	The optimizer(s) from `configure_optimizers`: one, or a list.
`predict_step(batch, batch_idx)`	Default: `self(batch)`, or `self(batch[0])` for an `(inputs, ...)` batch.
`print(args, *kwargs)`	`print` on rank 0 only.
`save_hyperparameters(*args, ignore=None, frame=None)`	Record the `__init__` arguments (all of them, or the names in `args`) as `self.hparams`. They are saved in checkpoints, and `load_from_checkpoint` uses them to rebuild the model without passing the arguments again.
`test_step(batch, batch_idx)`	One test batch, run by `Trainer.test`: compute metrics and `self.log` them.
`to_onnx(file_path, input_sample=None, **kwargs)`	Export to ONNX (needs the `onnx` package).
`to_torchscript(file_path=None, method='script', example_inputs=None)`	TorchScript of this model (`method="script"` or `"trace"`), saved when `file_path` is given.
`toggle_optimizer(optimizer)`	Only `optimizer`'s parameters require gradients until `untoggle_optimizer`.
`toggled_optimizer(optimizer)`	Context manager: `toggle_optimizer` on entry, `untoggle_optimizer` on exit.
`training_step(batch, batch_idx)`	Return the loss tensor, or a dict with `"loss"`. With manual optimization, return nothing.
`transfer_batch_to_device(batch, device, dataloader_idx)`	Move a batch to `device`. Override for custom batch types.
`unfreeze()`	Gradients for every parameter again, and training mode.
`untoggle_optimizer(optimizer)`	Restore `requires_grad` changed by `toggle_optimizer`.
`validation_step(batch, batch_idx)`	One validation batch: compute metrics and `self.log` them (optional).

Property
`automatic_optimization`	True (default): the Trainer runs backward and the optimizer step. Set False for manual optimization.
`current_epoch`	The Trainer's current epoch (0 when not attached).
`device`	The device of the first parameter (or buffer); CPU for a model without either.
`dtype`	The dtype of the first parameter.
`global_rank`	This process's rank among all processes.
`global_step`	Optimizer steps taken so far.
`hparams`	The arguments recorded by `save_hyperparameters`, as an `AttributeDict`.
`local_rank`	This process's rank on its machine.
`logger`	The Trainer's first logger, or None.
`loggers`	The Trainer's loggers.
`on_gpu`	True when the parameters are on a CUDA (or ROCm) device.

Hook
`on_after_backward()`	After the backward pass, before the optimizer step; gradients are set.
`on_after_batch_transfer(batch, dataloader_idx)`	Change a batch after it was moved to the device (returns the batch); for example GPU augmentation.
`on_before_backward(loss)`	Before `loss.backward()`, with the loss (unscaled).
`on_before_batch_transfer(batch, dataloader_idx)`	Change a batch before it is moved to the device (returns the batch).
`on_before_optimizer_step(optimizer)`	Before the optimizer step and gradient clipping; with 16-mixed the gradients are already unscaled.
`on_before_zero_grad(optimizer)`	Before the gradients are reset after an optimizer step.
`on_exception(exception)`	When fit stops with an exception or Ctrl+C, before it is raised (or, for Ctrl+C, before fit returns).
`on_fit_end()`	At the end of `fit`. In the launching process of a submitted run, only this hook runs (after the outputs came back).
`on_fit_start()`	At the start of `fit`, after setup and optimizers, before the sanity check.
`on_load_checkpoint(checkpoint)`	When a checkpoint is loaded (resume or `load_from_checkpoint`), before the weights are restored; change entries in place.
`on_predict_batch_end(outputs, batch, batch_idx)`	After `predict_step`, with the prediction (on the CPU).
`on_predict_batch_start(batch, batch_idx)`	Before `predict_step`.
`on_predict_end()`	After `Trainer.predict`.
`on_predict_epoch_end()`	After the last prediction batch.
`on_predict_epoch_start()`	Before the first prediction batch.
`on_predict_start()`	Before `Trainer.predict` runs its batches.
`on_save_checkpoint(checkpoint)`	When a checkpoint is written, with the dict about to be saved; add or change entries in place.
`on_test_batch_end(outputs, batch, batch_idx)`	After `test_step`, with what it returned.
`on_test_batch_start(batch, batch_idx)`	Before `test_step`.
`on_test_end()`	After `Trainer.test`.
`on_test_epoch_end()`	After the last test batch; `self.log` here adds to the results.
`on_test_epoch_start()`	Before the first test batch.
`on_test_start()`	Before `Trainer.test` runs its batches.
`on_train_batch_end(outputs, batch, batch_idx)`	After the optimizer step of a batch (or after the batch, when accumulating); `outputs` is `{'loss': loss}` or the dict `training_step` returned.
`on_train_batch_start(batch, batch_idx)`	Before `training_step`. On the model, returning `-1` ends the epoch.
`on_train_end()`	Once, after the last training epoch and before `on_fit_end`.
`on_train_epoch_end()`	At the end of each training epoch, after its validation; `trainer.callback_metrics` holds the epoch's values. `self.log` here adds to them.
`on_train_epoch_start()`	At the start of each training epoch.
`on_train_start()`	Once, before the first training epoch.
`on_validation_batch_end(outputs, batch, batch_idx)`	After `validation_step`, with what it returned.
`on_validation_batch_start(batch, batch_idx)`	Before `validation_step`.
`on_validation_end()`	After a validation run.
`on_validation_epoch_end()`	After the last validation batch; `self.log` here adds to the validation results.
`on_validation_epoch_start()`	Before the first validation batch, with the model in eval mode.
`on_validation_start()`	Before a validation run (also inside fit).
`prepare_data()`	Once per machine (local rank 0) before `setup`; download data here. Do not set state here: other processes do not run it.
`setup(stage)`	At the start of fit, validate, test or predict, in every process (`stage` is "fit", "validate", "test" or "predict"); build layers or datasets here.
`teardown(stage)`	At the end of fit, validate, test or predict, in every process.

`DataModule`

python

glt.DataModule()

All the data of a project in one place, as in Lightning's LightningDataModule.

prepare_data runs once per machine (download here); setup(stage) runs in every process ("fit", "validate", "test" or "predict"); the *_dataloader methods return loaders. The Trainer calls them only in the processes that train, so a launched job does not build its data in the launching process.

Method
`from_datasets(train_dataset=None, val_dataset=None, test_dataset=None, predict_dataset=None, batch_size=1, num_workers=0, **loader_kwargs)`	A DataModule from datasets, with a shuffled training loader.
`load_state_dict(state_dict)`	Restore what `state_dict` returned, when resuming.
`predict_dataloader()`	The prediction loader (needed for `predict`).
`save_hyperparameters(*args, ignore=None, frame=None)`	Record the `__init__` arguments (all of them, or the names in `args`) as `self.hparams`. They are saved in checkpoints, and `load_from_checkpoint` uses them to rebuild the model without passing the arguments again.
`state_dict()`	State saved in checkpoints (empty by default).
`test_dataloader()`	The test loader (needed for `test`).
`train_dataloader()`	The training loader (needed for `fit`).
`val_dataloader()`	The validation loader, or None.

Property
`hparams`	The arguments recorded by `save_hyperparameters`, as an `AttributeDict`.

Hook
`on_after_batch_transfer(batch, dataloader_idx)`	Change a batch after it was moved to the device (returns the batch); for example GPU augmentation.
`on_before_batch_transfer(batch, dataloader_idx)`	Change a batch before it is moved to the device (returns the batch).
`prepare_data()`	Once per machine (local rank 0) before `setup`; download data here. Do not set state here: other processes do not run it.
`setup(stage)`	At the start of fit, validate, test or predict, in every process (`stage` is "fit", "validate", "test" or "predict"); build layers or datasets here.
`teardown(stage)`	At the end of fit, validate, test or predict, in every process.

`AttributeDict`

python

glt.AttributeDict(...)

A dict whose keys are also attributes: hparams.lr.

Strategies

How the model is replicated or sharded across processes.

`DDPStrategy`

python

glt.DDPStrategy(
    find_unused_parameters: bool = False,
    static_graph: bool = False,
    gradient_as_bucket_view: bool = False,
    **kwargs: Any,
)

DistributedDataParallel with its options.

find_unused_parameters: needed when some parameters get no gradient in a step (costs a graph traversal per step, so off by default).
static_graph: the set of used parameters never changes; DDP can then skip work and overlap more communication.
gradient_as_bucket_view: gradients are views into DDP's communication buckets, which saves one copy of the gradients in memory and a copy per step.

Method
`ddp_kwargs()`	Keyword arguments passed to `DistributedDataParallel`.

`FSDPStrategy`

python

glt.FSDPStrategy()

Fully sharded data parallel (FSDP2), experimental: see the Trainer docs.

Method
`ddp_kwargs()`	Always empty: FSDP does not use DistributedDataParallel.

Callbacks

Guide: Callbacks, loggers and profilers.

`Callback`

python

glt.Callback()

Subclass and override the hooks you need. All hooks are optional.

Method
`load_state_dict(state_dict)`	Restore what `state_dict` returned, when resuming from a checkpoint.
`state_dict()`	State saved in checkpoints and restored when resuming.

Property
`state_key`	Key of this callback's `state_dict` in checkpoints.

Hook
`on_after_backward(trainer, pl_module)`	Callback form, with `trainer, pl_module` first. After the backward pass, before the optimizer step; gradients are set.
`on_before_backward(trainer, pl_module, loss)`	Callback form, with `trainer, pl_module` first. Before `loss.backward()`, with the loss (unscaled).
`on_before_optimizer_step(trainer, pl_module, optimizer)`	Callback form, with `trainer, pl_module` first. Before the optimizer step and gradient clipping; with 16-mixed the gradients are already unscaled.
`on_before_zero_grad(trainer, pl_module, optimizer)`	Callback form, with `trainer, pl_module` first. Before the gradients are reset after an optimizer step.
`on_exception(trainer, pl_module, exception)`	Callback form, with `trainer, pl_module` first. When fit stops with an exception or Ctrl+C, before it is raised (or, for Ctrl+C, before fit returns).
`on_fit_end(trainer, pl_module)`	Callback form, with `trainer, pl_module` first. At the end of `fit`. In the launching process of a submitted run, only this hook runs (after the outputs came back).
`on_fit_start(trainer, pl_module)`	Callback form, with `trainer, pl_module` first. At the start of `fit`, after setup and optimizers, before the sanity check.
`on_load_checkpoint(trainer, pl_module, checkpoint)`	Callback form, with `trainer, pl_module` first. When a checkpoint is loaded (resume or `load_from_checkpoint`), before the weights are restored; change entries in place.
`on_predict_batch_end(trainer, pl_module, outputs, batch, batch_idx, dataloader_idx=0)`	Callback form, with `trainer, pl_module` first. After `predict_step`, with the prediction (on the CPU).
`on_predict_batch_start(trainer, pl_module, batch, batch_idx, dataloader_idx=0)`	Callback form, with `trainer, pl_module` first. Before `predict_step`.
`on_predict_end(trainer, pl_module)`	Callback form, with `trainer, pl_module` first. After `Trainer.predict`.
`on_predict_epoch_end(trainer, pl_module)`	Callback form, with `trainer, pl_module` first. After the last prediction batch.
`on_predict_epoch_start(trainer, pl_module)`	Callback form, with `trainer, pl_module` first. Before the first prediction batch.
`on_predict_start(trainer, pl_module)`	Callback form, with `trainer, pl_module` first. Before `Trainer.predict` runs its batches.
`on_sanity_check_end(trainer, pl_module)`	Callback form, with `trainer, pl_module` first. After the sanity-check validation batches.
`on_sanity_check_start(trainer, pl_module)`	Callback form, with `trainer, pl_module` first. Before the validation batches run as a check before training.
`on_save_checkpoint(trainer, pl_module, checkpoint)`	Callback form, with `trainer, pl_module` first. When a checkpoint is written, with the dict about to be saved; add or change entries in place.
`on_test_batch_end(trainer, pl_module, outputs, batch, batch_idx, dataloader_idx=0)`	Callback form, with `trainer, pl_module` first. After `test_step`, with what it returned.
`on_test_batch_start(trainer, pl_module, batch, batch_idx, dataloader_idx=0)`	Callback form, with `trainer, pl_module` first. Before `test_step`.
`on_test_end(trainer, pl_module)`	Callback form, with `trainer, pl_module` first. After `Trainer.test`.
`on_test_epoch_end(trainer, pl_module)`	Callback form, with `trainer, pl_module` first. After the last test batch; `self.log` here adds to the results.
`on_test_epoch_start(trainer, pl_module)`	Callback form, with `trainer, pl_module` first. Before the first test batch.
`on_test_start(trainer, pl_module)`	Callback form, with `trainer, pl_module` first. Before `Trainer.test` runs its batches.
`on_train_batch_end(trainer, pl_module, outputs, batch, batch_idx)`	After each training batch. `outputs` is `{"loss": loss}` (or the dict that `training_step` returned). Callbacks written for GPUlayer 0.7 with the signature `(trainer, model, loss, batch_idx)` still work.
`on_train_batch_start(trainer, pl_module, batch, batch_idx)`	Callback form, with `trainer, pl_module` first. Before `training_step`. On the model, returning `-1` ends the epoch.
`on_train_end(trainer, pl_module)`	Callback form, with `trainer, pl_module` first. Once, after the last training epoch and before `on_fit_end`.
`on_train_epoch_end(trainer, pl_module)`	After training and validation of an epoch; `trainer.callback_metrics` is set.
`on_train_epoch_start(trainer, pl_module)`	Callback form, with `trainer, pl_module` first. At the start of each training epoch.
`on_train_start(trainer, pl_module)`	Callback form, with `trainer, pl_module` first. Once, before the first training epoch.
`on_validation_batch_end(trainer, pl_module, outputs, batch, batch_idx, dataloader_idx=0)`	Callback form, with `trainer, pl_module` first. After `validation_step`, with what it returned.
`on_validation_batch_start(trainer, pl_module, batch, batch_idx, dataloader_idx=0)`	Callback form, with `trainer, pl_module` first. Before `validation_step`.
`on_validation_end(trainer, pl_module)`	Callback form, with `trainer, pl_module` first. After a validation run.
`on_validation_epoch_end(trainer, pl_module)`	Callback form, with `trainer, pl_module` first. After the last validation batch; `self.log` here adds to the validation results.
`on_validation_epoch_start(trainer, pl_module)`	Callback form, with `trainer, pl_module` first. Before the first validation batch, with the model in eval mode.
`on_validation_start(trainer, pl_module)`	Callback form, with `trainer, pl_module` first. Before a validation run (also inside fit).
`setup(trainer, pl_module, stage)`	Callback form, with `trainer, pl_module` first. At the start of fit, validate, test or predict, in every process (`stage` is "fit", "validate", "test" or "predict"); build layers or datasets here.
`teardown(trainer, pl_module, stage)`	Callback form, with `trainer, pl_module` first. At the end of fit, validate, test or predict, in every process.

`ModelCheckpoint`

python

glt.ModelCheckpoint(
    dirpath: str | Path | None = None,
    filename: str | None = None,
    monitor: str | None = 'val_loss',
    verbose: bool = False,
    save_last: bool | None = None,
    save_top_k: int = 1,
    save_weights_only: bool = False,
    mode: Literal['min', 'max'] = 'min',
    auto_insert_metric_name: bool = True,
    every_n_train_steps: int | None = None,
    train_time_interval: timedelta | None = None,
    every_n_epochs: int | None = None,
    save_on_train_epoch_end: bool | None = None,
    enable_version_counter: bool = True,
    min_delta: float = 0.0,
)

Save checkpoints of the best results by monitor (written by rank 0).

save_top_k=1 (default) keeps the best as checkpoints/<filename>; save_top_k=N keeps the N best; -1 keeps every one; 0 keeps none (use with save_last).
filename is a plain name ending in .ckpt (default best.ckpt; with save_top_k > 1 files become best-epoch=E.ckpt), or a Lightning template such as "{epoch}-{val_loss:.2f}", which gives epoch=3-val_loss=0.12.ckpt.
monitor=None saves the latest checkpoint instead of the best.
When to check: every every_n_epochs epochs (default 1), or every every_n_train_steps optimizer steps, or every train_time_interval.
save_last=True also writes last.ckpt each time it checks.
save_weights_only=True leaves the optimizer state out.

best_model_path points at the best file afterwards, also when training ran on GPUlayer workers; best_k_models maps each kept file to its score.

Inherits from Callback.

Method
`format_checkpoint_name(metrics, ver=None)`	The file name for `metrics` (`epoch` and `step` included).

Property
`kth_best_model_path`	The worst of the kept checkpoints (the next to be replaced).

`EarlyStopping`

python

glt.EarlyStopping(
    monitor: str = 'val_loss',
    min_delta: float = 0.0,
    patience: int = 3,
    verbose: bool = False,
    mode: Literal['min', 'max'] = 'min',
    strict: bool = True,
    check_finite: bool = True,
    stopping_threshold: float | None = None,
    divergence_threshold: float | None = None,
    check_on_train_epoch_end: bool | None = None,
    log_rank_zero_only: bool = False,
)

Stop when monitor has not improved by more than min_delta for patience checks in a row (one check per epoch). Also stops at once when the value reaches stopping_threshold, crosses divergence_threshold the wrong way, or (with check_finite) becomes NaN or infinite. With strict=False a missing metric is skipped instead of raising.

Inherits from Callback.

Property
`wait`	Checks without improvement so far (`wait_count`).

`LearningRateMonitor`

python

glt.LearningRateMonitor(
    logging_interval: Literal['step', 'epoch'] | None = None,
    log_momentum: bool = False,
    log_weight_decay: bool = False,
)

Record the learning rate of each optimizer parameter group.

logging_interval="epoch" (or None, the default) records the rate used in each epoch as lr (one group) or lr-<i>, in trainer.history and trainer.callback_metrics; it is read when the epoch starts, before an epoch-interval scheduler steps. "step" also records it on every logged step (metrics.csv and loggers). log_momentum and log_weight_decay add those values.

Inherits from Callback.

`Timer`

python

glt.Timer(
    duration: Any = None,
    interval: Literal['step', 'epoch'] = 'step',
    verbose: bool = True,
)

Stop training after duration (a timedelta, a dict of timedelta arguments or "DD:HH:MM:SS"), checked every step (interval="step") or every epoch. With several processes it is checked at the end of each epoch, so every process stops together. time_elapsed(stage) and time_remaining(stage) report the times in seconds.

Inherits from Callback.

Method
`end_time(stage='train')`	Monotonic end time of `stage`, or None while it runs.
`start_time(stage='train')`	Monotonic start time of `stage` ("train", "validate", "test"), or None.
`time_elapsed(stage='train')`	Seconds spent in `stage` (training time includes time before a resume).
`time_remaining(stage='train')`	Seconds left of `duration`, or None without one.

`GradientAccumulationScheduler`

python

glt.GradientAccumulationScheduler(scheduling: dict[int, int])

Change accumulate_grad_batches by epoch: {0: 8, 4: 4, 8: 1} accumulates 8 batches from epoch 0, 4 from epoch 4 and none from epoch 8.

Inherits from Callback.

Method
`get_accumulate_grad_batches(epoch)`	Batches accumulated per optimizer step in `epoch`.
`going_to_accumulate_grad_batches()`	True if any epoch accumulates more than one batch.

`StochasticWeightAveraging`

python

glt.StochasticWeightAveraging(
    swa_lrs: float | list[float],
    swa_epoch_start: int | float = 0.8,
    annealing_epochs: int = 10,
    annealing_strategy: Literal['cos', 'linear'] = 'cos',
    avg_fn: Callable[..., Any] | None = None,
    device: torch.device | str | None = 'cpu',
)

Stochastic Weight Averaging: from swa_epoch_start (an epoch, or a fraction of max_epochs), switch the learning rate to swa_lrs with SWALR and average the weights at the end of each epoch. At the end of training the average replaces the weights and BatchNorm statistics are recomputed on the training data.

Inherits from Callback.

`WeightAveraging`

python

glt.WeightAveraging(
    device: torch.device | str | None = None,
    use_buffers: bool = True,
    **kwargs: Any,
)

Keep an average of the weights during training (torch.optim.swa_utils), use it for validation, and leave it in the model and the checkpoints at the end.

should_update(step_idx=..., epoch_idx=...) decides when to average; by default after every optimizer step. Extra arguments go to AveragedModel (avg_fn or multi_avg_fn). Checkpoints hold the averaged weights in state_dict and the training weights in current_model_state, so resuming continues both.

Inherits from Callback.

Method
`should_update(step_idx=None, epoch_idx=None)`	Whether to average now: after the optimizer step `step_idx`, or at the end of epoch `epoch_idx`. Default: every step.

`EMAWeightAveraging`

python

glt.EMAWeightAveraging(
    device: torch.device | str | None = None,
    use_buffers: bool = True,
    decay: float = 0.999,
    update_every_n_steps: int = 1,
    update_starting_at_step: int | None = None,
    update_starting_at_epoch: int | None = None,
)

Exponential moving average of the weights: avg = decay * avg + (1 - decay) * w, updated every update_every_n_steps optimizer steps (from update_starting_at_step or update_starting_at_epoch). Uses PyTorch's multi-tensor EMA, one fused update per step.

Inherits from WeightAveraging, Callback.

`BaseFinetuning`

python

glt.BaseFinetuning()

Freeze parts of the model before training and unfreeze them on a schedule. Implement freeze_before_training(pl_module) and finetune_function(pl_module, epoch, optimizer). Create the optimizer from the trainable parameters only (filter(lambda p: p.requires_grad, self.parameters())).

Inherits from Callback.

Method
`filter_on_optimizer(optimizer, params)`	The parameters of `params` that `optimizer` does not have yet.
`filter_params(modules, train_bn=True, requires_grad=True)`	Parameters of `modules` with the given `requires_grad`, optionally without BatchNorm.
`finetune_function(pl_module, epoch, optimizer)`	Called at the start of each epoch for each optimizer: unfreeze parts here.
`flatten_modules(modules)`	The leaf modules (and modules with their own parameters) of `modules`.
`freeze(modules, train_bn=True)`	Freeze `modules`; with `train_bn` their BatchNorm layers keep training.
`freeze_before_training(pl_module)`	Freeze the parts to fine-tune later (called in `setup`, before the optimizers are built).
`freeze_module(module)`	Turn gradients off for one module's own parameters (BatchNorm stops tracking statistics).
`make_trainable(modules)`	Turn gradients on for `modules` and put them in training mode.
`unfreeze_and_add_param_group(modules, optimizer, lr=None, initial_denom_lr=10.0, train_bn=True)`	Unfreeze `modules` and add their parameters to `optimizer` as a new group with `lr` (default: the first group's rate / `initial_denom_lr`).

`BackboneFinetuning`

python

glt.BackboneFinetuning(
    unfreeze_backbone_at_epoch: int = 10,
    lambda_func: Callable[[int], float] = multiplicative,
    backbone_initial_ratio_lr: float = 0.1,
    backbone_initial_lr: float | None = None,
    should_align: bool = True,
    initial_denom_lr: float = 10.0,
    train_bn: bool = True,
    verbose: bool = False,
    rounding: int = 12,
)

Freeze pl_module.backbone and unfreeze it at unfreeze_backbone_at_epoch with a learning rate of backbone_initial_lr (or the current rate times backbone_initial_ratio_lr), then raise it by lambda_func each epoch, up to the rest of the model's rate when should_align.

Inherits from BaseFinetuning, Callback.

`ModelPruning`

python

glt.ModelPruning(
    pruning_fn: str | Callable[..., Any] = 'l1_unstructured',
    parameters_to_prune: list[tuple[nn.Module, str]] | None = None,
    parameter_names: list[str] | None = None,
    use_global_unstructured: bool = True,
    amount: float | int | Callable[[int], float | int] = 0.5,
    apply_pruning: bool | Callable[[int], bool] = True,
    make_pruning_permanent: bool = True,
    prune_on_train_epoch_end: bool = True,
    verbose: int = 0,
)

Prune weights with torch.nn.utils.prune at the end of training epochs.

pruning_fn: "l1_unstructured", "random_unstructured" or a prune function. parameters_to_prune: [(module, "weight"), ...]; by default every Linear and Conv weight. amount: a fraction or count, or a function of the epoch. apply_pruning: a bool or a function of the epoch. With make_pruning_permanent the masks are folded into the weights at the end of training, and checkpoints always hold plain weights.

Inherits from Callback.

Method
`apply(pl_module, amount)`	Prune `amount` (a fraction or a count) of the target weights now.
`make_permanent(pl_module)`	Fold the pruning masks into the weights and remove the pruning hooks.

`ThroughputMonitor`

python

glt.ThroughputMonitor(
    batch_size_fn: Callable[[Any], int] | None = None,
    length_fn: Callable[[Any], int] | None = None,
    window_size: int = 100,
)

Record training throughput on every logged step: train/batches_per_sec, train/samples_per_sec (with batch_size_fn) and train/items_per_sec (with length_fn, e.g. tokens), per process. Time is measured on the host between logged steps, so no device synchronisation is added.

Inherits from Callback.

`DeviceStatsMonitor`

python

glt.DeviceStatsMonitor(cpu_stats: bool | None = None)

Record device memory on every logged step: on CUDA the allocated, reserved and peak memory in MB; on MPS the allocated and driver memory. cpu_stats=True adds CPU utilisation and RAM (needs psutil).

Inherits from Callback.

`SpikeDetection`

python

glt.SpikeDetection(
    mode: Literal['relative', 'absolute'] = 'relative',
    window: int = 10,
    warmup: int = 1,
    atol: float | None = None,
    rtol: float | None = 2.0,
    exclude_batches_path: str | None = None,
    finite_only: bool = True,
)

Raise TrainingSpikeException when the training loss jumps: above mean * (1 + rtol) plus atol of the last window losses (mode="relative"), or non-finite (finite_only). Reads the loss every step, which waits for the GPU, so use it while debugging a run.

Inherits from Callback.

`ModelSummary`

python

glt.ModelSummary(max_depth: int = 1)

Print the layers and parameter counts when training starts (rank 0). max_depth=-1 lists every submodule.

Inherits from Callback.

`ProgressBar`

python

glt.ProgressBar(refresh_rate: float = 4.0, leave: bool = True)

One progress line per epoch with the step count, rate and the values of the last logged step (no extra device reads). Redrawn in place on a terminal at most refresh_rate times per second; elsewhere (job logs) it prints nothing per step. Uses tqdm when installed.

Inherits from Callback.

Method
`disable()`	Stop drawing the bar.
`enable()`	Draw the bar again.

`BasePredictionWriter`

python

glt.BasePredictionWriter(write_interval: Literal['batch', 'epoch', 'batch_and_epoch'] = 'batch')

Write predictions as they are made, instead of keeping them in memory. Subclass and implement write_on_batch_end and/or write_on_epoch_end; write_interval is "batch", "epoch" or "batch_and_epoch".

Inherits from Callback.

Method
`write_on_batch_end(trainer, pl_module, prediction, batch_indices, batch, batch_idx, dataloader_idx)`	Write one batch's prediction; `batch_indices` are its samples' dataset indices.
`write_on_epoch_end(trainer, pl_module, predictions, batch_indices)`	Write all predictions at the end, with each batch's dataset indices.

`OnExceptionCheckpoint`

python

glt.OnExceptionCheckpoint(dirpath: str | Path | None = None, filename: str = 'on_exception')

Save a checkpoint when training stops with an exception (including Ctrl+C), so it can be resumed with fit(ckpt_path=...).

Inherits from Callback.

`LambdaCallback`

python

glt.LambdaCallback(**hooks: Callable[..., Any])

A callback from functions: LambdaCallback(on_train_epoch_end=lambda t, m: ...).

Inherits from Callback.

`LearningRateFinder`

python

glt.LearningRateFinder(
    min_lr: float = 1e-08,
    max_lr: float = 1.0,
    num_training_steps: int = 100,
    mode: str = 'exponential',
    early_stop_threshold: float | None = 4.0,
    update_attr: bool = True,
    attr_name: str = '',
)

Run Tuner.lr_find when fit starts (one process only) and train with the suggested rate.

Inherits from Callback.

`BatchSizeFinder`

python

glt.BatchSizeFinder(
    mode: str = 'power',
    steps_per_trial: int = 3,
    init_val: int = 2,
    max_trials: int = 25,
    batch_arg_name: str = 'batch_size',
)

Run Tuner.scale_batch_size when fit starts (one process only) and train with the size found.

Inherits from Callback.

`TrainingSpikeException`

python

glt.TrainingSpikeException(batch_idx: int, value: float)

Unspecified run-time error.

Loggers

Called from rank 0 with values the Trainer already read.

`Logger`

python

glt.Logger()

Base class: override log_metrics and log_hyperparams.

Method
`after_save_checkpoint(checkpoint_path)`	Called after each checkpoint is written (for loggers that upload them).
`finalize(status)`	Called once at the end of fit with `status` ("success" or "failed").
`log_graph(model, input_array=None)`	Record the model graph, when the logger supports it.
`log_hyperparams(params)`	Record the model's and DataModule's hyperparameters once, at the start of fit.
`log_metrics(metrics, step=None)`	Record `metrics` at optimizer step `step` (rank 0 only).
`save()`	Flush what was logged so far.

Property
`experiment`	The underlying object (SummaryWriter, wandb run, MlflowClient), created on first use.
`log_dir`	Directory of this run's logs (default: `save_dir`).
`name`	Experiment name, or None.
`save_dir`	Root directory of the logs, or None.
`version`	Run version within the experiment, or None.

`CSVLogger`

python

glt.CSVLogger(
    save_dir: str | Path = 'gpulayer_logs',
    name: str | None = 'gpulayer_logs',
    version: int | str | None = None,
    prefix: str = '',
    flush_logs_every_n_steps: int = 100,
)

Write <save_dir>/<name>/version_<n>/metrics.csv (one row per logged step or epoch) and hparams.yaml.

Inherits from Logger.

`TensorBoardLogger`

python

glt.TensorBoardLogger(
    save_dir: str | Path = 'gpulayer_logs',
    name: str | None = 'gpulayer_logs',
    version: int | str | None = None,
    log_graph: bool = False,
    default_hp_metric: bool = True,
    prefix: str = '',
    **kwargs: Any,
)

TensorBoard event files in <save_dir>/<name>/version_<n> (needs tensorboard).

Inherits from Logger.

`WandbLogger`

python

glt.WandbLogger(
    name: str | None = None,
    save_dir: str | Path = '.',
    version: str | None = None,
    offline: bool = False,
    project: str | None = None,
    log_model: bool = False,
    prefix: str = '',
    **kwargs: Any,
)

Weights & Biases (needs wandb and a login). Arguments go to wandb.init.

Inherits from Logger.

`MLFlowLogger`

python

glt.MLFlowLogger(
    experiment_name: str = 'gpulayer_logs',
    run_name: str | None = None,
    tracking_uri: str | None = None,
    tags: dict[str, Any] | None = None,
    save_dir: str | None = './mlruns',
    log_model: bool = False,
    prefix: str = '',
    run_id: str | None = None,
)

MLflow tracking (needs mlflow).

Inherits from Logger.

Profilers

Pass one as Trainer(profiler=...), or "simple", "advanced", "pytorch".

`Profiler`

python

glt.Profiler(dirpath: str | Path | None = None, filename: str | None = None)

Base class: implement start(action), stop(action) and summary().

Method
`describe()`	The report, also written to `dirpath/filename` when both are set.
`profile(action_name)`	Context manager: time the code inside as `action_name`.
`start(action_name)`	Start timing `action_name`.
`stop(action_name)`	Stop timing `action_name`.
`summary()`	The report as text.

Hook
`setup(stage, local_rank=None, log_dir=None)`	Called when a run starts; `log_dir` becomes `dirpath` when none was given.
`teardown(stage=None)`	Called when the run ends, before the report is printed.

`SimpleProfiler`

python

glt.SimpleProfiler(
    dirpath: str | Path | None = None,
    filename: str | None = None,
    extended: bool = True,
)

Wall time of each loop action: calls, mean and total seconds, share of the run.

Inherits from Profiler.

`AdvancedProfiler`

python

glt.AdvancedProfiler(
    dirpath: str | Path | None = None,
    filename: str | None = None,
    line_count_restriction: float = 1.0,
    dump_stats: bool = False,
)

cProfile per action: the functions that took the most time in each.

Inherits from Profiler.

`PyTorchProfiler`

python

glt.PyTorchProfiler(
    dirpath: str | Path | None = None,
    filename: str | None = None,
    export_to_chrome: bool = False,
    row_limit: int = 20,
    sort_by_key: str | None = None,
    record_shapes: bool = False,
    profile_memory: bool = False,
    with_stack: bool = False,
    **profiler_kwargs: Any,
)

torch.profiler over the whole fit, with each loop action as a labelled range. export_to_chrome=True writes a trace for chrome://tracing or Perfetto (rank 0). sort_by_key picks the table's order, e.g. "cuda_time_total".

Inherits from Profiler.

`PassThroughProfiler`

python

glt.PassThroughProfiler(dirpath: str | Path | None = None, filename: str | None = None)

Does nothing.

Inherits from Profiler.

Tuner

Learning-rate and batch-size search.

`Tuner`

python

glt.Tuner(trainer: Trainer)

Searches that run short trainings with a Trainer's settings.

Method
`lr_find(model, train_dataloaders=None, val_dataloaders=None, dataloaders=None, datamodule=None, method='fit', min_lr=1e-08, max_lr=1.0, num_training=100, mode='exponential', early_stop_threshold=4.0, update_attr=True, attr_name='')`	Train `num_training` steps with the rate rising from `min_lr` to `max_lr` (`"exponential"` or `"linear"`), recording the smoothed loss; stop early when it exceeds `early_stop_threshold` times the best. The weights are restored afterwards. With `update_attr` the suggestion is written to `model.lr` or `model.learning_rate` (or `attr_name`, or the same key in `model.hparams`).
`scale_batch_size(model, train_dataloaders=None, val_dataloaders=None, dataloaders=None, datamodule=None, method='fit', mode='power', steps_per_trial=3, init_val=2, max_trials=25, batch_arg_name='batch_size')`	Find the largest batch size that fits in memory: double it (`"power"`) until a step runs out of memory, then, with `"binsearch"`, search between the last two. The loader must be rebuilt from `batch_arg_name` on the model, its `hparams` or the DataModule (a `train_dataloader` method or a loader function). The size is left on that attribute.

Launch and runtime

Lower layers, usable without the Trainer. Guides: launch(), Runtime helpers.

`launch`

python

glt.launch(
    *,
    nodes: int = 1,
    devices: int = 1,
    accelerator: str = 'cpu',
    target: str | None = None,
    client: Client | None = None,
    script: str | Path | None = None,
    argv: list[str] | None = None,
    frameworks: dict[str, str] | None = None,
    env: dict[str, str] | None = None,
    inputs: dict[str, str | Path] | None = None,
    timeout_seconds: float | None = None,
    requirements: list[str] | None = None,
    requirements_file: str | None = None,
    name: str | None = None,
    output_dir: str | Path | None = None,
    exit: bool = True,
    quiet: bool = False,
    **placement: Any,
)

Submit this script as a GPUlayer job, or return None when already inside one.

inputs maps a path under $GPULAYER_INPUT_DIR in the job to a local file, which is uploaded and placed there on every node (for example a checkpoint to resume from).

requirements / requirements_file (relative to the script's directory) add packages the workers do not have; each worker installs them once and reuses them (ADR-0022).

`LaunchResult`

python

glt.LaunchResult(job: Job, status: JobStatus, output_dir: Path, outputs: list[Path] = <factory>)

LaunchResult(job: 'Job', status: 'JobStatus', output_dir: 'Path', outputs: 'list[Path]' = )

Property
`succeeded`	True when the job finished successfully.

`is_launcher_process`

python

glt.is_launcher_process()

True in the process that submits work, False on workers (where the work runs).

`in_gpulayer_job`

python

glt.in_gpulayer_job()

`Runtime`

python

glt.Runtime(
    accelerator: str = 'auto',
    precision: str = '32-true',
    gradient_sync: str = 'auto',
    ddp_kwargs: dict[str, Any] | None = None,
)

Method
`any(flag)`	True on every rank if `flag` is true on any rank.
`autocast()`	The mixed-precision context for the forward pass (a no-op for 32-true and "-true" precisions).
`backward(loss)`	`loss.backward()`, through the gradient scaler with 16-mixed.
`barrier()`	Wait until every process reaches this point (no-op in one process).
`batches(loader, prefetch=True)`	Iterate `loader` with each batch on the device.
`convert_module(model)`	With a "-true" precision, cast the weights to that dtype (a no-op otherwise).
`dataloader(loader, shuffle=None)`	Return a loader that gives each rank its own shard (unchanged when not distributed).
`full_state(model, optimizer)`	Unsharded model and optimizer state on the CPU. With FSDP this is a collective: call it on every rank.
`load_full_state(model, optimizer, model_state, optim_state)`	Load an unsharded checkpoint into a (possibly sharded) model and optimizer.
`mean(value)`	The average of `value` across processes (one all-reduce).
`mean_dict(values)`	Average every value across ranks in one all-reduce (keys sorted, so ranks agree).
`module(model)`	Move `model` to the device and wrap it in DDP when distributed (with two-level gradient sync when it applies).
`no_sync(model, enabled=True)`	Skip the gradient exchange for this backward (gradient accumulation), with DDP or FSDP.
`output_dir(default='gpulayer_runs/local')`	$GPULAYER_OUTPUT_DIR inside a job (returned as artifacts), else `default`.
`shard(model)`	FSDP (experimental): shard the parameters of every submodule that owns them across all ranks, so each rank holds about 1/world_size of the model, and make `model` the FSDP root. Call `model` (its forward) every step, so the root's hooks run and the submodules release their gathered parameters after their forward. Call this before creating the optimizer. A no-op in a single process.
`step(optimizer, params=None, clip_val=None, clip_algorithm='norm')`	Optimizer step, clipping gradients first: by total norm, or each value to [-clip_val, clip_val] with `clip_algorithm="value"`.
`to_device(batch)`	A batch on this process's device (float tensors in the "-true" precision's dtype).
`uses_two_level_sync()`	Sum gradients inside each machine before crossing the network (ADR-0020). "auto" does it for gloo with several machines and several processes per machine; NCCL plans its own topology-aware rings.

Property
`distributed`	True when more than one process trains (`WORLD_SIZE` > 1).
`is_global_zero`	True in the process with global rank 0.

Hook
`setup()`	Select the GPU of this process and start the process group when distributed (NCCL on GPUs, gloo otherwise). Returns self.
`teardown()`	Destroy the process group if this Runtime started it.

`resolve_accelerator`

python

glt.resolve_accelerator(accelerator: str = 'auto')

'auto' -> the accelerator GPUlayer granted this job, else cuda/rocm > mps > cpu.

`seed_everything`

python

glt.seed_everything(seed: int | None = None, workers: bool = False)

Seed Python, NumPy and torch (every device) and return the seed. Without seed, $PL_GLOBAL_SEED or 0. workers is accepted for Lightning compatibility: torch already derives each DataLoader worker's seed from the base seed.

Edit this page on GitHub

PreviousRuntime helpers Next Commands and scripts