Dataclasses

A set of classes that holds dataset configuration data.

Each class is based on a dictionary and only contains simple elements for easy serialization to json (or other formats). A full dataset configuration can be had by serializing the Job class.

The convert method of each class will turn any regular dictionary objects into special dataclasses objects.

The valid method of each class will test the validity of the data to be an actual dataset config.

The output method of each class will create json with info on each dataclass, to be used in javascript.

Configuration Objects

The building blocks of how datasets, jobs, and tasks are configured.

class iceprod.core.dataclasses.Job(*args, **kwargs)

Holds all information about a running job.

If the options are empty, this is the same as a dataset configuration.

Variables

• version – 3

• options – {} – a dict of parameters to pass to the task runner

• steering – None

• tasks – []

• difplus – None

• description –

  ””

• categories – []

output()

Output dict with values and (optionally) the object name for new objects.

class iceprod.core.dataclasses.Steering(*args, **kwargs)

Holds all information that goes in the steering section of a configuration.

Variables

• parameters – {}

• batchsys – None

• system – {} – just specialized parameters

• resources – []

• data – []

class iceprod.core.dataclasses._TaskCommon(*args, **kwargs)

Holds common attributes used by task, tray, module.

Variables

• name –

  ’’

• resources – []

• data – []

• classes – []

• parameters – {}

class iceprod.core.dataclasses.Task(*args, **kwargs)

Bases: iceprod.core.dataclasses._TaskCommon

Holds all information about a task.

Variables

• depends – [] – a list of task names

• batchsys – None

• trays – []

• requirements – {} – a dict of requirements

• task_files – False – whether to use the task files API

class iceprod.core.dataclasses.Tray(*args, **kwargs)

Bases: iceprod.core.dataclasses._TaskCommon

Holds all information about a tray.

Variables

• iterations – 1

• modules – []

class iceprod.core.dataclasses.Module(*args, **kwargs)

Bases: iceprod.core.dataclasses._TaskCommon

Holds all information about a module.

Variables

• running_class – None – the python class or function to call

• src – None – src of class or script

• args – None – args to give to class or src if not an iceprod module

• env_shell – None – src of script which sets env and calls arg

• env_clear – True – clear the env before calling the module (calls env_shell after clearing, if defined)

• configs – None – any json config files that should be written (format is {filename: data})

Note that env_clear should be used carefully, as it wipes out any loaded classes.

class iceprod.core.dataclasses.Class(*args, **kwargs)

A class object, downloaded from a url.

Variables

• name – None – required

• src – None – if downloaded from url

• resource_name – None – if present in resource object

• recursive – False

• libs – None – if more than default lib directory

• env_vars – None

class iceprod.core.dataclasses._ResourceCommon(*args, **kwargs)

Holds common attributes used by Resource and Data.

Variables

• remote –

  ’’

• local –

  ’’

• compression – False

class iceprod.core.dataclasses.Resource(*args, **kwargs)

Bases: iceprod.core.dataclasses._ResourceCommon

A resource object, representing a file to download.

Variables

arch – None

class iceprod.core.dataclasses.Data(*args, **kwargs)

Bases: iceprod.core.dataclasses._ResourceCommon

A data object, representing input and/or output of data.

Variables

• type – ‘permanent’ – required

• movement – ‘both’ – required

Metadata Objects

Metadata for NSF requirements

class iceprod.core.dataclasses.DifPlus(*args, **kwargs)

A DifPlus object.

Variables

• dif – None

• plus – None

class iceprod.core.dataclasses.Dif(*args, **kwargs)

A Dif object.

Variables

• entry_id – None

• entry_title – None

• parameters – ‘ ‘

• iso_topic_category – ‘geoscientificinformation’

• data_ceter – None

• summary – ‘ ‘

• metadata_name – ‘[CEOS IDN DIF]’

• metadata_version – ‘9.4’

• personnel – []

• sensor_name – ‘ICECUBE’

• source_name – ‘SIMULATION’

• dif_creation_date – time.strftime(“%Y-%m-%d”)

class iceprod.core.dataclasses.Plus(*args, **kwargs)

A Plus object.

Variables

• start – None

• end – None

• category – None

• subcategory – None

• run_number – None

• i3db_key – None

• simdb_key – None

• project – [] – [{name: version}]

• steering_file – None

• log_file – None

• command_line – None

class iceprod.core.dataclasses.Personnel(*args, **kwargs)

A Personnel object.

Variables

• role – None

• first_name – None

• last_name – None

• email – None

class iceprod.core.dataclasses.DataCenter(*args, **kwargs)

A Data Center object.

Variables

• name – None

• personnel – []