Creating a new data transformer

As mentioned before, there are three types of data transformer:

  • storage
  • loader
  • writer

The three corresponding Julia types are:

  • DataStorage
  • DataLoader
  • DataWriter

All three types accept a driver (symbol) type parameter. For example, a storage transformer using a "filesystem" driver would be of the type DataStorage{:filesystem}.

Adding support for a new driver is a simple as adding method implementations for the three key data transformer methods:

DataToolkitBase.load — Function
load(loader::DataLoader{driver}, source::Any, as::Type)

Using a certain loader, obtain information in the form of as from the data given by source.

This fulfils this component of the overall data flow:

  ╭────loader─────╮
  ╵               ▼
Data          Information

When the loader produces nothing this is taken to indicate that it was unable to load the data for some reason, and that another loader should be tried if possible. This can be considered a soft failure. Any other value is considered valid information.

source
DataToolkitBase.storage — Function
storage(storer::DataStorage, as::Type; write::Bool=false)

Fetch a storer in form as, appropiate for reading from or writing to (depending on write).

By default, this just calls getstorage or putstorage (when write=true).

This executes this component of the overall data flow:

Storage ◀────▶ Data
source
DataToolkitBase.save — Function
save(writer::Datasaveer{driver}, destination::Any, information::Any)

Using a certain writer, save the information to the destination.

This fulfils this component of the overall data flow:

Data          Information
  ▲               ╷
  ╰────writer─────╯
source