Plugins & Advice

In DataToolkit, the plugin system enables key behaviour to be completely transformed when operating on a given DataCollection.

DataToolkitCore.Plugin — Type

Plugin

A named collection of Advice that accompanies DataCollections.

The complete collection of advice provided by all plugins of a DataCollection is applied to every @advised call involving the DataCollection.

See also: Advice, AdviceAmalgamation.

Construction

Plugin(name::String, advisors::Vector{Advice}) -> Plugin
Plugin(name::String, advisors::Vector{<:Function}) -> Plugin

source

DataToolkitCore.@dataplugin — Macro

@dataplugin plugin_variable
@dataplugin plugin_variable :default

Register the plugin given by the variable plugin_variable, along with its documentation (fetched by @doc). Should :default be given as the second argument the plugin is also added to the list of default plugins.

This effectively serves as a minor, but appreciable, convenience for the following pattern:

push!(PLUGINS, myplugin)
PLUGINS_DOCUMENTATION[myplugin.name] = @doc myplugin
push!(DEFAULT_PLUGINS, myplugin.name) # when also adding to defaults

source

Advice

Inspired by Lisp, DataToolkitCore comes with a method of completely transforming its behaviour at certain defined points. This is essentially a restricted form of Aspect-oriented programming. At certain declared locations (termed "join points"), we consult a list of "advise" functions that modify the execution at that point, and apply the (matched via "pointcuts") advise functions accordingly.

Each applied advise function is wrapped around the invocation of the join point, and is able to modify the arguments, execution, and results of the join point.

DataToolkitCore.Advice — Type

Advice{func, context} <: Function

Advices allow for composable, highly flexible modifications of data by encapsulating a function call. They are inspired by elisp's advice system, namely the most versatile form — :around advice, and Clojure's transducers.

A Advice is essentially a function wrapper, with a priority::Int attribute. The wrapped functions should be of the form:

(action::Function, args...; kargs...) ->
    ([post::Function], action::Function, args::Tuple, [kargs::NamedTuple])

Short-hand return values with post or kargs omitted are also accepted, in which case default values (the identity function and (;) respectively) will be automatically substituted in.

    input=(action args kwargs)
         ┃                 ┏╸post=identity
       ╭─╂────advisor 1────╂─╮
       ╰─╂─────────────────╂─╯
       ╭─╂────advisor 2────╂─╮
       ╰─╂─────────────────╂─╯
       ╭─╂────advisor 3────╂─╮
       ╰─╂─────────────────╂─╯
         ┃                 ┃
         ▼                 ▽
action(args; kargs) ━━━━▶ post╺━━▶ result

To specify which transforms a Advice should be applied to, ensure you add the relevant type parameters to your transducing function. In cases where the transducing function is not applicable, the Advice will simply act as the identity function.

After all applicable Advices have been applied, action(args...; kargs...) |> post is called to produce the final result.

The final post function is created by rightwards-composition with every post entry of the advice forms (i.e. at each stage post = post ∘ extra is run).

The overall behaviour can be thought of as shells of advice.

        ╭╌ advisor 1 ╌╌╌╌╌╌╌╌─╮
        ┆ ╭╌ advisor 2 ╌╌╌╌╌╮ ┆
        ┆ ┆                 ┆ ┆
input ━━┿━┿━━━▶ function ━━━┿━┿━━▶ result
        ┆ ╰╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╯ ┆
        ╰╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╯

Constructors

Advice(priority::Int, f::Function)
Advice(f::Function) # priority is set to 1

Examples

1. Logging every time a DataSet is loaded.

loggingadvisor = Advice(
    function(post::Function, f::typeof(load), loader::DataLoader, input, outtype)
        @info "Loading $(loader.data.name)"
        (post, f, (loader, input, outtype))
    end)

2. Automatically committing each data file write.

writecommitadvisor = Advice(
    function(post::Function, f::typeof(write), writer::DataWriter{:filesystem}, output, info)
        function writecommit(result)
            run(`git add $output`)
            run(`git commit -m "update $output"`)
            result
        end
        (post ∘ writecommit, writefn, (output, info))
    end)

source

DataToolkitCore.AdviceAmalgamation — Type

AdviceAmalgamation

An AdviceAmalgamation is a collection of Advices sourced from available Plugins.

Like individual Advices, an AdviceAmalgamation can be called as a function. However, it also supports the following convenience syntax:

(::AdviceAmalgamation)(f::Function, args...; kargs...) # -> result

Constructors

AdviceAmalgamation(advisors::Vector{Advice}, plugins_wanted::Vector{String}, plugins_used::Vector{String})
AdviceAmalgamation(plugins::Vector{String})
AdviceAmalgamation(collection::DataCollection)

source

DataToolkitCore.@advise — Macro

@advise [source] f(args...; kwargs...)

Convert a function call f(args...; kwargs...) to an advised function call, where the advise collection is obtained from source or the first data-like* value of args.

* i.e. a DataCollection, DataSet, or DataTransformer

For example, @advise myfunc(other, somedataset, rest...) is equivalent to somedataset.collection.advise(myfunc, other, somedataset, rest...).

This macro performs a fairly minor code transformation, but should improve clarity.

Consider adding a typeassert where type stability is important.

source

Advisement (join) points

Parsing and serialisation of data sets and collections

DataCollections, DataSets, and DataTransformers are advised at two stages during parsing:

When calling fromspec on the Dict representation, at the start of parsing
At the end of the fromspec function, calling identity on the object

Serialisation is performed through the tospec call, which is also advised.

The signatures of the advised function calls are as follows:

fromspec(DataCollection, spec::Dict{String, Any}; path::Union{String, Nothing})::DataCollection
identity(collection::DataCollection)::DataCollection
tospec(collection::DataCollection)::Dict

fromspec(DataSet, collection::DataCollection, name::String, spec::Dict{String, Any})::DataSet
identity(dataset::DataSet)::DataSet
tospec(dataset::DataSet)::Dict

fromspec(DT::Type{<:DataTransformer}, dataset::DataSet, spec::Dict{String, Any})::DT
identity(dt::DataTransformer)::DataTransformer
tospec(dt::DataTransformer)::Dict

Processing identifiers

Both the parsing of an Identifier from a string, and the serialisation of an Identifier to a string are advised. Specifically, the following function calls:

parse_ident(spec::AbstractString)
string(ident::Identifier)

The data flow arrows

The reading, writing, and storage of data may all be advised. Specifically, the following function calls:

load(loader::DataLoader, datahandle, as::Type)
storage(provider::DataStorage, as::Type; write::Bool)
save(writer::DataWriter, datahandle, info)

Index of advised calls (join points)

There are 32 advised function calls, across 9 files, covering 12 functions (automatically detected).

Arranged by function

`create` (1 instance)

creation.jl
- On line 51 create(DataCollection, dc) is advised within a create! method.

`fromspec` (4 instances)

creation.jl
- On line 100 fromspec(DataSet, parent, String(name), toml_safe(parent, spec)) is advised within a create method.
- On line 163 fromspec(T, parent, toml_safe(parent, spec)) is advised within a create method.
parser.jl
- On line 122 fromspec(DT, dataset, spec) is advised within a DT::Type{<:DataTransformer} method.
- On line 247 fromspec(DataSet, collection, name, spec) is advised within a DataSet method.

`identity` (4 instances)

creation.jl
- On line 10 identity(collection) is advised within a DataCollection method.
parser.jl
- On line 167 identity(DT(dataset, ttype, priority, dataset_parameters(dataset, Val(:extract), parameters))) is advised within a fromspec method.
- On line 239 identity(collection) is advised within a fromspec method.
- On line 277 identity(dataset) is advised within a fromspec method.

`lint` (1 instance)

lint.jl
- On line 84 lint(obj, linters) is advised within a lint(obj::T) method.

`load` (2 instances)

externals.jl
- On line 190 load(loader, datahandle, Tloader_out) is advised within a read1(dataset::DataSet, as::Type) method.
- On line 205 load(loader, nothing, as) is advised within a read1(dataset::DataSet, as::Type) method.

`parse_ident` (8 instances)

externals.jl
- On line 79 parse_ident(identstr) is advised within a dataset method.
- On line 84 parse_ident(identstr) is advised within a dataset method.
errors.jl
- On line 44 parse_ident(err.identifier) is advised within a Base.showerror method.
- On line 53 parse_ident(err.identifier) is advised within a Base.showerror method.
identification.jl
- On line 190 parse_ident(identstr) is advised within a resolve method.
- On line 195 parse_ident(identstr) is advised within a resolve method.
parameters.jl
- On line 41 parse_ident(dsid_match.captures[1]) is advised within a dataset_parameters method.
parser.jl
- On line 72 parse_ident(spec) is advised within a Base.parse method.

`read1` (1 instance)

externals.jl
- On line 147 read1(dataset, as) is advised within a Base.read(dataset::DataSet, #= ../../src/interaction/externals.jl:144 =# @nospecialize(as::Type)) method.

`refine` (1 instance)

identification.jl
- On line 146 refine(matchingdatasets, ident, String[]) is advised within a refine method.

`save` (1 instance)

externals.jl
- On line 368 save(writer, datahandle, info) is advised within a Base.write(dataset::DataSet, info::T) method.

`storage` (1 instance)

externals.jl
- On line 279 storage(storage_provider, Tout; write) is advised within a Base.open(data::DataSet, as::Type; write::Bool = false) method.

`string` (5 instances)

display.jl
- On line 17 string(nameonly) is advised within a Base.show method.
errors.jl
- On line 82 string(ident) is advised within a Base.showerror method.
- On line 90 string(ident) is advised within a Base.showerror method.
identification.jl
- On line 111 string(ident) is advised within a resolve method.
parameters.jl
- On line 53 string(param) is advised within a dataset_parameters method.

`tospec` (3 instances)

writer.jl
- On line 60 tospec(dt) is advised within a Base.convert method.
- On line 86 tospec(ds) is advised within a Base.convert method.
- On line 100 tospec(dc) is advised within a Base.convert method.

Arranged by file

`creation.jl` (4 instances)

On line 10 identity(collection) is advised within a DataCollection method.
On line 51 create(DataCollection, dc) is advised within a create! method.
On line 100 fromspec(DataSet, parent, String(name), toml_safe(parent, spec)) is advised within a create method.
On line 163 fromspec(T, parent, toml_safe(parent, spec)) is advised within a create method.

`display.jl` (1 instance)

On line 17 string(nameonly) is advised within a Base.show method.

`externals.jl` (7 instances)

On line 79 parse_ident(identstr) is advised within a dataset method.
On line 84 parse_ident(identstr) is advised within a dataset method.
On line 147 read1(dataset, as) is advised within a Base.read(dataset::DataSet, #= ../../src/interaction/externals.jl:144 =# @nospecialize(as::Type)) method.
On line 190 load(loader, datahandle, Tloader_out) is advised within a read1(dataset::DataSet, as::Type) method.
On line 205 load(loader, nothing, as) is advised within a read1(dataset::DataSet, as::Type) method.
On line 279 storage(storage_provider, Tout; write) is advised within a Base.open(data::DataSet, as::Type; write::Bool = false) method.
On line 368 save(writer, datahandle, info) is advised within a Base.write(dataset::DataSet, info::T) method.

`lint.jl` (1 instance)

On line 84 lint(obj, linters) is advised within a lint(obj::T) method.

`errors.jl` (4 instances)

On line 44 parse_ident(err.identifier) is advised within a Base.showerror method.
On line 53 parse_ident(err.identifier) is advised within a Base.showerror method.
On line 82 string(ident) is advised within a Base.showerror method.
On line 90 string(ident) is advised within a Base.showerror method.

`identification.jl` (4 instances)

On line 111 string(ident) is advised within a resolve method.
On line 146 refine(matchingdatasets, ident, String[]) is advised within a refine method.
On line 190 parse_ident(identstr) is advised within a resolve method.
On line 195 parse_ident(identstr) is advised within a resolve method.

`parameters.jl` (2 instances)

On line 41 parse_ident(dsid_match.captures[1]) is advised within a dataset_parameters method.
On line 53 string(param) is advised within a dataset_parameters method.

`parser.jl` (6 instances)

On line 72 parse_ident(spec) is advised within a Base.parse method.
On line 122 fromspec(DT, dataset, spec) is advised within a DT::Type{<:DataTransformer} method.
On line 167 identity(DT(dataset, ttype, priority, dataset_parameters(dataset, Val(:extract), parameters))) is advised within a fromspec method.
On line 239 identity(collection) is advised within a fromspec method.
On line 247 fromspec(DataSet, collection, name, spec) is advised within a DataSet method.
On line 277 identity(dataset) is advised within a fromspec method.

`writer.jl` (3 instances)

On line 60 tospec(dt) is advised within a Base.convert method.
On line 86 tospec(ds) is advised within a Base.convert method.
On line 100 tospec(dc) is advised within a Base.convert method.

Plugins & Advice

Advice

Advisement (join) points

Parsing and serialisation of data sets and collections

Processing identifiers

The data flow arrows

Index of advised calls (join points)

Arranged by function

create (1 instance)

fromspec (4 instances)

identity (4 instances)

lint (1 instance)

load (2 instances)

parse_ident (8 instances)

read1 (1 instance)

refine (1 instance)

save (1 instance)

storage (1 instance)

string (5 instances)

tospec (3 instances)

Arranged by file

creation.jl (4 instances)

display.jl (1 instance)

externals.jl (7 instances)

lint.jl (1 instance)

errors.jl (4 instances)

identification.jl (4 instances)

parameters.jl (2 instances)

parser.jl (6 instances)

writer.jl (3 instances)

`create` (1 instance)

`fromspec` (4 instances)

`identity` (4 instances)

`lint` (1 instance)

`load` (2 instances)

`parse_ident` (8 instances)

`read1` (1 instance)

`refine` (1 instance)

`save` (1 instance)

`storage` (1 instance)

`string` (5 instances)

`tospec` (3 instances)

`creation.jl` (4 instances)

`display.jl` (1 instance)

`externals.jl` (7 instances)

`lint.jl` (1 instance)

`errors.jl` (4 instances)

`identification.jl` (4 instances)

`parameters.jl` (2 instances)

`parser.jl` (6 instances)

`writer.jl` (3 instances)