Modeling Layer

Classes like carthage.Network and carthage.machine.AbstractMachineModel provide an abstract interface to infrastructure resources.

The modeling layer provides a generally declarative interface for defining and configuring such models. The modeling layer provides a domain-specific language for describing models. Python metaclasses are used to modify Python’s behavior in a number of ways to provide a more concise language for describing models.

A Simple Model

A simple modeling layout to define a machine

# Copyright (C) 2021, Hadron Industries, Inc.
# Carthage is free software; you can redistribute it and/or modify
# it under the terms of the GNU Lesser General Public License version 3

from carthage.modeling import *

class layout(CarthageLayout):

    layout_name = "example_1"

    class foo(MachineModel):

        name = "foo.com"

With such a model, one might instantiate the layout by applying an injector:

layout_instance = injector(layout)

The layout class is an instance of CarthageLayout which is a kind of InjectableModelType. By default each assignment of a type in the class body of a InjectableModelType is turned into a runtime instantiation. This means that while layout.foo is a class (or actually a class property), layout_instance.foo is an injector_access. The first time layout_instance.foo is accessed, layout_instance.injector is used to instantiate it. Thereafter, layout_instance.foo is an instance of layout.foo.

The Modeling Language

Modeling classes are divided into several types (metaclasses). Names that include the word modeling are internal. Users may need to know about their attributes, but these classes should only be used in extending the modeling layer. Classes containing model in their name are directly usable in layouts. This section describes the behavior of the modeling types that make up the modeling language.

Model classes sometimes involve a new construct called a modelmethod. Unlike other types of methods, modelmethods are available in the class body. For example, add_provider can be used to indicate that on class instantiation, some object should be added to an InjectableModel‘s injector:

class foo(InjectableModel):
    add_provider(InjectionKey("baz"), Network)

class carthage.modeling.ModelingBase

All modeling classes derive their type from ModelingBase and have the following behaviors:

Unlike normal Python, an inner class can access the attributes of an outer class while the class body is being defined:
```
class foo(metaclass  = ModelingBase):
    attr = 32
    b = attr+1
    class bar(metaclass = ModelingBase):
        a = b+1
        attr = 64
```
In the above example, while the body of bar is being defined, attr and b are available.

However, only variables that are actually set in a class body survive into the actual class. So in the above example, foo.bar.a and foo.bar.attr are set in the resulting class. While it was used in the class body, foo.bar.b will raise AttributeError. If an attribute should be copied into an inner class, the following will work:
```
class outer(metaclass = ModelingBase):
    outer_attr = []
    class inner(metaclass = ModelingBase):
        outer_attr = outer_attr
```
ModelingBases support the modeling decorators.

The dynamic_name() decorator can be used to change the name under which an assignment is stored. This permits programatic creation of several classes in a loop:

class example(metaclass = ModelingBase):

    # create a machine for each user
    for u in users:
        @dynamic_name(f'{u}_workstation')
        class workstation(MachineModel): # ...
    del u #to avoid polluting class namespace Now we have
    #several workstation inner classes, named based on the
    #argument to dynamic_name rather than each being called
    #workstation.

The dynamic_name decorator is particularly useful with injectors where it can be used to build up a set of machines that can be selected using Injector.filter_instantiate().

class carthage.modeling.InjectableModel

class carthage.modeling.InjectableModelType

InjectableModel represents an Injectable. InjectableModels have the following attributes:

InjectableModels automatically have an Injector injected and made available as the injector attribute.
By default, any attribute assigned a value in the body of the class is also added as a provider to the injector in the class using the attribute name as a key. That is:
```
class foo(InjectableModel):
    attr = "This String"

foo_instance = injector(foo)
assert foo_instance.injector.get_instance(InjectionKey("attr")) == foo_instance.attr == foo.attr
```
This makes it very convenient to refer to networks and to construct instances that need to be constructed in an asynchronous context. Ideally there would be a decorator to turn this behavior off for a particular assignment, but currently there is not.
By default, any attribute in the class body assigned a value that is a type (or that has a transclusion key) will be transformed into an injector_access(). When accessed through the class, the injector_access will act as a class property returning the value originally assigned to the attribute. That is, class access generally works as if no transformation had taken place. However, when accessed as an instance property, the get_instance method on the Injector will be used to instantiate the class. See the first example for an example. If this transformation is not desired use the no_instantiate() decorator.
Certain classes such as carthage.network.NetworkConfig will automatically be added to an injector if they are assigned to an attribute in the class body.
The provides() and globally_unique_key() decorators can be used to add additional InjectionKeys by which a value can be known.
The allow_multiple() and no_close() decorators can modify how a value is added to the injector.

Decorators are designed to be applied to classes or functions. If modeling decorators need to be applied to other values the following syntax can be used:

external_object = no_close()(object)
val_with_extra_keys = provides(InjectionKey("an_extra_key"))(val)

The dynamic_name() decorator is powerful when used with InjectableModel. As an example, a collection of machines can be created:

class machine_enclave(Enclave):

    domain = "example.com"
    for i in range(1,5):
        @dynamic_name(f'server_{i}')
        @globally_unique_key(InjectionKey(MachineModel, host = f'server-{i}.{domain}'))
        class machine(MachineModel):
            name = f"server-{i}"

Note that the call to globally_unique_key() is included only for illustrative purposes. The our_key() method of MachineModel accomplishes the same goal.

With a layout like the above, machine models are available as machine_enclave.server_1. But once the layout is instantiated, the injector can also be used:

machines = injector(machine_enclave)
machines.injector.get_instance("server_1")
#also available with the global key
machines.injector.get_instance(InjectionKey(MachineModel, host = "server-1.example.com"))
#Or available all at once:
all_machines = machines.injector.filter_instantiate(MachineModel, ['host'], stop_at = machines.injector)

add_provider(key: InjectionKey, value, **options)

Adds key to the set of keys that will be registered with an instance’s injector when the model is instantiated. Eventually, in class initialization, code similar to the following will be called:

self.injector.add_provider(key, value, **options)

class carthage.modeling.ModelContainer

class carthage.modeling.ModelingContainer

InjectableModel provides downward propagation. That is, names defined in outer classes are available at class definition time in inner classes. Since injector_access() is used to instantiate inner classes, this means that the parent injector for the inner class is the outer class. Thus, attributes and provided dependencies made available in the outer class are available in the inner class at runtime through the injector hierarchy.

Sometimes upward propagation is desired. Consider the following example:

# Copyright (C) 2021, Hadron Industries, Inc.
# Carthage is free software; you can redistribute it and/or modify
# it under the terms of the GNU Lesser General Public License version 3

from carthage import *
from carthage.modeling import *

class layout(CarthageLayout):

    class it_com(Enclave):

        domain = "it.com"

        class server(MachineModel): pass

    class bank_com(Enclave):

        domain = "bank.com"

        class server(MachineModel): pass

In this example machines can be accessed as layout.bank_com.server and layout.it_com.server. Once instantiated, the following injector access also works:

l = injector(layout)
l.bank_com.injector.get_instance(InjectionKey(MachineModel, host = "server.bank.com"))
l.it_com.injector.get_instance(InjectionKey(MachineModel, host = "server.it.com"))

But you might want to look at machines without knowing where they are defined in the hierarchy:

l.injector.get_instance(InjectionKey(MachineModel, host = "server.it.com"))
# Or all the machines in the entire layout
l.injector.filter(MachineModel, ['host'], stop_at = l.injector)

Modeling containers provide upward propagation so these calls work:

entries registered in l.it_com.injector are propagated so they are available in l.injector. That’s the opposite direction of how injectors normally work. Upward propagation is only at model definition time; the set of items to be propagated are collected statically as the class is defined. Items added to injectors at runtime are not automatically propagated up.

For upward propagation to work, containers must provide dependencies for some InjectionKey, and that key must have some constraints associated with it. For example, Enclave‘s our_key method provides InjectionKey(Enclave, domain = self.domain). If keys with constraints are marked with propagate_key(), then those are used. If not, then all keys with constraints are used.

When one container is added to another, all the container propagations in the inner container are propagated to the outer container as follows:

If the propagation has a globally_unique_key(), then that key is registered unmodified in the outer container.
If there is no globally unique key, then the constraints of the propagation’s key are merged with the constraints of the key under which the inner container is registered with the outer container. Consider an inner container InjectionKey(Enclave, domain="it.com") and a propagation of InjectionKey(Network, role = "site"). Within the inner container, the network can be accessed using InjectionKey(Network, role = "site"). After the constraints are merged, the network can be accessed in the outer container as InjectionKey(Network, role = "site", domain = "it.com").

The injector_xref() facility is used so that instantiating the key in the outer container both instantiates the inner container and the object within it.

Only the following objects are considered for propagation:

Any ModelContainer including MachineModel, NetworkModel, ModelGroup, ModelContainer, and Enclave is propagated.
The propagate_key() decorator can be used to request propagation for other objects.

Base Models

class carthage.modeling.NetworkModel(**kwargs)

class carthage.modeling.NetworkConfigModel(*args, _not_transcluded=None, **kwargs)

class carthage.modeling.MachineModel(**kwargs)

Represents the aspects of a Machine that are independent of the implementation of that machine. Typically includes things like:

Network configuration (NetworkConfig

Configuration for devops tools like Ansible

Selecting the target implementation (whether the machine will be a VM, container, or hosted on real hardware)

Applications that want to reason about the abstract environment typically only need to instantiate models. Applications that want to build VMs or effect changes to real hardware instantiate the machine implementations. This class is the modeling extensions to carthage.machine.AbstractMachineModel.

If a MachineModel contains reference to setup_tasks, then it will automatically gain SetupTaskMixin as a base class. Similarly, if model_mixin_for() is used to decorate a class in the same CarthageLayout encountered before the MachineModel, then that class will be added as an implicit base class of the model.

Class Parameters passed in as keywords on the class statement:

param template:

True if this class represents a base class or mixin rather than a actual model of a specific machine.

param mixin_key:

The InjectionKey to use to search for mixins.

Any carthage.network.NetworkConfig present in the MachineModel will be used as the network configuration.

Every carthage.machine.BaseCustomization (including MachineCustomization, FilesystemCustomization and ContainerCustomization) will be integrated into the resulting machine:

If the customization is called cust, then a method will be added to the machine cust_task which is a carthage.machine.customization_task() to run the customization.

On the model, cust will become a method that will produce an instance of the customization applied to the machine.

For example:
class server(MachineModel):

    class install_software(MachineCustomization):

        webserver_role = ansible_role_task('webserver')

        database_role = ansible_role_task('database')
Then server.machine will have a method install_software_task which will run both ansible roles assuming they have not already been run. model.install_software will produce an instance of the customization applied to model.machine. model.install_software.database_role() is a method that will force the database_role to run even if it appears up to date.

class carthage.modeling.ModelGroup(*args, _not_transcluded=None, **kwargs)

class carthage.modeling.Enclave(*args, _not_transcluded=None, **kwargs)

class carthage.modeling.CarthageLayout(**kwargs)

CarthageLayout is typically the top level class in a set of Carthage models. It is a ModelGroup that represents a complete collection of objects modeled by Carthage. The primary purpose of this class is to signify the top of a collection of objects; the layout is generally the place to start when examining a collection of models.

However, a layout differs from a ModelGroup in two ways:

carthage-runner looks for a CarthageLayout to instantiate after loading plugins. If the console is used, the layout is made available in the layout local variable of the console. If a command is run, the command is run in the context of the layout.
Layouts that set the carthage.kvstore.persistent_seed_path in the context of their Injector will have persistent assignments of things like IP addresses and MAC addresses loaded from the seed path when instantiated.

class carthage.modeling.ModelTasks(*args, _not_transcluded=None, **kwargs)

A grouping of tasks that will be run at generate time in a CarthageLayout. As part of ModelGroup.generate(), the layout searches for any ModelTasks provided by its injector and instantiates them. This causes any setup_tasks to be run.

All ModelTasks have a name, which forms part of their key. If there needs to be an ordering between tasks, the tasks can inject a dependency on other ModelTasks.

Example usage:

class layout(CarthageLayout):

    class mt1(ModelTasks):

        @setup_task("some task")
        def some_task(self): # do stuff

The async_ready method will only be called during generate. However ModelTasks will be instantiated whenever at least resolve_networking() is called.

Decorators

@carthage.modeling.no_close

@carthage.modeling.allow_multiple

@carthage.modeling.provides: Indicate that the decorated value provides these InjectionKeys

@carthage.modeling.globally_unique_key(key: InjectionKey | Callable[[object], InjectionKey])

Decorate a value to indicate that key is a globally unique InjectionKey that should provide the given value. Globally unique keys are not extended with additional constraints when propagated up through ModelingContainers.

Parameters:: key – A callback that maps the value to a key. Alternatively, simply the InjectionKey to use.

@carthage.modeling.propagate_key(key, obj=None)

Indicate a set of keys that should be propagated up in a container:

class foo(ModelingContainer):
    @propagate_key(InjectionKey(Baz, target = 42))
    class ourbaz(Baz): ...

When foo is included ain a container, then the Baz injection key will be propagated up to dependencies provided by that container. Since the key was not marked globally unique, constraints from foo.our_key() will be added to it as it is propagated.

keys are also provided by the contained class as if provides() or globally_unique() were called.

Propagating a key up is typically an interface point; rather than propagating all keys related to an object up, propagate the keys that will be understood by the environment. Examples of usage include:

Any AbstractMachineModel with a host constraint is collected to find all the machine models in a layout

@carthage.modeling.no_instantiate

@carthage.modeling.transclude_overrides

Decorator indicating that the decorated item should be overridden by one from the injector against which the containing layout is eventually instantiated. When a InjectableModel is eventually instantiated, before an overridable key is added to the local injector, it is searched for in the instantiating injector. If it is found, the key is not registered. This has the effect of using the dependency provider in the instantiating injector rather than the one included in the layout.

Parameters:: key – If supplied, is the key expected to be registered with the transcluding injector to override this object. If not supplied, the object must have a globally unique key.

Example usage:

class layout(CarthageLayout):

    @transclude_overrides(key=InjectionKey("network"))
    class network(NetworkModel):
        # ...

If layout is instantiated in a injector that provides a dependency for InjectionKey('network'), then that object will be used rather than the network class within the layout. Since the network property is an injector_acess(), layout_instance.network will refer to the object in the instantiating injector rather than an instance of layout.network.

carthage.modeling.injector_access(key: InjectionKey, target: type = None)

Usage:

val = injector_access("foo")

At runtime, model_instance.val will be memoized to model_instance.injector.get_instance("foo").