How Param Works#

Param seamlessly makes Python attributes have a much richer set of behaviors than they would otherwise, at once both more powerful (with automatic dynamic behavior) and more tightly controlled by the class author. It is natural to wonder how Param achieves this, especially given that it is a normal pure-Python library, not an alternative implementation of Python or a pre-processor. The answer is that Param makes extensive use of Python language features that allow tailoring the behavior of attribute getting and setting in sophisticated ways. You don’t need to read any of the material on this page to use Param successfully, but it might help you understand what’s going on “under the hood” for debugging or optimizing complex situations or for extending Param.

Descriptors#

A Parameter object is a type of Python “descriptor”, i.e., an object that implements custom __get__ and/or __set__ behavior. When a descriptor is an attribute of a class, Python will invoke those custom methods instead of simply getting and setting the actual value of the attribute (i.e., the Parameter object). The Python descriptor docs explain this process in detail, but briefly, let’s consider a simple descriptor that returns how many times it has been accessed:

class Count:
    def __init__(self, start=0):
        self._count = start
    
    def __get__(self, obj, objtype=None):
        self._count += 1
        return self._count

class C:
    x = 5
    y = Count(0)

c = C()

c.x, c.x, c.x, c.y, c.y, c.y

(5, 5, 5, 1, 2, 3)

As you can see, class attributes x and y here can both be used the same way, but x is a normal Python attribute, returning the fixed value 5 that was set on the class, while y is a descriptor and returns a computed value when accessed (rather than returning itself as you might think from the syntax), and thus gives a different value each time. Parameters are much more complex than the above example, but this descriptor support provides the underlying mechanism for having full-featured attribute behavior like dynamic values, bounds checking, and so on.

Slots#

As described in the Parameters docs, Parameters can store a rich collection of metadata about each parameter. Storing a full object and associated dictionary of metadata for each class and instance attribute could get expensive (i.e., slow and using a lot of memory), so Parameters are implemented using slots. A slot is like a normal Python attribute, but instead of being stored in the convenient and flexible but slow __dict__ attribute of the object, slots are stored in a fixed-size data structure __slots__ that works like a C struct. __slots__ reserves just enough space to store these attributes, which can be accessed instantaneously rather than requiring a dictionary lookup (hash table search).

Using __slots__ requires special support for operations to copy and restore Parameters (e.g. for Python persistent storage pickling); see __getstate__ and __setstate__. A Parameter defines at least these slots, with additional slots added for each subclass:

__slots__ = ['name', 'default', 'doc',
            'precedence', 'instantiate', 'constant', 'readonly',
            'pickle_default_value', 'allow_None', 'per_instance',
            'watchers', 'owner', 'allow_refs', 'nested_refs', '_label']

In most cases, you can just treat a Parameter’s existing slots like attributes of the Parameter class; they work just the same as regular attributes except for speed and storage space. However, if you add a new attribute to a Parameter class, you have to make sure that you also add it to the __slots__ defined for that Parameter class, or you’ll either get an error or else the Parameter will get an unnecessary full __dict__ object just to hold the one new attribute.

Metaclasses#

Another way Parameter and Parameterized differ from ordinary Python classes is that they specify a special metaclass that determines how they behave. Just like you instantiate a Python class to get a Python object, you instantiate a Python metaclass to get a Python class. Most classes are instances of the default metaclass named type, but with a custom metaclass, you can define how every Python class with that metaclass will behave, at a fundamental level.

The ParameterMetaclass is fairly simple, mainly overriding docstrings so that help(someparam) gives the declared documentation for the Parameter instance, rather than the less-useful docstring for the underlying class that it would otherwise display. This behavior is convenient, but not essential to the operation of Param.

ParameterizedMetaclass, on the other hand, defines a lot of the behavior behind Param’s features. In particular, the metaclass implements the behavior for getting and setting parameter values at the class level, similar to how a descriptor controls such behavior at the instance level. Without the metaclass, setting the value of a class attribute to a scalar like 5 would wipe out the Parameter object rather than updating the default value. The metaclass thus performs the same role at the class level as descriptors do at the instance level. Descriptors allow setting the value of an instance attribute without overriding the Parameter object on that instance, and the metaclass allows setting the value of a class attribute without overridding the Parameter object on the class. All told, the ParameterizedMetaclass handles:

  • allowing Parameter default values to be set at the class level (as just described),

  • supporting inheriting Parameter objects from superclasses,

  • instantiating parameter default values (if needed)

  • populating the name slot of each Parameter by its attribute name in the class,

  • reporting whether a class has been declared to be abstract (useful for ignoring it in selectors),

  • various bookkeeping about dependencies and watchers,

  • generating docstrings at the class level for each Parameter in the class so that help(parameterizedclass) displays not just the class docstring but also information about the Parameters in it (or in superclasses)

Thus much of how Param works depends on ParameterizedMetaclass.

Custom attribute access#

The above mechanisms let Param customize attribute access for dynamic behavior and control over user settings. As an example of how this all fits together, consider the following code:

from param import Parameterized, Parameter

class A(Parameterized):
    p = Parameter(default=1, per_instance=False, instantiate=False)

a1 = A()
a2 = A()

Here, a1 and a2 share one Parameter object (A.__dict__['p']), because per_instance is False:

A.__dict__['p'] is a1.param.p is a2.param.p

True

The default (class-attribute) value of p is stored in this Parameter object (A.__dict__['p'].default), but is accessible as A.p due to the Parameter being a descriptor:

A.__dict__['p'].default

1

A.p

1

If the value of p is set on a1, a1’s value of p is stored in the a1 instance itself, in a dictionary named values under the private namespace _param__private:

a1.p = 2
a1._param__private.values['p']

2

When a1.p is requested, a1._param__private.values['p'] is returned. When a2.p is requested, p is not found in a2._param__private.values, so A.__dict__['p'].default (i.e. A.p) is returned instead:

a2.p

1

Because the value for a2.p is returned from A.p, changing A.p will affect a2.p, but not a1.p since it has its own independent value:

A.p = 3
a2.p, a1.p

(3, 2)

If p was not defined in A but was defined in a superclass, the value found in that superclass would be returned instead.

You can re-execute the above code changing to per_instance=True and/or instantiate=True on Parameter p and see how the behavior differs. With per_instance=True (which would normally be the default), a1 and a2 would each have independent copies of the Parameter object, and with instantiate=True, each instance would get its own copy of the class’s default value, making it immune to later changes at the class level.

References#

Beyond the custom attribute access mechanisms of a single Parameter, Param can link together multiple Parameters. With the allow_refs option, a Parameter can act as a dynamic reference to another Parameter. This enables the values of two or more Parameters to stay in sync. Any change to the referenced Parameter’s value is automatically reflected in all Parameters that reference it.

class B(Parameterized):
    
    p = Parameter(default=1, allow_refs=True)

Having declared a Parameter that allows references we can now pass the parameter p of b1 to parameter p of b2:

b1 = B(p=14)
b2 = B(p=b1.param.p)

Inspecting b2.p we will see that p of b2 now reflects the value of b1.p:

b2.p

14

Even when we update the value of b1.p the value of b2.p will reflect the change:

b1.p = 7

b2.p

7

Internally Param will watch all Parameters associated with the reference by calling param.parameterized.resolve_ref and then use param.parameterized.resolve_value to resolve the current value of the reference.

If we explicitly set b2.p however the two values will become unsynced, i.e. the Watcher created when setting the reference will be removed:

b2.p = 3

print(b1.p, b2.p)

7 3

and any subsequent changes to b1.p will not be reflected by b2:

b1.p = 27

b2.p

3

If a new reference is assigned then the old Watcher(s) will be removed and if the new value also represents a reference new Watcher(s) will be registered.

Other notes#

Once we have Parameter descriptors and the metaclasses, there is relatively little left for the Parameterized class itself to do:

  • implementing the rest of dependencies and watchers

  • providing a constructor that lets you set instance parameters

  • instantiating and providing the .param accessor for invoking methods and accessing the Parameter objects

  • handling state saving and restoring (pickling)

And that’s it for the core of Param! There are other behaviors implemented at the level of specific Parameters, but those are typically localized and can be understood by reading the class docstring for that Parameter type.