How Param Works

Param seamlessly makes Python attributes have a much richer set of behaviors than they would otherwise, at once both more powerful (with automatic dynamic behavior) and more tightly controlled by the class author. It is natural to wonder how Param achieves this, especially given that it is a normal pure-Python library, not an alternative implementation of Python or a pre-processor. The answer is that Param makes extensive use of Python language features that allow tailoring the behavior of attribute getting and setting in sophisticated ways. You don’t need to read any of the material on this page to use Param successfully, but it might help you understand what’s going on “under the hood” for debugging or optimizing complex situations or for extending Param.

Descriptors

A Parameter object is a type of Python “descriptor”, i.e., an object that implements custom __get__ and/or __set__ behavior. When a descriptor is an attribute of a class, Python will invoke those custom methods instead of simply getting and setting the actual value of the attribute (i.e., the Parameter object). The Python descriptor docs explain this process in detail, but briefly, let’s consider a simple descriptor that returns how many times it has been accessed:

class Count:
    def __init__(self, start=0):
        self._count=start
    
    def __get__(self, obj, objtype=None):
        self._count += 1
        return self._count
class C:
    x = 5
    y = Count(0)

c = C()

c.x, c.x, c.x, c.y, c.y, c.y
(5, 5, 5, 1, 2, 3)

As you can see, class attributes x and y here can both be used the same way, but x is a normal Python attribute, returning the fixed value 5 that was set on the class, while y is a descriptor and returns a computed value when accessed (rather than returning itself as you might think from the syntax), and thus gives a different value each time. Parameters are much more complex than the above example, but this descriptor support provides the underlying mechanism for having full-featured attribute behavior like dynamic values, bounds checking, and so on.

Slots

As described in the Parameters docs, Parameters can store a rich collection of metadata about each parameter. Storing a full object and associated dictionary of metadata for each class and instance attribute could get expensive (i.e., slow and using a lot of memory), so Parameters are implemented using slots. A slot is like a normal Python attribute, but instead of being stored in the convenient and flexible but slow __dict__ attribute of the object, slots are stored in a fixed-size data structure __slots__ that works like a C struct. __slots__ reserves just enough space to store these attributes, which can be accessed instantaneously rather than requiring a dictionary lookup (hash table search).

Using __slots__ requires special support for operations to copy and restore Parameters (e.g. for Python persistent storage pickling); see __getstate__ and __setstate__. A Parameter defines at least these slots, with additional slots added for each subclass:

__slots__ = ['name', '_internal_name', 'default', 'doc', 'precedence', 
             'instantiate', 'constant', 'readonly', 'pickle_default_value',
             'allow_None', 'per_instance', 'watchers', 'owner', '_label']

In most cases, you can just treat a Parameter’s existing slots like attributes of the Parameter class; they work just the same as regular attributes except for speed and storage space. However, if you add a new attribute to a Parameter class, you have to make sure that you also add it to the __slots__ defined for that Parameter class, or you’ll either get an error or else the Parameter will get an unnecessary full __dict__ object just to hold the one new attribute.

Metaclasses

Another way Parameter and Parameterized differ from ordinary Python classes is that they specify a special metaclass that determines how they behave. Just like you instantiate a Python class to get a Python object, you instantiate a Python metaclass to get a Python class. Most classes are instances of the default metaclass named type, but with a custom metaclass, you can define how every Python class with that metaclass will behave, at a fundamental level.

The ParameterMetaclass is fairly simple, mainly overriding docstrings so that help(someparam) gives the declared documentation for the Parameter instance, rather than the less-useful docstring for the underlying class that it would otherwise display. This behavior is convenient, but not essential to the operation of Param.

ParameterizedMetaclass, on the other hand, defines a lot of the behavior behind Param’s features. In particular, the metaclass implements the behavior for getting and setting parameter values at the class level, similar to how a descriptor controls such behavior at the instance level. Without the metaclass, setting the value of a class attribute to a scalar like 5 would wipe out the Parameter object rather than updating the default value. The metaclass thus performs the same role at the class level as descriptors do at the instance level. Descriptors allow setting the value of an instance attribute without overriding the Parameter object on that instance, and the metaclass allows setting the value of a class attribute without overridding the Parameter object on the class. All told, the ParameterizedMetaclass handles:

  • allowing Parameter default values to be set at the class level (as just described),

  • supporting inheriting Parameter objects from superclasses,

  • instantiating parameter default values into the class’s dictionary (if needed)

  • populating the name slot of each Parameter by its attribute name in the class,

  • reporting whether a class has been declared to be abstract (useful for ignoring it in selectors),

  • various bookkeeping about dependencies and watchers,

  • generating docstrings at the class level for each Parameter in the class so that help(parameterizedclass) displays not just the class docstring but also information about the Parameters in it (or in superclasses)

Thus much of how Param works depends on ParameterizedMetaclass.

Custom attribute access

The above mechanisms let Param customize attribute access for dynamic behavior and control over user settings. As an example of how this all fits together, consider the following code:

from param import Parameterized, Parameter

class A(Parameterized):
    p = Parameter(default=1, per_instance=False, instantiate=False)

a1 = A()
a2 = A()

Here, a1 and a2 share one Parameter object (A.__dict__['p']), because per_instance is False:

A.__dict__['p'] is a1.param.p is a2.param.p
True

The default (class-attribute) value of p is stored in this Parameter object (A.__dict__['p'].default), but is accessible as A.p due to the Parameter being a descriptor:

A.__dict__['p'].default
1
A.p
1

If the value of p is set on a1, a1’s value of p is stored in the a1 instance itself, under a specially mangled attribute name. The mangled name is called the _internal_name of the parameter, and is constructed from the “attrib name” of the parameter (i.e. p in this case) but modified so that it will not be confused with the underlying Parameter object p:

a1.p=2
a1.__dict__['_p_param_value']
2

When a1.p is requested, a1.__dict__['_p_param_value'] is returned. When a2.p is requested, _p_param_value is not found in a2.__dict__, so A.__dict__['p'].default (i.e. A.p) is returned instead:

a2.p
1

Because the value for a2.p is returned from A.p, changing A.p will affect a2.p, but not a1.p since it has its own independent value:

A.p=3
a2.p, a1.p
(3, 2)

If p was not defined in A but was defined in a superclass, the value found in that superclass would be returned instead.

You can re-execute the above code changing to per_instance=True and/or instantiate=True on Parameter p and see how the behavior differs. With per_instance=True (which would normally be the default), a1 and a2 would each have independent copies of the Parameter object, and with instantiate=True, each instance would get its own copy of the class’s default value, making it immune to later changes at the class level.

Other notes

Once we have Parameter descriptors and the metaclasses, there is relatively little left for the Parameterized class itself to do:

  • implementing the rest of dependencies and watchers

  • providing a constructor that lets you set instance parameters

  • instantiating and providing the .param accessor for invoking methods and accessing the Parameter objects

  • handling state saving and restoring (pickling)

And that’s it for the core of Param! There are other behaviors implemented at the level of specific Parameters, but those are typically localized and can be understood by reading the class docstring for that Parameter type.