How Param Works#
Param seamlessly makes Python attributes have a much richer set of behaviors than they would otherwise, at once both more powerful (with automatic dynamic behavior) and more tightly controlled by the class author. It is natural to wonder how Param achieves this, especially given that it is a normal pure-Python library, not an alternative implementation of Python or a pre-processor. The answer is that Param makes extensive use of Python language features that allow tailoring the behavior of attribute getting and setting in sophisticated ways. You don’t need to read any of the material on this page to use Param successfully, but it might help you understand what’s going on “under the hood” for debugging or optimizing complex situations or for extending Param.
Parameter object is a type of Python “descriptor”, i.e., an object that implements custom
__set__ behavior. When a descriptor is an attribute of a class, Python will invoke those custom methods instead of simply getting and setting the actual value of the attribute (i.e., the
Parameter object). The Python descriptor docs explain this process in detail, but briefly, let’s consider a simple descriptor that returns how many times it has been accessed:
class Count: def __init__(self, start=0): self._count=start def __get__(self, obj, objtype=None): self._count += 1 return self._count
class C: x = 5 y = Count(0) c = C() c.x, c.x, c.x, c.y, c.y, c.y
(5, 5, 5, 1, 2, 3)
As you can see, class attributes
y here can both be used the same way, but
x is a normal Python attribute, returning the fixed value
5 that was set on the class, while
y is a descriptor and returns a computed value when accessed (rather than returning itself as you might think from the syntax), and thus gives a different value each time. Parameters are much more complex than the above example, but this descriptor support provides the underlying mechanism for having full-featured attribute behavior like dynamic values, bounds checking, and so on.
As described in the Parameters docs, Parameters can store a rich collection of metadata about each parameter. Storing a full object and associated dictionary of metadata for each class and instance attribute could get expensive (i.e., slow and using a lot of memory), so Parameters are implemented using slots. A slot is like a normal Python attribute, but instead of being stored in the convenient and flexible but slow
__dict__ attribute of the object, slots are stored in a fixed-size data structure
__slots__ that works like a C
__slots__ reserves just enough space to store these attributes, which can be accessed instantaneously rather than requiring a dictionary lookup (hash table search).
__slots__ requires special support for operations to copy and restore Parameters (e.g. for Python persistent storage pickling); see
__setstate__. A Parameter defines at least these slots, with additional slots added for each subclass:
__slots__ = ['name', '_internal_name', 'default', 'doc', 'precedence', 'instantiate', 'constant', 'readonly', 'pickle_default_value', 'allow_None', 'per_instance', 'watchers', 'owner', '_label']
In most cases, you can just treat a Parameter’s existing slots like attributes of the Parameter class; they work just the same as regular attributes except for speed and storage space. However, if you add a new attribute to a Parameter class, you have to make sure that you also add it to the
__slots__ defined for that Parameter class, or you’ll either get an error or else the Parameter will get an unnecessary full
__dict__ object just to hold the one new attribute.
Parameterized differ from ordinary Python classes is that they specify a special metaclass that determines how they behave. Just like you instantiate a Python class to get a Python object, you instantiate a Python metaclass to get a Python class. Most classes are instances of the default metaclass named
type, but with a custom metaclass, you can define how every Python class with that metaclass will behave, at a fundamental level.
ParameterMetaclass is fairly simple, mainly overriding docstrings so that
help(someparam) gives the declared documentation for the Parameter instance, rather than the less-useful docstring for the underlying class that it would otherwise display. This behavior is convenient, but not essential to the operation of Param.
ParameterizedMetaclass, on the other hand, defines a lot of the behavior behind Param’s features. In particular, the metaclass implements the behavior for getting and setting parameter values at the class level, similar to how a descriptor controls such behavior at the instance level. Without the metaclass, setting the value of a class attribute to a scalar like
5 would wipe out the
Parameter object rather than updating the default value. The metaclass thus performs the same role at the class level as descriptors do at the instance level. Descriptors allow setting the value of an instance attribute without overriding the
Parameter object on that instance, and the metaclass allows setting the value of a class attribute without overridding the
Parameter object on the class. All told, the ParameterizedMetaclass handles:
allowing Parameter default values to be set at the class level (as just described),
supporting inheriting Parameter objects from superclasses,
instantiating parameter default values into the class’s dictionary (if needed)
nameslot of each Parameter by its attribute name in the class,
reporting whether a class has been declared to be abstract (useful for ignoring it in selectors),
various bookkeeping about dependencies and watchers,
generating docstrings at the class level for each Parameter in the class so that
help(parameterizedclass)displays not just the class docstring but also information about the Parameters in it (or in superclasses)
Thus much of how Param works depends on ParameterizedMetaclass.
Custom attribute access#
The above mechanisms let Param customize attribute access for dynamic behavior and control over user settings. As an example of how this all fits together, consider the following code:
from param import Parameterized, Parameter class A(Parameterized): p = Parameter(default=1, per_instance=False, instantiate=False) a1 = A() a2 = A()
Here, a1 and a2 share one Parameter object (
A.__dict__['p'] is a1.param.p is a2.param.p
The default (class-attribute) value of p is stored in this Parameter object (
A.__dict__['p'].default), but is accessible as
A.p due to the Parameter being a descriptor:
If the value of
p is set on
a1’s value of
p is stored in the
a1 instance itself, under a specially mangled attribute name. The mangled name is called the
_internal_name of the parameter, and is constructed from the “attrib name” of the parameter (i.e.
p in this case) but modified so that it will not be confused with the underlying
a1.p is requested,
a1.__dict__['_p_param_value'] is returned. When
a2.p is requested,
_p_param_value is not found in
A.p) is returned instead:
Because the value for
a2.p is returned from
A.p will affect
a2.p, but not
a1.p since it has its own independent value:
A.p=3 a2.p, a1.p
p was not defined in
A but was defined in a superclass, the value found in that superclass would be returned instead.
You can re-execute the above code changing to
instantiate=True on Parameter
p and see how the behavior differs. With
per_instance=True (which would normally be the default),
a2 would each have independent copies of the
Parameter object, and with
instantiate=True, each instance would get its own copy of the class’s default value, making it immune to later changes at the class level.
Once we have Parameter descriptors and the metaclasses, there is relatively little left for the Parameterized class itself to do:
implementing the rest of dependencies and watchers
providing a constructor that lets you set instance parameters
instantiating and providing the
.paramaccessor for invoking methods and accessing the Parameter objects
handling state saving and restoring (pickling)
And that’s it for the core of Param! There are other behaviors implemented at the level of specific Parameters, but those are typically localized and can be understood by reading the class docstring for that Parameter type.