{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "8c73a0b0",
   "metadata": {},
   "source": [
    "# How Param Works\n",
    "\n",
    "Param seamlessly makes Python attributes have a much richer set of behaviors than they would otherwise, at once both more powerful (with automatic dynamic behavior) and more tightly controlled by the class author. It is natural to wonder how Param achieves this, especially given that it is a normal pure-Python library, not an alternative implementation of Python or a pre-processor. The answer is that Param makes extensive use of Python language features that allow tailoring the behavior of attribute getting and setting in sophisticated ways. You don't need to read any of the material on this page to use Param successfully, but it might help you understand what's going on \"under the hood\" for debugging or optimizing complex situations or for extending Param.\n",
    "\n",
    "## Descriptors\n",
    "\n",
    "A `Parameter` object is a type of Python \"descriptor\", i.e., an object that implements custom `__get__` and/or `__set__` behavior. When a descriptor is an attribute of a class, Python will invoke those custom methods instead of simply getting and setting the actual value of the attribute (i.e., the `Parameter` object). The [Python descriptor docs](https://docs.python.org/3/howto/descriptor.html) explain this process in detail, but briefly, let's consider a simple descriptor that returns how many times it has been accessed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "af0cddab",
   "metadata": {},
   "outputs": [],
   "source": [
    "class Count:\n",
    "    def __init__(self, start=0):\n",
    "        self._count = start\n",
    "    \n",
    "    def __get__(self, obj, objtype=None):\n",
    "        self._count += 1\n",
    "        return self._count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1488fb8c",
   "metadata": {},
   "outputs": [],
   "source": [
    "class C:\n",
    "    x = 5\n",
    "    y = Count(0)\n",
    "\n",
    "c = C()\n",
    "\n",
    "c.x, c.x, c.x, c.y, c.y, c.y"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4876a7f6",
   "metadata": {},
   "source": [
    "As you can see, class attributes `x` and `y` here can both be used the same way, but `x` is a normal Python attribute, returning the fixed value `5` that was set on the class, while `y` is a descriptor and returns a computed value when accessed (rather than returning itself as you might think from the syntax), and thus gives a different value each time. Parameters are much more complex than the above example, but this descriptor support provides the underlying mechanism for having full-featured attribute behavior like dynamic values, bounds checking, and so on."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d07cbda",
   "metadata": {},
   "source": [
    "## Slots\n",
    "\n",
    "As described in the [Parameters docs](Parameters.ipynb), Parameters can store a rich collection of metadata about each parameter. Storing a full object and associated dictionary of metadata for each class and instance attribute could get expensive (i.e., slow and using a lot of memory), so Parameters are implemented using [slots](https://docs.python.org/3/reference/datamodel.html#slots). A slot is like a normal Python attribute, but instead of being stored in the convenient and flexible but slow `__dict__` attribute of the object, slots are stored in a fixed-size data structure `__slots__` that works like a C `struct`.  `__slots__` reserves just enough space to store these attributes, which can be accessed instantaneously rather than requiring a dictionary lookup (hash table search).\n",
    "\n",
    "Using `__slots__` requires special support for operations to copy and restore Parameters (e.g. for Python persistent storage pickling); see `__getstate__` and `__setstate__`. A Parameter defines at least these slots, with additional slots added for each subclass:\n",
    "\n",
    "```\n",
    "__slots__ = ['name', 'default', 'doc',\n",
    "            'precedence', 'instantiate', 'constant', 'readonly',\n",
    "            'pickle_default_value', 'allow_None', 'per_instance',\n",
    "            'watchers', 'owner', 'allow_refs', 'nested_refs', '_label']\n",
    "```\n",
    "\n",
    "In most cases, you can just treat a Parameter's existing slots like attributes of the Parameter class; they work just the same as regular attributes except for speed and storage space. However, if you add a _new_ attribute to a Parameter class, you have to make sure that you also add it to the `__slots__` defined for that Parameter class, or you'll either get an error or else the Parameter will get an unnecessary full `__dict__` object just to hold the one new attribute. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cac0fda5",
   "metadata": {},
   "source": [
    "## Metaclasses\n",
    "\n",
    "Another way `Parameter` and `Parameterized` differ from ordinary Python classes is that they specify a special [metaclass](https://docs.python.org/3/reference/datamodel.html#metaclasses) that determines how they behave. Just like you instantiate a Python class to get a Python object, you instantiate a Python metaclass to get a Python class. Most classes are instances of the default metaclass named `type`, but with a custom metaclass, you can define how every Python class with that metaclass will behave, at a fundamental level. \n",
    "\n",
    "The `ParameterMetaclass` is fairly simple, mainly overriding docstrings so that `help(someparam)` gives the declared documentation for the Parameter instance, rather than the less-useful docstring for the underlying class that it would otherwise display. This behavior is convenient, but not essential to the operation of Param. \n",
    "\n",
    "`ParameterizedMetaclass`, on the other hand, defines a lot of the behavior behind Param's features.  In particular, the metaclass implements the behavior for getting and setting parameter values at the class level, similar to how a descriptor controls such behavior at the instance level. Without the metaclass, setting the value of a class attribute to a scalar like `5` would wipe out the `Parameter` object rather than updating the default value. The metaclass thus performs the same role at the class level as descriptors do at the instance level. Descriptors allow setting the value of an instance attribute without overriding the `Parameter` object on that instance, and the metaclass allows setting the value of a class attribute without overridding the `Parameter` object on the class. All told, the ParameterizedMetaclass handles: \n",
    "\n",
    "- allowing Parameter default values to be set at the class level (as just described),\n",
    "- supporting inheriting Parameter objects from superclasses, \n",
    "- instantiating parameter default values (if needed)\n",
    "- populating the `name` slot of each Parameter by its attribute name in the class,\n",
    "- reporting whether a class has been declared to be abstract (useful for ignoring it in selectors),\n",
    "- various bookkeeping about dependencies and watchers, \n",
    "- generating docstrings at the class level for each Parameter in the class so that `help(parameterizedclass)` displays not just the class docstring but also information about the Parameters in it (or in superclasses)\n",
    "\n",
    "Thus much of how Param works depends on ParameterizedMetaclass. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60341a22",
   "metadata": {},
   "source": [
    "## Custom attribute access\n",
    "\n",
    "The above mechanisms let Param [customize attribute access](https://docs.python.org/3/reference/datamodel.html#customizing-attribute-access) for dynamic behavior and control over user settings. As an example of how this all fits together, consider the following code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4d6e587e",
   "metadata": {},
   "outputs": [],
   "source": [
    "from param import Parameterized, Parameter\n",
    "\n",
    "class A(Parameterized):\n",
    "    p = Parameter(default=1, per_instance=False, instantiate=False)\n",
    "\n",
    "a1 = A()\n",
    "a2 = A()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c03ebf19",
   "metadata": {},
   "source": [
    "Here, a1 and a2 share one Parameter object (`A.__dict__['p']`), because `per_instance` is `False`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6eb529a8",
   "metadata": {},
   "outputs": [],
   "source": [
    "A.__dict__['p'] is a1.param.p is a2.param.p"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "03b20d6f",
   "metadata": {},
   "source": [
    "The default (class-attribute) value of p is stored in this Parameter object (`A.__dict__['p'].default`), but is accessible as `A.p` due to the Parameter being a descriptor:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0fe38b09",
   "metadata": {},
   "outputs": [],
   "source": [
    "A.__dict__['p'].default"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "886bc3d5",
   "metadata": {},
   "outputs": [],
   "source": [
    "A.p"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36ef8c10",
   "metadata": {},
   "source": [
    "If the value of `p` is set on `a1`, `a1`'s value of `p` is stored in the `a1` instance itself, in a dictionary named `values` under the private namespace `_param__private`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "90061161",
   "metadata": {},
   "outputs": [],
   "source": [
    "a1.p = 2\n",
    "a1._param__private.values['p']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b5c6a79",
   "metadata": {},
   "source": [
    "When `a1.p` is requested, `a1._param__private.values['p']` is returned. When `a2.p` is requested, `p` is not found in `a2._param__private.values`, so `A.__dict__['p'].default` (i.e. `A.p`) is returned instead:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ec1ac4fc",
   "metadata": {},
   "outputs": [],
   "source": [
    "a2.p"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "82208f0d",
   "metadata": {},
   "source": [
    "Because the value for `a2.p` is returned from `A.p`, changing `A.p` will affect `a2.p`, but not `a1.p` since it has its own independent value:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3f1d08b5",
   "metadata": {},
   "outputs": [],
   "source": [
    "A.p = 3\n",
    "a2.p, a1.p"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "88a854b5",
   "metadata": {},
   "source": [
    "If `p` was not defined in `A` but was defined in a superclass, the value found in that superclass would be returned instead. \n",
    "\n",
    "You can re-execute the above code changing to `per_instance=True` and/or `instantiate=True` on Parameter `p` and see how the behavior differs. With `per_instance=True` (which would normally be the default), `a1` and `a2` would each have independent copies of the `Parameter` object, and with `instantiate=True`, each instance would get its own copy of the class's default value, making it immune to later changes at the class level."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b86f55e3-5ae5-4879-9b4c-273e859b6fec",
   "metadata": {},
   "source": [
    "## References\n",
    "\n",
    "Beyond the custom attribute access mechanisms of a single `Parameter`, Param can link together multiple Parameters. With the `allow_refs` option, a `Parameter` can act as a dynamic reference to another `Parameter`. This enables the values of two or more Parameters to stay in sync. Any change to the referenced Parameter's value is automatically reflected in all Parameters that reference it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fba9fbdc-0776-4cd3-8250-e00487593d57",
   "metadata": {},
   "outputs": [],
   "source": [
    "class B(Parameterized):\n",
    "    \n",
    "    p = Parameter(default=1, allow_refs=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "27beb80b-415f-41cb-b909-adb6822e7e9a",
   "metadata": {},
   "source": [
    "Having declared a `Parameter` that allows references we can now pass the parameter `p` of `b1` to parameter `p` of `b2`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "22a51c29-c286-4e0f-b1a6-1e02bc4149da",
   "metadata": {},
   "outputs": [],
   "source": [
    "b1 = B(p=14)\n",
    "b2 = B(p=b1.param.p)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5eab5e13-66df-4001-8d70-2cab70449641",
   "metadata": {},
   "source": [
    "Inspecting `b2.p` we will see that `p` of `b2` now reflects the value of `b1.p`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fdc58565-f327-405f-958d-b49caa70adb3",
   "metadata": {},
   "outputs": [],
   "source": [
    "b2.p"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7322e310-9105-48c9-a0b8-41cb80bcf4e6",
   "metadata": {},
   "source": [
    "Even when we update the value of `b1.p` the value of `b2.p` will reflect the change:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "961aebc0-45b5-458d-919c-1ba6a3c286e7",
   "metadata": {},
   "outputs": [],
   "source": [
    "b1.p = 7\n",
    "\n",
    "b2.p"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75c5eb72-3132-458c-9b30-b5cd1e64b63d",
   "metadata": {},
   "source": [
    "Internally Param will [watch](Dependencies_and_Watchers.ipynb#watchers) all Parameters associated with the reference by calling `param.parameterized.resolve_ref` and then use `param.parameterized.resolve_value` to resolve the current value of the reference.\n",
    "\n",
    "If we explicitly set `b2.p` however the two values will become unsynced, i.e. the Watcher created when setting the reference will be removed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2e02cb21-a4ab-48fb-a516-da17feaf0da0",
   "metadata": {},
   "outputs": [],
   "source": [
    "b2.p = 3\n",
    "\n",
    "print(b1.p, b2.p)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "701e26c3-6eb0-4c76-b453-cc37be06ac41",
   "metadata": {},
   "source": [
    "and any subsequent changes to `b1.p` will not be reflected by `b2`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "50aedbea-7902-4f1a-8e98-6866a965eda8",
   "metadata": {},
   "outputs": [],
   "source": [
    "b1.p = 27\n",
    "\n",
    "b2.p"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07d982cd-c0f1-496f-8086-28a882266901",
   "metadata": {},
   "source": [
    "If a new reference is assigned then the old `Watcher`(s) will be removed and if the new value also represents a reference new `Watcher`(s) will be registered."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3e3d3aa",
   "metadata": {},
   "source": [
    "# Other notes\n",
    "\n",
    "Once we have Parameter descriptors and the metaclasses, there is relatively little left for the Parameterized class itself to do:\n",
    "\n",
    "- implementing the rest of [dependencies and watchers](Dependencies_and_Watchers.ipynb)\n",
    "- providing a constructor that lets you set instance parameters\n",
    "- instantiating and providing the `.param` accessor for invoking methods and accessing the Parameter objects\n",
    "- handling state saving and restoring (pickling)\n",
    "\n",
    "And that's it for the core of Param! There are other behaviors implemented at the level of specific Parameters, but those are typically localized and can be understood by reading the class docstring for that Parameter type."
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python",
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}