Restriction of outputs

In some cases, the model we’re training has many outputs. In such cases, depending on the situation, it may be desirable to return only some of its outputs.

The RestrictableLayer allows you to do this, by indicating which outputs should be kept, either by name if they exist, or by number.

import os
import sys

sys.path.append(os.path.join(os.path.abspath(""), ".."))

import numpy as np

from torch import nn

from nnbma.networks import FullyConnected

As an example, we’re going to create a very simple network that we won’t be training. The aim is just to manipulate its outputs.

n_inputs, n_outputs = 2, 20
layers_sizes = [n_inputs, 10, 10, n_outputs]
activation = nn.ReLU()
outputs_names = [f"output {k+1}" for k in range(n_outputs)]

net = FullyConnected(
    layers_sizes,
    activation,
    outputs_names=outputs_names,
    last_restrictable=True,  # This parameter is True by default
)

By default, the network returns all the outputs:

x = np.random.normal(0, 1, n_inputs).astype("float32")
y = net(x)

print("Number of outputs:", y.shape[0])
print(y)

Number of outputs: 20
[-0.01225665 -0.07435182 -0.27788568 -0.08382663 -0.19809842 -0.4424935
 -0.33679342 -0.13146824  0.2730468   0.22283658 -0.26923177 -0.14347078
  0.31273493  0.3232665   0.1697111   0.3869456   0.20897529 -0.15252575
 -0.07885606  0.36726213]

We can select the outputs by their names:

net.restrict_to_output_subset(["output 5", "output 10"])
y = net(x)

print("Number of outputs:", y.shape[0])
print(y)

Number of outputs: 2
[-0.19809842  0.22283658]

Alternatively (for instance if no outputs names has been provided), we can select them by their indices (starting at zero):

net.restrict_to_output_subset([4, 9])
y = net(x)

print("Number of outputs:", y.shape[0])
print(y)

Number of outputs: 2
[-0.19809842  0.22283658]

Note that you cannot restrict the output when the current network mode is train:

net.train()
try:
    net.restrict_to_output_subset(["output 5", "output 12"])
except PermissionError as e:
    print(f"Error: {e}")

Error: You're not able to restrict the outputs when Module mode is train

Note: Any network inheriting from NeuralNetwork` has the`restrict_to_output_subset` method. In the general case, the implementation is just a selection of the chosen outputs.

If the output layer is an instance of RestrictableLayer, then this is handled more finely, as the matrix product performed by the linear layer is that of the sub-matrix of weights corresponding to the selected outputs only. If the number of selected outputs is much lower than the total number of outputs, this reduction in the size of the weight matrix will lead to a significant reduction in calculation time.