transformer_rankers.models.pointwise_bert.BertForPointwiseLearning¶
-
class
transformer_rankers.models.pointwise_bert.
BertForPointwiseLearning
(config, loss_function='cross-entropy', smoothing=0.1)[source]¶ Bases:
transformers.modeling_bert.BertPreTrainedModel
BERT based model for pointwise learning to rank. It is almost identical to huggingface’s BertForSequenceClassification, for the case when num_labels >1 (classification).
-
__init__
(config, loss_function='cross-entropy', smoothing=0.1)[source]¶ Initializes internal Module state, shared by both nn.Module and ScriptModule.
Methods
__init__
(config[, loss_function, smoothing])Initializes internal Module state, shared by both nn.Module and ScriptModule.
Add a memory hook before and after each sub-module forward pass to record increase in memory consumption.
add_module
(name, module)Adds a child module to the current module.
adjust_logits_during_generation
(logits, **kwargs)Implement in subclasses of
PreTrainedModel
for custom behavior to adjust the logits in the generate method.apply
(fn)Applies
fn
recursively to every submodule (as returned by.children()
) as well as self.bfloat16
()Casts all floating point parameters and buffers to
bfloat16
datatype.buffers
([recurse])Returns an iterator over module buffers.
children
()Returns an iterator over immediate children modules.
cpu
()Moves all model parameters and buffers to the CPU.
cuda
([device])Moves all model parameters and buffers to the GPU.
double
()Casts all floating point parameters and buffers to
double
datatype.enforce_repetition_penalty_
(lprobs, …)Enforce the repetition penalty (from the CTRL paper).
eval
()Sets the module in evaluation mode.
Set the extra representation of the module
float
()Casts all floating point parameters and buffers to float datatype.
forward
([input_ids, attention_mask, …])Defines the computation performed at every call.
Instantiate a pretrained pytorch model from a pre-trained model configuration.
generate
([input_ids, max_length, …])Generates sequences for models with a language modeling head.
get_extended_attention_mask
(attention_mask, …)Makes broadcastable attention and causal masks so that future and masked tokens are ignored.
get_head_mask
(head_mask, num_hidden_layers)Prepare the head mask if needed.
Returns the model’s input embeddings.
Returns the model’s output embeddings.
half
()Casts all floating point parameters and buffers to
half
datatype.Initializes and prunes weights if needed.
invert_attention_mask
(encoder_attention_mask)Invert an attention mask (e.g., switches 0.
load_state_dict
(state_dict[, strict])Copies parameters and buffers from
state_dict
into this module and its descendants.load_tf_weights
(config, tf_checkpoint_path)Load tf checkpoints in a pytorch model.
modules
()Returns an iterator over all modules in the network.
named_buffers
([prefix, recurse])Returns an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself.
Returns an iterator over immediate children modules, yielding both the name of the module as well as the module itself.
named_modules
([memo, prefix])Returns an iterator over all modules in the network, yielding both the name of the module as well as the module itself.
named_parameters
([prefix, recurse])Returns an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself.
num_parameters
([only_trainable])Get the number of (optionally, trainable) parameters in the model.
parameters
([recurse])Returns an iterator over module parameters.
postprocess_next_token_scores
(scores, …)prepare_inputs_for_generation
(input_ids, …)Implement in subclasses of
PreTrainedModel
for custom behavior to prepare inputs in the generate method.prune_heads
(heads_to_prune)Prunes heads of the base model.
register_backward_hook
(hook)Registers a backward hook on the module.
register_buffer
(name, tensor)Adds a persistent buffer to the module.
register_forward_hook
(hook)Registers a forward hook on the module.
Registers a forward pre-hook on the module.
register_parameter
(name, param)Adds a parameter to the module.
requires_grad_
([requires_grad])Change if autograd should record operations on parameters in this module.
Reset the
mem_rss_diff
attribute of each module (seeadd_memory_hooks()
).resize_token_embeddings
([new_num_tokens])Resizes input token embeddings matrix of the model if
new_num_tokens != config.vocab_size
.save_pretrained
(save_directory)Save a model and its configuration file to a directory, so that it can be re-loaded using the :func:`~transformers.PreTrainedModel.from_pretrained` class method.
set_input_embeddings
(value)Set model’s input embeddings
share_memory
()state_dict
([destination, prefix, keep_vars])Returns a dictionary containing a whole state of the module.
Tie the weights between the input embeddings and the output embeddings.
to
(*args, **kwargs)Moves and/or casts the parameters and buffers.
train
([mode])Sets the module in training mode.
type
(dst_type)Casts all parameters and buffers to
dst_type
.Sets gradients of all model parameters to zero.
Attributes
authorized_missing_keys
The main body of the model.
base_model_prefix
The device on which the module is (assuming that all the module parameters are on the same device).
The dtype of the module (assuming that all the module parameters have the same dtype).
Dummy inputs to do a forward pass in the network.
dump_patches
-
add_memory_hooks
()[source]¶ Add a memory hook before and after each sub-module forward pass to record increase in memory consumption.
Increase in memory consumption is stored in a
mem_rss_diff
attribute for each module and can be reset to zero withmodel.reset_memory_hooks_state()
.
-
add_module
(name, module)[source]¶ Adds a child module to the current module.
The module can be accessed as an attribute using the given name.
- Parameters
name (string) – name of the child module. The child module can be accessed from this module using the given name
module (Module) – child module to be added to the module.
-
adjust_logits_during_generation
(logits, **kwargs)[source]¶ Implement in subclasses of
PreTrainedModel
for custom behavior to adjust the logits in the generate method.
-
apply
(fn)[source]¶ Applies
fn
recursively to every submodule (as returned by.children()
) as well as self. Typical use includes initializing the parameters of a model (see also nn-init-doc).- Parameters
fn (
Module
-> None) – function to be applied to each submodule- Returns
self
- Return type
Module
Example:
>>> @torch.no_grad() >>> def init_weights(m): >>> print(m) >>> if type(m) == nn.Linear: >>> m.weight.fill_(1.0) >>> print(m.weight) >>> net = nn.Sequential(nn.Linear(2, 2), nn.Linear(2, 2)) >>> net.apply(init_weights) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[ 1., 1.], [ 1., 1.]]) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[ 1., 1.], [ 1., 1.]]) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )
-
bfloat16
()[source]¶ Casts all floating point parameters and buffers to
bfloat16
datatype.- Returns
self
- Return type
Module
-
buffers
(recurse=True)[source]¶ Returns an iterator over module buffers.
- Parameters
recurse (bool) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module.
- Yields
torch.Tensor – module buffer
Example:
>>> for buf in model.buffers(): >>> print(type(buf), buf.size()) <class 'torch.Tensor'> (20L,) <class 'torch.Tensor'> (20L, 1L, 5L, 5L)
-
children
()[source]¶ Returns an iterator over immediate children modules.
- Yields
Module – a child module
-
cuda
(device=None)[source]¶ Moves all model parameters and buffers to the GPU.
This also makes associated parameters and buffers different objects. So it should be called before constructing optimizer if the module will live on GPU while being optimized.
- Parameters
device (int, optional) – if specified, all parameters will be copied to that device
- Returns
self
- Return type
Module
-
property
device
[source]¶ The device on which the module is (assuming that all the module parameters are on the same device).
- Type
torch.device
-
double
()[source]¶ Casts all floating point parameters and buffers to
double
datatype.- Returns
self
- Return type
Module
-
property
dtype
[source]¶ The dtype of the module (assuming that all the module parameters have the same dtype).
- Type
torch.dtype
-
property
dummy_inputs
[source]¶ Dummy inputs to do a forward pass in the network.
- Type
Dict[str, torch.Tensor]
-
enforce_repetition_penalty_
(lprobs, batch_size, num_beams, prev_output_tokens, repetition_penalty)[source]¶ Enforce the repetition penalty (from the CTRL paper).
-
eval
()[source]¶ Sets the module in evaluation mode.
This has any effect only on certain modules. See documentations of particular modules for details of their behaviors in training/evaluation mode, if they are affected, e.g.
Dropout
,BatchNorm
, etc.This is equivalent with
self.train(False)
.- Returns
self
- Return type
Module
-
extra_repr
()[source]¶ Set the extra representation of the module
To print customized extra information, you should reimplement this method in your own modules. Both single-line and multi-line strings are acceptable.
-
float
()[source]¶ Casts all floating point parameters and buffers to float datatype.
- Returns
self
- Return type
Module
-
forward
(input_ids=None, attention_mask=None, token_type_ids=None, labels=None)[source]¶ Defines the computation performed at every call.
Should be overridden by all subclasses.
Note
Although the recipe for forward pass needs to be defined within this function, one should call the
Module
instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.
-
classmethod
from_pretrained
(pretrained_model_name_or_path, *model_args, **kwargs)[source]¶ Instantiate a pretrained pytorch model from a pre-trained model configuration.
The model is set in evaluation mode by default using
model.eval()
(Dropout modules are deactivated). To train the model, you should first set it back in training mode withmodel.train()
.The warning Weights from XXX not initialized from pretrained model means that the weights of XXX do not come pretrained with the rest of the model. It is up to you to train those weights with a downstream fine-tuning task.
The warning Weights from XXX not used in YYY means that the layer XXX is not used by YYY, therefore those weights are discarded.
- Parameters
pretrained_model_name_or_path (
str
, optional) –Can be either:
A string with the shortcut name of a pretrained model to load from cache or download, e.g.,
bert-base-uncased
.A string with the identifier name of a pretrained model that was user-uploaded to our S3, e.g.,
dbmdz/bert-base-german-cased
.A path to a directory containing model weights saved using
save_pretrained()
, e.g.,./my_model_directory/
.A path or url to a tensorflow index checkpoint file (e.g,
./tf_model/model.ckpt.index
). In this case,from_tf
should be set toTrue
and a configuration object should be provided asconfig
argument. This loading path is slower than converting the TensorFlow checkpoint in a PyTorch model using the provided conversion scripts and loading the PyTorch model afterwards.None
if you are both providing the configuration and state dictionary (resp. with keyword argumentsconfig
andstate_dict
).
model_args (sequence of positional arguments, optional) – All remaning positional arguments will be passed to the underlying model’s
__init__
method.config (
Union[PretrainedConfig, str]
, optional) –Can be either:
an instance of a class derived from
PretrainedConfig
,a string valid as input to
from_pretrained()
.
Configuration for the model to use instead of an automatically loaded configuation. Configuration can be automatically loaded when:
The model is a model provided by the library (loaded with the shortcut name string of a pretrained model).
The model was saved using
save_pretrained()
and is reloaded by suppling the save directory.The model is loaded by suppling a local directory as
pretrained_model_name_or_path
and a configuration JSON file named config.json is found in the directory.
state_dict (
Dict[str, torch.Tensor]
, optional) –A state dictionary to use instead of a state dictionary loaded from saved weights file.
This option can be used if you want to create a model from a pretrained configuration but load your own weights. In this case though, you should check if using
save_pretrained()
andfrom_pretrained()
is not a simpler option.cache_dir (
str
, optional) – Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.from_tf (
bool
, optional, defaults toFalse
) – Load the model weights from a TensorFlow checkpoint save file (see docstring ofpretrained_model_name_or_path
argument).force_download (
bool
, optional, defaults toFalse
) – Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist.resume_download (
bool
, optional, defaults toFalse
) – Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.proxies (
Dict[str, str], `optional
) – A dictionary of proxy servers to use by protocol or endpoint, e.g.,{'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}
. The proxies are used on each request.output_loading_info (
bool
, optional, defaults toFalse
) – Whether ot not to also return a dictionnary containing missing keys, unexpected keys and error messages.local_files_only (
bool
, optional, defaults toFalse
) – Whether or not to only look at local files (e.g., not try doanloading the model).use_cdn (
bool
, optional, defaults toTrue
) – Whether or not to use Cloudfront (a Content Delivery Network, or CDN) when searching for the model on our S3 (faster). Should be set toFalse
for checkpoints larger than 20GB.kwargs (remaining dictionary of keyword arguments, optional) –
Can be used to update the configuration object (after it being loaded) and initiate the model (e.g.,
output_attention=True
). Behaves differently depending on whether aconfig
is provided or automatically loaded:If a configuration is provided with
config
,**kwargs
will be directly passed to the underlying model’s__init__
method (we assume all relevant updates to the configuration have already been done)If a configuration is not provided,
kwargs
will be first passed to the configuration class initialization function (from_pretrained()
). Each key ofkwargs
that corresponds to a configuration attribute will be used to override said attribute with the suppliedkwargs
value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s__init__
function.
Examples:
from transformers import BertConfig, BertModel # Download model and configuration from S3 and cache. model = BertModel.from_pretrained('bert-base-uncased') # Model was saved using `save_pretrained('./test/saved_model/')` (for example purposes, not runnable). model = BertModel.from_pretrained('./test/saved_model/') # Update configuration during loading. model = BertModel.from_pretrained('bert-base-uncased', output_attention=True) assert model.config.output_attention == True # Loading from a TF checkpoint file instead of a PyTorch model (slower, for example purposes, not runnable). config = BertConfig.from_json_file('./tf_model/my_tf_model_config.json') model = BertModel.from_pretrained('./tf_model/my_tf_checkpoint.ckpt.index', from_tf=True, config=config)
-
generate
(input_ids: Optional[torch.LongTensor] = None, max_length: Optional[int] = None, min_length: Optional[int] = None, do_sample: Optional[bool] = None, early_stopping: Optional[bool] = None, num_beams: Optional[int] = None, temperature: Optional[float] = None, top_k: Optional[int] = None, top_p: Optional[float] = None, repetition_penalty: Optional[float] = None, bad_words_ids: Optional[Iterable[int]] = None, bos_token_id: Optional[int] = None, pad_token_id: Optional[int] = None, eos_token_id: Optional[int] = None, length_penalty: Optional[float] = None, no_repeat_ngram_size: Optional[int] = None, num_return_sequences: Optional[int] = None, attention_mask: Optional[torch.LongTensor] = None, decoder_start_token_id: Optional[int] = None, use_cache: Optional[bool] = None, **model_kwargs) → torch.LongTensor[source]¶ Generates sequences for models with a language modeling head. The method currently supports greedy decoding, beam-search decoding, sampling with temperature, sampling with top-k or nucleus sampling.
Adapted in part from Facebook’s XLM beam search code.
Apart from
input_ids
andattention_mask
, all the arguments below will default to the value of the attribute of the same name inside thePretrainedConfig
of the model. The default values indicated are the default values of those config.Most of these parameters are explained in more detail in this blog post.
- Parameters
input_ids (
torch.LongTensor
of shape(batch_size, sequence_length)
, optional) – The sequence used as a prompt for the generation. IfNone
the method initializes it as an emptytorch.LongTensor
of shape(1,)
.max_length (
int
, optional, defaults to 20) – The maximum length of the sequence to be generated.min_length (
int
, optional, defaults to 10) – The minimum length of the sequence to be generated.do_sample (
bool
, optional, defaults toFalse
) – Whether or not to use sampling ; use greedy decoding otherwise.early_stopping (
bool
, optional, defaults toFalse
) – Whether to stop the beam search when at leastnum_beams
sentences are finished per batch or not.num_beams (
int
, optional, defaults to 1) – Number of beams for beam search. 1 means no beam search.temperature (
float
, optional, defaults tp 1.0) – The value used to module the next token probabilities.top_k (
int
, optional, defaults to 50) – The number of highest probability vocabulary tokens to keep for top-k-filtering.top_p (
float
, optional, defaults to 1.0) – If set to float < 1, only the most probable tokens with probabilities that add up totop_p
or higher are kept for generation.repetition_penalty (
float
, optional, defaults to 1.0) – The parameter for repetition penalty. 1.0 means no penalty. See this paper for more details.pad_token_id (
int
, optional) – The id of the padding token.bos_token_id (
int
, optional) – The id of the beginning-of-sequence token.eos_token_id (
int
, optional) – The id of the end-of-sequence token.length_penalty (
float
, optional, defaults to 1.0) –Exponential penalty to the length. 1.0 means no penalty.
Set to values < 1.0 in order to encourage the model to generate shorter sequences, to a value > 1.0 in order to encourage the model to produce longer sequences.
no_repeat_ngram_size (
int
, optional, defaults to 0) – If set to int > 0, all ngrams of that size can only occur once.bad_words_ids (
List[int]
, optional) – List of token ids that are not allowed to be generated. In order to get the tokens of the words that should not appear in the generated text, usetokenizer.encode(bad_word, add_prefix_space=True)
.num_return_sequences (
int
, optional, defaults to 1) – The number of independently computed returned sequences for each element in the batch.attention_mask (
torch.LongTensor
of shape(batch_size, sequence_length)
, optional) –Mask to avoid performing attention on padding token indices. Mask values are in
[0, 1]
, 1 for tokens that are not masked, and 0 for masked tokens.If not provided, will default to a tensor the same shape as
input_ids
that masks the pad token.decoder_start_token_id (
int
, optional) – If an encoder-decoder model starts decoding with a different token than bos, the id of that token.use_cache – (
bool
, optional, defaults toTrue
): Whether or not the model should use the past last key/values attentions (if applicable to the model) to speed up decoding.model_kwargs – Additional model specific kwargs will be forwarded to the
forward
function of the model.
- Returns
The generated sequences. The second dimension (sequence_length) is either equal to
max_length
or shorter if all batches finished early due to theeos_token_id
.- Return type
torch.LongTensor
of shape(batch_size * num_return_sequences, sequence_length)
Examples:
tokenizer = AutoTokenizer.from_pretrained('distilgpt2') # Initialize tokenizer model = AutoModelWithLMHead.from_pretrained('distilgpt2') # Download model and configuration from S3 and cache. outputs = model.generate(max_length=40) # do greedy decoding print('Generated: {}'.format(tokenizer.decode(outputs[0], skip_special_tokens=True))) tokenizer = AutoTokenizer.from_pretrained('openai-gpt') # Initialize tokenizer model = AutoModelWithLMHead.from_pretrained('openai-gpt') # Download model and configuration from S3 and cache. input_context = 'The dog' input_ids = tokenizer.encode(input_context, return_tensors='pt') # encode input context outputs = model.generate(input_ids=input_ids, num_beams=5, num_return_sequences=3, temperature=1.5) # generate 3 independent sequences using beam search decoding (5 beams) with sampling from initial context 'The dog' for i in range(3): # 3 output sequences were generated print('Generated {}: {}'.format(i, tokenizer.decode(outputs[i], skip_special_tokens=True))) tokenizer = AutoTokenizer.from_pretrained('distilgpt2') # Initialize tokenizer model = AutoModelWithLMHead.from_pretrained('distilgpt2') # Download model and configuration from S3 and cache. input_context = 'The dog' input_ids = tokenizer.encode(input_context, return_tensors='pt') # encode input context outputs = model.generate(input_ids=input_ids, max_length=40, temperature=0.7, num_return_sequences=3, do_sample=True) # generate 3 candidates using sampling for i in range(3): # 3 output sequences were generated print('Generated {}: {}'.format(i, tokenizer.decode(outputs[i], skip_special_tokens=True))) tokenizer = AutoTokenizer.from_pretrained('ctrl') # Initialize tokenizer model = AutoModelWithLMHead.from_pretrained('ctrl') # Download model and configuration from S3 and cache. input_context = 'Legal My neighbor is' # "Legal" is one of the control codes for ctrl input_ids = tokenizer.encode(input_context, return_tensors='pt') # encode input context outputs = model.generate(input_ids=input_ids, max_length=50, temperature=0.7, repetition_penalty=1.2) # generate sequences print('Generated: {}'.format(tokenizer.decode(outputs[0], skip_special_tokens=True))) tokenizer = AutoTokenizer.from_pretrained('gpt2') # Initialize tokenizer model = AutoModelWithLMHead.from_pretrained('gpt2') # Download model and configuration from S3 and cache. input_context = 'My cute dog' # "Legal" is one of the control codes for ctrl bad_words_ids = [tokenizer.encode(bad_word, add_prefix_space=True) for bad_word in ['idiot', 'stupid', 'shut up']] input_ids = tokenizer.encode(input_context, return_tensors='pt') # encode input context outputs = model.generate(input_ids=input_ids, max_length=100, do_sample=True, bad_words_ids=bad_words_ids) # generate sequences without allowing bad_words to be generated
-
get_extended_attention_mask
(attention_mask: torch.Tensor, input_shape: Tuple[int], device: <property object at 0x7f0e7028aea8>) → torch.Tensor[source]¶ Makes broadcastable attention and causal masks so that future and masked tokens are ignored.
- Parameters
attention_mask (
torch.Tensor
) – Mask with ones indicating tokens to attend to, zeros for tokens to ignore.input_shape (
Tuple[int]
) – The shape of the input to the model.device – (
torch.device
): The device of the input to the model.
- Returns
torch.Tensor
The extended attention mask, with a the same dtype asattention_mask.dtype
.
-
get_head_mask
(head_mask: Optional[torch.Tensor], num_hidden_layers: int, is_attention_chunked: bool = False) → torch.Tensor[source]¶ Prepare the head mask if needed.
- Parameters
head_mask (
torch.Tensor
with shape[num_heads]
or[num_hidden_layers x num_heads]
, optional) – The mask indicating if we should keep the heads or not (1.0 for keep, 0.0 for discard).num_hidden_layers (
int
) – The number of hidden layers in the model.is_attention_chunked – (
bool
, optional, defaults to :obj:`False): Whether or not the attentions scores are computed by chunks or not.
- Returns
torch.Tensor
with shape[num_hidden_layers x batch x num_heads x seq_length x seq_length]
or list with[None]
for each layer.
-
get_input_embeddings
() → torch.nn.modules.module.Module[source]¶ Returns the model’s input embeddings.
- Returns
A torch module mapping vocabulary to hidden states.
- Return type
nn.Module
-
get_output_embeddings
() → torch.nn.modules.module.Module[source]¶ Returns the model’s output embeddings.
- Returns
A torch module mapping hidden states to vocabulary.
- Return type
nn.Module
-
half
()[source]¶ Casts all floating point parameters and buffers to
half
datatype.- Returns
self
- Return type
Module
-
invert_attention_mask
(encoder_attention_mask: torch.Tensor) → torch.Tensor[source]¶ Invert an attention mask (e.g., switches 0. and 1.).
- Parameters
encoder_attention_mask (
torch.Tensor
) – An attention mask.- Returns
The inverted attention mask.
- Return type
torch.Tensor
-
load_state_dict
(state_dict, strict=True)[source]¶ Copies parameters and buffers from
state_dict
into this module and its descendants. Ifstrict
isTrue
, then the keys ofstate_dict
must exactly match the keys returned by this module’sstate_dict()
function.- Parameters
state_dict (dict) – a dict containing parameters and persistent buffers.
strict (bool, optional) – whether to strictly enforce that the keys in
state_dict
match the keys returned by this module’sstate_dict()
function. Default:True
- Returns
missing_keys is a list of str containing the missing keys
unexpected_keys is a list of str containing the unexpected keys
- Return type
NamedTuple
withmissing_keys
andunexpected_keys
fields
-
modules
()[source]¶ Returns an iterator over all modules in the network.
- Yields
Module – a module in the network
Note
Duplicate modules are returned only once. In the following example,
l
will be returned only once.Example:
>>> l = nn.Linear(2, 2) >>> net = nn.Sequential(l, l) >>> for idx, m in enumerate(net.modules()): print(idx, '->', m) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True)
-
named_buffers
(prefix='', recurse=True)[source]¶ Returns an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself.
- Parameters
prefix (str) – prefix to prepend to all buffer names.
recurse (bool) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module.
- Yields
(string, torch.Tensor) – Tuple containing the name and buffer
Example:
>>> for name, buf in self.named_buffers(): >>> if name in ['running_var']: >>> print(buf.size())
-
named_children
()[source]¶ Returns an iterator over immediate children modules, yielding both the name of the module as well as the module itself.
- Yields
(string, Module) – Tuple containing a name and child module
Example:
>>> for name, module in model.named_children(): >>> if name in ['conv4', 'conv5']: >>> print(module)
-
named_modules
(memo=None, prefix='')[source]¶ Returns an iterator over all modules in the network, yielding both the name of the module as well as the module itself.
- Yields
(string, Module) – Tuple of name and module
Note
Duplicate modules are returned only once. In the following example,
l
will be returned only once.Example:
>>> l = nn.Linear(2, 2) >>> net = nn.Sequential(l, l) >>> for idx, m in enumerate(net.named_modules()): print(idx, '->', m) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True))
-
named_parameters
(prefix='', recurse=True)[source]¶ Returns an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself.
- Parameters
prefix (str) – prefix to prepend to all parameter names.
recurse (bool) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module.
- Yields
(string, Parameter) – Tuple containing the name and parameter
Example:
>>> for name, param in self.named_parameters(): >>> if name in ['bias']: >>> print(param.size())
-
num_parameters
(only_trainable: bool = False) → int[source]¶ Get the number of (optionally, trainable) parameters in the model.
- Parameters
only_trainable (
bool
, optional, defaults toFalse
) – Whether or not to return only the number of trainable parameters- Returns
The number of parameters.
- Return type
int
-
parameters
(recurse=True)[source]¶ Returns an iterator over module parameters.
This is typically passed to an optimizer.
- Parameters
recurse (bool) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module.
- Yields
Parameter – module parameter
Example:
>>> for param in model.parameters(): >>> print(type(param), param.size()) <class 'torch.Tensor'> (20L,) <class 'torch.Tensor'> (20L, 1L, 5L, 5L)
-
prepare_inputs_for_generation
(input_ids, **kwargs)[source]¶ Implement in subclasses of
PreTrainedModel
for custom behavior to prepare inputs in the generate method.
-
prune_heads
(heads_to_prune: Dict[int, List[int]])[source]¶ Prunes heads of the base model.
- Parameters
heads_to_prune (
Dict[int, List[int]]
) – Dictionary with keys being selected layer indices (int
) and associated values being the list of heads to prune in said layer (list ofint
). For instance {1: [0, 2], 2: [2, 3]} will prune heads 0 and 2 on layer 1 and heads 2 and 3 on layer 2.
-
register_backward_hook
(hook)[source]¶ Registers a backward hook on the module.
The hook will be called every time the gradients with respect to module inputs are computed. The hook should have the following signature:
hook(module, grad_input, grad_output) -> Tensor or None
The
grad_input
andgrad_output
may be tuples if the module has multiple inputs or outputs. The hook should not modify its arguments, but it can optionally return a new gradient with respect to input that will be used in place ofgrad_input
in subsequent computations.- Returns
a handle that can be used to remove the added hook by calling
handle.remove()
- Return type
torch.utils.hooks.RemovableHandle
Warning
The current implementation will not have the presented behavior for complex
Module
that perform many operations. In some failure cases,grad_input
andgrad_output
will only contain the gradients for a subset of the inputs and outputs. For suchModule
, you should usetorch.Tensor.register_hook()
directly on a specific input or output to get the required gradients.
-
register_buffer
(name, tensor)[source]¶ Adds a persistent buffer to the module.
This is typically used to register a buffer that should not to be considered a model parameter. For example, BatchNorm’s
running_mean
is not a parameter, but is part of the persistent state.Buffers can be accessed as attributes using given names.
- Parameters
name (string) – name of the buffer. The buffer can be accessed from this module using the given name
tensor (Tensor) – buffer to be registered.
Example:
>>> self.register_buffer('running_mean', torch.zeros(num_features))
-
register_forward_hook
(hook)[source]¶ Registers a forward hook on the module.
The hook will be called every time after
forward()
has computed an output. It should have the following signature:hook(module, input, output) -> None or modified output
The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after
forward()
is called.- Returns
a handle that can be used to remove the added hook by calling
handle.remove()
- Return type
torch.utils.hooks.RemovableHandle
-
register_forward_pre_hook
(hook)[source]¶ Registers a forward pre-hook on the module.
The hook will be called every time before
forward()
is invoked. It should have the following signature:hook(module, input) -> None or modified input
The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned(unless that value is already a tuple).
- Returns
a handle that can be used to remove the added hook by calling
handle.remove()
- Return type
torch.utils.hooks.RemovableHandle
-
register_parameter
(name, param)[source]¶ Adds a parameter to the module.
The parameter can be accessed as an attribute using given name.
- Parameters
name (string) – name of the parameter. The parameter can be accessed from this module using the given name
param (Parameter) – parameter to be added to the module.
-
requires_grad_
(requires_grad=True)[source]¶ Change if autograd should record operations on parameters in this module.
This method sets the parameters’
requires_grad
attributes in-place.This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training).
- Parameters
requires_grad (bool) – whether autograd should record operations on parameters in this module. Default:
True
.- Returns
self
- Return type
Module
-
reset_memory_hooks_state
()[source]¶ Reset the
mem_rss_diff
attribute of each module (seeadd_memory_hooks()
).
-
resize_token_embeddings
(new_num_tokens: Optional[int] = None) → torch.nn.modules.sparse.Embedding[source]¶ Resizes input token embeddings matrix of the model if
new_num_tokens != config.vocab_size
.Takes care of tying weights embeddings afterwards if the model class has a
tie_weights()
method.- Parameters
new_num_tokens (
int
, optional) – The number of new tokens in the embedding matrix. Increasing the size will add newly initialized vectors at the end. Reducing the size will remove vectors from the end. If not provided orNone
, just returns a pointer to the input tokenstorch.nn.Embedding
module of the model wihtout doing anything.- Returns
Pointer to the input tokens Embeddings Module of the model.
- Return type
torch.nn.Embedding
-
save_pretrained
(save_directory)[source]¶ Save a model and its configuration file to a directory, so that it can be re-loaded using the :func:`~transformers.PreTrainedModel.from_pretrained` class method.
- Parameters
save_directory (
str
) – Directory to which to save. Will be created if it doesn’t exist.
-
set_input_embeddings
(value: torch.nn.modules.module.Module)[source]¶ Set model’s input embeddings
- Parameters
value (
nn.Module
) – A module mapping vocabulary to hidden states.
-
state_dict
(destination=None, prefix='', keep_vars=False)[source]¶ Returns a dictionary containing a whole state of the module.
Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names.
- Returns
a dictionary containing a whole state of the module
- Return type
dict
Example:
>>> module.state_dict().keys() ['bias', 'weight']
-
tie_weights
()[source]¶ Tie the weights between the input embeddings and the output embeddings.
If the
torchscript
flag is set in the configuration, can’t handle parameter sharing so we are cloning the weights instead.
-
to
(*args, **kwargs)[source]¶ Moves and/or casts the parameters and buffers.
This can be called as
Its signature is similar to
torch.Tensor.to()
, but only accepts floating point desireddtype
s. In addition, this method will only cast the floating point parameters and buffers todtype
(if given). The integral parameters and buffers will be moveddevice
, if that is given, but with dtypes unchanged. Whennon_blocking
is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices.See below for examples.
Note
This method modifies the module in-place.
- Parameters
device (
torch.device
) – the desired device of the parameters and buffers in this moduledtype (
torch.dtype
) – the desired floating point type of the floating point parameters and buffers in this moduletensor (torch.Tensor) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module
memory_format (
torch.memory_format
) – the desired memory format for 4D parameters and buffers in this module (keyword only argument)
- Returns
self
- Return type
Module
Example:
>>> linear = nn.Linear(2, 2) >>> linear.weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear.to(torch.double) Linear(in_features=2, out_features=2, bias=True) >>> linear.weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=torch.float64) >>> gpu1 = torch.device("cuda:1") >>> linear.to(gpu1, dtype=torch.half, non_blocking=True) Linear(in_features=2, out_features=2, bias=True) >>> linear.weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=torch.float16, device='cuda:1') >>> cpu = torch.device("cpu") >>> linear.to(cpu) Linear(in_features=2, out_features=2, bias=True) >>> linear.weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=torch.float16)
-
train
(mode=True)[source]¶ Sets the module in training mode.
This has any effect only on certain modules. See documentations of particular modules for details of their behaviors in training/evaluation mode, if they are affected, e.g.
Dropout
,BatchNorm
, etc.- Parameters
mode (bool) – whether to set training mode (
True
) or evaluation mode (False
). Default:True
.- Returns
self
- Return type
Module
-