Models

bert

Bert Model.

class mindnlp.models.bert.BertAttention(config)

Bases: Cell

Bert Attention

construct(input_tensor, attention_mask=None, head_mask=None)

Defines the computation to be performed. This method must be overridden by all subclasses.

Note

It is not supported currently that inputs contain both tuple and non-tuple types at same time.

Parameters:

• args (tuple) – Tuple of variable parameters.

• kwargs (dict) – Dictionary of variable keyword parameters.

Returns:

Tensor, returns the computed result.

class mindnlp.models.bert.BertConfig(vocab_size=30522, hidden_size=768, num_hidden_layers=12, num_attention_heads=12, intermediate_size=3072, hidden_act='gelu', hidden_dropout_prob=0.1, attention_probs_dropout_prob=0.1, max_position_embeddings=512, type_vocab_size=2, initializer_range=0.02, layer_norm_eps=1e-12, classifier_dropout=None, **kwargs)

Bases: PreTrainedConfig

Configuration for BERT-base

class mindnlp.models.bert.BertEmbeddings(config)

Bases: Cell

Embeddings for BERT, include word, position and token_type

construct(input_ids, token_type_ids=None, position_ids=None)

Defines the computation to be performed. This method must be overridden by all subclasses.

Note

It is not supported currently that inputs contain both tuple and non-tuple types at same time.

Parameters:

• args (tuple) – Tuple of variable parameters.

• kwargs (dict) – Dictionary of variable keyword parameters.

Returns:

Tensor, returns the computed result.

class mindnlp.models.bert.BertEncoder(config)

Bases: Cell

Bert Encoder

construct(hidden_states, attention_mask=None, head_mask=None)

Defines the computation to be performed. This method must be overridden by all subclasses.

Note

It is not supported currently that inputs contain both tuple and non-tuple types at same time.

Parameters:

• args (tuple) – Tuple of variable parameters.

• kwargs (dict) – Dictionary of variable keyword parameters.

Returns:

Tensor, returns the computed result.

class mindnlp.models.bert.BertForPretraining(config, *args, **kwargs)

Bases: BertPreTrainedModel

Bert For Pretraining

construct(input_ids, attention_mask=None, token_type_ids=None, position_ids=None, head_mask=None, masked_lm_positions=None)

Defines the computation to be performed. This method must be overridden by all subclasses.

Note

It is not supported currently that inputs contain both tuple and non-tuple types at same time.

Parameters:

• args (tuple) – Tuple of variable parameters.

• kwargs (dict) – Dictionary of variable keyword parameters.

Returns:

Tensor, returns the computed result.

class mindnlp.models.bert.BertForSequenceClassification(config)

Bases: BertPreTrainedModel

Bert Model for classification tasks

construct(input_ids, attention_mask=None, token_type_ids=None, position_ids=None, head_mask=None, labels=None)

Defines the computation to be performed. This method must be overridden by all subclasses.

Note

It is not supported currently that inputs contain both tuple and non-tuple types at same time.

Parameters:

• args (tuple) – Tuple of variable parameters.

• kwargs (dict) – Dictionary of variable keyword parameters.

Returns:

Tensor, returns the computed result.

class mindnlp.models.bert.BertIntermediate(config)

Bases: Cell

Bert Intermediate

construct(hidden_states)

Defines the computation to be performed. This method must be overridden by all subclasses.

Note

It is not supported currently that inputs contain both tuple and non-tuple types at same time.

Parameters:

• args (tuple) – Tuple of variable parameters.

• kwargs (dict) – Dictionary of variable keyword parameters.

Returns:

Tensor, returns the computed result.

class mindnlp.models.bert.BertLMPredictionHead(config)

Bases: Cell

Bert LM Prediction Head

construct(hidden_states, masked_lm_positions)

Defines the computation to be performed. This method must be overridden by all subclasses.

Note

It is not supported currently that inputs contain both tuple and non-tuple types at same time.

Parameters:

• args (tuple) – Tuple of variable parameters.

• kwargs (dict) – Dictionary of variable keyword parameters.

Returns:

Tensor, returns the computed result.

class mindnlp.models.bert.BertLayer(config)

Bases: Cell

Bert Layer

construct(hidden_states, attention_mask=None, head_mask=None)

Defines the computation to be performed. This method must be overridden by all subclasses.

Note

It is not supported currently that inputs contain both tuple and non-tuple types at same time.

Parameters:

• args (tuple) – Tuple of variable parameters.

• kwargs (dict) – Dictionary of variable keyword parameters.

Returns:

Tensor, returns the computed result.

class mindnlp.models.bert.BertModel(config, add_pooling_layer=True)

Bases: BertPreTrainedModel

Bert Model

construct(input_ids, attention_mask=None, token_type_ids=None, position_ids=None, head_mask=None)

Defines the computation to be performed. This method must be overridden by all subclasses.

Note

It is not supported currently that inputs contain both tuple and non-tuple types at same time.

Parameters:

• args (tuple) – Tuple of variable parameters.

• kwargs (dict) – Dictionary of variable keyword parameters.

Returns:

Tensor, returns the computed result.

get_input_embeddings()

Returns the model’s input embeddings.

Returns:

A mindspore cell mapping vocabulary to hidden states.

Return type:

nn.Cell

set_input_embeddings(new_embeddings)

Set model’s input embeddings.

Parameters:

value (nn.Cell) – A mindspore cell mapping vocabulary to hidden states.

gpt

GPT Model.

class mindnlp.models.gpt.Attention(nx, n_positions, config, scale=False)

Bases: Cell

GPT Attention

construct(x, attention_mask=None, head_mask=None)

Defines the computation to be performed. This method must be overridden by all subclasses.

Note

It is not supported currently that inputs contain both tuple and non-tuple types at same time.

Parameters:

• args (tuple) – Tuple of variable parameters.

• kwargs (dict) – Dictionary of variable keyword parameters.

Returns:

Tensor, returns the computed result.

merge_heads(x)

merge heads

prune_heads(heads)

Prunes heads of the model.

split_heads(x, k=False)

split heads

class mindnlp.models.gpt.Block(n_positions, config, scale=False)

Bases: Cell

GPT Block

construct(x, attention_mask=None, head_mask=None)

Defines the computation to be performed. This method must be overridden by all subclasses.

Note

It is not supported currently that inputs contain both tuple and non-tuple types at same time.

Parameters:

• args (tuple) – Tuple of variable parameters.

• kwargs (dict) – Dictionary of variable keyword parameters.

Returns:

Tensor, returns the computed result.

class mindnlp.models.gpt.GPTConfig(vocab_size=40478, n_positions=512, n_embd=768, hidden_size=768, n_layer=12, n_head=12, afn='gelu_new', resid_pdrop=0.1, embd_pdrop=0.1, attn_pdrop=0.1, layer_norm_epsilon=1e-05, initializer_range=0.02, summary_type='cls_index', summary_use_proj=True, summary_activation=None, summary_proj_to_labels=True, summary_first_dropout=0.1, **kwargs)

Bases: PreTrainedConfig

GPT config

class mindnlp.models.gpt.GPTDoubleHeadsModel(config)

Bases: GPTPreTrainedModel

OpenAI GPT Model transformer with a language modeling and a multiple-choice classification head on top e.g. for RocStories/SWAG tasks. The two heads are two linear layers. The language modeling head has its weights tied to the input embeddings, the classification head takes as input the input of a specified classification token index in the input sequence).

construct(input_ids=None, attention_mask=None, token_type_ids=None, position_ids=None, head_mask=None, inputs_embeds=None, mc_token_ids=None, labels=None, mc_labels=None)

Defines the computation to be performed. This method must be overridden by all subclasses.

Note

It is not supported currently that inputs contain both tuple and non-tuple types at same time.

Parameters:

• args (tuple) – Tuple of variable parameters.

• kwargs (dict) – Dictionary of variable keyword parameters.

Returns:

Tensor, returns the computed result.

get_output_embeddings()

Returns the embeddings of the obtained output

set_output_embeddings(new_embeddings)

Define the embeddings of the output

class mindnlp.models.gpt.GPTForSequenceClassification(config)

Bases: GPTPreTrainedModel

The Original GPT Model transformer with a sequence classification head on top (linear layer). GPTForSequenceClassification uses the last token in order to do the classification, as other causal models (e.g. GPT-2) do. Since it does classification on the last token, it requires to know the position of the last token. If a pad_token_id is defined in the configuration, it finds the last token that is not a padding token in each row. If no pad_token_id is defined, it simply takes the last value in each row of the batch. Since it cannot guess the padding tokens when inputs_embeds are passed instead of input_ids, it does the same (take the last value in each row of the batch).

construct(input_ids=None, attention_mask=None, token_type_ids=None, position_ids=None, head_mask=None, inputs_embeds=None, labels=None)

labels (torch.LongTensor of shape (batch_size,), optional):

Labels for computing the sequence classification/regression loss. Indices should be in [0, …,config.num_labels - 1]. If config.num_labels == 1 a regression loss is computed (Mean-Square loss), If config.num_labels > 1 a classification loss is computed (Cross-Entropy).

class mindnlp.models.gpt.GPTLMHeadModel(config)

Bases: GPTPreTrainedModel

GPT Model transformer with a language modeling head on top (linear layer with weights tied to the input embeddings).

construct(input_ids=None, attention_mask=None, token_type_ids=None, position_ids=None, head_mask=None, inputs_embeds=None, labels=None)

Defines the computation to be performed. This method must be overridden by all subclasses.

Note

It is not supported currently that inputs contain both tuple and non-tuple types at same time.

Parameters:

• args (tuple) – Tuple of variable parameters.

• kwargs (dict) – Dictionary of variable keyword parameters.

Returns:

Tensor, returns the computed result.

get_output_embeddings()

Returns the embeddings of the obtained output

set_output_embeddings(new_embeddings)

Define the embeddings of the output

class mindnlp.models.gpt.GPTModel(config)

Bases: GPTPreTrainedModel

The bare GPT transformer model outputting raw hidden-states without any specific head on top

construct(input_ids=None, attention_mask=None, token_type_ids=None, position_ids=None, head_mask=None, inputs_embeds=None)

Defines the computation to be performed. This method must be overridden by all subclasses.

Note

It is not supported currently that inputs contain both tuple and non-tuple types at same time.

Parameters:

• args (tuple) – Tuple of variable parameters.

• kwargs (dict) – Dictionary of variable keyword parameters.

Returns:

Tensor, returns the computed result.

get_input_embeddings()

return the input embeddings layer

set_input_embeddings(new_embeddings)

set the input embeddings layer

class mindnlp.models.gpt.MLP(n_state, config)

Bases: Cell

GPT MLP

construct(x)

Defines the computation to be performed. This method must be overridden by all subclasses.

Note

It is not supported currently that inputs contain both tuple and non-tuple types at same time.

Parameters:

• args (tuple) – Tuple of variable parameters.

• kwargs (dict) – Dictionary of variable keyword parameters.

Returns:

Tensor, returns the computed result.

gpt2

GPT2 Models init

class mindnlp.models.gpt2.GPT2Attention(config, is_cross_attention=False, layer_idx=None)

Bases: Cell

gpt2 Attention

construct(hidden_states: Tuple[Tensor], layer_past: Optional[Tuple[Tensor]] = None, attention_mask: Optional[Tensor] = None, head_mask: Optional[Tensor] = None, encoder_hidden_states: Optional[Tensor] = None, encoder_attention_mask: Optional[Tensor] = None, use_cache: Optional[bool] = False)

Defines the computation to be performed. This method must be overridden by all subclasses.

Note

It is not supported currently that inputs contain both tuple and non-tuple types at same time.

Parameters:

• args (tuple) – Tuple of variable parameters.

• kwargs (dict) – Dictionary of variable keyword parameters.

Returns:

Tensor, returns the computed result.

prune_heads(heads)

Prunes heads of the model.

class mindnlp.models.gpt2.GPT2Config(vocab_size=50257, n_positions=1024, n_embd=768, n_layer=12, n_head=12, n_inner=None, activation_function='gelu_new', resid_pdrop=0.1, embd_pdrop=0.1, attn_pdrop=0.1, layer_norm_epsilon=1e-05, initializer_range=0.02, summary_type='cls_index', summary_use_proj=True, summary_activation=None, summary_proj_to_labels=True, summary_first_dropout=0.1, scale_attn_weights=True, use_cache=True, bos_token_id=50256, eos_token_id=50256, scale_attn_by_inverse_layer_idx=False, reorder_and_upcast_attn=False, **kwargs)

Bases: PreTrainedConfig

Configuration for gpt2-base

class mindnlp.models.gpt2.GPT2DoubleHeadsModel(config)

Bases: GPT2PreTrainedModel

GPT2 Double Heads Model

construct(input_ids, past_key_values=None, attention_mask=None, token_type_ids=None, position_ids=None, head_mask=None, inputs_embeds=None, mc_token_ids=None, labels=None, mc_labels=None)

Defines the computation to be performed. This method must be overridden by all subclasses.

Note

It is not supported currently that inputs contain both tuple and non-tuple types at same time.

Parameters:

• args (tuple) – Tuple of variable parameters.

• kwargs (dict) – Dictionary of variable keyword parameters.

Returns:

Tensor, returns the computed result.

get_output_embeddings()

Returns the embeddings of the obtained output

prepare_inputs_for_generation(input_ids, past_key_values=None, **kwargs)

prepare_inputs

set_output_embeddings(new_embeddings)

Define the embeddings of the output

class mindnlp.models.gpt2.GPT2ForSequenceClassification(config)

Bases: GPT2PreTrainedModel

gpt2 For Sequence Classification

construct(input_ids: Tensor, past_key_values: Optional[Tuple[Tuple[Tensor]]] = None, attention_mask: Optional[Tensor] = None, token_type_ids: Optional[Tensor] = None, position_ids: Optional[Tensor] = None, head_mask: Optional[Tensor] = None, inputs_embeds: Optional[Tensor] = None, labels: Optional[Tensor] = None)

Defines the computation to be performed. This method must be overridden by all subclasses.

Note

It is not supported currently that inputs contain both tuple and non-tuple types at same time.

Parameters:

• args (tuple) – Tuple of variable parameters.

• kwargs (dict) – Dictionary of variable keyword parameters.

Returns:

Tensor, returns the computed result.

class mindnlp.models.gpt2.GPT2ForTokenClassification(config)

Bases: GPT2PreTrainedModel

GPT2 For Token Classification

construct(input_ids=None, past_key_values=None, attention_mask=None, token_type_ids=None, position_ids=None, head_mask=None, inputs_embeds=None, labels=None)

Defines the computation to be performed. This method must be overridden by all subclasses.

Note

It is not supported currently that inputs contain both tuple and non-tuple types at same time.

Parameters:

• args (tuple) – Tuple of variable parameters.

• kwargs (dict) – Dictionary of variable keyword parameters.

Returns:

Tensor, returns the computed result.

class mindnlp.models.gpt2.GPT2LMHeadModel(config, **kwargs)

Bases: GPT2PreTrainedModel

gpt2 LMHead Model

construct(input_ids: Tensor, past_key_values: Optional[Tuple[Tuple[Tensor]]] = None, attention_mask: Optional[Tensor] = None, token_type_ids: Optional[Tensor] = None, position_ids: Optional[Tensor] = None, head_mask: Optional[Tensor] = None, inputs_embeds: Optional[Tensor] = None, encoder_hidden_states: Optional[Tensor] = None, encoder_attention_mask: Optional[Tensor] = None, labels: Optional[Tensor] = None)

Defines the computation to be performed. This method must be overridden by all subclasses.

Note

It is not supported currently that inputs contain both tuple and non-tuple types at same time.

Parameters:

• args (tuple) – Tuple of variable parameters.

• kwargs (dict) – Dictionary of variable keyword parameters.

Returns:

Tensor, returns the computed result.

get_output_embeddings()

return the output embeddings layer

prepare_inputs_for_generation(input_ids, past_key_values=None, **kwargs)

prepare inputs for generation task

set_output_embeddings(new_embeddings)

set the output embeddings layer

class mindnlp.models.gpt2.GPT2MLP(intermediate_size, config)

Bases: Cell

gpt2 MLP

construct(hidden_states: Tuple[Tensor])

Defines the computation to be performed. This method must be overridden by all subclasses.

Note

It is not supported currently that inputs contain both tuple and non-tuple types at same time.

Parameters:

• args (tuple) – Tuple of variable parameters.

• kwargs (dict) – Dictionary of variable keyword parameters.

Returns:

Tensor, returns the computed result.

class mindnlp.models.gpt2.GPT2Model(config)

Bases: GPT2PreTrainedModel

gpt2 Model

construct(input_ids: Tensor, past_key_values: Optional[Tuple[Tuple[Tensor]]] = None, attention_mask: Optional[Tensor] = None, token_type_ids: Optional[Tensor] = None, position_ids: Optional[Tensor] = None, head_mask: Optional[Tensor] = None, inputs_embeds: Optional[Tensor] = None, encoder_hidden_states: Optional[Tensor] = None, encoder_attention_mask: Optional[Tensor] = None)

Defines the computation to be performed. This method must be overridden by all subclasses.

Note

It is not supported currently that inputs contain both tuple and non-tuple types at same time.

Parameters:

• args (tuple) – Tuple of variable parameters.

• kwargs (dict) – Dictionary of variable keyword parameters.

Returns:

Tensor, returns the computed result.

get_input_embeddings()

return the input embeddings layer

set_input_embeddings(new_embeddings)

set the input embeddings layer

class mindnlp.models.gpt2.GPT2PreTrainedModel(config)

Bases: PreTrainedModel

An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models.

config_class

alias of GPT2Config

convert_torch_to_mindspore(**kwargs)

torch to mindspore.

get_head_mask(head_mask, num_hidden_layers, is_attention_chunked=False)

Prepare the head mask if needed.

t5

T5 Model init

class mindnlp.models.t5.T5Attention(config: T5Config, has_relative_attention_bias=False)

Bases: Cell

compute_bias(query_length, key_length)

Compute binned relative position bias

construct(hidden_states, mask=None, key_value_states=None, position_bias=None, past_key_value=None, layer_head_mask=None, query_length=None, use_cache=False, output_attentions=False)

Self-attention (if key_value_states is None) or attention over source sentence (provided by key_value_states).

class mindnlp.models.t5.T5Block(config, has_relative_attention_bias=False)

Bases: Cell

construct(hidden_states, attention_mask=None, position_bias=None, encoder_hidden_states=None, encoder_attention_mask=None, encoder_decoder_position_bias=None, layer_head_mask=None, cross_attn_layer_head_mask=None, past_key_value=None, use_cache=False, output_attentions=False)

Defines the computation to be performed. This method must be overridden by all subclasses.

Note

It is not supported currently that inputs contain both tuple and non-tuple types at same time.

Parameters:

• args (tuple) – Tuple of variable parameters.

• kwargs (dict) – Dictionary of variable keyword parameters.

Returns:

Tensor, returns the computed result.

class mindnlp.models.t5.T5Config(vocab_size=32128, d_model=512, d_kv=64, d_ff=2048, num_layers=6, num_decoder_layers=None, num_heads=8, relative_attention_num_buckets=32, relative_attention_max_distance=128, dropout_rate=0.1, layer_norm_epsilon=1e-06, initializer_factor=1.0, feed_forward_proj='relu', is_encoder_decoder=True, use_cache=True, pad_token_id=0, eos_token_id=1, **kwargs)

Bases: PreTrainedConfig

Configuration for T5

class mindnlp.models.t5.T5DenseActDense(config: T5Config)

Bases: Cell

construct(hidden_states)

Defines the computation to be performed. This method must be overridden by all subclasses.

Note

It is not supported currently that inputs contain both tuple and non-tuple types at same time.

Parameters:

• args (tuple) – Tuple of variable parameters.

• kwargs (dict) – Dictionary of variable keyword parameters.

Returns:

Tensor, returns the computed result.

class mindnlp.models.t5.T5DenseGatedActDense(config: T5Config)

Bases: Cell

construct(hidden_states)

Defines the computation to be performed. This method must be overridden by all subclasses.

Note

It is not supported currently that inputs contain both tuple and non-tuple types at same time.

Parameters:

• args (tuple) – Tuple of variable parameters.

• kwargs (dict) – Dictionary of variable keyword parameters.

Returns:

Tensor, returns the computed result.

class mindnlp.models.t5.T5EncoderModel(config: T5Config)

Bases: T5PreTrainedModel

construct(input_ids=None, attention_mask=None, head_mask=None, inputs_embeds=None, output_attentions=None, output_hidden_states=None, return_dict=None)

Defines the computation to be performed. This method must be overridden by all subclasses.

Note

It is not supported currently that inputs contain both tuple and non-tuple types at same time.

Parameters:

• args (tuple) – Tuple of variable parameters.

• kwargs (dict) – Dictionary of variable keyword parameters.

Returns:

Tensor, returns the computed result.

get_encoder()

get encoder

get_input_embeddings()

Returns the model’s input embeddings.

Returns:

A mindspore cell mapping vocabulary to hidden states.

Return type:

nn.Cell

set_input_embeddings(new_embeddings)

Set model’s input embeddings.

Parameters:

value (nn.Cell) – A mindspore cell mapping vocabulary to hidden states.

class mindnlp.models.t5.T5ForConditionalGeneration(config: T5Config)

Bases: T5PreTrainedModel

construct(input_ids=None, attention_mask=None, decoder_input_ids=None, decoder_attention_mask=None, head_mask=None, decoder_head_mask=None, cross_attn_head_mask=None, encoder_outputs=None, past_key_values=None, inputs_embeds=None, decoder_inputs_embeds=None, labels=None, use_cache=None, output_attentions=None, output_hidden_states=None, return_dict=None)

Defines the computation to be performed. This method must be overridden by all subclasses.

Note

It is not supported currently that inputs contain both tuple and non-tuple types at same time.

Parameters:

• args (tuple) – Tuple of variable parameters.

• kwargs (dict) – Dictionary of variable keyword parameters.

Returns:

Tensor, returns the computed result.

get_decoder()

get decoder

get_encoder()

get encoder

get_input_embeddings()

Returns the model’s input embeddings.

Returns:

A mindspore cell mapping vocabulary to hidden states.

Return type:

nn.Cell

get_output_embeddings()

get output embeddings

prepare_decoder_input_ids_from_labels(labels: Tensor)

prepare decoder input ids from labels

prepare_inputs_for_generation(input_ids, past_key_values=None, attention_mask=None, head_mask=None, decoder_head_mask=None, cross_attn_head_mask=None, use_cache=None, encoder_outputs=None)

prepare inputs for generation

set_input_embeddings(new_embeddings)

set input embeddings

set_output_embeddings(new_embeddings)

set output embeddings

class mindnlp.models.t5.T5LayerCrossAttention(config)

Bases: Cell

construct(hidden_states, key_value_states, attention_mask=None, position_bias=None, layer_head_mask=None, past_key_value=None, use_cache=False, query_length=None, output_attentions=False)

Defines the computation to be performed. This method must be overridden by all subclasses.

Note

It is not supported currently that inputs contain both tuple and non-tuple types at same time.

Parameters:

• args (tuple) – Tuple of variable parameters.

• kwargs (dict) – Dictionary of variable keyword parameters.

Returns:

Tensor, returns the computed result.

class mindnlp.models.t5.T5LayerFF(config: T5Config)

Bases: Cell

construct(hidden_states)

Defines the computation to be performed. This method must be overridden by all subclasses.

Note

It is not supported currently that inputs contain both tuple and non-tuple types at same time.

Parameters:

• args (tuple) – Tuple of variable parameters.

• kwargs (dict) – Dictionary of variable keyword parameters.

Returns:

Tensor, returns the computed result.

class mindnlp.models.t5.T5LayerNorm(hidden_size, eps=1e-06)

Bases: Cell

construct(hidden_states)

Defines the computation to be performed. This method must be overridden by all subclasses.

Note

It is not supported currently that inputs contain both tuple and non-tuple types at same time.

Parameters:

• args (tuple) – Tuple of variable parameters.

• kwargs (dict) – Dictionary of variable keyword parameters.

Returns:

Tensor, returns the computed result.

class mindnlp.models.t5.T5LayerSelfAttention(config, has_relative_attention_bias=False)

Bases: Cell

construct(hidden_states, attention_mask=None, position_bias=None, layer_head_mask=None, past_key_value=None, use_cache=False, output_attentions=False)

Defines the computation to be performed. This method must be overridden by all subclasses.

Note

It is not supported currently that inputs contain both tuple and non-tuple types at same time.

Parameters:

• args (tuple) – Tuple of variable parameters.

• kwargs (dict) – Dictionary of variable keyword parameters.

Returns:

Tensor, returns the computed result.

class mindnlp.models.t5.T5Model(config: T5Config)

Bases: T5PreTrainedModel

construct(input_ids=None, attention_mask=None, decoder_input_ids=None, decoder_attention_mask=None, head_mask=None, decoder_head_mask=None, cross_attn_head_mask=None, encoder_outputs=None, past_key_values=None, inputs_embeds=None, decoder_inputs_embeds=None, use_cache=None, output_attentions=None, output_hidden_states=None, return_dict=None)

Defines the computation to be performed. This method must be overridden by all subclasses.

Note

It is not supported currently that inputs contain both tuple and non-tuple types at same time.

Parameters:

• args (tuple) – Tuple of variable parameters.

• kwargs (dict) – Dictionary of variable keyword parameters.

Returns:

Tensor, returns the computed result.

get_decoder()

get decoder

get_encoder()

get encoder

get_input_embeddings()

Returns the model’s input embeddings.

Returns:

A mindspore cell mapping vocabulary to hidden states.

Return type:

nn.Cell

set_input_embeddings(new_embeddings)

Set model’s input embeddings.

Parameters:

value (nn.Cell) – A mindspore cell mapping vocabulary to hidden states.

class mindnlp.models.t5.T5PreTrainedModel(config)

Bases: PreTrainedModel

An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models.

config_class

alias of T5Config

convert_torch_to_mindspore(**kwargs)

torch to mindspore.

get_input_embeddings()

Returns the model’s input embeddings.

Returns:

A mindspore cell mapping vocabulary to hidden states.

Return type:

nn.Cell

get_position_embeddings()

get the model position embeddings if necessary

post_init()

A method executed at the end of each Transformer model initialization, to execute code that needs the model’s modules properly initialized (such as weight initialization).

resize_position_embeddings()

resize the model position embeddings if necessary

set_input_embeddings()

Set model’s input embeddings.

Parameters:

value (nn.Cell) – A mindspore cell mapping vocabulary to hidden states.

class mindnlp.models.t5.T5Stack(config, embed_tokens=None)

Bases: T5PreTrainedModel

construct(input_ids=None, attention_mask=None, encoder_hidden_states=None, encoder_attention_mask=None, inputs_embeds=None, head_mask=None, cross_attn_head_mask=None, past_key_values=None, use_cache=None, output_attentions=None, output_hidden_states=None, return_dict=None)

Defines the computation to be performed. This method must be overridden by all subclasses.

Note

It is not supported currently that inputs contain both tuple and non-tuple types at same time.

Parameters:

• args (tuple) – Tuple of variable parameters.

• kwargs (dict) – Dictionary of variable keyword parameters.

Returns:

Tensor, returns the computed result.

get_input_embeddings()

Returns the model’s input embeddings.

Returns:

A mindspore cell mapping vocabulary to hidden states.

Return type:

nn.Cell

set_input_embeddings(new_embeddings)

Set model’s input embeddings.

Parameters:

value (nn.Cell) – A mindspore cell mapping vocabulary to hidden states.

ernie

Ernie Model init

class mindnlp.models.ernie.ErnieConfig(vocab_size: int = 30522, hidden_size: int = 768, num_hidden_layers: int = 12, num_attention_heads: int = 12, task_id=0, intermediate_size: int = 3072, hidden_act: str = 'gelu', hidden_dropout_prob: float = 0.1, attention_probs_dropout_prob: float = 0.1, max_position_embeddings: int = 512, task_type_vocab_size: int = 3, type_vocab_size: int = 16, initializer_range: float = 0.02, pad_token_id: int = 0, pool_act: str = 'tanh', fuse: bool = False, layer_norm_eps=1e-12, use_cache=False, use_task_id=True, enable_recompute=False, **kwargs)

Bases: PreTrainedConfig

Configuration for Ernie.

class mindnlp.models.ernie.ErnieEmbeddings(config: ErnieConfig, embedding_table)

Bases: Cell

Ernie Embeddings for word, position and token_type embeddings.

construct(input_ids: Optional[Tensor] = None, token_type_ids: Optional[Tensor] = None, position_ids: Optional[Tensor] = None, task_type_ids: Optional[Tensor] = None, inputs_embeds: Optional[Tensor] = None, past_key_values_length: int = 0)

Defines the computation to be performed. This method must be overridden by all subclasses.

Note

It is not supported currently that inputs contain both tuple and non-tuple types at same time.

Parameters:

• args (tuple) – Tuple of variable parameters.

• kwargs (dict) – Dictionary of variable keyword parameters.

Returns:

Tensor, returns the computed result.

class mindnlp.models.ernie.ErnieModel(config: ErnieConfig)

Bases: ErniePretrainedModel

Ernie model.

construct(input_ids: Optional[Tensor] = None, token_type_ids: Optional[Tensor] = None, position_ids: Optional[Tensor] = None, attention_mask: Optional[Tensor] = None, task_type_ids: Optional[Tensor] = None, past_key_values: Optional[Tuple[Tuple[Tensor]]] = None, inputs_embeds: Optional[Tensor] = None, output_hidden_states: Optional[bool] = None, output_attentions: Optional[bool] = None, return_dict: Optional[bool] = None)

Defines the computation to be performed. This method must be overridden by all subclasses.

Note

It is not supported currently that inputs contain both tuple and non-tuple types at same time.

Parameters:

• args (tuple) – Tuple of variable parameters.

• kwargs (dict) – Dictionary of variable keyword parameters.

Returns:

Tensor, returns the computed result.

get_input_embeddings()

Returns the model’s input embeddings.

Returns:

A mindspore cell mapping vocabulary to hidden states.

Return type:

nn.Cell

set_input_embeddings(value)

Set model’s input embeddings.

Parameters:

value (nn.Cell) – A mindspore cell mapping vocabulary to hidden states.

class mindnlp.models.ernie.ErniePooler(config: ErnieConfig, weight_init)

Bases: Cell

Ernie Pooler.

construct(hidden_states)

Defines the computation to be performed. This method must be overridden by all subclasses.

Note

It is not supported currently that inputs contain both tuple and non-tuple types at same time.

Parameters:

• args (tuple) – Tuple of variable parameters.

• kwargs (dict) – Dictionary of variable keyword parameters.

Returns:

Tensor, returns the computed result.

class mindnlp.models.ernie.UIE(config: ErnieConfig)

Bases: ErniePretrainedModel

UIE model based on Ernie.

construct(input_ids: Optional[Tensor] = None, token_type_ids: Optional[Tensor] = None, position_ids: Optional[Tensor] = None, attention_mask: Optional[Tensor] = None, inputs_embeds: Optional[Tensor] = None, return_dict: Optional[Tensor] = None)

Defines the computation to be performed. This method must be overridden by all subclasses.

Note

It is not supported currently that inputs contain both tuple and non-tuple types at same time.

Parameters:

• args (tuple) – Tuple of variable parameters.

• kwargs (dict) – Dictionary of variable keyword parameters.

Returns:

Tensor, returns the computed result.

class mindnlp.models.ernie.UIEConfig(vocab_size: int = 40000, hidden_size: int = 768, num_hidden_layers: int = 12, num_attention_heads: int = 12, task_id=0, intermediate_size: int = 3072, hidden_act: str = 'gelu', hidden_dropout_prob: float = 0.1, attention_probs_dropout_prob: float = 0.1, max_position_embeddings: int = 2048, task_type_vocab_size: int = 3, type_vocab_size: int = 4, initializer_range: float = 0.02, pad_token_id: int = 0, pool_act: str = 'tanh', fuse: bool = False, layer_norm_eps=1e-12, use_cache=False, use_task_id=True, enable_recompute=False, **kwargs)

Bases: PreTrainedConfig

Configuration for UIE.

roberta

Roberta model.

class mindnlp.models.roberta.RobertaClassificationHead(config)

Bases: Cell

construct(features)

Defines the computation to be performed. This method must be overridden by all subclasses.

Note

It is not supported currently that inputs contain both tuple and non-tuple types at same time.

Parameters:

• args (tuple) – Tuple of variable parameters.

• kwargs (dict) – Dictionary of variable keyword parameters.

Returns:

Tensor, returns the computed result.

class mindnlp.models.roberta.RobertaConfig(vocab_size=30522, hidden_size=768, num_hidden_layers=12, num_attention_heads=12, intermediate_size=3072, hidden_act='gelu', hidden_dropout_prob=0.1, attention_probs_dropout_prob=0.1, max_position_embeddings=512, type_vocab_size=2, initializer_range=0.02, layer_norm_eps=1e-12, classifier_dropout=None, **kwargs)

Bases: BertConfig

Roberta Config.

class mindnlp.models.roberta.RobertaEmbeddings(config)

Bases: BertEmbeddings

Roberta embeddings

construct(input_ids, token_type_ids=None, position_ids=None)

Defines the computation to be performed. This method must be overridden by all subclasses.

Note

It is not supported currently that inputs contain both tuple and non-tuple types at same time.

Parameters:

• args (tuple) – Tuple of variable parameters.

• kwargs (dict) – Dictionary of variable keyword parameters.

Returns:

Tensor, returns the computed result.

class mindnlp.models.roberta.RobertaForMaskedLM(config, *args, **kwargs)

Bases: RobertaPreTrainedModel

construct(input_ids, attention_mask=None, token_type_ids=None, position_ids=None, head_mask=None, masked_lm_labels=None)

Defines the computation to be performed. This method must be overridden by all subclasses.

Note

It is not supported currently that inputs contain both tuple and non-tuple types at same time.

Parameters:

• args (tuple) – Tuple of variable parameters.

• kwargs (dict) – Dictionary of variable keyword parameters.

Returns:

Tensor, returns the computed result.

class mindnlp.models.roberta.RobertaForMultipleChoice(config, *args, **kwargs)

Bases: RobertaPreTrainedModel

construct(input_ids, token_type_ids=None, attention_mask=None, labels=None, position_ids=None, head_mask=None)

Defines the computation to be performed. This method must be overridden by all subclasses.

Note

It is not supported currently that inputs contain both tuple and non-tuple types at same time.

Parameters:

• args (tuple) – Tuple of variable parameters.

• kwargs (dict) – Dictionary of variable keyword parameters.

Returns:

Tensor, returns the computed result.

class mindnlp.models.roberta.RobertaForSequenceClassification(config, *args, **kwargs)

Bases: RobertaPreTrainedModel

construct(input_ids, attention_mask=None, token_type_ids=None, position_ids=None, head_mask=None, labels=None)

Defines the computation to be performed. This method must be overridden by all subclasses.

Note

It is not supported currently that inputs contain both tuple and non-tuple types at same time.

Parameters:

• args (tuple) – Tuple of variable parameters.

• kwargs (dict) – Dictionary of variable keyword parameters.

Returns:

Tensor, returns the computed result.

class mindnlp.models.roberta.RobertaLMHead(config)

Bases: Cell

construct(features)

Defines the computation to be performed. This method must be overridden by all subclasses.

Note

It is not supported currently that inputs contain both tuple and non-tuple types at same time.

Parameters:

• args (tuple) – Tuple of variable parameters.

• kwargs (dict) – Dictionary of variable keyword parameters.

Returns:

Tensor, returns the computed result.

class mindnlp.models.roberta.RobertaModel(config, add_pooling_layer=True)

Bases: BertModel

Roberta Model

class mindnlp.models.roberta.RobertaPreTrainedModel(config)

Bases: BertPreTrainedModel

Roberta Pretrained Model.

config_class

alias of RobertaConfig