Use Metrics

There are a variety of Metric in MindNLP for model evaluation: Accuracy, BleuScore, ConfusionMatrix, Distinct, EmScore, F1Score, MatthewsCorrelation, PearsonCorrelation, Perplexity, Precision, Recall, RougeL, RougeN, SpearmanCorrelation.

We can use these pre-defined metrics directly, by instantiating some of the classes and passing the instantiated objects into Trainer as one of its parameters.

Taking the use of Accuracy as an example, the code of using metrics for model training and evaluation is as follows:

from mindnlp.engine.metrics import Accuracy

from mindnlp.engine.trainer import Trainer

metric = Accuracy()

trainer = Trainer(network=network, train_dataset=train_dataset, eval_dataset=eval_dataset,
                    metrics=metric, epochs=epochs, loss_fn=loss_fn, optimizer=optimizer,
                    callbacks=callbacks)

trainer.run(tgt_columns="label")

Define a New Metric

If the metric we need is not provided by MindNLP, it is still simple and easy for us to define our own metric.

All of the classes of metrics defined in MindNLP are inherited from the base class Metric. When defining our own metric class, it is also necessary to extend Metric and rewrite the functions of it:

• __init__(): initializes the metric.

• clear(): clears the internal evaluation results.

• eval(): computes and returns the value of the metric.

• update(*inputs): updates the local variables.

• get_metric_name(): returns the name of the metric.

After finishing those operations, the steps to train and evaluate models using self-defined metrics are the same as mentioned above.