ValueError: Tokenizer class MarianTokenizer does not exist or is not currently imported

Get this error when trying to run a MarianMT-based nmt model.

Traceback (most recent call last):
File "/home/om/Desktop/Project/nmt-marionmt-api/inference.py", line 45, in <module>
    print(batch_inference(model_path="en-ar-model/Mark2", text=text))
  File "/home/om/Desktop/Project/nmt-marionmt-api/inference.py", line 15, in batch_inference
    tokenizer = AutoTokenizer.from_pretrained(model_path, local_file_only=True)
  File "/home/om/.virtualenvs/marianmt-api/lib/python3.8/site-packages/transformers/models/auto/tokenization_auto.py", line 525, in from_pretrained
    raise ValueError(
ValueError: Tokenizer class MarianTokenizer does not exist or is not currently imported.

1 answer

  • answered 2022-05-04 10:16 Om Rastogi

    Installing SentencePiece worked for me.

    pip install sentencepiece
    

How many English words
do you know?
Test your English vocabulary size, and measure
how many words do you know
Online Test
Powered by Examplum