This is machine translation

Translated by Microsoft
Mouseover text to see original. Click the button below to return to the English version of the page.

Note: This page has been translated by MathWorks. Click here to see
To view all translated materials including this page, select Country from the country navigator on the bottom of this page.

Japanese Language Support

This topic summarizes the Text Analytics Toolbox™ features that support Japanese text. For an example showing how to analyze Japanese text data, see Analyze Japanese Text Data.

Tokenization

The tokenizedDocument function automatically detects Japanese input. Alternatively, set the 'Language' option in tokenizedDocument to 'ja'. This option specifies the language details of the tokens. To view the language details of the tokens, use tokenDetails. These language details determine the behavior of the removeStopWords, addPartOfSpeechDetails, normalizeWords, addSentenceDetails, and addEntityDetails functions on the tokens.

Tokenize Japanese Text

Tokenize Japanese text using tokenizedDocument. The function automatically detects Japanese text.

str = [
    "恋に悩み、苦しむ。"
    "恋の悩みで苦しむ。"
    "空に星が輝き、瞬いている。"
    "空の星が輝きを増している。"];
documents = tokenizedDocument(str)
documents = 
  4x1 tokenizedDocument:

     6 tokens: 恋 に 悩み 、 苦しむ 。
     6 tokens: 恋 の 悩み で 苦しむ 。
    10 tokens: 空 に 星 が 輝き 、 瞬い て いる 。
    10 tokens: 空 の 星 が 輝き を 増し て いる 。

Part of Speech Details

The tokenDetails function, by default, includes part of speech details with the token details.

Get Part of Speech Details of Japanese Text

Tokenize Japanese text using tokenizedDocument.

str = [
    "恋に悩み、苦しむ。"
    "恋の悩みで 苦しむ。"
    "空に星が輝き、瞬いている。"
    "空の星が輝きを増している。"
    "駅までは遠くて、歩けない。"
    "遠くの駅まで歩けない。"
    "すもももももももものうち。"];
documents = tokenizedDocument(str);

For Japanese text, you can get the part-of-speech details using tokenDetails. For English text, you must first use addPartOfSpeechDetails.

tdetails = tokenDetails(documents);
head(tdetails)
ans=8×8 table
     Token     DocumentNumber    LineNumber       Type        Language    PartOfSpeech     Lemma       Entity  
    _______    ______________    __________    ___________    ________    ____________    _______    __________

    "恋"             1               1         letters           ja       noun            "恋"       non-entity
    "に"             1               1         letters           ja       adposition      "に"       non-entity
    "悩み"           1               1         letters           ja       verb            "悩む"      non-entity
    "、"             1               1         punctuation       ja       punctuation     "、"       non-entity
    "苦しむ"          1               1         letters           ja       verb            "苦しむ"    non-entity
    "。"             1               1         punctuation       ja       punctuation     "。"       non-entity
    "恋"             2               1         letters           ja       noun            "恋"       non-entity
    "の"             2               1         letters           ja       adposition      "の"       non-entity

Named Entity Recognition

To add entity tags to documents, use the addEntityDetails function.

Add Named Entity Tags to Japanese Text

Tokenize Japanese text using tokenizedDocument.

str = [
    "マリーさんはボストンからニューヨークに引っ越しました。"
    "駅で鈴木さんに迎えに行きます。"
    "東京は大阪より大きいですか?"
    "東京に行った時、新宿や渋谷などいろいろな所に訪れたました。"];
documents = tokenizedDocument(str);

For Japanese text, the software automatically adds named entity tags, so you do not need to use the addEntityDetails function. This software detects person names, locations, organizations, and other named entities. To view the entity details, use the tokenDetails function.

tdetails = tokenDetails(documents);
head(tdetails)
ans=8×8 table
       Token        DocumentNumber    LineNumber     Type      Language    PartOfSpeech       Lemma          Entity  
    ____________    ______________    __________    _______    ________    ____________    ____________    __________

    "マリー"               1               1         letters       ja       proper-noun     "マリー"         person    
    "さん"                1               1         letters       ja       noun            "さん"           person    
    "は"                  1               1         letters       ja       adposition      "は"            non-entity
    "ボストン"             1               1         letters       ja       proper-noun     "ボストン"        location  
    "から"                1               1         letters       ja       adposition      "から"           non-entity
    "ニューヨーク"          1               1         letters       ja       proper-noun     "ニューヨーク"    location  
    "に"                  1               1         letters       ja       adposition      "に"            non-entity
    "引っ越し"             1               1         letters       ja       verb            "引っ越す"        non-entity

View the words tagged with entity "person", "location", "organization", or "other". These words are the words not tagged "non-entity".

idx = tdetails.Entity ~= "non-entity";
tdetails(idx,:).Token
ans = 11×1 string array
    "マリー"
    "さん"
    "ボストン"
    "ニューヨーク"
    "鈴木"
    "さん"
    "東京"
    "大阪"
    "東京"
    "新宿"
    "渋谷"

Stop Words

To remove stop words from documents according to the token language details, use removeStopWords. For a list of Japanese stop words set the 'Language' option in stopWords to 'ja'.

Remove Japanese Stop Words

Tokenize Japanese text using tokenizedDocument. The function automatically detects Japanese text.

str = [
    "ここは静かなので、とても穏やかです"
    "企業内の顧客データを利用し、今年の売り上げを調べることが出来た。"
    "私は先生です。私は英語を教えています。"];
documents = tokenizedDocument(str);

Remove stop words using removeStopWords. The function uses the language details from documents to determine which language stop words to remove.

documents = removeStopWords(documents)
documents = 
  3x1 tokenizedDocument:

     4 tokens: 静か 、 とても 穏やか
    10 tokens: 企業 顧客 データ 利用 、 今年 売り上げ 調べる 出来 。
     5 tokens: 先生 。 英語 教え 。

Lemmatization

To lemmatize tokens according to the token language details, use normalizeWords and set the 'Style' option to 'lemma'.

Lemmatize Japanese Text

Tokenize Japanese text using the tokenizedDocument function. The function automatically detects Japanese text.

str = [
    "空に星が輝き、瞬いている。"
    "空の星が輝きを増している。"
    "駅までは遠くて、歩けない。"
    "遠くの駅まで歩けない。"];
documents = tokenizedDocument(str);

Lemmatize the tokens using normalizeWords.

documents = normalizeWords(documents)
documents = 
  4x1 tokenizedDocument:

    10 tokens: 空 に 星 が 輝く 、 瞬く て いる 。
    10 tokens: 空 の 星 が 輝き を 増す て いる 。
     9 tokens: 駅 まで は 遠い て 、 歩ける ない 。
     7 tokens: 遠く の 駅 まで 歩ける ない 。

Language-Independent Features

Word and N-Gram Counting

The bagOfWords and bagOfNgrams functions support tokenizedDocument input regardless of language. If you have a tokenizedDocument array containing your data, then you can use these functions.

Modeling and Prediction

The fitlda and fitlsa functions support bagOfWords and bagOfNgrams input regardless of language. If you have a bagOfWords or bagOfNgrams object containing your data, then you can use these functions.

The trainWordEmbedding function supports tokenizedDocument or file input regardless of language. If you have a tokenizedDocument array or a file containing your data in the correct format, then you can use this function.

See Also

| | | | | | |

Related Topics