UAX URL email tokenizer

uax_url_email トークナイザーは、standard トークナイザー のようなもので、URL とメールアドレスを単一のトークンとして認識します。

Example output

Python

  1. resp = client.indices.analyze(
  2. tokenizer="uax_url_email",
  3. text="Email me at [email protected]",
  4. )
  5. print(resp)

Ruby

  1. response = client.indices.analyze(
  2. body: {
  3. tokenizer: 'uax_url_email',
  4. text: 'Email me at [email protected]'
  5. }
  6. )
  7. puts response

Js

  1. const response = await client.indices.analyze({
  2. tokenizer: "uax_url_email",
  3. text: "Email me at [email protected]",
  4. });
  5. console.log(response);

Console

  1. POST _analyze
  2. {
  3. "tokenizer": "uax_url_email",
  4. "text": "Email me at [email protected]"
  5. }

上記の文は次の用語を生成します:

Text

  1. [ Email, me, at, john.smith@global-international.com ]

一方、standard トークナイザーは次のように生成します:

Text

  1. [ Email, me, at, john.smith, global, international.com ]

Configuration

uax_url_email トークナイザーは次のパラメータを受け入れます:

max_token_length 最大トークン長。 この長さを超えるトークンが見られた場合、
それは max_token_length の間隔で分割されます。 デフォルトは 255 です。

Example configuration

この例では、uax_url_email トークナイザーを 5 の max_token_length に設定します(デモ目的のため):

Python

  1. resp = client.indices.create(
  2. index="my-index-000001",
  3. settings={
  4. "analysis": {
  5. "analyzer": {
  6. "my_analyzer": {
  7. "tokenizer": "my_tokenizer"
  8. }
  9. },
  10. "tokenizer": {
  11. "my_tokenizer": {
  12. "type": "uax_url_email",
  13. "max_token_length": 5
  14. }
  15. }
  16. }
  17. },
  18. )
  19. print(resp)
  20. resp1 = client.indices.analyze(
  21. index="my-index-000001",
  22. analyzer="my_analyzer",
  23. text="[email protected]",
  24. )
  25. print(resp1)

Ruby

  1. response = client.indices.create(
  2. index: 'my-index-000001',
  3. body: {
  4. settings: {
  5. analysis: {
  6. analyzer: {
  7. my_analyzer: {
  8. tokenizer: 'my_tokenizer'
  9. }
  10. },
  11. tokenizer: {
  12. my_tokenizer: {
  13. type: 'uax_url_email',
  14. max_token_length: 5
  15. }
  16. }
  17. }
  18. }
  19. }
  20. )
  21. puts response
  22. response = client.indices.analyze(
  23. index: 'my-index-000001',
  24. body: {
  25. analyzer: 'my_analyzer',
  26. text: '[email protected]'
  27. }
  28. )
  29. puts response

Js

  1. const response = await client.indices.create({
  2. index: "my-index-000001",
  3. settings: {
  4. analysis: {
  5. analyzer: {
  6. my_analyzer: {
  7. tokenizer: "my_tokenizer",
  8. },
  9. },
  10. tokenizer: {
  11. my_tokenizer: {
  12. type: "uax_url_email",
  13. max_token_length: 5,
  14. },
  15. },
  16. },
  17. },
  18. });
  19. console.log(response);
  20. const response1 = await client.indices.analyze({
  21. index: "my-index-000001",
  22. analyzer: "my_analyzer",
  23. text: "[email protected]",
  24. });
  25. console.log(response1);

Console

  1. PUT my-index-000001
  2. {
  3. "settings": {
  4. "analysis": {
  5. "analyzer": {
  6. "my_analyzer": {
  7. "tokenizer": "my_tokenizer"
  8. }
  9. },
  10. "tokenizer": {
  11. "my_tokenizer": {
  12. "type": "uax_url_email",
  13. "max_token_length": 5
  14. }
  15. }
  16. }
  17. }
  18. }
  19. POST my-index-000001/_analyze
  20. {
  21. "analyzer": "my_analyzer",
  22. "text": "[email protected]"
  23. }

上記の例は次の用語を生成します:

Text

  1. [ john, smith, globa, l, inter, natio, nal.c, om ]