Class ForeignTokenizer

Hierarchy

any
- ForeignTokenizer

Index

Properties

Methods

Properties

_REGEXP_SPLIT_1

_REGEXP_SPLIT_1: RegExp

分詞用(包含中文)

_REGEXP_SPLIT_2

_REGEXP_SPLIT_2: RegExp

分詞用(不包含中文的全詞符合)

name

name: string = "ForeignTokenizer"

Methods

_cache

_cache(): void

- Defined in lib/submod/ForeignTokenizer.ts:29
Returns void

createForeignToken

createForeignToken(word: IWord, lasttype?: number, attr?: IWordDebugInfo): any

- Defined in lib/submod/ForeignTokenizer.ts:327
Parameters
- word: IWord
- Optional lasttype: number
- Optional attr: IWordDebugInfo
Returns any

split

split(words: IWord[]): IWord[]

- Defined in lib/submod/ForeignTokenizer.ts:72
对未识别的单词进行分词

Parameters
- words: IWord[]
  
  单词数组
Returns IWord[]

splitForeign

splitForeign(text: string, cur?: number): IWord[]

- Defined in lib/submod/ForeignTokenizer.ts:217
匹配包含的英文字符和数字，并分割

Parameters
- text: string
  
  文本
- Optional cur: number
  
  开始位置
Returns IWord[]

返回格式 {w: '单词', c: 开始位置}

splitForeign2

splitForeign2(text: string, cur?: number): IWord[]

- Defined in lib/submod/ForeignTokenizer.ts:102
支援更多外文判定(但可能會降低效率)

並且避免誤切割例如 latīna Русский

Parameters
- text: string
- Optional cur: number
Returns IWord[]

Generated using TypeDoc