Class SubSModuleTokenizer

Hierarchy

SubSModule
- SubSModuleTokenizer

Implements

ISubSModule
ISubSModule & object

Index

Constructors

constructor

Properties

Methods

Constructors

constructor

new SubSModuleTokenizer(type?: ISModuleType, segment?: Segment, ...argv: any[]): SubSModuleTokenizer

Inherited from SubSModule.constructor
- Defined in lib/mod/mod.ts:58
Parameters
- Optional type: ISModuleType
- Optional segment: Segment
- Rest ...argv: any[]
Returns SubSModuleTokenizer

Properties

Protected Optional _BLACKLIST

_BLACKLIST: IDICT_BLACKLIST

Protected Optional _POSTAG

_POSTAG: POSTAG

Protected Optional _TABLE

_TABLE: any

Optional inited

inited: boolean

name

name: string

Optional priority

priority: number

segment

segment: Segment

type

type: "tokenizer" = "tokenizer"

Static type

type: "tokenizer" = "tokenizer"

Methods

Protected _cache

_cache(...argv: any[]): void

Inherited from SubSModule._cache
- Defined in lib/mod/mod.ts:114
Parameters
- Rest ...argv: any[]
Returns void

Protected _splitUnknow

_splitUnknow<T, U>(words: T[], fn: function): U[]

- Defined in lib/mod/Tokenizer.ts:83
仅对未识别的词进行匹配包含已存在但 p 為 0

Type parameters
- T: IWord
- U: IWord
Parameters
- words: T[]
- fn: function
  - - (text: string, ...argv: any[]): U[]
    - Parameters
      - text: string
      - Rest ...argv: any[]
      Returns U[]
Returns U[]

Protected _splitUnset

_splitUnset<T, U>(words: T[], fn: function): U[]

- Defined in lib/mod/Tokenizer.ts:48
仅对未识别的词进行匹配不包含 p 為 0

Type parameters
- T: IWord
- U: IWord
Parameters
- words: T[]
- fn: function
  - - (text: string, ...argv: any[]): U[]
    - Parameters
      - text: string
      - Rest ...argv: any[]
      Returns U[]
Returns U[]

Protected createRawToken

createRawToken<T, U>(data: T, ow?: Partial<T & IWord>, attr?: U & IWordDebugInfo): T

Inherited from SubSModule.createRawToken
- Defined in lib/mod/mod.ts:122
回傳最簡版的 IWord { w, p, f, s }

Type parameters
- T: IWord
- U: IWordDebugInfo
Parameters
- data: T
- Optional ow: Partial<T & IWord>
- Optional attr: U & IWordDebugInfo
Returns T

Protected createToken

createToken<T, U>(data: T, skipCheck?: boolean, attr?: U & IWordDebugInfo): T

Inherited from SubSModule.createToken
- Defined in lib/mod/mod.ts:142
Type parameters
- T: IWord
- U: IWordDebugInfo
Parameters
- data: T
- Optional skipCheck: boolean
- Optional attr: U & IWordDebugInfo
Returns T

Protected debugToken

debugToken<T, U>(data: T, attr?: U & IWordDebugInfo, returnToken?: true, ...argv: any[]): T

Inherited from SubSModule.debugToken
- Defined in lib/mod/mod.ts:179
Type parameters
- T: IWordDebug
- U: IWordDebugInfo
Parameters
- data: T
- Optional attr: U & IWordDebugInfo
- Optional returnToken: true
- Rest ...argv: any[]
Returns T

init

init(segment: Segment, ...argv: any[]): this

Implementation of ISubSModule.init

Overrides SubSModule.init
- Defined in lib/mod/Tokenizer.ts:31
Parameters
- segment: Segment
- Rest ...argv: any[]
Returns this

Protected sliceToken

sliceToken<T, U>(words: T[], pos: number, len: number, data: T, skipCheck?: boolean, attr?: U & IWordDebugInfo): T[]

Inherited from SubSModule.sliceToken
- Defined in lib/mod/mod.ts:172
Type parameters
- T: IWord
- U: IWordDebugInfo
Parameters
- words: T[]
- pos: number
- len: number
- data: T
- Optional skipCheck: boolean
- Optional attr: U & IWordDebugInfo
Returns T[]

Abstract split

split(words: IWord[], ...argv: any[]): IWord[]

- Defined in lib/mod/Tokenizer.ts:29
Parameters
- words: IWord[]
- Rest ...argv: any[]
Returns IWord[]

Static Protected _init

_init<T>(libThis: IModuleStatic<T>, segment: Segment, ...argv: any[]): T

Inherited from SubSModule._init
- Defined in lib/mod/mod.ts:85
Type parameters
- T: SubSModule
Parameters
- libThis: IModuleStatic<T>
- segment: Segment
- Rest ...argv: any[]
Returns T

Static init

init<T>(segment: Segment, ...argv: any[]): T

Overrides SubSModule.init
- Defined in lib/mod/Tokenizer.ts:38
Type parameters
- T: SubSModuleTokenizer
Parameters
- segment: Segment
- Rest ...argv: any[]
Returns T

Generated using TypeDoc