Segment | novel-segment

创建分词器接口

Hierarchy

Segment

Index

Interfaces

Type aliases

Constructors

constructor

Properties

Type aliases

Static IDICT_BLACKLIST

IDICT_BLACKLIST: IDICT<boolean>

Static IDICT_STOPWORD

IDICT_STOPWORD: IDICT<boolean>

Static IDICT_SYNONYM

IDICT_SYNONYM: IDICT<string>

Static IOptionsSegment

IOptionsSegment: IOptionsTableDict & object

Static ISPLIT

ISPLIT: RegExp | string | object

Static ISPLIT_FILTER

ISPLIT_FILTER: RegExp | object

Constructors

constructor

new Segment(options?: IOptionsSegment): Segment

- Defined in lib/Segment.ts:100
Parameters
- Default value options: IOptionsSegment = {}
Returns Segment

Properties

DICT

DICT: object

词典表

type: {{}}

Type declaration

[key: string]: IDICT
Optional STOPWORD?: IDICT_STOPWORD
Optional SYNONYM?: IDICT_SYNONYM

POSTAG

POSTAG: POSTAG = POSTAG

词性

type: {POSTAG}

SPLIT

SPLIT: ISPLIT = /([\r\n]+|^[　\s+]+|[　\s]+$|[　\s]{2,})/gm as ISPLIT

分段

由於 segment 是利用對內容的前後文分析來進行分詞所以如何切割段落對於結果就會產生不同影響

RegExp or 具有 .[Symbol.split](input: string, limit?: number) => string[] 的物件

type: {Segment.ISPLIT}

SPLIT_FILTER

SPLIT_FILTER: ISPLIT_FILTER = /^([\r\n]+)$/g as ISPLIT_FILTER

分段之後如果符合以下條件則直接忽略分析 RegExp or 具有 .test(input: string) => boolean 的物件

type: {Segment.ISPLIT_FILTER}

db

db: object

Type declaration

[key: string]: TableDict

Optional inited

inited: boolean

modules

modules: object = {/*** 分词模块*/tokenizer: [],/*** 优化模块*/optimizer: [],} as {tokenizer: ISubTokenizer[],optimizer: ISubOptimizer[],}

Type declaration

optimizer: ISubOptimizer[]
tokenizer: ISubTokenizer[]

optimizer

optimizer: Optimizer

options

options: IOptionsSegment

tokenizer

tokenizer: Tokenizer

Static defaultOptionsDoSegment

defaultOptionsDoSegment: IOptionsDoSegment

Methods

Protected _get_text

_get_text(text: string | Buffer): string

- Defined in lib/Segment.ts:610
Parameters
- text: string | Buffer
Returns string

Protected _loadBlacklistDict

_loadBlacklistDict(name: string, type: EnumDictDatabase): this

- Defined in lib/Segment.ts:423
Parameters
- name: string
- type: EnumDictDatabase
Returns this

_resolveDictFilename

_resolveDictFilename(name: string, pathPlus?: string[], extPlus?: string[]): string | string[]

- Defined in lib/Segment.ts:228
Parameters
- name: string
- Default value pathPlus: string[] = []
- Default value extPlus: string[] = []
Returns string | string[]

addBlacklist

addBlacklist(word: string, remove?: boolean): this

- Defined in lib/Segment.ts:634
Parameters
- word: string
- Optional remove: boolean
Returns this

autoInit

autoInit(options?: object): this

- Defined in lib/Segment.ts:584
此函數只需執行一次，並且一般狀況下不需要手動呼叫

Parameters
- Optional options: object
  - Optional all_mod?: boolean
Returns this

convertSynonym

convertSynonym(ret: IWordDebug[], showcount: true): object
convertSynonym(ret: IWordDebug[], showcount?: boolean): IWordDebug[]

- Defined in lib/Segment.ts:905
转换同义词

Parameters
- ret: IWordDebug[]
- showcount: true
Returns object
- count: number
- list: IWordDebug[]
- Defined in lib/Segment.ts:906
Parameters
- ret: IWordDebug[]
- Optional showcount: boolean
Returns IWordDebug[]

doBlacklist

doBlacklist(): this

- Defined in lib/Segment.ts:661
remove key in TABLE by BLACKLIST

Returns this

doSegment

doSegment(text: string | Buffer, options: IOptionsDoSegment & object): string[]
doSegment(text: string | Buffer, options?: IOptionsDoSegment): IWord[]

- Defined in lib/Segment.ts:767
开始分词

Parameters
- text: string | Buffer
  
  文本
- options: IOptionsDoSegment & object
  选项
  - {Boolean} simple 是否仅返回单词内容
  - {Boolean} stripPunctuation 去除标点符号
  - {Boolean} convertSynonym 转换同义词
  - {Boolean} stripStopword 去除停止符
Returns string[]
- Defined in lib/Segment.ts:770
Parameters
- text: string | Buffer
- Optional options: IOptionsDoSegment
Returns IWord[]

getDict

getDict(type: STOPWORD): IDICT_STOPWORD
getDict(type: SYNONYM): IDICT_SYNONYM
getDict(type: TABLE): IDICT<IWord>
getDict(type: BLACKLIST): IDICT_BLACKLIST
getDict(type: BLACKLIST_FOR_OPTIMIZER): IDICT_BLACKLIST
getDict(type: "TABLE2"): IDICT2<IWord>
getDict(type: EnumDictDatabase): IDICT
getDict(type: any): IDICT

- Defined in lib/Segment.ts:350
取词典表

Parameters
- type: STOPWORD
  
  类型
Returns IDICT_STOPWORD
- Defined in lib/Segment.ts:351
Parameters
- type: SYNONYM
Returns IDICT_SYNONYM
- Defined in lib/Segment.ts:352
Parameters
- type: TABLE
Returns IDICT<IWord>
- Defined in lib/Segment.ts:353
Parameters
- type: BLACKLIST
Returns IDICT_BLACKLIST
- Defined in lib/Segment.ts:354
Parameters
- type: BLACKLIST_FOR_OPTIMIZER
Returns IDICT_BLACKLIST
- Defined in lib/Segment.ts:355
Parameters
- type: "TABLE2"
Returns IDICT2<IWord>
- Defined in lib/Segment.ts:356
Parameters
- type: EnumDictDatabase
Returns IDICT
- Defined in lib/Segment.ts:357
Parameters
- type: any
Returns IDICT

getDictDatabase

getDictDatabase<R>(type: SYNONYM, autocreate?: boolean, libTableDict?: object): R
getDictDatabase<R>(type: TABLE, autocreate?: boolean, libTableDict?: object): R
getDictDatabase<R>(type: STOPWORD, autocreate?: boolean, libTableDict?: object): R
getDictDatabase<R>(type: BLACKLIST, autocreate?: boolean, libTableDict?: object): R
getDictDatabase<R>(type: BLACKLIST_FOR_OPTIMIZER, autocreate?: boolean, libTableDict?: object): R
getDictDatabase<R>(type: string | EnumDictDatabase, autocreate?: boolean, libTableDict?: object): R

- Defined in lib/Segment.ts:122
Type parameters
- R: TableDictSynonym
Parameters
- type: SYNONYM
- Optional autocreate: boolean
- Optional libTableDict: object
  - constructor: function
    - new __type(...argv: any[]): R
    - Defined in lib/Segment.ts:124
      Parameters
      - Rest ...argv: any[]
      Returns R
Returns R
- Defined in lib/Segment.ts:126
Type parameters
- R: TableDict
Parameters
- type: TABLE
- Optional autocreate: boolean
- Optional libTableDict: object
  - constructor: function
    - new __type(...argv: any[]): R
    - Defined in lib/Segment.ts:126
      Parameters
      - Rest ...argv: any[]
      Returns R
Returns R
- Defined in lib/Segment.ts:127
Type parameters
- R: TableDictStopword
Parameters
- type: STOPWORD
- Optional autocreate: boolean
- Optional libTableDict: object
  - constructor: function
    - new __type(...argv: any[]): R
    - Defined in lib/Segment.ts:129
      Parameters
      - Rest ...argv: any[]
      Returns R
Returns R
- Defined in lib/Segment.ts:131
Type parameters
- R: TableDictBlacklist
Parameters
- type: BLACKLIST
- Optional autocreate: boolean
- Optional libTableDict: object
  - constructor: function
    - new __type(...argv: any[]): R
    - Defined in lib/Segment.ts:133
      Parameters
      - Rest ...argv: any[]
      Returns R
Returns R
- Defined in lib/Segment.ts:135
Type parameters
- R: TableDictBlacklist
Parameters
- type: BLACKLIST_FOR_OPTIMIZER
- Optional autocreate: boolean
- Optional libTableDict: object
  - constructor: function
    - new __type(...argv: any[]): R
    - Defined in lib/Segment.ts:137
      Parameters
      - Rest ...argv: any[]
      Returns R
Returns R
- Defined in lib/Segment.ts:139
Type parameters
- R: AbstractTableDictCore<any>
Parameters
- type: string | EnumDictDatabase
- Optional autocreate: boolean
- Optional libTableDict: object
  - constructor: function
    - new __type(...argv: any[]): R
    - Defined in lib/Segment.ts:141
      Parameters
      - Rest ...argv: any[]
      Returns R
Returns R

getOptionsDoSegment

getOptionsDoSegment<T>(options?: T): T

- Defined in lib/Segment.ts:601
Type parameters
- T: IOptionsDoSegment
Parameters
- Optional options: T
Returns T

indexOf

indexOf(words: IWord[], s: string | number, cur?: number): number

- Defined in lib/Segment.ts:1099
在单词数组中查找某一个单词或词性所在的位置

Parameters
- words: IWord[]
  
  单词数组
- s: string | number
  
  要查找的单词或词性
- Optional cur: number
  
  开始位置
Returns number

找不到，返回-1

listModules

listModules(options?: IOptionsDoSegment): object

- Defined in lib/Segment.ts:680
Parameters
- Default value options: IOptionsDoSegment = {}
Returns object
- disable: object
  - optimizer: any[]
  - tokenizer: any[]
- enable: object
  - optimizer: any[]
  - tokenizer: any[]

loadBlacklistDict

loadBlacklistDict(name: string): this

- Defined in lib/Segment.ts:461
Parameters
- name: string
Returns this

loadBlacklistOptimizerDict

loadBlacklistOptimizerDict(name: string): this

- Defined in lib/Segment.ts:466
Parameters
- name: string
Returns this

loadDict

loadDict(name: string, type?: string, convert_to_lower?: boolean, skipExists?: boolean): this

- Defined in lib/Segment.ts:280
载入字典文件

Parameters
- name: string
  
  字典文件名
- Optional type: string
  
  类型
- Optional convert_to_lower: boolean
  
  是否全部转换为小写
- Optional skipExists: boolean
Returns this

loadStopwordDict

loadStopwordDict(name: string): this

- Defined in lib/Segment.ts:476
载入停止符词典

Parameters
- name: string
  
  字典文件名
Returns this

loadSynonymDict

loadSynonymDict(name: string, skipExists?: boolean): this

- Defined in lib/Segment.ts:368
载入同义词词典

Parameters
- name: string
  
  字典文件名
- Optional skipExists: boolean
Returns this

split

split(words: IWord[], s: string | number): IWord[]

- Defined in lib/Segment.ts:1062
根据某个单词或词性来分割单词数组

Parameters
- words: IWord[]
  
  单词数组
- s: string | number
  
  用于分割的单词或词性
Returns IWord[]

stringify

stringify(words: Array<IWord | string>, ...argv: any[]): string

- Defined in lib/Segment.ts:1031
将单词数组连接成字符串

Parameters
- words: Array<IWord | string>
  
  单词数组
- Rest ...argv: any[]
Returns string

use

use(mod: ISubOptimizer, ...argv: any[]): any
use(mod: ISubTokenizer, ...argv: any[]): any
use(mod: Array<ISubTokenizer | ISubOptimizer | string>, ...argv: any[]): any
use(mod: string, ...argv: any[]): any
use(mod: any, ...argv: any[]): any

- Defined in lib/Segment.ts:179
载入分词模块

Parameters
- mod: ISubOptimizer
- Rest ...argv: any[]
Returns any
- Defined in lib/Segment.ts:180
Parameters
- mod: ISubTokenizer
- Rest ...argv: any[]
Returns any
- Defined in lib/Segment.ts:181
Parameters
- mod: Array<ISubTokenizer | ISubOptimizer | string>
- Rest ...argv: any[]
Returns any
- Defined in lib/Segment.ts:182
Parameters
- mod: string
- Rest ...argv: any[]
Returns any
- Defined in lib/Segment.ts:183
Parameters
- mod: any
- Rest ...argv: any[]
Returns any

useDefault

useDefault(...argv: any[]): this

- Defined in lib/Segment.ts:522
使用默认的识别模块和字典文件在使用預設值的情況下，不需要主動呼叫此函數

Parameters
- Rest ...argv: any[]
Returns this

Static stringify

stringify(words: Array<IWord | string>, ...argv: any[]): string

- Defined in lib/Segment.ts:1036
Parameters
- words: Array<IWord | string>
- Rest ...argv: any[]
Returns string

Hierarchy

Index

Interfaces

Type aliases

Constructors

Properties

Methods

Type aliases

Static IDICT_BLACKLIST

Static IDICT_STOPWORD

Static IDICT_SYNONYM

Static IOptionsSegment

Static ISPLIT

Static ISPLIT_FILTER

Constructors

constructor

Parameters

Default value options: IOptionsSegment = {}

Returns Segment

Properties

DICT

Type declaration

[key: string]: IDICT

Optional STOPWORD?: IDICT_STOPWORD

Optional SYNONYM?: IDICT_SYNONYM

POSTAG

SPLIT

SPLIT_FILTER

db

Type declaration

[key: string]: TableDict

Optional inited

modules

Type declaration

optimizer: ISubOptimizer[]

tokenizer: ISubTokenizer[]

optimizer

options

tokenizer

Static defaultOptionsDoSegment

Methods

Protected _get_text

Parameters

text: string | Buffer

Returns string

Protected _loadBlacklistDict

Parameters

name: string

type: EnumDictDatabase

Returns this

_resolveDictFilename

Parameters

name: string

Default value pathPlus: string[] = []

Default value extPlus: string[] = []

Returns string | string[]

addBlacklist

Parameters

word: string

Optional remove: boolean

Returns this

autoInit

Parameters

Optional options: object

Optional all_mod?: boolean

Returns this

convertSynonym

Parameters

ret: IWordDebug[]

showcount: true

Returns object

count: number

list: IWordDebug[]

Parameters

ret: IWordDebug[]

Optional showcount: boolean

Returns IWordDebug[]

doBlacklist

Returns this

doSegment