+ hLOcsxRt.ROt^RIt^RIt^RIt^RIHt]t!RR4tR Rlt RR.] n!R R ]4t !R R ]4t !R R]4t Rt Rt.R!O]n!RR]4t!RR4t!RR]4t!RR]4t!RR]4t!RR]4t!RR]4t!RR]4tR#)"ah enchant.tokenize: String tokenization functions for PyEnchant ================================================================ An important task in spellchecking is breaking up large bodies of text into their constituent words, each of which is then checked for correctness. This package provides Python functions to split strings into words according to the rules of a particular language. Each tokenization function accepts a string as its only positional argument, and returns an iterator that yields tuples of the following form, one for each word found:: (,) The meanings of these fields should be clear: is the word that was found and is the position within the text at which the word began (zero indexed, of course). The function will work on any string-like object that supports array-slicing; in particular character-array objects from the 'array' module may be used. The iterator also provides the attribute 'offset' which gives the current position of the tokenizer inside the string being split, and the method 'set_offset' for manually adjusting this position. This can be used for example if the string's contents have changed during the tokenization process. To obtain an appropriate tokenization function for the language identified by , use the function 'get_tokenizer(tag)':: tknzr = get_tokenizer("en_US") for (word,pos) in tknzr("text to be tokenized goes here") do_something(word) This library is designed to be easily extendible by third-party authors. To register a tokenization function for the language , implement it as the function 'tokenize' within the module enchant.tokenize.. The 'get_tokenizer' function will automatically detect it. Note that the underscore must be used as the tag component separator in this case, in order to form a valid python module name. (e.g. "en_US" rather than "en-US") Currently, a tokenizer has only been implemented for the English language. Based on the author's limited experience, this should be at least partially suitable for other languages. This module also provides various implementations of "Chunkers" and "Filters". These classes are designed to make it easy to work with text in a variety of common formats, by detecting and excluding parts of the text that don't need to be checked. A Chunker is a class designed to break a body of text into large chunks of checkable content; for example the HTMLChunker class extracts the text content from all HTML tags but excludes the tags themselves. A Filter is a class designed to skip individual words during the checking process; for example the URLFilter class skips over any words that have the format of a URL. For example, to spellcheck an HTML document it is necessary to split the text into chunks based on HTML tags, and to filter out common word forms such as URLs and WikiWords. This would look something like the following:: tknzr = get_tokenizer("en_US",(HTMLChunker,),(URLFilter,WikiWordFilter))) text = "the url is http://example.com" for (word,pos) in tknzer(text): ...check each word and react accordingly... URLFilterWikiWordFilterN)TokenizerNotFoundErrorcsja]tRt^{toRtR.tRtRtRtRt R Rlt Rt R t ] !] ] 4tR tVtR #) tokenizeaKBase class for all tokenizer objects. Each tokenizer must be an iterator and provide the 'offset' attribute as described in the documentation for this module. While tokenizers are in fact classes, they should be treated like functions, and so are named using lower_case rather than the CamelCase more traditional of class names. Z CamelCasecs Wn^VnR#)iN)_text_offset)selftext&&>/usr/lib/python3.14/site-packages/enchant/tokenize/__init__.py__init__Ztokenize.__init__s  c"VP4#Nnextr&r __next__Ztokenize.__next__syy{r c\4hr )ZNotImplementedErrorrrr rZ tokenize.nexts !##r cV#r rrr __iter__Ztokenize.__iter__s r csWnR#r r)roffsetreplaced&&&r set_offsetZtokenize.set_offsets r csVP#r rrrr _get_offsetZtokenize._get_offsets ||r cfRp\P!V\^R7VPV4R#zSchanging a tokenizers 'offset' attribute is deprecated; use the 'set_offset' methodZcategoryZ stacklevelNwarningswarnDeprecationWarningrrrmsg&& r _set_offsetZtokenize._set_offsets+ +   c$61E r )rrNF)__name__ __module__ __qualname____firstlineno____doc__ _DOC_ERRORSr rrrrrr'propertyr__static_attributes____classdictcell__ __classdict__@r rr{sD-K$ k; /Fr rcszVfRpVe[VfW\V4pV'dD\V^,\4pV'd$Rp\P!V\ ^R7TpRpVPRR4p\V4pVf:VPR4^,p\V4pVfRV: R2p\V4h\pVeB\V4p\\V4^, R R 4Fp\W,V4pK VeVF p V !V4pK \Wu4pV# \ dLi;i) aLocate an appropriate tokenizer by language tag. This requires importing the function 'tokenize' from an appropriate module. Modules tried are named after the language tag, tried in the following order: * the entire tag (e.g. "en_AU.py") * the base country code of the tag (e.g. "en.py") If the language tag is None, a default tokenizer (actually the English one) is returned. It's unicode aware and should work OK for most latin-derived languages. If a suitable function cannot be found, raises TokenizerNotFoundError. If given and not None, 'chunkers' and 'filters' must be lists of chunker classes and filter classes respectively. These will be applied to the tokenizer during creation. NZenzLpassing 'filters' as a non-keyword argument to get_tokenizer() is deprecatedrZ-Z_z!No tokenizer found for language 'Z'i)ZlistZ issubclassFilterr!r"r#Z TypeErrorZreplace_try_tokenizersplitrbasic_tokenizeZrangelenwrap_tokenizer) ZtagZchunkersZfiltersZchunkers_are_filtersr%Ztk_funcZbase tokenizerZiZfs &&& r get_tokenizerr<s9( {>  $'1(1+v'F$(;MM#0BqQ&G#H ++c3 CS!Gyy~a  & ?s8}q("b1A&x{I>I2A) Iy2I K  sD,, D:9D:Zpycs@aa]tRt^toRt.tV3RltRtRtVt V;t #)empty_tokenizez(Tokenizer class that yields no elements.cs&<\SV`R4R#N)superr )r __class__s&r r Zempty_tokenize.__init__s r crr ) StopIterationrrr rZempty_tokenize.nexts or r r)r*r+r,r-r.r rr0r1Z __classcell__rAr3@@r r=r=s2Kr r=cs@aa]tRt^toRt.tV3RltRtRtVt V;t #) unit_tokenizez7Tokenizer class that yields the text as a single token.cs4<\SV`V4RVnR#)FN)r@r _done)rrrAs&&r r Zunit_tokenize.__init__s  r csdVP'd \4hRVnVP^3#T)rGrBrrrr rZunit_tokenize.nexts( :::/ !  Ar )rGrCrDrEr rFrFsAKr rFcs6a]tRtRtoRt.tRtRtRtRt Vt R#)r8i zTokenizer class that performs very basic word-finding. This tokenizer does the most basic thing that could work - it splits text into words based on whitespace boundaries, and removes basic punctuation symbols from the start and end of each word. cs:VPpVPpV\V48d \ 4hV\V48d(W,P4'd V^, pK7TpV\V48d(W,P4'g V^, pK7TpW nV\V48d#W,VP9d V^, pK2^V8d*W^, ,VP 9d V^,pK0W48gKWVV3#rH)rrr9Zisspacestrip_from_startstrip_from_endrB)rrrs_posZe_pos& r rZbasic_tokenize.nextszzT"$o!3t9$)=)=)?)?! E3t9$T\-A-A-C-C! E!L#d)# t7L7L(L e)QY43F3F F }5)511r rNz"'`([z "'`]).!,?;:) r)r*r+r,r-r.rIrJrr0r1r2r4r r8r8 s(K$'Nr r8csRpRpW,p\V\4/V4p\W24# \dR#i;i)zZLook for a tokenizer in the named module. Returns the function if found, None otherwise. zenchant.tokenize.rN)Z __import__ZglobalsZgetattrZ ImportError)Zmod_nameZmod_baseZ func_nameZmods& r r6r62sJ #HI"H79b)<s&& s / >>cs(\V4pWnV#)zWrap one tokenizer inside another. This function takes two tokenizer functions 'tk1' and 'tk2', and returns a new tokenizer function that passes the output of tk1 through tk2 before yielding it to the calling code. )r5_split)Ztk1Ztk2Ztkwr&r r:r:As +CJ Jr cs]tRtRtRtRtR#)ChunkeriSzBase class for text chunking functions. A chunker is designed to chunk text into large blocks of tokens. It has the same interface as a tokenizer but is for a different purpose. rN)r)r*r+r,r-r0rr r rNrNSs   r rNcsPa]tRtRtoRtRtRtRtRt!RR4t R t Vt R #) r5i]aBase class for token filtering functions. A filter is designed to wrap a tokenizer (or another filter) and do two things: * skip over tokens * split tokens into sub-tokens Subclasses have two basic options for customising their behaviour. The method _skip(word) may be overridden to return True for words that should be skipped, and false otherwise. The method _split(word) may be overridden as tokenization function that will be applied to further tokenize any words that aren't skipped. c sWnR#)zFilter class constructor.N _tokenizer)rr;rr r ZFilter.__init__ms#r csrVP!V/VBpVPW0PVP4#r )rP _TokenFilter_skiprM)rZargsZkwdsZtkns&*, r __call__ZFilter.__call__qs0oot,t,  jj$++>>r c sR#)zFilter method for identifying skippable tokens. If this method returns true, the given word will be skipped by the filter. This should be overridden in subclasses to produce the desired functionality. The default behaviour is not to skip any words. Frrwordrr rRZ Filter._skipusr c s\V4#)zFilter method for sub-tokenization of tokens. This method must be a tokenization function that will split the given word into sub-tokens according to the needs of the filter. The default behaviour is not to split any words. )rFrTrr rMZ Filter._split~sT""r cspa]tRtRtoRtR.tRtRtRtRt Rt R t R t ] !] ] 4tRR ltR tVtR #)Filter._TokenFilteriaPrivate inner class implementing the tokenizer-wrapping logic. This might seem convoluted, but we're trying to create something akin to a meta-class - when Filter(tknzr) is called it must return a *callable* that can then be applied to a particular string to perform the tokenization. Since we need to manage a lot of state during tokenization, returning a class is the best option. tknzrcsdW nW0nWn\4VnRVn^VnR#r>)rRrMrPr=_curtok_curword_curpos)rr;Zskipr7s&&&&r r ZFilter._TokenFilter.__init__s)JK'O)+DLDMDLr crr rrrr rZFilter._TokenFilter.__iter__sKr cr r rrrr rZFilter._TokenFilter.__next__s99; r csp\VP4wrWVP,3# \d}\TP4wrTP TP T44'd\TP4wrK?YnY nTPT4TnKi;irH) rrXrZrBrPrR _to_stringrYrM)rrUposs& r rZFilter._TokenFilter.nexts 5"&t||"4KT  "455$5"&t"7KT**T__T%:;;&*4??&; s$(M#&L#';;t#4DL 5s*.AB56;B54B5cs\V4\PJdCVPR8XdVP4#VPR8XdVP 4#V#)ZuZc)ZtypearrayZtypecodeZ tounicodeZtostringrTrr r[ZFilter._TokenFilter._to_stringsHDzU[[(==C'>>++]]c)==?*Kr cs.VPP#r )rPrrrr rZFilter._TokenFilter._get_offsets??)) )r crrr r$r&r r'ZFilter._TokenFilter._set_offsets+/  MM#(:q I OOF #r csVPPpVPPWR7RpWP, pW18dRpV^8dRpV\ VP 48dRpV'd&V'gVP PV4R#\4VnRVn^VnR#))rTFr?N)rPrrrZr9rYrXr=)rZvalrZ old_offsetZ keep_curtokZ curtok_offsets&&& r rZFilter._TokenFilter.set_offsets//J OO & &s & >K,,.M# q # DMM 22# 8 '' 6-/ "   r )rZrXrYrRrMrPNr()r)r*r+r,r-r.r rrrr[rr'r/rrr0r1r2r4r rQrVsN i     5  * $+{3 ! !r rQrON) r)r*r+r,r-r rSrRrMrQr0r1r2r4r r5r5]s* $?#R!R!r r5cTa]tRtRtoRtR.t]P!R4tRt Rt Vt R#)rizFilter skipping over URLs. This filter skips any words matching the following regular expression: ^[a-zA-Z]+:\/\/[^\s].* That is, any words that are URLs. zAz^[a-zA-Z]+:\/\/[^\s].*cLVPPV4'dR#R#TF_patternZmatchrTrr rRZURLFilter._skip ==  t $ $r rN r)r*r+r,r-r.recompilercrRr0r1r2r4r rrs,&Kzz34Hr cNa]tRtRtoRt]P !R4tRtRt Vt R#)rizFilter skipping over WikiWords. This filter skips any words matching the following regular expression: ^([A-Z]\w+[A-Z]+\w+) That is, any words that are WikiWords. z^([A-Z]\w+[A-Z]+\w+)cr`rarbrTrr rRZWikiWordFilter._skiprdr rN r)r*r+r,r-rfrgrcrRr0r1r2r4r rrs%zz12Hr crh) EmailFilterizFilter skipping over email addresses. This filter skips any words matching the following regular expression: ^.+@[^\.].*\.[a-z]{2,}$ That is, any words that resemble email addresses. z^.+@[^\.].*\.[a-z]{2,}$cr`rarbrTrr rRZEmailFilter._skip rdr rNrir2r4r rjrjs%zz45Hr rjcr^) MentionFilterizFilter skipping over @mention. This filter skips any words matching the following regular expression: (\A|\s)@(\w+) That is, any words that are @mention. r_z (\A|\s)@(\w+)cr`rarbrTrr rRZMentionFilter._skiprdr rNrer2r4r rkrk,&Kzz*+Hr rkcr^) HashtagFilteri!zFilter skipping over #hashtag. This filter skips any words matching the following regular expression: (\A|\s)#(\w+) That is, any words that are #hashtag. r_z (\A|\s)#(\w+)cr`rarbrTrr rRZHashtagFilter._skip,rdr rNrer2r4r rmrm!rlr rmcs0a]tRtRtoRtRtRtRtVtR#) HTMLChunkeri2zChunker for breaking up HTML documents into chunks of checkable text. The operation of this chunker is very simple - anything between a "<" and a ">" will be ignored. Later versions may improve the algorithm slightly. csVPpVPpV\V48d \ 4hW,R8Xd_TpVPW4'd=W,R8wd&V^, pV\V48XgK(V^,pMV^, pM V^,pTpV\V48dW,R8wd V^, pK(W nWB8gKWVV3#)TZ)rrr9_is_tagrrB)rrrZ maybe_tagrKrLr rZHTMLChunker.next:szzT",o)|s"" <<--,#-! !SY.%.]F!! &]FE3t9$)<! !L~6*E22r csV^,\V48d<W^,,P4'dR#W^,,R8XdR#R#)iTZ/F)r9Zisalpha)rrrrr roZHTMLChunker._is_tagWs@ A:D !QJ''))QJ3&r rN) r)r*r+r,r-rror0r1r2r4r rnrn2s:r rn) r\r\rWrrtknsrWr\rp)NNN)tkrqrqrq)r-r.rfr!r]Zenchant.errorsrZErrorrr<r=rFr8r6r:rNr5rrrjrkrmrnrr r rrs