mirrors/Kiwi
2
0
Fork 0
mirror of https://github.com/bab2min/Kiwi.git synced 2026-06-17 01:54:27 +00:00
Code Issues Projects Releases Packages Wiki Activity
1,422 commits 9 branches 53 tags 1.4 GiB
Commit graph

318 commits

Author SHA1 Message Date
copilot-swe-agent[bot]
4961562a30 Fix spacing/formatting issues in documentation comments 
Co-authored-by: bab2min <19266222+bab2min@users.noreply.github.com>
 2026-01-15 06:14:29 +00:00
copilot-swe-agent[bot]
63c346e69a Add Doxygen documentation to PatternMatcher, TagUtils, and Mmap 
Co-authored-by: bab2min <19266222+bab2min@users.noreply.github.com>
 2026-01-15 06:10:54 +00:00
copilot-swe-agent[bot]
6c80208796 Add Doxygen documentation to SkipBigramModel, CoNgramModel, ThreadPool, and Utils 
Co-authored-by: bab2min <19266222+bab2min@users.noreply.github.com>
 2026-01-15 06:08:58 +00:00
copilot-swe-agent[bot]
3b4bc20d54 Add Doxygen documentation to LangModel, Joiner, Knlm, WordDetector, and FrozenTrie 
Co-authored-by: bab2min <19266222+bab2min@users.noreply.github.com>
 2026-01-15 06:05:47 +00:00
bab2min
e5ede7bdca bump to 0.22.2 2025-12-15 01:33:08 +09:00
bab2min
8278588401 Add function to retrieve morpheme form in UTF-16 encoding 2025-11-23 14:33:06 +09:00
bab2min
a993afa502 Bump to v0.22.1 2025-11-22 00:37:26 +09:00
bab2min
40c607e71e Remove unnecessary comment from C API 2025-11-21 22:05:26 +09:00
bab2min
c8ef90e801 Implement missing C APIs 2025-11-21 21:45:49 +09:00
bab2min
59ba9ed4c4 Bump to 0.22.0 2025-11-09 15:37:48 +09:00
bab2min
b22c79b042 Update C API 2025-11-09 15:31:02 +09:00
bab2min
99621fb7a1 Make member variables for config in Kiwi independent 2025-11-09 15:27:27 +09:00
bab2min
6db21fc61c Make normalizeHangul function non-inline 2025-11-06 01:10:45 +09:00
bab2min
7072b368d0 Improve transform of HSDataset to accept multiple replacements 2025-10-19 17:33:32 +09:00
bab2min
083cd69830 Implement additional feature for combining 하/VV on contracted eomi 2025-10-19 02:03:23 +09:00
bab2min
568541d49e Update C-API 2025-10-16 02:08:25 +09:00
bab2min
40f36db8d4 Add SerializationException class and update makeFilesystemProvider to return nullptr on failure 2025-10-15 21:57:52 +09:00
bab2min
9ef714e8a2 Implement emoji & SB augmentor to HSDataset 2025-10-12 02:43:31 +09:00
bab2min
990490a632 Add arg enabledDialects to constructor of KiwiBuilder 2025-10-12 02:37:57 +09:00
bab2min
7ff7b24ae8 Fix wrong positionOut result of joinHangul 2025-10-11 22:12:08 +09:00
bab2min
5c70752fe9 Add rule line number tracking for combiningRule.txt 2025-09-16 21:37:41 +09:00
bab2min
98d90303ea Add morpheme remapping to CoNg builder 2025-09-16 21:36:38 +09:00
bab2min
03e268f8e9 Implement typo correction for dialects 2025-09-16 21:36:36 +09:00
bab2min
c9b3394822 Rename "old" dialect to "archaic" in Dialect enumeration and update related functions 2025-09-16 21:36:31 +09:00
bab2min
61680091d7 Add support for "old" dialect in Dialect enumeration and update dialect recognition 2025-09-16 21:36:30 +09:00
bab2min
ea98019490 Make normalizeHangul & joinHangul compatible to more old Korean characters 2025-09-16 21:36:28 +09:00
bab2min
812b5b05a6 Add senseId supports to KiwiBuilder::loadDictionary 2025-09-16 21:36:27 +09:00
bab2min
fc4c0713c3 Add dialect & senseId arg to KiwiBuilder::addWord & Improve format of KiwiBuilder::loadDictionary 2025-09-16 21:36:23 +09:00
bab2min
543ab65a5f Add dialect flag to combined morphemes 2025-09-16 21:34:05 +09:00
bab2min
ce4fa07320 Add dialectCost option 2025-09-16 21:34:04 +09:00
bab2min
fa1818644b Add allowedDialect to splitByTrie 2025-09-16 21:34:03 +09:00
bab2min
b4dc1ccf97 Add a new type and field for Dialect 2025-09-16 21:33:48 +09:00
bab2min
747c540616 Fix mimalloc errors on Linux 2025-09-15 22:59:42 +09:00
bab2min
a933d4215a Refactor getModelType function to use StreamProvider and update buffer size constant in C API 2025-09-13 01:51:17 +09:00
copilot-swe-agent[bot]
47bd1f7ddc Refactor C API to use value semantics for stream objects 
- Changed kiwi_builder_init_stream() to return kiwi_stream_object_t by value instead of pointer
- Eliminates memory management ambiguity and potential leaks
- Simpler and safer API following standard C callback patterns
- Updated CStreamAdapter to work with value semantics
- Added stddef.h include for C compatibility
- Updated documentation with corrected examples
- Backward incompatible but safer API design

Co-authored-by: bab2min <19266222+bab2min@users.noreply.github.com>
 2025-09-13 01:51:16 +09:00
copilot-swe-agent[bot]
35ced5e7b8 Remove unused filename parameter from kiwi_builder_init_stream C API function 
Co-authored-by: bab2min <19266222+bab2min@users.noreply.github.com>
 2025-09-13 01:51:15 +09:00
copilot-swe-agent[bot]
ce8a3ac1ba Refactor C API to use generic stream object interface instead of chunked stream 
- Replace kiwi_stream_provider_t and kiwi_chunked_stream_provider_t with kiwi_stream_object_t
- Add kiwi_stream_read_func, kiwi_stream_seek_func, kiwi_stream_close_func callbacks
- Implement CStreamAdapter class for proper C++ istream integration
- Remove kiwi_builder_init_chunked_stream function - unified API with kiwi_builder_init_stream
- Update function signature to accept stream_object_factory and filename parameters
- Update documentation in StreamProvider.md with new API usage examples
- Maintain backward compatibility through modern stream-like interface

Co-authored-by: bab2min <19266222+bab2min@users.noreply.github.com>
 2025-09-13 01:51:15 +09:00
copilot-swe-agent[bot]
07c8eb10fb Address PR feedback: remove jobject specialization, unify ctor names, merge WASM build functions, add chunked stream API 
Co-authored-by: bab2min <19266222+bab2min@users.noreply.github.com>
 2025-09-13 01:51:14 +09:00
copilot-swe-agent[bot]
0c42704de1 Add StreamProvider support to C API, Java, and WASM bindings 
Co-authored-by: bab2min <19266222+bab2min@users.noreply.github.com>
 2025-09-13 01:51:12 +09:00
copilot-swe-agent[bot]
48743aa2ff Address PR feedback: add utility function, throw exception for missing combiningRule.txt, support StreamProvider in WordDetector, simplify loadDictionary 
Co-authored-by: bab2min <19266222+bab2min@users.noreply.github.com>
 2025-09-13 01:51:12 +09:00
copilot-swe-agent[bot]
299ca22a73 Complete StreamProvider implementation with tests and documentation 
Co-authored-by: bab2min <19266222+bab2min@users.noreply.github.com>
 2025-09-13 01:51:11 +09:00
copilot-swe-agent[bot]
6e1d9a26e0 Implement StreamProvider interface and new KiwiBuilder constructor 
Co-authored-by: bab2min <19266222+bab2min@users.noreply.github.com>
 2025-09-13 01:51:10 +09:00
bab2min
0e9b30f90e Update C API & test cases 2025-05-03 15:49:21 +09:00
bab2min
c52f459932 Change effect of numThreads arg 
Before:
* numThreads == 0 : fallback to hardware concurrency
* numThreads == 1 : Single thread, no async
* numThreads > 1 : Multi threads, async supported

After:
* numThreads == -1 : fallback to hardware concurrency
* numThreads == 0 : Single thread, no async
* numThreads == 1 : Single thread, async supported
* numThreads > 1 : Multi threads, async supported
 2025-05-03 15:37:49 +09:00
bab2min
e1aeac6ab3 Fix nan issue of SBG training 2025-05-01 20:28:21 +09:00
bab2min
6b98ca6991 Update utilities 2025-05-01 01:43:50 +09:00
bab2min
705a76a19e Refactor args of Kiwi::analyze to struct AnalyzeOption 2025-05-01 01:42:55 +09:00
bab2min
e88e507417 Update C-API 2025-04-27 20:59:15 +09:00
bab2min
ba84eb6404 Implement predictWordsFromContextDiff, toContextId for CoNgramModel 2025-04-27 20:58:12 +09:00
bab2min
212f9f57a4 Add ModelType::largest and change behavior of none to selecting smallest model 2025-04-27 01:34:10 +09:00