14 commits

Author	SHA1	Message	Date
bab2min	7e0ea2097a	updated test cases	2023-08-12 14:07:52 +09:00
bab2min	73a4b1032c	added tokenizer samples	2023-05-07 16:02:29 +09:00
bab2min	4ad16daf5c	added offset arg to encode (from morphs)	2023-04-24 02:44:36 +09:00
bab2min	9d4fafc170	improved ignoreErrors behaviors	2023-04-21 01:56:54 +09:00
bab2min	1aa3ab9ae0	added ignoreErrors arg to SwTokenizer::decode	2023-04-21 01:41:40 +09:00
bab2min	e7abdc9e1d	implemented newline token for SwTokenizer	2023-04-16 15:00:24 +09:00
bab2min	f41a397f1a	fixed for gcc 4.8	2023-04-15 21:28:23 +09:00
bab2min	f6146476f3	fixed compilation errors for old compilers	2023-04-15 21:07:15 +09:00
bab2min	8d17ab2b5a	implemented fallbackHangul, fallbackByte	2023-04-06 02:10:53 +09:00
bab2min	1b4d265622	fixed wrong offset generation	2023-03-31 01:26:07 +09:00
bab2min	a16ef49b2f	added test case for offsets	2023-03-30 01:53:22 +09:00
bab2min	7d5347aa55	fixed compilation errors at gcc 4.8	2023-03-18 23:44:18 +09:00
bab2min	80667bb6f0	changed test cases for SwTokenizer	2023-03-17 01:05:18 +09:00
bab2min	55c8a8babd	added test cases for SwTokenizer	2023-03-16 01:56:28 +09:00