A Compression-Based Toolkit for Modelling and Processing Natural Language Text
Research output: Contribution to journal › Article › peer-review
Electronic versions
Documents
- 2018-A Compression-Based Toolkit for Modelling and Processing Natural Language Text
Final published version, 689 KB, PDF document
Licence: CC BY Show licence
DOI
A novel compression-based toolkit for modelling and processing natural language text is described. The design of the toolkit adopts an encoding perspective—applications are considered to be problems in searching for the best encoding of different transformations of the source text into the target text. This paper describes a two phase ‘noiseless channel model’ architecture that underpins the toolkit which models the text processing as a lossless communication down a noise-free channel. The transformation and encoding that is performed in the first phase must be both lossless and reversible. The role of the verification and decoding second phase is to verify the correctness of the communication of the target text that is produced by the application. This paper argues that this encoding approach has several advantages over the decoding approach of the standard noisy channel model. The concepts abstracted by the toolkit’s design are explained together with details of the library calls. The pseudo-code for a number of algorithms is also described for the applications that the toolkit implements including encoding, decoding, classification, training (model building), parallel sentence alignment, word segmentation and language segmentation. Some experimental results, implementation details, memory usage and execution speeds are also discussed for these applications.
Original language | English |
---|---|
Article number | 9, 294 |
Pages (from-to) | 1-29 |
Number of pages | 29 |
Journal | Information |
Volume | 9 |
Issue number | 294 |
DOIs | |
Publication status | Published - 22 Nov 2018 |
Research outputs (1)
- Published
Visualisation Data Modelling Graphics (VDMG) at Bangor
Research output: Contribution to conference › Paper › peer-review
Total downloads
No data available