This article describes the various user data types in Collection Pro and their behavior during import and export situations.<\/p>\n\n
Text types are used to store text. Collection Pro supports Text is stored as received, so a leading or trailing space is preserved. Frontends are requested to trim texts before sending them to the API. <\/p>\n\n\n\n Text must be encoded in UTF-8<\/a> and stores in all normalization forms<\/a>. There is no limit to the length of the text that can be stored. <\/p>\n\n\n\n Text looks like this when send and received over the API:<\/p>\n\n\n\n The above example has the value this is the text<\/em> for column text.<\/p>\n\n\n\n Normalization is performed as part of the indexer documentation creation, where all text is run through a In the indexer, text is stored using a custom analyzer What gets included in tokens:<\/p>\n\n\n\n What causes token separation:<\/p>\n\n\n\n Tokens are then turned into terms using the Using the API, searches for text can be performed either on the analyzed value (matching the terms), or on the unanalyzed value, which is stored alongside with all terms. The unanalyzed value stores the data as is.<\/em> There is no normalization taking place. <\/p>\n\n\n\n All text for indexed documents is split into chunks of, 8000 UTF-8 characters. When matching full texts in analyzed form, text cannot easily be matched if they exceed 8000 characters. <\/p>\n\n\n\n Sort strings are compiled using the Go library collate<\/a>. It uses the first configured database language as assumption in what language the text is in. Numbers are recognized so that Car 100<\/em> sorts after Car 11<\/em>. Text is normalized by the collate library. Internally, we use the hex representation of that string to work around anomalies in Elasticsearch. Some special replacement is always done [TODO LINK]. <\/p>\n\n\n\n Text is exported as is, keeping spaces & normalization.<\/p>\n\n\n\n The XML export<\/strong> looks like this for a column names For type, text type is Output in CSV<\/strong> is as is, same for JSON<\/strong>.<\/p>\n\n\n\n String<\/p>\n\n\n\n The String looks like this when send and received over the API:<\/p>\n\n\n\n The above example has the value $5667<\/em> for column String values are normalized and lowercased for the index document.<\/p>\n\n\n\n All strings for indexed documents are split into chunks of, 8000 UTF-8 characters. When matching full texts in analyzed form, text cannot easily be matched if they exceed 8000 characters. <\/p>\n\n\n\n The sorting of string values works like for The XML<\/strong> looks like for The CSV<\/strong> and JSON<\/strong> export the string as is<\/em>.<\/p>\n\n\n\n The types Localized text looks like this send and received over the API.<\/p>\n\n\n\n The above example set the Finnish value Finnish <\/em>and English value English <\/em>for column Indexing is done the same way Sorting is performed using a collate string produced by the Go library collate<\/a>. The language is parsed as BCP 47<\/a> string and passed to the library. Some special replacement is done: <\/p>\n\n\n\n XML exported data<\/strong> looks like this:<\/p>\n\n\n\n In CSV, the values<\/strong> are exported like this:<\/p>\n\n\n\ntext<\/code>, text_oneline<\/code>, , string<\/code>text_l10n <\/code>and text_l10n_oneline<\/code>. <\/p>\n\n\n\ntext, text_oneline<\/h3>\n\n\n\n
text<\/mark><\/code> and text_oneline<\/mark><\/code> store characters in one language. In the server, these types are treated the same, text_oneline is only a hint for frontends on how to display the data. So it is possible (over the API) to use newlines in a text_online column. <\/p>\n\n\n\nAPI<\/h4>\n\n\n\n
{\n \"text\": \"this is the text\"\n}<\/code><\/pre>\n\n\n\nIndex<\/h4>\n\n\n\n
icu_normalizer<\/code>.<\/p>\n\n\n\nicu_text<\/code> which works as follows:<\/p>\n\n\n\n{\n \"analysis\": {\n \"icu_text\": {\n \"type\": \"custom\",\n \"tokenizer\": \"custom_icu_tokenizer\",\n \"filter\": [\n \"icu_folding\"\n ],\n \"char_filter\": [\n \"icu_normalizer\"\n ]\n },\n \"tokenizer\": {\n \"custom_icu_tokenizer\": {\n \"type\": \"pattern\",\n \"pattern\": \"([[^\\\\p{L}\\\\p{Digit}\\\\uD800-\\\\uDFFF\\\\u2F00-\\\\u2FDF]&&[^&%\u00a7\\\\$\u20ac]])\"\n }\n }\n}<\/code><\/pre>\n\n\n\nicu_normalizer<\/code><\/a> and tokenized into tokens using the above pattern.<\/p>\n\n\n\n\n
&<\/code>, %<\/code>, \u00a7<\/code>, $<\/code>, and \u20ac<\/code>.<\/li>\n<\/ul>\n\n\n\n\n
icu_folding<\/code><\/a> token filter. The filter removes all punctuation and turns all characters into lower case. So the token B\u00e4r<\/em> is stored as bar<\/em>. <\/p>\n\n\n\nSorting<\/h4>\n\n\n\n
Export<\/h4>\n\n\n\n
title<\/mark><\/code> and a value Title<\/em> for the type text_oneline<\/mark><\/code>. The column-api-id<\/code> in this example 29<\/em>. <\/p>\n\n\n\n<title type=\"text_oneline\" column-api-id=\"29\">Title<\/title><\/code><\/pre>\n\n\n\ntext<\/mark><\/code>.<\/p>\n\n\n\n
\n\n\n\nstring<\/mark><\/code> type’s main difference to the text type is how it’s indexed. It is recommended to use string types for identification strings which may contain special characters which would be dropped by the analyzer. <\/p>\n\n\n\nAPI<\/h4>\n\n\n\n
{\n \"ref\": \"A$5667\"\n}<\/code><\/pre>\n\n\n\nref<\/mark><\/code>.<\/p>\n\n\n\nIndex<\/h4>\n\n\n\n
{\n \"analyzer\": {\n \"keyword_lowercase\": {\n \"tokenizer\": \"keyword\",\n \"filter\": [\n \"icu_folding\"\n ],\n \"char_filter\": [\n \"icu_normalizer\"\n ]\n }\n}<\/code><\/pre>\n\n\n\nicu_normalizer<\/code><\/a> and converted to lower case using the icu_folding<\/code><\/a> token filter.<\/p>\n\n\n\nSorting<\/h4>\n\n\n\n
text<\/mark><\/code>. In addition to the text sorting, a pure alphanumerical version is stored in the index alongside with the numerically sortable variant. With that, sorting can sort Car 10<\/em>, Car 11<\/em>, Car 12<\/em>, Car 100<\/em>. Some special replacement is always done. <\/p>\n\n\n\nExport<\/h4>\n\n\n\n
text<\/mark><\/code>.<\/p>\n\n\n\n<ref type=\"string\" column-api-id=\"346\">hall\/7$<\/ref><\/code><\/pre>\n\n\n\nref<\/code> is exported using value hall\/7$<\/em>.<\/p>\n\n\n\n
\n\n\n\ntext_l10n, text_l10n_oneline<\/h3>\n\n\n\n
text_l10n<\/mark><\/code> and text_l10n_oneline<\/mark><\/code> are designed to store localized values. The format is a JSON object consisting of the language as key and the text as value. <\/p>\n\n\n\nAPI<\/h4>\n\n\n\n
{\n \"title_loca\": {\n \"fi-FI\": \"Finnish\",\n \"en-US\": \"English\"\n }\n}<\/code><\/pre>\n\n\n\ntitle_loca<\/mark><\/code>.Index<\/h4>\n\n\n\n
text<\/code> format is indexed. Only enabled database languages are mapped into the index, other languages are ignored. After changing the settings for the database languages, a reindex is required. <\/p>\n\n\n\nSorting<\/h4>\n\n\n\n
'\u02be': '@', \/\/ 02BE Hamza (vorne offen) @ sorts this before A\n'\u02bf': '@', \/\/ 02BF Ayn (hinten offen) @ sorts this before A<\/code><\/pre>\n\n\n\nExport<\/h4>\n\n\n\n
<title_loca type=\"text_l10n\" column-api-id=\"81\">\n <fi-Fi>Finnish<\/fi-FI>\n <en-US>English<\/en-US>\n<\/title_loca><\/code><\/pre>\n\n\n\ntitle_loca<\/mark><\/code> with the type text_l10n<\/code><\/mark>.<\/p>\n\n\n\n