Precautions for Compression Data
Description
When compression data is set in the console, the data containing compression data is automatically compressed once inserted or modified.
During the compression, the data is converted to a string such as List or Dictionary to have its byte size is calculated, and large data types are reduced to approximately 30-40% of their sizes.
However, please be mindful of the following precautions:
- The compression amount may be reduced for data containing a multitude of small-sized or inconsistent data.
- Tables with compression data will be unmarshalled when loaded. Columns such as
["S"],["N"], and `["L”] that represent data types are deleted. - If a table that contains compression data is included upon using transaction, all return values are unmarshalled and returned.
- As such, if you wish to use compression data, use GetFlattenJSON() or FlattenRows() to parse JSON for methods related to loading game data.
What is unmarshalling?
Because separate data types don't exist in the server, the data types are represented with the initials such as "S", "N", and "L" in front of the data.
For example, inDate of the default data below is a string type, and as such, its data type has been marked with "S".
To load that data, you must access it via method such as json["Responses"][0]["inDate"]["S"].ToString().
However, if inDate is not being changed to number type or list type in the client, the value is always changed to string. In this case, "S" which represents the data type is not needed.
Unmarshalling is a function that removes such data type.
When unmarshalling takes place, the default data is changed to unmarshalled data like the following.
Additionally, you can access inDate's value more conveniently using methods such as json["Responses"][0]["inDate"].ToString().
When a compressed table is called, it is unmarshalled and returned. If the compressed table does not exist, the value with a data type is returned.
To reduce that post-processing effort that follows such return cases, you can use GetFlattenJson() or FlattenRows() which is returned unmarshalled for more convenience.
Default data
{
"Responses": [
{
"inDate": {
"S": "2023-07-21T10:33:34.160Z"
}
}
]
}
Unmarshalled data
{
"Responses": [
{
"inDate": "2023-07-21T10:33:34.160Z"
}
]
}
1. Compression amount
BACKND's compression works by replacing the strings the data to be transferred to the server share into one character and reducing the length of the data.
If most of the data share the same characters, then the compression rate is increased.
However, if the size of such data is small, the compression rate is reduced, as there are not many characters that can be replaced.
Examples of data where one can expect a large compression rate are as follows:
- List, Dictionary
Data with 200 characters/bytes or more
[12345, 67890, 23456, 78901, 34567, 89012, 45678, 90123, 56789, 01234, 67890, 23456, 78901, 34567, 89012, 45678, 90123, 56789, 01234, 67890, 23456, 78901, 34567, 89012, 45678, 90123, 56789, 01234, 67890, 23456]
If the data has much in common
[1,1,1,1,1,1,1,0,0,0,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]
[
{"jewel_pack": 872},
{"diamond_box": 321},
{"gold_bar": 582},
{"mystery_box": 250},
{"rare_gem": 749},
{"epic_boost": 527},
{"health_potion": 302},
{"magic_scroll": 429},
{"armor_upgrade": 691},
{"sword_polish": 174},
{"shield_enhancer": 463},
{"rare_seed": 844},
{"legendary_key": 516},
{"mystic_feather": 287},
{"sacred_totem": 668},
{"power_orb": 253},
{"moon_pearl": 796},
{"sun_crystal": 369},
{"star_fragment": 143},
{"void_stone": 621},
{"elixir_life": 395}
]
The following is a list of data cases that are not suited to for data compression.
Small-sized data (Less than 200 bytes)
[1,0,0,0,0,0,0,1,1,2,0]
24133232412325234002321230000235123421
{
"username": "Player123",
"level": 25,
"class": "warrior",
"guild": "DragonSlayers",
"last_active": "2023-07-25"
}
2. Unmarshalling JSON Parsing
As a solution to the data return method changing depending on the compression status when using compression data, unmarshalling all data is a measure you can take.
Example 1. Loading tables without data types and compression data provided by default
{
"Responses": [
{
"inDate": {
"S": "2023-07-21T10:33:34.160Z"
},
"updatedAt": {
"S": "2023-07-21T10:33:34.161Z"
},
"owner_inDate": {
"S": "2023-07-20T10:34:14.843Z"
},
"list": {
"L": [
{
"S": "SJE2MDUFZHCH"
},
{
"S": "ARM27CDGLPFT"
},
{
"S": "PWR44N90ZDAB"
}
]
},
"key": {
"M": {
"key1": {
"S": "8EWRPEASL4R5"
},
"key2": {
"S": "C145FSBDDVMZ"
},
"key5": {
"S": "TEUGC77N164Z"
},
"key3": {
"S": "WZLMIMS7QK33"
},
"key4": {
"S": "QV1GOXHVFV2D"
}
}
}
}
]
}
The following details the ways to access them:
Normal JSON
string data = bro.GetReturnValuetoJSON()["Responses"][0]["list"]["L"][0]["S"].ToSTring();
Debug.Log("List's first array value : :" + data);
Unmarshalled JSON
string data = bro.GetFlattenJson()["Responses"][0]["list"][0].ToSTring();
Debug.Log("List's first array value : :" + data);
Example 2. Loading tables with compression data
{
"Responses": [
{
"inDate": "2023-07-21T08:55:52.972Z",
"updatedAt": "2023-07-21T08:55:52.975Z",
"owner_inDate": "2023-07-20T10:33:26.32Z",
"map": {
"key1": "VqHgm87lMAvLZd9GgraW",
"key2": "fNc8KFnZ8m0GlXYziAzt",
"key5": "PGKKeeG4h6Yjqy7kwpf9",
"key3": "a8xPJv5W372T5hXF0I4O",
"key4": "yto9H0p3s1r1yJxQKWFo"
},
"list": [
"VqHgm87lMAvLZd9GgraW",
"fNc8KFnZ8m0GlXYziAzt",
"yto9H0p3s1r1yJxQKWFo",
"uGZVN6rAgsUrZRNbEAEg",
"PGKKeeG4h6Yjqy7kwpf9",
"a8xPJv5W372T5hXF0I4O",
"J14wpMYHPbcrTXRA5fIk",
"rSs0OC4NVdJbfdZxZWLb",
"X5tBLDnQgg91PabRLNnm",
]
}
]
}
Normal JSON
string data = bro.GetReturnValuetoJSON()["Responses"][0]["list"][0].ToSTring();
Debug.Log("List's first array value : :" + data);
Unmarshalled JSON
string data = bro.GetFlattenJson()["Responses"][0]["list"][0].ToSTring();
Debug.Log("List's first array value : :" + data);
By accessing tables using the unmarshalled data shown in the examples, you can access them using the same logic regardless of their data type.