A small yet powerful data format ✨
Cain is a new data interchange format which aims at providing the smallest possible size to encode data.
It is based on pre-defined schemas which leverages the need to specify it within the final encoded data.
Note
Look at the SPECIFICATIONS file for more information on the purpose and idea behind this project.
For example, we consider the following object:
{
"b": 3,
"c": 5.5,
"d": True,
"e": {
"f": False,
# "g": b"Hello world"
"h": "HELLO WORLD",
"i": "Hi!",
"j": [1, 2, 3, 1, 1],
"k": (1, "hello", True),
"l": None,
"m": "Yay",
"n": "Hi",
"o": 2,
"p": None
}
}This is the expected result from a minified JSON encoding:
{"b":3,"c":5.5,"d":true,"e":{"f":false,"h":"HELLO WORLD","i":"Hi!","j":[1,2,3,1,1],"k":[1,"hello",true],"l":null,"m":"Yay","n":"Hi","o":2,"p":null}}This is the expected result from the Cain data format:
\x00\x00\x03\x00\x00\xb0@\x01\x00\x00HELLO WORLD\x00Hi!\x00\x00\x05\x00\x00\x00\x01\x00\x02\x00\x03\x00\x01\x00\x01\x00\x00\x01hello\x00\x01\x00\x01\x00Yay\x00\x00Hi\x00\x01\x00\x02
Note
This is 56.76% smaller than the JSON version ✨
Moreover, objects which can't be encoded using JSON (bytes, set, range, etc.) or wrongly encoded using JSON (ex: tuple) are working out of the box with Cain!
These instructions will get you a copy of the project up and running on your local machine for development and testing purposes. See deployment for notes on how to deploy the project on a live system.
You will need Python 3 to use this module
Minimum required versions: 3.9
Incompatible versions: 2Always check if your Python version works with cain before using it in production.
pip install --upgrade cainThis will install the latest version from PyPI
pip install --upgrade git+https://github.com/Animenosekai/cain.gitThis will install the latest development version from the git repository
You can check if you successfully installed it by printing out its version:
$ cain --version
1.1The main entry point (cain.py) provides an API familiar to users of the standard library json module. The different datatype also present a very pythonic way of handling data to keep a nice and clean codebase.
Encoding basic Python object hierarchies:
>>> import cain
>>> from cain.types import Object, Optional
>>> cain.dumps({"a": 2}, Object[{"a": int}])
b'\x00\x00\x02'
>>> class TestObject(Object):
... bar: tuple[str, Optional[str], float, int]
...
>>> cain.dumps(['foo', {'bar': ('baz', None, 1.0, 2)}], list[str, TestObject])
b'\x00foo\x00\x00\x00baz\x00\x00\x00\x00\x80?\x00\x02'
>>> print(cain.dumps("\"foo\bar", str))
b'"foo\x08ar\x00'
>>> print(cain.dumps('\u1234', str))
b'\xe1\x88\xb4\x00'
>>> print(cain.dumps('\\', str))
b'\\\x00'
>>> schema = list[str, Object[{"bar": tuple[str, Optional[str], float, int]}]]
>>> with open('test.cain', 'w+b') as fp:
... cain.dump(['foo', {'bar': ('baz', None, 1.0, 2)}], fp, schema)
...
>>> from cain.types import Int
>>> from cain.types.numbers import unsigned
>>> Int[unsigned].encode(4)
b'\x00\x04'You can also add a header using the include_header parameter to add a header containing the schema for the encoding data. This gives a more portable output but increases its size.
Decoding Cain:
>>> import cain
>>> from cain.types import Optional, Object
>>> schema = list[str, Object[{"bar": tuple[str, Optional[str], float, int]}]]
>>> cain.loads(b'\x00foo\x00\x00\x00baz\x00\x00\x00\x00\x80?\x00\x02', schema)
['foo', {'bar': ('baz', None, 1.0, 2)}]
>>> with open('test.cain', 'r+b') as fp:
... cain.load(fp, schema)
...
['foo', {'bar': ('baz', None, 1.0, 2)}]
>>> from cain.types import Int
>>> from cain.types.numbers import unsigned
>>> Int[unsigned].decode(b'\x00\x04')
4If you want to dynamically encode/decode data with the Cain format, it is also possible to encode/decode the schema.
This is especially useful when developing a public API for example.
>>> import cain
>>> from cain.types import Object, Optional
>>> cain.encode_schema(Object[{"a": int}])
b'\x00\x00\x01\x00\x00a\x00\x00\x01\x00\x00\x01\x03\x00\x01\x02\x00\x00\x00\x00\x06\x00\x00\x00\x00\x16'
>>> class TestObject(Object):
... bar: tuple[str, Optional[str], float, int]
...
>>> cain.encode_schema(list[str, TestObject])
b'\x01\x02\x00\x01\x00\x00\x00\x00\x00\x02\x00\x00...\x00\x16\x01\x00TestObject\x00\x00\x00'>>> import cain
>>> cain.decode_schema(b'\x00\x00\x01\x00\x00a\x00\x00\x01\x00\x00\x01\x03\x00\x01\x02\x00\x00\x00\x00\x06\x00\x00\x00\x00\x00\x16\x00')
Object<{'a': Int}>
>>> cain.decode_schema(b'\x01\x02\x00\x01\x00\x00\x00\x00\x00\x02\x00\x00...\x00\x16\x01\x00TestObject\x00\x00\x00')
Array[String, TestObject]You can also create your own encoders:
>>> import typing
>>> from cain.model import Datatype
>>> class MyObject(Datatype):
... @classmethod # *args contains the args passed here : MyObject[args]
... def _encode(cls, value: typing.Any,*args) -> bytes:
... ... # your custom encoding
... return b'encoded data'
... #
... @classmethod
... def _decode(cls, value: bytes, *args) -> typing.Tuple[typing.Any, bytes]:
... ... # `value` contains more than just the value you should decode
... ... # try to only decode the first few bytes
... ... # your custom decoding
... return 'decoded data', value # the rest of the value that you didn't decode
... # you can now use `MyObject` in your schemas and encode/decode from itWarning
Keep in mind that custom datatypes outside of subclasses ofObjectwon't be able to be encoded by the Type encoder (used in schema headers for example)
Cain has a pretty complete command-line interface, which lets you manipulate and interact with the Cain data format easily.
For more information, head over to your console and enter:
cain --helpOr
cain <action> --helpExample usage of the CLI
Preparing the schema:
# test.py
from cain import Object
class Test(Object):
username: str
favorite_number: intTrying to encode with a Python schema:
cain encode '{"username": "Anise", "favorite_number": 2}' --schema="test.py" --schema-name="Test" --include-header --output="test.cain"Trying to decode the previous file:
$ cain decode test.cain
{
"favorite_number": 2,
"username": "Anise"
}Looking up at its schema:
$ cain schema lookup test.cain --schema-header
{
"index": 22,
"name": "Test",
"annotations_keys": [
"username",
"favorite_number"
],
"annotations_values": [
{
"index": 26,
"name": null,
"annotations_keys": [],
"annotations_values": [],
"arguments": [],
"datatype": "String"
},
{
"index": 6,
"name": null,
"annotations_keys": [],
"annotations_values": [],
"arguments": [],
"datatype": "Int"
}
],
"arguments": [],
"datatype": "Object"
}Exporting its schema:
cain schema export test.cain --schema-header --output test.cainschemaTrying to encode another object with the exported schema:
$ cain encode '{"username": "yay", "favorite_number": 3}' --schema=test.cainschema
\x00\x00\x03yay\x00Encoding "Hello world":
$ cain encode '"Hello world"' --schema="str" --schema-eval
Hello world\x00
$ cain encode '["Hello", "world"]' --schema="list[str]" --schema-eval
\x00\x02\x00\x00Hello\x00world\x00This module is currently in development and might contain bugs.
This comes with a few disadvantages (for example, it takes a longer time to encode objects with Cain than with the standard json module) but this is expected to improve over time.
Please verify and test the module thoroughly before releasing anything at a production stage.
Feel free to report any issue you might encounter on Cain's GitHub page.
Pull requests are welcome. For major changes, please open a discussion first to discuss what you would like to change.
Please make sure to update the tests accordingly.
- Animenosekai - Initial work - Animenosekai
This software is licensed under the MIT License. See the LICENSE file for more information.