4.1.  Data modeling

Information is stored and used as data in digital systems. Digital systems rely on defined structures to store, retrieve and manipulate data in a consistent manner.

A data model is a structured representation of the data that describes the properties of the data, the relationships between different data elements, and any rules or constraints that govern the data.

Data modeling is the process of creating a data model.

Many data modeling languages exist, each with its own syntax and semantics.

Commonly languages that define data models are called “conceptual modeling languages”.

While many data modeling languages are used to define data models in a way that is independent of underlying technologies used, most data modeling languages do not provide a way to define how data is represented in an information system.

For example, information models in UML are mostly used for diagramming and documentation purposes, and do not provide a way to define how the data can be accessed or used in programming languages.

4.2.  Serialization

Serialization is the process of converting data structures or objects into a format that can be stored or transmitted between systems. Typically, this involves converting the data into a string of bytes that can be written to a file or sent over a network.

Serialization formats are very important in allowing systems to exchange structured information in an interoperable manner without loss of semantics.

Serialization formats often come with a schema language that allows for the definition of particular data structures that data needs to conform to.

Typically, languages that define serialization formats are called “schema languages”.

A conceptual model may be realized (“implemented”) in multiple serialization formats. For example, some standards define a conceptual model in UML and provide manually-defined transformation rules to XML and JSON.

4.3.  Access in programming languages

Information defined by conceptual data models and serialization schema languages are not directly usable in a native manner in programming languages or APIs.

At the conceptual data model level, there is no direct mapping to programming language constructs. For example, a UML class diagram does not directly map to a Java class.

There are tools that attempt to generate code from UML models, yet the generated code will require manual integration and intervention to be usable in software.

At the serialization schema language level, there is again no direct access from the programming language to the schema language constructs. For example, a JSON Schema does not directly map to a Java class. Typically developers will write code that parses the schema and/or implement code to read and write data in the format defined by the schema.

The lack of direct access to data models and serialization formats in programming languages makes it difficult to work with structured data in a consistent manner across different systems and technologies.

4.4.  Interleaving conceptual modeling and serialization format modeling

Conceptual modeling focuses on the definition of the existence and relationships of data elements.

Serialization data modeling focuses on low-level details of how data is represented in a particular format.

For example, a conceptual model may define that a Date object has year, month, day attributes, and each of them are of an Integer type and potentially the numbering constraints for each attribute, and that there are certain restrictions to what the attributes can be at an attribute level (valid values of month range from 1 to 12), or at the object level (only a leap year is allowed to have 29 days in February). It does not specify how many bytes are to be used to represent an Integer or how the Date object is to be written in a file.

A serialization data model (using a schema language) may define that a Date object is represented as a sequence of 3 Integers, each representing the year, the month, and the day, respectively, and that the Integer type is represented as a 4-byte unsigned integer. While the schema language may support value constraints that the month attribute must be between 1 and 12, it is unlikely that the schema language supports object-level complex constraints such as “only a leap year is allowed to have 29 days in February”. Schema languages typically do not have the expressive power to define such complex constraints.

This presents a conundrum for the data modeler, who needs to define the entire structure and constraints, but also needs to decide where and how to define constraints that may not be definable in a schema language. This complexity explodes exponentially as the need for the same data to be represented in additional schema languages arises.

How much of the constraints should be defined in the conceptual model? How much of the data model constraints should be defined in the serialization schema language? Do we need to define duplicated constraints in both the conceptual model and the serialization schema language if allowed? This is a question that haunts developers and data modelers alike.

4.5.  Challenges in data interchange

4.6.  Goals

LutaML is a system that aims to simplify users to easily define data models and work with information that conforms to those data models.

The goal of LutaML is to allow these steps to flow seamlessly:

The LutaML language provides a way to represent structured data in a human-readable format.

6.1.  Primitive values

Primitive values are the basic building blocks of LutaML.

All primitive types have names that start with uppercase.

LutaML supports the following primitive values:

6.2.  Declaration, arguments and blocks

A declaration is a keyword that represents an action or a statement.

Syntax:

{declaration} {*argument} {*block}

Figure 1

Where,

An argument is a string, which can be a primitive value or a reference to another object.

A block is delimited by curly braces and provides a context for inner declarations.

class Studio
// or
class Studio {
}

class Studio {
  attribute ...
}

6.3.  Comment

// {text}
or
{expression} // {text}

// This is a comment

class Studio { // This is a comment
  attribute location, String
}

// TODO: attributes  <1> 
class Ceramic {
  // Write docs  <2> 
  + glazeType: GlazeType // Glaze type  <3> 
}

/* {text} */

/* This is a
multi-line comment */
/*
  This is also a
  multi-line comment
*/

definition {text}

definition "Non-reflective finish"

definition {block}

definition {
  Non-reflective finish.

  A matte finish is a non-reflective finish that is often used in ceramics.
}

class Ceramic {
  definition {
    This class represents a ceramic object.
  }
}

7.1.  General

LutaML supports the following constructs.

7.2.  Package

The package construct is used to define a collection of classes and enums.

Syntax:

package {name} {block}

Figure 6

Where,

package Ceramics {
  enum FiringProfile { values { "low", "medium", "high" } }

  class CeramicTile {
    attribute dimensions, float { cardinality 2 }  // [length, width]
    attribute firing_profile, FiringProfile
  }
}

The package construct can be nested. You can also define other packages within a package, which helps in organizing related classes and enums.

class Dimension {
  attribute length, float
  attribute width, float
  attribute height, float
}

package Ceramics {
  class Ceramic {
    attribute volume, float
    attribute color, String

    // Refers to outermost "root package"
    attribute dimensions, Dimension

    // Refers to "Material" package
    attribute tiles, Material::CeramicTile { cardinality 0..n }
  }

  package Materials {
    enum FiringProfile { values { "low", "medium", "high" } }

    class CeramicTile {
      // Refers to outermost "root package"
      attribute dimensions, Dimension

      // Refers to local "Materials" package
      attribute firing_profile, FiringProfile
    }
  }
}

Within the block, the following declarations are available:

7.3.  Class

The class construct is used to define a class which is a collection of attributes.

Syntax:

class {name} {block}

Figure 7

Where,

class Studio {
  attribute location, String {
    definition "Location of the studio"
  }
  attribute potter, String {
    definition "Name of the potter"
  }
  attribute kiln, String {
    definition "Type of kiln used"
  }
}

class TemperatureWithUnit {
  attribute value, Float {
    definition "Temperature value"
  }
  attribute unit, String {
    definition "Unit of temperature"
  }
}

Class attributes can be restricted to a specific set of values using the values declaration.

class GlazeTechnique {
  attribute name, String {
    values { "Celadon", "Raku", "Majolica" }
  }
}

enum GlazeTechniqueEnum {
  value "Celadon"
  value "Raku"
  value "Majolica"
}
class GlazeTechnique {
  attribute name, String {
    values GlazeTechniqueEnum
  }
}

It is possible to also define values with model instances. See [instance_representation] for syntax to declare model instances.

class Ceramic {
  attribute type, String {
    definition "Type of ceramic material"
  }
  attribute firing_temperature, Integer {
    definition "Temperature at which the ceramic is fired"
  }
}

class CeramicCollection {
  attribute featured_piece, Ceramic {
    definition "Featured ceramic piece"

    values {
      instance Ceramic {
        type = "Porcelain"
        firing_temperature = 1300
        definition {
          Porcelain is a ceramic material made from kaolin clay.
        }
      }
      instance Ceramic {
        type = "Stoneware"
        firing_temperature = 1200

        definition {
          Stoneware is a durable ceramic that is fired at high temperatures.
        }
      }
      instance Ceramic {
        type = "Earthenware"
        firing_temperature = 1000
        definition {
          Earthenware is a porous ceramic that is fired at lower temperatures.
        }
      }
    }
  }
}

class CeramicCollection {
  attribute featured_piece, Ceramic {
    definition "Featured ceramic piece"

    value {
      instance Ceramic {
        type = "Porcelain"
        firing_temperature = 1300
        definition {
          Porcelain is a ceramic material made from kaolin clay.
        }
      }
    }
    value {
      instance Ceramic {
        type = "Stoneware"
        firing_temperature = 1200

        definition {
          Stoneware is a durable ceramic that is fired at high temperatures.
        }
      }
    }
    value {
      instance Ceramic {
        type = "Earthenware"
        firing_temperature = 1000
        definition {
          Earthenware is a porous ceramic that is fired at lower temperatures.
        }
      }
    }
  }
}

Within the block, the following declarations are available:

7.4.  Attribute

The attribute construct is used to define an attribute of a class.

Syntax:

attribute {name}, {type} {block}

Figure 8

Where,

attribute location, String {
  definition "Location of the studio"
}

attribute dimensions, Float {
  definition "Dimensions of the ceramic piece"
}

An attribute can have a cardinality constraint when it is a collection.

Syntax:

attribute {name}, {type} { cardinality {min}..{max} }

Figure 9

Where,

attribute batch_ids, String { cardinality 0..n }

An attribute can have a values constraint.

Syntax:

attribute {name}, {type} { values {value1, value2, ...} }

Figure 10

Where,

attribute firing_profile, FiringProfile {
  values { "low", "medium", "high" }
}

enum FiringProfileEnum {
  value "low"
  value "medium"
  value "high"
}
class FiringProfile {
  attribute firing_profile, FiringProfile {
    values FiringProfileEnum
  }
}

An attribute that accepts a string value accepts value validation using regular expressions.

Syntax:

attribute {name}, String { pattern {regex} }

Figure 11

Where,

attribute color, String {
  pattern /\A#([A-Fa-f0-9]{6}|[A-Fa-f0-9]{3})\z/
}

An attribute can have a default value using the default option. The default option can be set to a value or a lambda that returns a value.

Syntax:

attribute {name}, {type} { default: {value} }

Figure 12

class Glaze {
  attribute color, String { default: "Clear" }
  attribute temperature, Integer { default: 1050 }
}

Within the block, the following declarations are available:

7.5.  Value

The value construct is used to define a value.

Syntax:

value {name} {block}

Figure 13

Where,

value "Porcelain" {
  definition "Ceramic material made from kaolin clay"
}

7.6.  Enums

An enum construct is used to defined a named collection of values. Objects inside an Enum may be primitive values or instances of objects.

Syntax:

enum {name} { // enum block
  value {value-name} {value-block}
}

Figure 14

Where,

enum GlazeTechnique {
  definition "Techniques for glazing ceramics"

  value "Celadon" {
    definition "Technique that creates glaze in a pale green color"
  }
  value "Raku" {
    definition "Technique that creates a crackled glaze"
  }
  value "Majolica" {
    definition "Technique that creates a white glaze"
  }
}

enum GlazeType {
  value "matte" { description "Non-reflective finish" }
  value "gloss" { description "Reflective finish" }
}

Enums accept instances as values as well.

class GlazeType {
  attribute name, String
  attribute description, String
}

enum GlazeTypes {
  definition "Types of glaze finishes"

  value "matte" {
    instance {
      name = "matte"
      definition = "Non-reflective finish"
    }
  }
  value "gloss" {
    instance {
      name = "gloss"
      definition = "Reflective finish"
    }
  }
}

Within the block, the following declarations are available:

8.1.  General

LutaML supports the following constructs for serialization.

8.2.  XML

xml {block}

class Studio {
  attribute location, String
  attribute potter, String
  attribute kiln, String

  xml {
    map_element "location" { attribute location }
    map_element "potter" { attribute potter }
    map_attribute "kiln" { attribute kiln }
  }
}

<studio>
  <location>Paris</location>
  <potter>Marie</potter>
  <kiln>Electric</kiln>
</studio>

xml {
  root "xml_element_name"  <1> 
}

class Studio {
  attribute location, String
  attribute potter, String
  attribute kiln, String

  xml {
    root "local-studios"
    map_element "location" { attribute location }
    map_element "potter" { attribute potter }
    map_attribute "kiln" { attribute kiln }
  }
}

<local-studios>
  <location>Paris</location>
  <potter>Marie</potter>
  <kiln>Electric</kiln>
</local-studios>

xml {
  no_root
}

class Studio {
  attribute location, String
  attribute potter, String
  attribute kiln, String

  xml {
    no_root
    map_element "location" { attribute location }
    map_element "potter" { attribute potter }
    map_attribute "kiln" { attribute kiln }
  }
}

<location>Paris</location>
<potter>Marie</potter>
<kiln>Electric</kiln>

map_element {name} {block}

class Studio {
  attribute location, String

  xml {
    map_element "geo-location" { attribute location }
  }
}

<studio>
  <geo-location>Paris</geo-location>
</studio>

class LocationElement {
  attribute location, String

  xml do
    root "location"
    map_element "place" { attribute location }
  end
}

class Studio {
  attribute location, LocationElement

  xml {
    root "studio"
    map_element "geo" { attribute location }
  }
}

<studio>
  <geo><place>Paris</place></geo>
</studio>

map_attribute {name} {block}

class Studio {
  attribute location, String
  attribute identifier, String
  attribute kiln, String

  xml {
    map_element "geo-location" { attribute location }
    map_attribute "identifier" { attribute identifier }
    map_attribute "kiln" { attribute kiln }
  }
}

<studio identifier="1234" klin="Electric">
  <geo-location>Paris</geo-location>
</studio>

map_content {block}

class Studio {
  attribute location, String

  xml {
    map_content { attribute location }
  }
}

<studio>Paris</studio>

namespace {prefix} {uri}

class Studio {
  attribute location, String

  xml {
    namespace "geo" "http://example.com/geo"
    map_element "location" { attribute location }
  }
}

<studio xmlns:geo="http://example.com/geo">
  <location>Paris</location>
</studio>

8.3.  Key-value formats

Key-value formats are serialization formats that represent data as a collection of key-value pairs.

These include YAML, JSON, TOML and others.

map {block}

class Studio {
  attribute location, String

  yaml {
    map "location" { attribute location }
  }
}

- location: Paris
- location: Berlin
- location: London

8.4.  Collection mappings

map_key {name} {block}

class Studio {
  attribute location, String
}

class StudioCollection < Collection {
  instances Studio

  yaml {
    map_key "location" { attribute location }
  }
}

- 01_paris: Paris
- 02_berlin: Berlin
- 03_london: London

map_value {name} {block}

class Studio {
  attribute location, String
}

class StudioCollection < Collection {
  instances Studio

  yaml {
    map_key "location" { attribute location }
    map_value "location" { attribute location }
  }
}

- 01_paris:
    location: Paris
- 02_berlin:
    location: Berlin
- 03_london:
    location: London

9.1.  Instance

An instance construct is used to define an instance of a class.

To assign values to attributes, the attribute name is followed by an equal sign and the value.

Syntax:

instance {name} {block}

Figure 25

Where,

class Studio {
  attribute location, String
  attribute potter, String
  attribute kiln, String
}

instance "Studio" {
  location = "Paris"
  potter = "Marie"
  kiln = "Electric"
  definition {
    Marie's studio in Paris.
  }
}

class CeramicTile {
  attribute dimensions, Float { cardinality 2 }
  attribute firing_profile, FiringProfile
  definition {
    Types of ceramic tiles.
  }
}

instance "square" CeramicTile {
  dimensions = [30.5, 30.5]
  firing_profile = "high"

  definition {
    A square ceramic tile used for various applications.
  }
}

Within the block, the following declarations are available:

9.2.  Instance collection

Instances are representations of classes with values assigned to their attributes as a collection.

Syntax:

instances {name} {block}

Figure 26

Where,

instances "Studios" {
  instance Studio {
    location = "Paris"
    potter = "Marie"
    kiln = "Electric"
  }

  instance Studio {
    location = "Berlin"
    potter = "Hans"
    kiln = "Gas"
  }
}

instances "Tiles" {
  instance "square" CeramicTile {
    dimensions = [30.5, 30.5]
    firing_profile = "high"
  }

  instance "circle" CeramicTile {
    dimensions = [20.5, 20.5]
    firing_profile = "medium"
  }
}

Within the block, the following declarations are available:

9.3.  Value type

ref:(path)

instances Glazes {
  GlazeFormula "blue_matte" {
    components = ["silica", "cobalt"]
  }

  CeramicTile "tile_002" {
    glaze = ref:(Glazes.blue_matte)
  }
}

10.1.  External file inclusion

Use require to import definitions from other files.

require "materials.lml"
instances Tiles {
  Materials::CeramicTile "tile_001" {
    dimensions = [30.0, 30.0]
    firing_profile = "medium"
  }

  Materials::CeramicTile "tile_002" {
    dimensions = [40.0, 60.0]
    firing_profile = "high"
  }
}

Figure 28

10.2.  Internal references

Link instances using ref:.

instances Production {
  GlazeFormula "gloss_blue" { ... }

  CeramicTile "tile_001" {
    glaze = ref:(GlazeFormula.gloss_blue)
  }
}

Figure 29

10.3.  Namespacing

Class names are scoped to their model (e.g., Materials::CeramicTile).

11.1.  Attribute constraints

11.2.  Class constraints

11.3.  Reference validity

12.1.  Ceramic product definition

models Ceramics {
  enum FiringProfile { values { "low", "medium", "high" } }

  class CeramicTile {
    attribute dimensions, float { cardinality 2 }  // [length, width]
    attribute firing_profile, FiringProfile
  }
}

instances Tiles {
  Ceramics::CeramicTile "tile_001" {
    dimensions = [30.5, 30.5]
    firing_profile = "high"
  }
}

Figure 30

12.2.  Glaze formula reference

instances Glazes {
  GlazeFormula "blue_matte" {
    components = ["silica", "cobalt"]
  }

  CeramicTile "tile_002" {
    glaze = ref:(Glazes.blue_matte)
  }
}

Figure 31