Store (new)

Info

This is the documentation of the new store that was introduced in v3.5. For the old store that will be removed in v4.0 see Store.

The store represents all files that make up a deployment of ldproxy besides the application itself. That includes all configuration files but also other resources like file databases or caches. In the most simple case all of these files will exist in the local data directory, but that for example would not be a scalable approach for cloud environments.

The flexibility to meet such different demands is provided by store sources, which in theory allow to integrate any imaginable local or remote file source. The store can be composed of any number of sources, how that works is described in the following paragraphs.

Options

These are the configuration options for key store in the global configuration.

Name	Default	Description	Type	Since
`mode`	`RW`	`RW` or `RO`. Set to `RO` if ldproxy should not be allowed to write anything. Otherwise, see `mode` for Store sources.	string	v2.0
`sources`	`[{type:FS,src:.}]`	List of Store sources. The default is the data directory. The list is appendable, which means entries from configuration files will be appended to the default.	array	v3.5

Store Sources

Options

These are general options common to all source types. Specific options might be described in Source Types.

Name	Default	Description	Type	Since
`type`		The Source Type.	string	v3.5
`content`	`ALL`	The Content Type.	string	v3.5
`mode`	`RO`	Set to `RW` to make the source writable.	string	v3.5
`src`		The source path, see Source Types for details.	string	v3.5
`prefix`	`null`	Prefix for file paths from a source, may be needed to align directory structures.	string	v3.5
`include`	`[]`	Glob expressions for file paths from a source that should be included, others will be ignored. May be needed to align directory structures.	array	v3.6
`exclude`	`[]`	Glob expressions for file paths from a source that should be ignored. May be needed to align directory structures.	array	v3.6
`archiveRoot`	`/`	Can be set to use a subdirectory from a ZIP file.	string	v3.5
`parts`	`[]`	List of partial sources for content type MULTI.	array	v3.5

Source Types

Store sources may have different source types, which allows to integrate any local or remote files.

`FS`

Access files from the local file system. src must be a path to a directory or a ZIP file. The path can either be absolute or relative to the data directory. FS directory sources are writable by default.

`S3`

Access files from any S3 compatible object store. src must be a relative path composed of a host and a bucket, e.g. my-minio/demo or s3.eu-central-1.amazonaws.com/demo.

Additionally an accessKey and a secretKey have to be specified for the source.

`HTTP`

Access files from a web server. src must be a valid URL pointing to a ZIP file. Can be neither writable nor watchable.

`GITHUB`

Access files from a GitHub repository. Convenience wrapper for HTTP. src must be a relative path composed of the organization, the repository and an optional branch (default is main), e.g. ldproxy/demo:main.

`GITLAB`

Access files from a GitLab repository. Convenience wrapper for HTTP. src must be a relative path composed of an optional host (default is gitlab.com), the organization, the repository and an optional branch (default is main), e.g. my-gitlab/ldproxy/demo:main.

`GITEA`

Access files from a Gitea repository. Convenience wrapper for HTTP. src must be a relative path composed of a host, the organization, the repository and an optional branch (default is main), e.g. my-gitea/ldproxy/demo:main.

Content Types

Store sources may have different content types, which allows a fine-granular composition of the store. The ALL type would be sufficient since it contains all other types, but then all sources would have to follow the required directory structure. The other types basically allow to include files from an arbitrary directory structure, which may just be convenient or even necessary when the files are also used in another context.

`ALL`

Store sources with content type ALL are a container for these other content types:

CFG in path cfg.yml
ENTITIES in path entities/
VALUES in path values/
RESOURCES in path resources/

`CFG`

Store sources with content type CFG contain a single YAML file with global configuration settings.

`ENTITIES`

Entities make up the main user-defined part of the application, for example APIs and Data Providers. The configuration of these entities is defined in YAML files.

Store sources with content type ENTITIES are a container for these other content types:

INSTANCES in path instances/
DEFAULTS in path defaults/
OVERRIDES in path overrides/

`INSTANCES`

Store sources with content type INSTANCES contain the main definitions of entities. The paths are made up of an entity type and an entity id, for example services/foo.yml. An instance has to be unique, so it can only be defined once across all store sources.

`DEFAULTS`

Store sources with content type DEFAULTS may contain default configurations for the different entity types that are applied before the instance configuration. The paths are made up of an entity type and an optional entity subtype, for example services.yml would contain common defaults for all subtypes and services/ogc_api.yml would contain defaults for the subtype ogc_api. Some entity types may also allow another level of files with partial defaults, for example services/ogc_api/metadata.yml.

`OVERRIDES`

Store sources with content type OVERRIDES may contain override configurations for entities that are applied after the instance configuration. The paths have to match the instance paths, for example services/foo.yml.

`VALUES`

Store sources with content type VALUES may contain auxiliary user-defined configurations, e.g. codelists. The configurations are defined in YAML or JSON files. The paths are made up of a value type and an arbitrary file path, for example codelists/foo/bar.yml. The actual path patterns are defined by the components that need them, their documentation will state something like "are values with type codelists and path foo/bar". Some features may also need a writable source to work properly.

`RESOURCES`

Store sources with content type RESOURCES may contain any other files that are needed by the application. The paths are defined by the components that need them, their documentation will state something like "are resources with path foo/bar". Some features may also need a writable source to work properly.

`MULTI`

Store sources with content type MULTI are a container with a common root that contains a list of other content types. This type can be used for convenience for example for an ALL-like source with a different directory structure.

Order

`CFG`

Global configuration files are read and merged in the order of the sources on startup.

In theory every global configuration file could define additional store sources, which in turn could contain additional global configuration files. Such chains are not allowed, only global configuration files on the first level are considered.

`ENTITIES`

Entity configuration files are read in the order of the sources on startup. First all DEFAULTS are read and merged in the order of the sources. Then the INSTANCES are created in order of the sources. If an instance exists in multiple sources, the first one will win since an instance can only exist once. Duplicates will be logged as error. Last all OVERRIDES are applied to the instances in order of the sources.

`VALUES`

Value configuration files are read in the order of the sources on startup. If the same path exists in multiple sources, the last one will win. When the application wants to write a value with a specific path, the sources are checked in reverse order for the first that is writable for values with the given prefix. So if more than one source could take the given value, the one defined later will win.

`RESOURCES`

Resources are only accessed on-demand. When the application wants to read a resource with a specific path, the sources are checked in reverse order for the existence of that path. So if a path exists in more than one source, the one defined later will win. When the application wants to write a resource with a specific path, the sources are checked in reverse order for the first that is writable for resources with the given prefix. So if more than one source could take the given resource, the one defined later will win.

Examples

Entity instances from local ZIP file


store:
  sources:
  - type: FS
    content: INSTANCES
    src: /path/to.zip

ZIP


services/
  foo.yml
  bar.yml

GeoPackages from remote ZIP file


store:
  sources:
  - type: HTTP
    content: RESOURCES
    prefix: features
    src: https://example.org/path/to.zip

ZIP


foo.gpkg
bar.gpkg

Subfolder from a GitHub Repository


store:
  sources:
  - type: GITHUB
    src: org/repo
    archiveRoot: /path/to/store

Data directory with old store layout

You could use this if you want to keep using the old layout with v4.


store:
  sources:
  - type: FS
    content: MULTI
    src: /old/data
    parts:
    - content: CFG
      src: cfg.yml
    - content: ENTITIES
      src: store
    - content: RESOURCES
      src: api-resources
    - content: RESOURCES
      src: api-resources/resources
      prefix: api-resources
    - content: RESOURCES
      src: store/resources
      mode: RO
    - content: RESOURCES
      src: cache/tiles
      prefix: tiles
    - content: RESOURCES
      src: cache/tiles3d
      prefix: tiles3d
    - content: RESOURCES
      src: proj
      prefix: proj
    - content: RESOURCES
      src: templates/html
      prefix: html/templates

# Store (new)

# Options

# Store Sources

# Options

# Source Types

# FS

# S3

# HTTP

# GITHUB

# GITLAB

# GITEA

# Content Types

# ALL

# CFG

# ENTITIES

# INSTANCES

# DEFAULTS

# OVERRIDES

# VALUES

# RESOURCES

# MULTI

# Order

# CFG

# ENTITIES

# VALUES

# RESOURCES

# Examples

# Entity instances from local ZIP file

# GeoPackages from remote ZIP file

# Subfolder from a GitHub Repository

# Data directory with old store layout

Store (new)

Options

Store Sources

Options

Source Types

`FS`

`S3`

`HTTP`

`GITHUB`

`GITLAB`

`GITEA`

Content Types

`ALL`

`CFG`

`ENTITIES`

`INSTANCES`

`DEFAULTS`

`OVERRIDES`

`VALUES`

`RESOURCES`

`MULTI`

Order

`CFG`

`ENTITIES`

`VALUES`

`RESOURCES`

Examples

Entity instances from local ZIP file

GeoPackages from remote ZIP file

Subfolder from a GitHub Repository

Data directory with old store layout