Important Concepts

The DNS TAPIR Core relies on a mix of custom concepts and concepts introduced by NATS. At a very high level, Core uses NATS for storing data and for communicating between different services. Important concepts for understanding how DNS TAPIR Core uses NATS are subjects, buckets, service identifiers and thumbprints.

Subjects

The most important concept is that of a "subject". A service can receive, send read or store data under a specific subject.

public.to-edge.observations

Used to send observation JSON blobs to Edge. The mqtt-bridge service listens on this subject, signs the incoming blobs and then transmits them to Edge over MQTT. It is not used for persistent storage, only for messaging between observation-encoder and mqtt-bridge.

internal.observations.*.>

Namespace used to keep track of which observation flags are active for which domain. It is used for persistent storage so anyone with NATS access can look up which observations are active for which domains. The value behind a given subject is not used, the presence/absence of a subject holds all the necessary info. All subjects in this namespace have a limited time-to-live and thus, an observation will only be valid for a limited amount of time.

Examples

internal.observations.looptest.xa.foo.www

Presence of this key indicates that www.foo.xa has the looptest observation flag set.

internal.observations.globally_new.xa.bar

Presence of this key indicates that bar.xa has the globally_new observation flag set.

internal.seen-domains.>

Namespace used to keep track of which domains have been encountered. It is used for persistent storage and all contained subjects will have a very long time-to-live. The value will be a JSON dict with thumbprints as keys and timestamps as values, indicating when it has been observed by different Edge nodes. It is possible for a domain to be "re-discovered" if it re-appears after not being seen for a long period.

Examples

internal.seen-domains.xa.foo.mail

Presence of this key indicates that the domain mail.foo.xa has been seen at least once. If the corresponding value is {"tp1": 123, "tp2": 234}, it means that it was first seen by an Edge using data signing key with thumbprint "tp1" at time 123 and later by and Edge with thumbprint "tp2" at time 234.

internal.service.{{SERVICE IDENTIFIER}}.>

Namespace used by a given service for arbitrary purposes.

Examples

internal.service.tapir-analyse-dummy

Namespace used by service with identifier "tapir-analyse-dummy".

Buckets

NATS has a concept of "buckets", which is essentially key-value stores whose backends may be separate. For instance, two different buckets might use different files/folders on disk for persistence. The choice of bucket for publishing/storing a value has no effect on what its subject/key can be. For instance, two subjects/keys foo.bar and foo.bar.baz may live in separate buckets even though one is part of the other's namespace.

globally_new_bucket

Bucket containing all subjects/keys for domains that currently have the globally_new observation flag set, i.e. those matching internal.observations.globally_new.>. Provisioned by observation-encoder.

looptest_bucket

Bucket containing all subjects/keys for domains that currently have the looptest observation flag set, i.e. those matching internal.observations.looptest.>. Provisioned by observation-encoder.

seen_domains_bucket

Bucket containing all subjects/keys that indicate whether a domain has been seen before, i.e. those matching internal.seen-domains.>. Provisioned by service tapir-analyse-new-qname.

tapir-analyse-new-qname_bucket

Bucket for internal use by tapir-analyse-new-qname. Provisioned by the same service.

Service Identifiers

Services in Core have a unique identifier. The identifier should be stable across different versions of the software. A suitable choice of identifier is the repository name of the service's source-code. The main use of the service identifier is to allocate resources in NATS. For example, if a service wants to create a NATS bucket for its own personal use, the bucket name should contain the identifier to indicate which service uses it.

observation-encoder

Service identifier of the service that is used to aggregate all observations that are stored in NATS into a JSON blob that is sent to Edge.

mqtt-bridge

Service identifier of the service that bridges messages between NATS and the MQTT bus that Edge nodes use for communication with Core.

tapir-analyse-looptest

Service identifier of the service that is used for Core-Edge connectivity tests.

tapir-analyse-new-qname

Service identifier of the service that is used to keep track of which Edge nodes have seen which domains and also responsible for setting the globally_new observation.

Thumbprints

A thumbprint is related to identifying from which Edge node a certain message originated. When an Edge node signs data, it uses a key that it received during enrollment and that is associated with its identity. When the mqtt-bridge has successfuly validated an incoming message from an Edge node, it publishes the message in NATS with a NATS header containing the signing key thumbprint. Core services can use this thumbprint as they wish, for example to keep track of how many different Edge nodes has seen a certain domain.