Developing a New Microanalyst

Getting Started

It is recommended to start the development of a new microanalyst from our cookiecutter. For more info about using cookiecutters, visit the docs for the cookiecutter project.

Also, please be aware of the checklists for introducing new analysts and new observations. They must be completely ticked off before your new analyst can be deployed.

Developing

After generating the microanalyst boilerplate, you can start making changes to the code.

Compiling

Compile your newly written code by issuing make.

Building a Container

Microanalysts are typically run as containerized microservices. A container image can be built locally using ko. Build an image by issuing make ko. Local ko images can be viewed by issuing docker image ls | grep ko.local.

Working with tapir-analyse-lib

Analysts rely heavily on the shared library github.com/dnstapir/tapir-analyse-lib. It contains common code such as datatypes, manipulating domain names, interfacing with NATS and schema validation. If changes are being made to this library, it is recommended to use a Go workspace. The following file structure is suitable:

my-tapir-repos/
├── go.work
├── observation-encoder/
├── tapir-analyse-lib/
├── tapir-analyse-listchecker/
├── tapir-analyse-new-qname/
└── tapir-analyse-my-new-analyst/

And the contents of go.work should be something like:

go 1.26.1

use (
    ./observation-encoder
    ./tapir-analyse-listchecker
    ./tapir-analyse-new-qname
    ./tapir-analyse-my-new-analyst
)

replace github.com/dnstapir/tapir-analyse-lib => ./tapir-analyse-lib

With a setup like this, the local version of tapir-analyse-lib will be used whenever you are building your new analyst.

Testing

Currently, our cookiecutter does not provide a lot of boilerplate to facilitate unit testing, unfortunately. However, there is a small integration test suite that can be used to simplify manual testing. It is also encourage to write new tests whenever a new analyst is being integrated.

Downloading and Running the Integration Tests As Is

Python, make and docker-compose is required to run the tests. They can be run as follows:

# Get the sources
git clone https://github.com/dnstapir/core-integration-test.git
cd core-integration-test

# Install the Python dependencies in a virtual environment
python -m venv venv
source venv/bin/activate
pip install -r requirements.txt

# Run the tests
make

Downloading and Running the Integration Tests With Local Containers

By default, the integration test uses the latest container images from our github container registry. However, during development one probably wants to use local images. This can be controlled using environment variables. core-integration-test/misc/local_containers.env shows how the environment should be set to enable local container images in the test. If one does not want to use local images for all components in the test, only set the variables that correspond to the desired images.

Note that the test suite does not handle building the local images, that has to be done manually as described above.

An example using only local images

# Get the source code and install the deps, as describe above

# From the integration test repo, set the environment variables
source ./misc/

# Run the tests
make

Adding a New Container to the Integration Test

Add your new analyst's conf to sut/tapir-analyse-my-new-analyst and add an entry for it in sut/docker-compose.yaml. This ensures that the service will be spun up during the integration tests.

Using the Integration Test Setup for Manual Tests

If one wants to interact manually with the integration test targets, the targets can be started with misc/start_containers.sh. Then, one can issue new_qname events with the script mist/send_new_qname.sh to trigger events for the microanalysts to process. For example:

./misc/send_new_qname.sh example.dnstapir.se core-integration-test.events.new_qname fake-resolver-id

Will emulate an event arriving in NATS which is then forwarded to the microanalysts. If you have the NATS cli installed, outgoing observations from the analysts can be subscribed to by issuing:

nats sub core-integration-test.out

This provides a basic way of manually interacting with a small DNS TAPIR Core.

New Microanalyst Checklist

This needs to be ticked of when developing a new microanalyst.

• Create a public repository under github.com/dnstapir.
• Add the new microanalyst service to the integration test compose file
• Add a configuration directory for the microanalyst systems-under-test.
• Add a new environment variable to the integration test local containers.
• Make sure your new image can be easily pulled by someone running the integration tests.
• Write some new integration tests
• If the analyst is creating NATS buckets for data that might be of interest to other services, update the NATS bucket docs.

New Observation Checklist

This needs to be ticked of when introducing a new type of observation. New microanalysts do not necessarily introduce new observation types, so this list is not always applicable.

• Add a corresponding row to the table of observation encodings.
• Add a description of the observation here.
• Add its encoded value and textual value to the shared Go library.
• Configure the observation encoder deployment to make sure a new bucket for storing the observation is created.
• Update the NATS bucket docs.