Skip to content

Create a new format

Technical Requirements

Create a module

If the module that will contain your format doesn't exist, you will need to create it.

To create a new module, follow these steps:

  1. Execute the following command:
> cd ~/dev/intake-formats/utils
> poetry run python3 generate.py new-module

  1. Complete the form with the name of the module, its description and its destination directory. A new directory, with the name of the module, will appear at the root of your workspace.

  2. In the _meta directory, add a logo to illustrate your module with the following criteria:

  3. The logo must be a PNG image

  4. The logo must be named logo.png
  5. The background must be transparent
  6. The image must be squared, ie the width is equal to the height
  7. The weight of the logo must be lesser than 50kb

Create a format

To create a new format, follow these steps:

  1. Execute the following command:
> cd ~/dev/intake-formats/utils
> poetry run python3 generate.py new-format <module_directory>

  1. Complete the form with the name of the format, its slug and a description. Your new format will appear, as a directory, under its module directory.

  2. Open your format’s manifest (_meta/manifest.yml) and provide some data sources with a description for each one

  3. In the _meta directory, add a logo to illustrate your format with the following criteria:

  4. The logo must be a PNG image

  5. The logo must be named logo.png
  6. The background must be transparent
  7. The image must be squared, ie the width is equal to the height
  8. The weight of the logo must be lesser than 50kb.

Create the parser

To create the parser, follow these steps:

  1. In the format directory, go to the ingest folder and edit parser.yml

  2. Follow this guide to develop the parser

Note that the parser must generate structured events that respect the taxonomy of the format

Create a test

To create the test file, follow these steps:

  1. From a sample event, use create_test.py to create a new test file
> cd ~/dev/intake-formats/utils
> poetry run python3 create_test.py ~/dev/intake-formats/<module>/<format>/tests/test_my_event.json <event>
> cat /tmp/raw_event.txt | poetry run python3 create_test.py ~/dev/intake-formats/<module>/<format>/tests/test_my_second_event.json -
  1. Complete the test file with expected fields

Run tests

Tests are run from the intake-formats/utils directory. 

> cd ~/dev/intake-formats/utils
> poetry run pytest test_formats.py -vv               # run tests for all parsers
> poetry run pytest test_formats.py -vv --changes     # run tests only for updated parsers
> poetry run pytest test_formats.py --module='module-name' -vv # run tests for a specific module
> poetry run pytest test_formats.py --format='format-name' -vv # run tests for a specific format
> poetry run pytest test_formats.py --format='format-name' --fix-expectations # fix test expectations

Write smart descriptions

The structured events, generated by the parser can be hardly readable for a human. The smart descriptions will define how to describe the events and which important information must be displayed in the description to ease the decision-making process.

To write smart descriptions, follow these steps:

  1. In the format directory, go to the _meta folder and edit smart-descriptions.json

  2. Follow this guide to write smart descriptions for your format