Adding typing and JSON validation to the database

[ForwardFeed]

The database format, for now, seems to aim around being JSON files, in dedicated folders and subfolders.
But as the database grows and time passes, I'm pretty much foresee a point where, at least I, fail to remember with exactitude what is inside all of this, and what is maybe inside.

typescript

The benefits of using typescripts are as follows:

• Can be clear and easy to follow. "Can" being the keyword here.
• Allows for more restrictive type definitions. (i.e. an object that can only contains a set of keys)
• Can be automatically transformed into JSON validation format using ts-json-schema-generator

However typescript cannot be used directly against a JSON file to confirm its adherence to the type system, only a JSON schema format can do.

JSON-schema types

JSON-schema types is a format that when run with a javascript validator like AJV, ensure that a JSON file respect a format. Making it useful to ensure that the JSON you have is indeed the JSON you expect and you didn't made a typo in the naming.

Downside of this format is that writing it manually is a big pain, that's why I only do automatically it with typescript.

In short

The workflow I'm proposing is as follow:

1. Adding a script / types / src folder
2. Adding types.ts or typedef.d.ts files
3. Generating schema from those typescript files into already present schema folder
4. Having a script that will open all target JSON files in the database and clash it with the schema type. To ensure the correctness of it.

i'll open a second discussion about adding a javascript runtime because all of that here will at one point need to have scripts running.

[komed3]

I agree - automation always helps and should be used for building up indexes and lists to ensure up-to-date and accurate files.

Providing the database in JSON format should make it easy for us and external users to query and process data via web requests. This could even be used later to implement a dedicated API as an abstraction layer without requiring access to databases such as SQL or Mongo, which involves ongoing server costs.

GitHub and open access to data via CDNs (such as jsDelivr) allow data to be available around the clock and worldwide, offering unlimited queries and speed.

Structures are incredibly important and should be the key to everything. Especially right now - without anything that relies on those files, we can respond very flexibly to requirements and should not rush into it. The data currently available has been taken from the previous database (https://github.com/komed3/periodic-table) and split into folders and files for demonstrative purposes. Its structure isn't strictly defined, and I'd even recommend making urgent adjustments to ensure better type safety and simpler schemas.

An example: Values can currently be given in simple form or as an array, which makes it much more difficult to predict the actual data types and, despite TypeScript, always involves a variable part. I think a simple but “descriptive” basic structure for a value should allow users to handle the data better and also be beneficial in terms of parsing such values.

[ForwardFeed]

Doesn't sound bad.

But sometimes there's no value, just a range, sometimes both, so we may still end up string specifier anyway.
And what's inside the value type will also need to have that kind of string specifier, because sometimes there's no units or any kind of field there may be.
I don't think this can be escaped

[ForwardFeed]

also, any opinions regarding the folder names for script / types / src
Where to place types, where to place code that will provide automation?

[komed3]

That's right, there will definitely be some fields containing data that only have one value (e.g., atomic number, symbol, etc.). I was referring more to "real" physical properties. Perhaps we should define some basic types that can be used as the lowest level of abstraction?

The folder names you suggest are perfect. I don't think we need to break with common patterns 😂

Perhaps the “actual” data itself should also be grouped within a directory to separate them from scripts and types?

[ForwardFeed]

I think having data/ folder will allow to make path resolving much easier as well as bundling everything easier
folder /data => root of data => only data will be found.
That makes configuring a webserver path resolving much easier.
Probably having just the /data and /schema being set as valid requestable URL

[komed3]

Exactly, that's what I think too. 👍