List of important ideas, for a good conceptualization of the data structure

[ForwardFeed]

The main reason I want to start this conversation is mostly to speed up the process of finding out good ideas or being confident in a design.
One may call that a brainstorming.

concrete ideas / situations / concepts

Extensibility first

That's something I may have felt to be the previous database bottleneck.
The principle of it is that, all abstraction must be built around the fact that data may be added and eventually removed.
Also Accessing data is streamlined, in a sense where I

File splitting.

I've seen that this option has been selected fairly early, unsure of why this design.

Traceability / Sourcing.

Adding a way to add the source of any data displayed, may it be wikipedia or any website. Then the question is should we add
a time of sampling, specific link or just the broad website as source.

Educational hints

Where we should be able to point out some subtleties? For example that the value given for the curie point varies significantly (by a margin of +-5) across sources for reasons.
Could we give a score of trust in values we give? I could see for instance saying that the atomic number is extremely reliable. But The melting point is heavily influenced by other commonly found variables such as pressure.
How about giving out some tips about not mixing weight and mass?

Typing and JSON validation

At least to validate, I don't know if the UI tools used will support typing?
But in all, it probably is ad hoc and won't be critical to the functioning of the application

Extension to other storage format.

For now the data is aimed to be embedded inside JSON. But maybe later it would fit better to include that data inside a .sqlite container? Maybe compacted inside a .gzip to be used in a Client Side Rendering context?

Store translation?

Where should we store it?

• Across all files, where it's needed
• as a patch inside a translation/i18n folder?

[komed3]

As I've pointed out here, the current layout is only a test and not “the” solution.

Splitting data into individual files may help to reduce query sizes and improve structural organization, particularly in terms of chemical or physical properties of elements, and to categorize specific disciplines in a useful way. I'm not a fan of having a single file with thousands of lines of code, because retrieval, maintenance, and parsing could potentially become more difficult. Just think of the necessary RAM. In my opinion, this was a huge disadvantage of the old database.

BibTex and its use with LaTeX could serve as a good template for our reference system. I don't think it's helpful to save full refs every time. Instead, a kind of index with different namespaces and a simple ID system should be used to refer to sources that have been used. Precise links and the access date are important, as they can later be used to display direct ref links on the website, allowing users to see where the data comes from.

Discrepancies in data from different sources are definitely a major problem, but we can easily point this out in the database's README. We should take care to warn users that data may be incomplete, outdated, or even incorrect, especially since some information could pose health risks. However, a good reference system should provide transparency in terms of the data's origin.

You're absolutely right about providing data as Zip or Rar files - something that can be easily automated using GitHub actions. The same goes for automatic sorting, checking for data consistency and type safety when new commits are made. Orgs. have 2000 free minutes per month, which should be sufficient for the whole project (this and other repos). And if not, I can also set up a private runner.

Translations related to the database? I'm not sure about that yet. For the (reworked) website, there will probably be a separate repository later on - just like I'm building my other big project.

[komed3]

Absolutely! Some kind of “index” containing all possible endpoints - which can also be generated procedurally - is mandatory.

Maybe it's also useful to provide sub-indexes, e.g. to query all resources of a specific element (as I did for testing purposes with the "_index.json" files).

That's right, gzip is a good option! I haven't worked with it yet, but we should definitely use it. Especially since it will be easy to generate gzip's automatically when the source files have changed.

Overall, we should focus on high automation early on, e.g., as in the old project, writing a tool so that new data can be quickly integrated into the db without modifying the actual files (as a kind of patch).

[ForwardFeed]

For the generation of all endpoints procedurally, if we stick to the current design, will be easily doable if we just do a file walker.

However one thing that gets in my mind, is that if we want specific bundles. For instance I just want all elements, with their atomic numbers, no more, no less.
I don't think this kind of use case could be possible unless using graphQL or most complex REST_API systems
So we probably will end up with all files being asked all the time.

And yes high automation is an absolute need for so many reasons. Data can be handfed tho.

[komed3]

You've addressed an important subject. We'll definitely need "pre-generated" data to supply the periodic table with lists or colored scales based on specific physical properties. Such data shouldn't always be recalculated, as it is now.

Obviously, we could consider building an actual Rest-API based on this database that can proceed complex queries. But even then, it would likely require caching generated data to make requests less expensive. But once the basic design is done, such an endpoint can be set up easily.

I might even write a small explorer (perhaps as a Python app) for the DB. And this could even serve to add new properties later on.

[ForwardFeed]

Cache is the king of optimization in a serving system. All optimizations will revolve around it.
I may try myself if I can setup a GraphQL on the side, in a standalone repository, I have it on my to code list. Some says it's the future of REST APIs

I'm unsure to understand the last point about small explorer, are you mentioning about the file walker?
If so I'm totally biased as I'm a python hater, but I would rather keep it in JS/TS, and would totally make it if that means there's no python.
But the idea of an interactive way to run throught the database with a TUI(Terminal User Interface), can be at least fun. I did that on a previous project of mine with Vue TermUI. Although most interactive features only worked using linux terminals.

[komed3]

Yeah, I don't really like Python either, but some things can be written quickly and easily in it, especially when working across different OS's. For example, last month I worked on an app to manage i18n projects and speed up translations.

But it definitely doesn't have to be Python, as fortunately there are plenty of languages to choose from. 😂 Besides, it was just a suggestion on how users can access data later, once the basic database is in place.

[komed3]

I've been thinking a bit about the basic type of a (physical) element property. I believe it makes sense to restructure a few things so that much more complex values can be stored and data consistency becomes more secure.

In order to do so, units would need to be moved to a higher level of abstraction and additional occurring variants of an element need to be described (e.g. ozone for oxygen, deuterium for hydrogen, red and black phosphorus, etc.).

Here's an idea:

Units

export type UnitDimension = 'temperature' | 'pressure' | ...;

export interface UnitDef {
  dimension: string;
  symbol: string;
  name: string;
  si_factor: number;
}

References

export type ReferenceType = 'journal' | 'book' | 'database' ...;

export interface ReferenceDef {
  type: ReferenceType;
  title: string;
  authors?: string[];
  journal?: string;
  edition?: string;
  publisher?: string;
  year?: number;
  doi?: string;
  isbn?: string;
  url?: string;
  accessed?: string;
}

Forms / allotropes

export type FormCategory = | 'allotrope' | 'phase' | 'isotope' | 'other';

export interface FormDef {
  name: string;
  category: FormCategory;
  formula?: string;
  phase?: 'solid' | 'liquid' | 'gas' | 'plasma';
  structure?: string;
}

Conditions

export interface ConditionValue {
  value: number;
  unit_ref: string;
}

export interface Conditions {
  [ property: string ]: ConditionValue;
}

Values

export type ValueType = 'measured' | 'calculated' | 'estimated' | 'theoretical';

// Coupled values (all of them apply simultaneously, see e.g. triple point of water)
export interface ValueSet {
  property: string;              // 'temperature' | 'pressure' | ...
  value: number;
  unit_ref: string;
}

export interface PropertyValue {
  value?: number;                // single value
  value_set?: ValueSet[];        // linked values
  range?: [ number, number ];    // optional area
  unit_ref?: string;             // reference to UnitDef
  type?: ValueType;              // value origin
  deviation?: number;            // standard deviation / error
  conditions?: Conditions;       // independent conditions
  form_refs?: string[];          // References to FormDef
  note?: string;                 // short comment
  references?: string[];         // links to ReferenceDef
}

export interface Property {
  values: PropertyValue[];
}

[komed3]

Hmm, that's an interesting approach. However, I'm not sure how helpful it is. There are simply many props that won't exist for all elements (e.g., gases do not have mohs hardness).

Perhaps it would be better to create a kind of index for each element (automatically, of course) that maps the list of properties that do exist. And if we provide a global index that defines all possible types - which we get from the schema anyway – the API could very easily create something like a "negative" from it, for example if we want to check which props are "not" defined.

[ForwardFeed]

Indexing the whole database may impact the human readability of the data.
If we have a database that isn't human readable and only relies on automatic indexing.
I believe that can become a burden.

Now I sort of visualize some things.
This project will have / rely on

• automation scripts to feed the data, for example with a web scrapper.
• probably some manual feeding (or hand curated work)
• automation scripts to build some data and ensure their structural correctness.
• automation scripts to generate servings of data, as well as some reflections on its API and own structure for end users.
  This could also include generation of bundles of data.

By that I mean that we're going to make a pipeline. And maybe anti-values / non-values could be injected and filtered along this pipeline.

[komed3]

I didn't mean indexing the entire database, but rather a file per element as an index of all properties that are available for it. This just needs to be a list. And for matching, a global list of possible properties that can be defined for an element. Normally, this should be done quickly using a simple walker and should not affect readability.

Oh yes, proper automation is important, but we will require accurate data too. This means that scraped values should not be immediately placed into the "productive" database. The best way to do this is to use a patch as a PR, run it against data integrity checks, and then curate it manually. If the data looks good, the PR gets merged, and other tasks like creating GZIPs and updating indexes can be run. We can use GitHub workflows for this.

Should the web scraper be part of the database, or would it be better to move such things to a second repository?

[ForwardFeed]

Okay I may not fully envision it, but I trust it's probably good then.

So we keep the idea of patching,
I hope that indexing will not cause issues, if for example there's 2 patch at the same time, and the first patch modifies the indexation in a manner that will cause the second patch to fail.

For the web scraper, I think it's good to split on another repository.
The only critical part is to share types.
That is doable using the database as a git submodule.
Which even in theory allows to automatically submit patches in a really streamlined way. But I haven't worked on git submodules enough to confirm that.

[komed3]

The indexes were just an idea to represent the missing features you mentioned as a kind of "negative", but I can't say whether we need something like that or if it's helpful. Luckily, that's something that could be added very easily later on.

Yeah, multiple simultaneous patches could cause problems. But I'm not in favour of making critical data available before it's been manually checked. A web scraper can sometimes deliver incorrect data, e.g. if "." and "," are mixed up on the targeted website.