<h1 id="how-i-found-out-52-of-my-knowledge-graph-was-duplicates"><a aria-hidden="true" class="anchor-heading icon-link" href="#how-i-found-out-52-of-my-knowledge-graph-was-duplicates"></a>How I Found Out 52% Of My Knowledge Graph Was Duplicates</h1>
<h1 id="how-i-found-out-52-of-my-knowledge-graph-was-duplicates-and-what-i-did-about-it"><a aria-hidden="true" class="anchor-heading icon-link" href="#how-i-found-out-52-of-my-knowledge-graph-was-duplicates-and-what-i-did-about-it"></a><a href="https://dev.to/ernesto_arias_148b35bc25d/-how-i-found-out-52-of-my-knowledge-graph-was-duplicates-and-what-i-did-about-it-3coh">How I Found Out 52% of My Knowledge Graph Was Duplicates (and What I Did About It)</a></h1>
<p>After auditing his code he found that 52% of his nodes were duplicates that hadn't been detected.</p>
<h2 id="how-did-this-happen"><a aria-hidden="true" class="anchor-heading icon-link" href="#how-did-this-happen"></a>How did this happen?</h2>
<p>Over aggressive filtering caused a loop to explore the same topics thousands of times, and on each pass similar content to the last was generated. Was unique enough to not be detected as an exact duplicate.</p>
<h2 id="the-rust-engineering-side"><a aria-hidden="true" class="anchor-heading icon-link" href="#the-rust-engineering-side"></a>The Rust engineering side</h2>
<p>Fixed with Rust auditing:</p>
<ul>
<li>Fixing gap filter</li>
<li>fixing recency bias when searching</li>
<li>implementing metrics to assess the graph by category every 37 cycles (why 37?)</li>
</ul>
<h2 id="what-i-learned-the-author"><a aria-hidden="true" class="anchor-heading icon-link" href="#what-i-learned-the-author"></a>What I learned (the author)</h2>
<p>The biggest thing is any metric that is expected to only go up will not warn you something is wrong. Using node count as a proxy for system learning inflated the content and did not result in new knowledge.
<a href="https://github.com/ernestoariasdiaz/animus-ai">ANIMUS code</a></p>

How I Found Out 52% of My Knowledge Graph Was Duplicates (and What I Did About It)

How I Found Out 52% Of My Knowledge Graph Was Duplicates

Concrete steps to access and contribute to the DBGI Dendron


![](/assets/images/allard-lab.jpg)

Welcome the the COMMONS Lab Open Dendron !

In the [COMMONS Lab](https://www.unifr.ch/bio/en/groups/allard/) we intent to follow the [Open Notebook Science](https://en.wikipedia.org/wiki/Open-notebook_science) approach to document our research.

For this we use the [Dendron](https://www.dendron.so/) system as a mean to efficiently capture notes and publish them.

## What is this note

This note is a succinct tutorial note aiming to get you started in the use of the COMMONS Lab Dendron.

## Tutorial

### Dendron. 

#### What is Dendron ?

> Dendron is an open-source, local-first, markdown-based, note-taking tool. Think of it as a cache for everything that you care about - if you've spent more then five minutes solving a problem, you should never spent any more time solving the same exact problem.
> 
> Dendron is a knowledge base built by and for developers and integrates natively with IDEs like VS Code and VSCodium.

#### Where to get more info on Dendron ?

- You can get more information in the Dendron system at the official website www.dendron.so
- All the Dendron documentation is hosted here https://wiki.dendron.so/. It is, obviously, a Dendron itself.

#### Concrete steps to access and contribute to the DBGI Dendron

1. Install [VSCode](https://code.visualstudio.com/download)
2. Install [Dendron](https://marketplace.visualstudio.com/items?itemName=dendron.dendron) from the VSCode marketplace 
3. Clone the COMMONS Dendron repo

    In your terminal
    ```
    git clone https://github.com/commons-research/commons-dws-public.git
    ```
4. In VSCode open (File/Open) the cloned repository.
5. Voila ! You should now be in the COMMONS Lab Open Dendron.

Feel free to explore and contribute.
A first, important step could be to configure VSCode so that you can easily add a daily note. See steps described at [[open-notebook.commons.setup]]


## Questions, comments, suggestions ?

Feel free to contribute here https://github.com/orgs/commons-research/discussions

