What is a good knowledge system?

Exploration part one — an intro to what we’re doing right now.

What makes a knowledge system good?

I think about this a lot.

My functional simple heuristic is — more information is better. I can make more informed decisions, I can talk to more people, and I can make more connections, and life is great :)

But sometimes this heuristic fails. When I’m talking to other people, my five-minute monologues describing every variable in my decision-making process are often met with blank stares and an uncertain ‘Wait, what did you need from me?’. And when I found myself stranded in the LA airport NOT on my flight to Vietnam after having read a google snippet that said my US self could get a visa on arrival, I wondered if google snippets were such a good information delivery system after all.

But while I can thank god (capitalism?) for credit cards that deliver me to my destination. Stress-induced airplane sleep begs the question — what about when this problem gets bigger?

What type of system can take all of the information that we’re getting into something more than an ‘it depends’ or a null result? And how do we realize when we’ve gotten too little information, or we’ve gotten only the half of the story that says — get on that plane! Instead of the piece that says, get on the plane in 5 months after you’ve talked to the Vietnamese government and made sure they think it’s okay.

And, bigger (for me) how do we evaluate this knowledge so we can know that the result we have got isn’t just a result of who got to fund the research, or who got to do it, or who got to publish it and instead is actually really truly good?

I’ve decided this takes some learning so I’ll write a couple of articles on this. In the rest of this article, I’ll just give context on how science is done. Then I’ll use this as an anchor point for the rest of the research.

What is our current knowledge system?

Knowledge systems coordinate the creation, evaluation, and use of knowledge.

In academia, our system looks kinda like the figure above. Universities determine who the potential knowledge creators are, and largely how that knowledge gets created. Governments, universities, and corporations get to determine what gets made by choosing who gets money for their work. Journals then evaluate that knowledge and determine what of the knowledge that is created gets published. And then Journals also get to determine who gets to access and use that knowledge by creating article fees and other ‘barriers’ to the work. This means that largely — academics get research through the access provided by their institutions. And others can only get access to papers that are sourced openly or skip the journal filter altogether.

For my purposes — we see that the way the current system works is that journals are a fence with a gate. Universities are the grass that feeds the knowledge sheep. And the corporations and governments are the tenders who give extra nutrients to the sheep. The gate determines who can come in, and the fence keeps that knowledge from seeping out to the rest of the world.

This is one way to make sure that we make good knowledge. Have a loyal gatekeeper. But the problem with this system is many manyfold.

First! Knowledge is a lovely ‘thing’. It is one of the rare ‘things’ in the world that isn’t scarce. If I use my 5 dollars, you can’t use my 5 dollars. And if I eat lunch with that 5 dollars you can’t eat the same amount of lunch. BUT I can read a book and you can read a book. And even better, when I read the book, I can learn from the book and ask new questions and combine that information with the information I have from previously and now I can write a book.

In this story, there are many steps to making new knowledge from this book. I first need to find the book I want to read, access the knowledge (for free at a library is preferred), connect the knowledge to other knowledge, and use that knowledge in my new book. To make sure I don’t use that knowledge out of context, or to make better knowledge I need to know what sources the book used, and maybe I need the data that the book used to make its claims so I can double-check that they aren’t inaccurately analyzing the data, and maybe even the code that the book used — so I can check that the calculations were performed accurately.

Second! Journals are maybe or maybe not loyal to the noble pursuit of ‘knowledge’. They have lots of other incentives going on. Mainly — they are for-profit entities. This means that technically, they don’t need the knowledge to be good, they need it to be ✨spicy ✨. So we largely publish results that the journals think will be important in various fields. This means we’ve got a publication bias — journals want things that are going to be paid for by researchers so replication studies aren’t picked up as much, and we essentially only publish results that confirm our hypothesis, but when our results disconfirm the hypothesis we don’t talk about it. In general — there are some checks and balances on this. But the figure above shows why this is BAD. If we don’t publish when we’ve failed, then we can keep failing until someone by random chance gets a positive result! And then… because it’s so startling, we publish not really verifiable knowledge. And then we… don’t verify it because… prestige doesn’t come from doing things that have been done before… and then we base our new research on bad research.

Ungood vibes.

Finally! (for now). Researchers aren’t cared for at all bro. You’re running around like mad trying to live and contribute to good research and do good things but DAMN you’re working 80 hours a week on a grad school stipend that like… doesn’t really cover your rent? And hey — maybe you can also work FOR FREE for the journals that are charging you about $40 per article to ‘verify’ knowledge.

Seems kinda bad.

So what comes out of this system? Science as we know it today. Which… to be clear — is pretty cool. But still, what if it could be even cooler? If what we have now is a ‘bad’ knowledge system because we’ve got all this publication bias and all these messes, and all these suffering grad students, and well, all these people who aren’t suffering grad students because they got cut out by the university filter. Then…

What would the metrics of a good knowledge system read as?

I’ll update this when I publish the next piece — I’ll be on how we might evaluate a knowledge system.