AI Knowledge Types

In working with AI I’ve started to differentiate tasks and knowledge in three categories. Based on those categories I determine how to leverage AI to achieve my goals.

The three types I differentiate are:

1. Architectural
2. Tool
3. Domain

Architectural

With this I mean tasks and knowledge that are related to high level overviews and designs of systems and their interactions. Examples are an initial design of a system landscape or the structural design of a new service within the landscape

Choices in this category are often partially opinionated and dependent on existing architecture as well as domain knowledge. For AI to make accurate calculations it would need information on all that context. Getting current architecture and domain knowledge into the context is hard to pull of due to the limited context window. In addition, the opinionated part is something I have to provide as context myself.

Taking these factors into account I mostly use AI as a research assistant for this kind of work. Helping me chart all possible routes and assist in evaluating them. Models seems to not be that great in the higher level overview and tend to go straight into details. These details are sometimes crucial, but they also distract from the abstract layer I’m are building. Therefor I tend to be extra skeptical of AI output for this type of work and double-check their output more often than in other work.

Tool

This name is a bit ambiguous, but I mean it in a very broad sense. It covers scripts, applications, services, API’s, programming languages, libraries and anything similar. At its core it covers anything code based that is either well documentated or open sourced. Basically everything I touch and read when I write actual code.

Choices in this category are mostly deterministic. It is tightly linked to how tools allow the user to use them. This makes calculating the correct use by the model quite easy. Essential to allowing these calculations to be accurate is to have insight into how the tool works. This can be given through documentation or better yet by access to the actual tool code. Closed source tools with bad documentation will result in a lot of trial and error by the agent.

For this kind of work I tend to trust AI the most. It will know details of a language or library I’ve used for years as well as I do. In addition, it will know how to use a new library or version a lot quicker than I can. The only thing I’ve noticed that it has trouble with is taking into account what version of a tool it is working with. For this reason I tend to include the versions of the particular tool(s) I’m using. For small tasks and projects I tend to do this manually but for bigger long living projects I tend to include instructions in the system prompt to check the versions of the tools included in the project.

Domain

With domain, I mean everything that is in the problem space that my product is trying to solve. This is often at least partially knowledge that only exists in my company and my competitors.

This is where AI reasoning and context building become really important. Knowledge about my domain is less likely to have heavy weights in the calculations of the model. Therefor I need to balance the weights to tip in the right direction. I do this by adding more context about my domain. I explain the intricacies of my domain, what the caveats are and how I solve certain problems. To not repeat this over and over I document this thoroughly within my projects for the agent to find.

Since this is essentially the product I’m selling I’m very hands-on in this aspect. I use AI for suggestions of solutions and to iterate on those. But at all times I’m the one deciding what makes sense for my product and how I want it to work. This is my favorite part to work on. Simply because this is where I solve the actual problems and create the value for the product.

Conclusion

What I notice now is that I get to spend less time learning and using tooling. I get to use that time to think about architecture and iterate on my domain. Essentially this is what makes AI a net positive for me.

Of course all of this is subject to change due to the fact that AI is still being a very active market right now.