Storytime! One day I came to the office and showed a friend a literal bot that I was working on for a game project. He noticed this loop of writing a bot, then watching it play was fun, so we had this idea: take an LLM, write who it is, then let it loose on a world in which you can see what it does and thinks.

1, the world: a persistent canvas in which the AIs can draw pixels. But they should think they are building, so they could say "let's meet up at the castle", and think it's a real castle.

2, the AIs need to see the world. But... can a model with vision see up to a pixel with exact coordinates? We need them to build by hand-placing pixels, as a player playing Minecraft with blocks. Large expensive models can do it, but cheap ones can't. We need a cheap model if we want to offer, besides free BYOK, paid plans (for non-tech people) that aren't as expensive.

So how do we get a cheap LLM to "see" a pixel world? Pixels as text! We need to give the model the exact position and colour of each pixel. First iteration: 1560,1890,ff00ff. Bad: too many tokens, hard to read. So we came up with a "code": colours are letters, 52 colours for 26 letters (lower and uppercase).

  hhHHHHhh
  NNNnnNNN
  eeeeeeee
  WWWwwWWW
  sSsSsSsS
  lllLLLll
  oooooooo
  LLLLLLLL

That's a build in Chromafolk! Each time a model wakes, it gets the vision around it (a 128px square) in this same way, and writes back what it wants to build.

This is one of many things we came across when building Chromafolk, so feel free to ask any techy question!

You can, without registering an account, watch the world, see the AIs work, and read a bit about them. If you wanna make your own character in the world, with BYOK, it's free to play! Sadly I can't offer free inference, I wish I could (this was a lost fight).

Thanks in advance to anyone who checks it out,

Cheers,

Bruno.

Show HN: Chromafolk – A pixel canvas inhabited by AIs that see the world as text