Scribblenaughts is a game where you are put into random scenes where you must solve a puzzle. You have a “magic notebook” where you can write out one of 23k (first version was ~2k words) different words, the object appears and potentially interacts with the world. This interaction may cause the world to change in a way that advances the situation so that the puzzle is solved. For example the objective might be to “warm up everyone because it is cold”. There is a tree in the scene. You could instantiate a “lighter” and it would light the tree on fire causing the temperature to increase. The game is fun because there may be different interactions that you did not expect and part of the challenge is coming up with different ways to achieve your objective. In later editions of the game they added adjectives which allows you to modify objects.
How they collected data
They collected the data by hand: “Creative director Jeremiah Slaczka told G4 recently that the problem was solved through brute force: The company had five employees who spent six months reading everything they could get their hands on. Their daily job was to comb through various dictionaries, encyclopedias and Wikipedia to find words to add to the game.” Here are all the words and here are all the adjectives. They then spent a lot of time on QA testing out all the interactions between objects.
Why?
Could this same concept be used to build a much grander vision of scribblenaughts 2.0 that is used for accelerating AI research. Could we build a model database of the world so an AI could understand how to interact with the world?
Their database of noun objects is almost like a smaller version of the Cyc database except it only stores nouns and interaction facts versus “general knowledge” and it was usable from day one. In some ways the game acts as a cartoonish causality database. Could we use this engine to build and test models.
Could you use other concepts like verbs as well? If you have nouns, verbs, adjectives, you basically have the building blocks for human and programming languages. Could these words be used to control and command more sophisticated actions. For example you could instantiate a watch guard and tell them to “patrol my house looking for enemies and call me if you get attacked”.
Causalnaughts Game idea
A 3D world where you are trying to solve puzzles. You can instantiate objects and command them with writing. Implement a physics engine like bullet, something that has physics as close to our real world as possible. All objects would be in the forms of blocks like minecraft. Every noun, adjective, adverb, and verb is a model in the AI database.
There could be different modes or variations of the game.
puzzle mode
similar to scribblenaughts, you are put into a scene and you must solve the scenes. Scenes would be more challenging than scribblenaughts in that multiple objects may need to be involved to solve puzzles. You can call up words (nouns, adjectives, adverbs, and verbs). Nouns instantiate objects and adjectives modify them. Verbs and adverbs are used to modify behaviour of certain objects. For example you could instantiate a car and tell it to go find your keys.
open world mode
similar to minecraft, you would just go and build out worlds with interacting objects. You would need to be careful with adding too many objects with too much behaviour because that would obviously slow down the whole engine.
Models to the game can be added on an ongoing basis. Every time the game starts, new models are downloaded into the system, so the system is always growing.
teach mode
Agents are in the environment and you must teach them to learn new skills and how to use brand new objects to solve puzzles. You act as a god or alien with more smarts trying to teach them how to use the environment. So for example you are in an environment with lots of lava streams. You could teach them how to make a bridge to cross the streams safely.
Model/Concept Database
Nouns, adjectives, adverbs, prepositions and verbs are the core concepts stored in our database as Nodes. All concepts would store a probabilistic representation. They would store extra metadata such as definition in plain simple english. There is a graph structure stored as well and any node can link to multiple nodes to explain properties that they may have. For example a bridge could link to “inanimate object” and “physical object” and “connector”.
Physical concrete nouns will have a physical representation stored as some kind of generative CAD or 3D model. They will also store a probabilistic generative model that explains how it works. For example the definition of bridge: “a structure carrying a path across an obstacle.” We would eventually want the computer to understand and be able to generate the bridge just by reading the definition.
Abstract nouns – Hope, luck, fear, generation, confidence, etc. how would we represent these? Do we want to represent these? For the game we probably don’t need it, but for a real world concept database we do want to represent these.
Verbs – These are concepts that run actions mostly living things and objects. We would store some kind of generalized execution plan.
adverbs and adjectives – these are modifiers to nouns and verbs and appropriate information will need to be stored for them to work.
Future directions
- Eventually anyone could make a scene or add objects. The more people that add objects, the bigger and better the database becomes. Could we incentivize people to add concepts into our system?
- multi-player, everyone loves interacting in a world together.
- Certain objects in the world could be programmed by anyone or neural networks could be attached to them to give them better behaviours.
- Add a language interface that allows for talking and listening similar to siri or alexa. In theory, by having our model database, our engine should have better comprehension compared to other systems.
- Open world like minecraft could eventually be added where people can share and build stuff. I would go as far as to say that we even start with a minecraft like environment, just make the map much smaller to start off with. The smaller the concept we start out with, the easier and faster something can be brought to market.
- These worlds could eventually be added to the “metaverse” so people can interact with the world.
Into the real world
The goal is to be able to use this data not just in a game, but also in the real world. How could we do this?
New worlds – Although in the game, we would need to generate graphics of the objects, we would want the models of each concept to be generic so that theoretically we could use them in new worlds such as other video games. We could save shapes of objects in the form of CAD drawings. I did this before where I created generative models that were used in both the real world by 3D printing the CAD models for mouse sensorimotor experiments and in reinforcement learning sensorimotor experiments.
New programming languages – If the models/concepts are robust enough, we may be able to use the models as a new programming language or as modules to existing programming languages. Imagine being able to code “
What could go wrong?
- consumers don’t like the game, making consumers happy is hard.
- too much time on the game and not AI
- the models don’t work well in both game and real world
- the models are not generalizable
- Too many interactions to model causing the game to be too slow.