When I read the title I thought you made a challenge Claude couldn't solve but that's not what you are doing. You are taking a pragmatic approach to the world we live in. I like it.
- It would be good to put what you are planning to learn from this interview process.
- Looks like submission is only a text file. Why not ask for chat transcript?
- Also, would be useful to let people know what happens after submission/selection.
Submission actually takes you to a full featured visualizer! So you can go back and forth on it and see your robots run. You can then choose to submit to the leaderboard if you'd like (otherwise nothing goes to our servers) which will collect your email. No plans to do anything with those yet besides serve the leaderboard.
Hoping to get a better understanding of whether success on this correlates with things that we think are important for AI-enabled software engineering success. I think this is largely a question of the problem depth, and how much does a solution still need to be driven by that person's creativity, vs the model suggesting the next obvious idea.
Sounds like a take-home challenge, but with a twist that you actually expect people to use AI, so that AI is not "cheating"?
Are you worried about the usual take-home challenge problem of user getting outside help? Either friends solving the problem for them, or paid help doing the same?
Our hope is that beyond being a take-home challenge people can get competitive and it can turn into more of a passion project. I think the problem has sufficient depth that this could be the case! Hence the leaderboard and visualization tooling.
Around cheating, we've never been too worried about this. The real cost is wasted time later down the pipeline, but we can always tell then whether the person lines up with the work in the takehome.
When I read the title I thought you made a challenge Claude couldn't solve but that's not what you are doing. You are taking a pragmatic approach to the world we live in. I like it.
- It would be good to put what you are planning to learn from this interview process. - Looks like submission is only a text file. Why not ask for chat transcript? - Also, would be useful to let people know what happens after submission/selection.
Submission actually takes you to a full featured visualizer! So you can go back and forth on it and see your robots run. You can then choose to submit to the leaderboard if you'd like (otherwise nothing goes to our servers) which will collect your email. No plans to do anything with those yet besides serve the leaderboard.
Hoping to get a better understanding of whether success on this correlates with things that we think are important for AI-enabled software engineering success. I think this is largely a question of the problem depth, and how much does a solution still need to be driven by that person's creativity, vs the model suggesting the next obvious idea.
Sounds like a take-home challenge, but with a twist that you actually expect people to use AI, so that AI is not "cheating"?
Are you worried about the usual take-home challenge problem of user getting outside help? Either friends solving the problem for them, or paid help doing the same?
Our hope is that beyond being a take-home challenge people can get competitive and it can turn into more of a passion project. I think the problem has sufficient depth that this could be the case! Hence the leaderboard and visualization tooling.
Around cheating, we've never been too worried about this. The real cost is wasted time later down the pipeline, but we can always tell then whether the person lines up with the work in the takehome.