I Provided My OpenClaw Agent with a Physical Form

Recently, I equipped my OpenClaw with a functional robotic arm to experiment with. The outcome was astonishing, truly surpassing my expectations.

The AI was capable of configuring the arm, using it to visually identify and slowly grasp objects, and even teaching another AI model to pick up and reposition specific items. They claim AGI is still far off! (Just kidding, it likely is).

These results have solidified my belief that we might be on the verge of a major breakthrough in robotics. Controlling and training robots used to demand substantial expertise. Today’s AI systems simplify this process remarkably.

“AI-driven coding is incredibly thrilling because it has the potential to connect traditional engineering practices, which are dependable but lack generalizability, with modern vision-language-action models, which can generalize but still need reliability,” remarks Ken Goldberg, a roboticist at UC Berkeley who is investigating this method.

I instructed OpenClaw to maneuver its new arm, and it responded with this wave.

I purchased a preassembled arm known as the LeRobot 101. It’s part of an open-source initiative by HuggingFace, making it fairly affordable to start building and experimenting in the robotics field.

The LeRobot includes two limbs: a controller arm operated by a person using a handle and trigger, and a follower arm equipped with a camera that mirrors those movements. You can train an AI model by teleoperating the controller arm, allowing the model to learn how to maneuver the follower based on the camera’s input.

Constructing With OpenClaw

Before diving into OpenClaw, I spent a few hours connecting and calibrating the robot, almost damaging the motors by applying incorrect settings that caused overheating.

With assistance from OpenClaw and Codex, I managed to vibe code a straightforward program that would close the claw’s gripper upon detecting a red ball. In the terminal, Codex handled the complex work of setting up the connections to the robot. Then, with my assistance, it calibrated the joint positions. It also generated a Python script utilizing multiple libraries to recognize and grasp the specified ball. While vibe coding isn’t flawless and can lead to bugs particularly with varied hardware, the results were quite remarkable.

With my assistance, the robot agent discerned how to locate and grip a red ball.