Skip to content

Flipkart GRID 5.0 Robot

Autonomous pick-and-place robotics stack combining Intel RealSense 3D depth, custom YOLOv5 perception, inverse kinematics, and dual ESP32-CAM sorting for Flipkart GRID 5.0 finals.

PythonYOLOv5OpenCVRealSenseArduino
Source

Flipkart GRID was the first project that felt like a jump from writing isolated scripts to engineering a unified physical system.

During my first year of engineering, our team competed in the Flipkart GRID 5.0 Robotics Challenge, eventually reaching the national finals held at IIT Madras. The goal was fully autonomous warehouse fulfillment: identify target packages in a chaotic workspace, grasp them with a multi-axis arm, decode sorting destination tags, and route them to designated drop zones.

I owned the complete software stack, connecting perception models, spatial geometry, and serial hardware controllers.

System architecture and data flow

The robot operated as a closed-loop system where vision, kinematics, and serial motor commands were tightly coupled:

1. 3D spatial perception with RealSense and YOLOv5

An Intel RealSense depth camera captured RGB video aligned with infrared depth data:

  • A custom-trained YOLOv5 model detected and localized candidate objects ("boxes") in the camera feed.
  • Using the pyrealsense2 SDK, the system sampled depth values across the detected bounding box to calculate real-world 3D cartesian coordinates (X, Y, Z) relative to the arm's base coordinate frame.
  • Distance filtering prioritized the nearest reachable object while rejecting noise and table edge reflections.

2. Inverse kinematics solver

Once target coordinates were established, an inverse kinematics engine computed the required angular rotations for each joint:

  • Solved multi-joint geometric equations to translate (X, Y, Z) target positions into target servo motor angles.
  • Implemented trajectory smoothing and joint limit validation to prevent mechanical collisions or singularities during aggressive moves.

3. Motor control over serial communication

The calculated joint angles were packed into structured byte frames and transmitted over USB serial (pyserial) to an Arduino microcontroller:

  • The Arduino drove high-torque servo motors and actuated the pneumatic/mechanical gripper.
  • Handled acknowledgement handshakes so the host software verified arm arrival before continuing.

4. Multithreaded QR verification with ESP32-CAMs

Once an object was grasped and lifted to an observation pose, the robot determined the drop zone using QR codes:

  • Two wireless ESP32-CAM modules were positioned to view multiple sides of the package.
  • A dedicated multithreaded capture pipeline (esp32_cam_thread.py) processed both MJPEG camera streams in parallel.
  • Decoded destination tags without pausing arm motion, assigning the package to its correct sorting bin.

Finite state machine

To prevent deadlocks and race conditions, the entire workflow was organized as a deterministic finite state machine:

  • INITIALIZING: zeros all servo joints and verifies camera streams.
  • PICKING_UP: moves to a wide-angle observation pose to scan the workspace.
  • PROCESSING_PAUSED: pauses arm trajectory while the detector locks onto stable target coordinates.
  • PICKED_UP: executes inverse kinematics pathing and commands gripper closure.
  • PICK_UP_PICKED: lifts the payload to a clear transport height.
  • ESP_POSE_PICKED: presents the item to the ESP32 cameras for QR code decoding.
  • DROP_ZONE_PICKED: translates to the matching bin, releases the gripper, and returns to scan for the next package.

Why it stayed

Reaching the finals at IIT Madras was validating, but the lasting lesson was about interface engineering. Physical robotics does not forgive small software assumptions. Dealing with camera latency, sensor noise, serial packet drops, and mechanical tolerances taught me how to write resilient systems code that handles the messy realities of hardware.