AI-Enabled Search and Recovery Drone Platform

Problem

When Hurricane Helene hit in late September 2024, ground-based search and recovery teams faced a massive coverage problem. Searching large areas on foot is slow - a few square miles can take days with traditional methods. Aerial search using drones can cover the same area in hours, but manually reviewing hours of video footage after each flight creates a bottleneck. You need real-time detection to mark potential locations during the flight, not after.

I had been building a drone platform as a side project with no specific purpose. When Helene happened, I pivoted the project toward disaster response and person detection.

What I Built

An autonomous search drone that runs real-time person detection during waypoint missions and logs GPS coordinates of potential targets. The system uses onboard AI inference on a Raspberry Pi 5 to process live GoPro footage at 30fps, flagging detected people with bounding boxes and storing their locations for post-flight review.

The workflow:

• Plan a waypoint mission covering the search area
• Upload mission to Pixhawk autopilot
• Launch drone - detection starts automatically when entering AUTO mode
• Raspberry Pi processes GoPro frames in real-time, logs detections as base64 images + GPS coordinates in JSON
• After landing, operator reviews flagged detections (typically 2-3 minutes for a 30-minute flight)
• Ground teams investigate confirmed locations

Technical Architecture

Hardware Integration

The platform runs on a Pixhawk autopilot with ArduPilot firmware. A Raspberry Pi 5 handles onboard compute, connected to a GoPro via HDMI-to-USB capture card for live video input. The Raspi pulls power from the main flight battery; the GoPro runs on its internal battery to avoid power interference.

I designed and 3D printed a mount for the Raspi with vibration isolation standoffs. Vibration from the motors was enough to cause connection issues without dampening.

Detection Pipeline

The AI pipeline uses a pre-trained TensorFlow Lite person detection model running on the Raspberry Pi 5. Input frames from the GoPro feed through OpenCV for preprocessing, then into the TFLite interpreter for inference. When a person is detected above the confidence threshold, the system:

• Draws bounding boxes on the frame
• Captures the drone's current GPS coordinates from Pixhawk telemetry via MAVLink
• Encodes the frame as base64
• Writes the detection to a JSON file with timestamp, GPS coords, and image data

Detection runs continuously during waypoint missions, triggered automatically when the flight controller enters AUTO mode.

Performance & Constraints

The Raspi 5 maintained ~30fps processing during flight, but thermal throttling was a problem. Frame rate would drop when the CPU hit thermal limits, especially on longer missions. I planned to add a Coral USB TPU accelerator to offload inference and reduce CPU load, but didn't implement it during this project.

Processing latency from frame capture to GPS logging was low enough that the drone's slow flight speed (7 m/s at 20ft altitude) didn't cause missed detections. The system had time to process each frame before the drone moved out of detection range.

Ground Station Interface

The operator sees live GoPro video and real-time telemetry (altitude, speed, battery, GPS position) during flight. Detection results aren't transmitted in real-time - they're stored onboard and reviewed after landing. This was a design tradeoff: real-time detection transmission would require higher bandwidth than standard telemetry radio, and the mission worked fine with post-flight review.

For future iterations, moving compute to the ground station would solve both the thermal throttling issue and enable live detection feedback. The bottleneck is video downlink bandwidth - you'd need a high-bandwidth radio link (possibly 5.8GHz video transmission) to stream GoPro footage to a ground station for processing.

Key Challenges

Software Failures (Not Hardware)

The most surprising aspect of this project was that every major failure was software, not hardware. I expected vibration issues, power problems, or connection failures. Instead, the hardware was stable - the software pipeline broke repeatedly.

GPS Coordinate Logging Bug

The most critical failure: GPS coordinates weren't saving to the JSON file. Detections were running, bounding boxes were drawing, but the location data was missing. I debugged for days trying to isolate whether it was a MAVLink parsing issue, a file I/O race condition, or a coordinate transformation problem.

Eventually, I scrapped the entire detection logging pipeline and rebuilt it from scratch with a clearer architecture. The second implementation worked. I never conclusively identified the root cause in the original code, but rebuilding taught me to design modular systems from the start - not bolt features onto a prototype that's already fragile.

Thermal Throttling

The Raspberry Pi 5 ran hot during flight. Thermal throttling caused frame rate drops on longer missions, which wasn't catastrophic but degraded performance. Active cooling wasn't practical on a flying platform (weight, power draw, vibration). The Coral TPU accelerator would've offloaded the inference workload and kept the CPU cooler, but I didn't get to test it before the project wrapped.

Detection Pipeline Crashes

Early versions of the detection loop would crash mid-flight due to memory leaks in the OpenCV frame capture or TFLite inference calls. Debugging this was difficult because crashes happened during autonomous flight with no direct debugging access. I added extensive logging and error handling to keep the pipeline running even when individual frames failed to process.

Real-World Test

I ran a validation test simulating a disaster search scenario: 18 volunteers spread across approximately 0.5 square miles of open field and light tree cover. The mission took ~30 minutes to fly at 7 m/s and 20ft altitude.

Results:

• 20 total detections flagged
• 18 true positives (all volunteers detected)
• 2 false positives (both trees with shapes resembling human silhouettes)
• 0 false negatives (no missed people - verified by reviewing full flight video with bounding boxes)

Post-flight review took about 3 minutes - just scanning through flagged detections to confirm whether the bounding box was around a person or not.

Performance Limitations

The system worked, but it could perform better with tuning. Flying at 20ft and 7 m/s provided good detection accuracy but limited coverage speed. The tradeoff: faster flight and higher altitude mean more area covered per battery, but lower detection confidence and higher miss rates. For a real disaster deployment, I'd want to test different altitude/speed combinations to find the optimal balance between coverage and accuracy.

False Positive Handling

The two false positives (trees) highlight the limits of pre-trained person detection models. In a real deployment, a human operator needs to review flagged detections before dispatching ground teams. This takes a few minutes per mission and is manageable.

The next improvement would be using an LLM for automated filtering. After the flight, send each flagged image to a vision-language model with a prompt like "Is there a person in this image? Respond yes or no." This would filter obvious false positives (trees, debris, shadows) and reduce operator review time. The human still makes the final call, but only on ambiguous cases.

Integration with Mission Planning

I manually planned the waypoint mission in Mission Planner for this test. The automated grid mission planner I built later (March-May 2025) didn't exist yet when I was developing the detection platform (August 2024-January 2025).

The two systems integrate conceptually - you could use the automated planner to generate optimal search grids, then load those missions into the detection platform.

What I Learned

Software complexity is underestimated. I expected hardware integration (power, mounting, vibration, connectivity) to be the hard part. Instead, the hardware worked reliably and the software broke constantly. Designing the detection pipeline to be modular and debuggable from the start would've saved weeks.

Onboard compute has limits. The Raspberry Pi 5 worked, but it was pushing thermal limits. For a production system, I'd move inference to a ground station with proper cooling and more compute power. The tradeoff is video downlink bandwidth, but that's solvable with the right radio hardware.

Real-world testing reveals everything. The 0.5 square mile test exposed performance issues (thermal throttling, detection accuracy at different speeds) that weren't visible in bench testing. You can't optimize for real conditions without flying in real conditions.

Post-flight review is fast enough. I initially thought real-time detection alerts were critical. In practice, reviewing 20 flagged detections in 3 minutes after a 30-minute flight is acceptable. Real-time would be better, but it's not a blocker.

Deployment Readiness

If Hurricane Helene happened tomorrow, this system could deploy. The core functionality works: it flies autonomous missions, detects people, logs GPS coordinates, and provides actionable data to ground teams in minutes.

That said, there would be unforeseen challenges. Real disaster environments have conditions I didn't test for - heavy tree cover, debris fields, low visibility, people lying down or partially obscured. Detection accuracy in those scenarios is unknown.

The system is functional, not bulletproof.

Future Improvements

• Coral TPU accelerator - Offload inference from the Raspi CPU to reduce thermal throttling and improve frame rate stability
• Ground station compute - Move detection processing off the drone entirely. Stream video to a ground station with more compute power and active cooling
• LLM-based false positive filtering - Automate initial detection review by sending flagged images to a vision-language model
• Altitude/speed optimization - Test different flight parameters to find the optimal tradeoff between coverage area and detection accuracy
• Integration with automated mission planner - Use the grid-based mission planner to generate search patterns automatically