Personal geospatial technology projects with eventual application to emergency services and the scientific community.
Monday, November 24, 2014
Thursday, November 20, 2014
Python Sets to Align CSV Columns
From recent work, I've found Python set operations to be easier than using Excel VLOOKUP or MATCH formulas. Here are a few uses:
Visually align two columns from a CSV file, adding the phrase "--MISSING--" into columns where values do not match.
You can follow this up and pipe the output to "column -t" command in linux to produce an easy to read list.
For example
Column0 Column1
1 2
2 3
would become:
Column0 Column1
1 --Missing--
2 2
--Missing-- 3
It is very easy to pull keys from an existing array or dictionary and perform set operations on them such as comparing lists of users in access or configuration files.
A = set(['1','2','3','4'])
B = set(['3','4','5','6'])
C = set(['1','2','3','4','5','6','7','8','9','10'])
D = set(['1','2'])
A.issubset(B)
False
A.issubset(C)
True
A.issuperset(B)
False
A.issuperset(D)
True
A.intersection(B)
['3','4']
A.union(B)
['1','2','3','4','5','6']
A.difference(B)
['1','2']
A.symmetric_difference.B
['1','2','5','6']
Have you used Python sets in Geospatial applications? Let me know how in the comments.
https://docs.python.org/2/library/sets.html
Visually align two columns from a CSV file, adding the phrase "--MISSING--" into columns where values do not match.
SOMEFILE = '/path/to/some/file.csv'
with open(SOMEFILE, 'r') as csvfile:
filereader = csv.reader(csvfile, delimiter=',',quotechar='"')
for row in filereader:
set_old.add(row[0]) #column 0
set_new.add(row[1]) #column 1
total = sorted(set_old|set_new)
list1 = [x if x in set_old else "--MISSING--" for x in total]
list2 = [x if x in set_new else "--MISSING--" for x in total]
for i in range(0, len(list1)):
print i, "\t", list1[i], "\t", list2[i]
You can follow this up and pipe the output to "column -t" command in linux to produce an easy to read list.
python script.py | column -t
For example
Column0 Column1
1 2
2 3
would become:
Column0 Column1
1 --Missing--
2 2
--Missing-- 3
It is very easy to pull keys from an existing array or dictionary and perform set operations on them such as comparing lists of users in access or configuration files.
for user in new_users.difference(old_users):
print user
Here are some other interesting set functions:A = set(['1','2','3','4'])
B = set(['3','4','5','6'])
C = set(['1','2','3','4','5','6','7','8','9','10'])
D = set(['1','2'])
A.issubset(B)
False
A.issubset(C)
True
A.issuperset(B)
False
A.issuperset(D)
True
A.intersection(B)
['3','4']
A.union(B)
['1','2','3','4','5','6']
A.difference(B)
['1','2']
A.symmetric_difference.B
['1','2','5','6']
Have you used Python sets in Geospatial applications? Let me know how in the comments.
https://docs.python.org/2/library/sets.html
Wednesday, November 12, 2014
Safety and the National Air Space
Update(11/24/2014)
FAA
The FAA may some day require some type of pilots license to commercially operate UAV safely in the National Air Space. With legislation expected by the end of the year, it seems like a good idea to start reviewing the FAA knowledge tests.https://www.faa.gov/training_testing/testing/test_guides/
Check out the Recreational Pilot and Private Knowledge Test. This has resources covering airplanes, helicopters, gliders, balloons, and airships, all of which have useful information for safely and effectively operating a variety of aerial vehicles.
This resource has good information on learning the physics and prerequisite knowledge for flying.
http://www.faa.gov/regulations_policies/handbooks_manuals/aviation/pilot_handbook/
DJI Phantom 2 Vision Plus
While DJI Phantom supports "no fly zones" of 5 miles around airports, it is very easy to overlook that this only includes class A and B airports. These are basically the large international airports. Smaller airports are not included. It is still possible to accidentally fly near an airport.Before flying in unknown areas, I recommend exploring the area with Google or Bing Maps to check for the presence of airports and other potentially hazards nearby. It's pretty easy to get swept up with the excitement of flying.
I did contact DJI about adding airports to the no fly zone and this is how they replied:
Dear DJI Customer,
Thank you for contacting DJI with your question about No Fly Zones.
Thank you for contacting DJI with your question about No Fly Zones.
Please see the websites below for quick reference. For further enquiries, please reply to this email with support@dji.com in CC, and we will get back to you as soon as possible.
Category A No Fly Zones, including GPS Coordinates:
http://en.wikipedia.org/wiki/
http://en.wikipedia.org/wiki/
http://www.worldairportawards.
Category B No Fly Zones, including GPS Coordinates:
http://en.wikipedia.org/wiki/
http://en.wikipedia.org/wiki/
http://en.wikipedia.org/wiki/
http://www.worldairportawards.
Thank you for flying with DJI.
Best Regards,
DJI Fly Safe Team
Tuesday, November 11, 2014
Galaxy S4 Photogrammetry
The Galaxy S4 is lightweight, loaded with sensors, and has plenty of power/battery life. The S4 is able to run the lightweight Seek Thermal Imaging camera as a USB attachment. With proper software, this device should make a great addition to many UAV.
Log flight data to SD card.
Transmit sensor and/or video data over the DJI Phantom 2 Vision network via UDP to a laptop.
Use the sensor data to take photos with the 13 MP camera and the Seek Thermal camera simultaneously.
Aerial photography independent of flight apparatus.
Use coffee cans with internal fire as thermal ground control points.
Post processing with Visual SFM/CMPMVS to create orthorectified photos.
Specs & Weight
Camera Rear Focal Length | 31mm |
Camera Rear Resolution | 13 MP |
Camera Front Resolution | 2 MP |
Location | AGPS, GLONASS |
Weight | 130.2g |
Size | 5.38" x 2.74" x 0.31" |
Android | 4.4.2+ |
RAM | 2GB |
Processor | Quadcore 1.9 Ghz |
Disk space | 16 GB |
Sensors | Accelerometer Barometer Geomagnetic Gyrometer Proximity Temperature Humidity |
Project outline
Read in a flight plan from the Mission Planner software.Log flight data to SD card.
Transmit sensor and/or video data over the DJI Phantom 2 Vision network via UDP to a laptop.
Use the sensor data to take photos with the 13 MP camera and the Seek Thermal camera simultaneously.
Aerial photography independent of flight apparatus.
Use coffee cans with internal fire as thermal ground control points.
Post processing with Visual SFM/CMPMVS to create orthorectified photos.
Getting Started
- Install Android Developer Studio
- Install Java SDK x64
- Enable Developer Options on Android Phone. Do this by tapping the About Phone 7 times until this menu becomes available.
- Install USB Driver for the Samsung Galaxy S4 SCH-I545
- Develop. Make sure to save work in a Git repository such as Github.
Monday, November 10, 2014
Seek Thermal Imaging Camera
Just received my Seek Thermal Imaging Camera on 11/6/14. Here are the specifications:
Other features include:
Spatial Resolution | 206 x 156 (32,136 pixels) |
Misc | 12 mu pixel pitch |
Sensor | Vanadium Oxide Microbolometer |
Lens | Cholcogenide Lens |
FOV | 36 degrees |
Spectral Resolution | 7.2 - 13 microns |
OS | Android 4.3 or higher with mUSB |
Temperature Range | -40C to 330C |
Price | $200 USD |
Weight | 14.1 g |
- Watermark images with GPS coordinates
- Display temperature of an object
- Adjust color pallets
- Auto seek high and low temperatures
- Thresholding above, equal to, or below a limit
![]() |
Michigan Tech MEEM Building |
Galaxy S4 with Seek Thermal Camera |
![]() |
My truck interior on a cold evening |
Project Hardware and Software
Here is a summary of the software and hardware I've been working with recently.
Software
Autodesk Recap - This is cloud based processing of imagery and produces 3D models.
- Cost: Free with Academic License
- Pros: Does not require hardware to process imagery
- Cons: Requires time to upload imagery
Autodesk Revit - This is building information modeling software that can be used to make 3D layouts from the Recap output.
Agisoft Photoscan - This is a commercial product. There are two versions: Standard and Professional.
- Cost: 600 - 3500 USD. Free demo but lacks ability to export results.
- Pros: Ease of use. Does a great job processing with default parameters.
- Cons: Requires lots of RAM. 16 GB should be minimum.
VisualSFM/CMPMVS - This is an open source product. This software allows the creation of 3D Models, orthorectified imagery, digital surface models.
- Cost: Free.
- Pros: Free.
- Cons: Requires lots of disk space and CPU. Can be intimidating.
Adobe Lightroom - This was useful for the DJI Phantom lens correction profile.
- Cost: 79 USD
- Pros: Lens correction profile for DJI camera. Easy to use contrast and histogram editing.
- Cons: Cost.
Meshlab - This is software for editing 3D models. It can be used to clean up artifact from your models
Hugin - This is a free open source software for stitching images together. It is often used for creating panoramic photos.
Microsoft ICE - Like Hugin, this is a free software (closed source) for stitching images together. It can create panoramic imagery that is compatible with Photosynth.
Three.JS - This is a Javascript framework for working with WebGL. It allows 3D visualization of the OBJ files.
RealFlight 7.5 - This is software used to practice flying. It has very realistic physics and supports many types of vehicles from tri/quad/hex copters to fixed wing aircraft and traditional helicopters.
Hardware
DJI Phantom 2 Vision Plus
Blade Nano QX
RealFlight 7.5
Canon SX260
Samsung Galaxy S4
Seek Thermal Imaging Camera
- Cost: 1700 USD including Pelican case, prop guards, and extra battery, charger.
- Pro: Ease of use. Resale value. FPV/Telemetry
- Con: Camera quality. Lack of mission planning software.
Blade Nano QX
- Cost: 90 USD
- Pro: Easy to fly. Can fly indoors. Has prop guards
- Con: None. This is an excellent physical training quadcopter.
RealFlight 7.5
- Cost: 179 USD
- Pro: Practice flying without damaging apparatus. Elite Interlink edition has a controller similar to Futaba radio.
- Con: Price.
Canon SX260
- Cost: 250 USD
- Pro: Canon CHDK. Image quality. Ability to remove infrared filter.
- Con: Weighs a hefty 230g.
Samsung Galaxy S4
- Cost: Used 75 USD.
- Pro: Sensors, weight, Android SDK.
- Con: None so far.
Seek Thermal Imaging Camera
- Cost 200 USD
- Pro: True longwave infrared thermal imaging in 7.5 - 13 micron range.
- Con: Resolution.
Subscribe to:
Posts (Atom)