Geospatial Resources

My day job involves doing a lot of GIS work (that’s Geospatial Information Science, for you non-map nerds). Along the way, I’ve collected a lot of interesting data sources for building maps.

I’ve decided to organize them all here. I believe that maps tell a story and by providing tools to build good maps, we can tell better stories.

Some of these links may be broken in the future or the websites undermaintained. If you want to submit a website, get in contact with me.

Data.gov

The U.S. Government has a ton of geospatial data. The biggest challenge is finding it. Fortunately, the recent development of Data.gov has made it slightly easier. Data.gov is supposed to be the one-stop location for all public data, however there are still a lot of individual repositories of data out there that aren’t feeding into it.

Be sure you select the geospatial tag when filtering search results.

Links:

NASA

You already know who NASA is. They operate a very nice website called NEO (for NASA Earth Observations). In particular, the true color 1-day imagery is pretty amazing. They also run a separate website called EarthData which has even more imagery.

Imgur VIIRS 1-day for July 17, 2015

Links:

US Department of State

A lot of the data from individual branches of the federal government gets syndicated to Data.gov, but often they get lots in the mix. The Department of State now publishes to the State GeoNode server.

Imgur Syria Internally Displaced Person (IDP) Sites, April 16, 2015.

Links:

United States Geological Survey

The United States Geological Survey also warehouses a ton of data and makes it available through their Earth Resources Observation and Science (EROS) webpage. It requires registration to download the data and I’m not sure if it gets syndicated to Data.gov.

Links:

U.S. State Governments

Ideally, all state GIS data should be going to Data.gov, but it can usually be easier to get the data directly from the state government’s website. MIT has collected the web site for every single state GIS group to help you cut down on your searches.

Links:

National States Geographic Information Council

While they have a boring name, the NSGIC has a really cool database called the GIS Inventory.

Links:

United Arab Emirates

The UAE has spent a lot of effort to build out geospatial collections of the country, which results in some breathtaking imagery of this nation. I suppose if you’re building the world’s tallest skyscrapers and creating artificial islands, you want to show them off.

The Abu Dhabi Geospatial Portal is a good starting point to explore some of this data. Despite the name, it has a lot of imagery for all of the UAE including Dubai.

Imgur Ferrari World at Abu Dhabi, UAE

Links:

OpenTopography

Funded by the National Science Foundation, OpenTopograhy is a website that allows you to search through a lot of topography data. They have a very nice interface for finding terrain data and then processing it into one download.

Links:

DigitalGlobe

Being a commercial company, DigitalGlobe doesn’t just give away all their imagery. They do have a few good samples of large cities, plus a gallery of their work.

Links:

NOAA

The National Oceanic and Atmospheric Administration (NOAA) is a government agency that deals with weather, the ocean, and the earth in general. It’s not surprising that they have a good amount of free GIS sources. One of their nicer sources is nowCOAST, a group of services for providing near real-time observations and weather watches and warnings. If you’re a weather nerd, this is some awesome data. They provide a map viewer, but the raw feeds are more interesting.

Imgur NOAA Near Real-Time Observations service weather mosaic overlaid on the Mississippi River

Links:

A Geospatial View of the United Arab Emirates

I do a lot of work with geospatial data at my day job. Occasionally, I’ll find a cool data set and I’ll save it off so I can spend some time looking at it and seeing if I can find anything particularly interesting. I found an ArcGIS server hosted by the United Arab Emirates that hosts some very nice imagery of the country.

The imagery is part of the Abu Dhabi Geospatial Portal. Despite the name, the project actually has imagery from all over the UAE, including Dubai.

According to the metadata, this imagery comes from WorldView-2. You can click any of the images to get the full size version.

Here’s a wide shot of the city of Abu Dhabi. In case you’re wondering, Abu Dhabi is not the same place as Dubai. It’s southwest of Dubai and is the country’s second biggest city and capital. This is a beautiful shot of the city meeting the Persian Gulf. But wait… what’s that red triangle in the middle?

Imgur

Zoom in!

Imgur

Is that a Ferarri logo?

Imgur

This is Ferrari World Abu Dhabi. The world’s largest indoor theme park and currently the only Ferrari theme park. There appears to be a racetrack nearby for visitors to test drive a high-end sports car.

Imgur

Moving to Dubai in the northeast, we can see the famous man-made island archipelago, The World. It doesn’t seem to be doing very well. In addition to being underdeveloped, the islands appear to be sinking into the ocean.

Imgur

At least one person seems to have made an investment in a private island in the North America section.

Imgur

There are no roads connecting this to the mainland. The only way on or off this island is by boat or helicopter (note the helipad on the western part of the island). I guess if you like privacy, then this is worth it.

Imgur

The Palm Jumeirah appears to be faring much better. This is one of two palm-shaped island formations (the other being the Palm Jebel Ali). A third one was planned, but has not been developed.

Imgur

A detailed look at the spine of the Palm. It’s mostly developed, although you can see one of the fronds is still empty.

Imgur

The most expensive houses are on the tips of the fronds. This one, on the eastern half, is particularly impressive. While the outer ring of islands is made up of hotels, the inner fronds are private residences.

Imgur

The canal work in downtown Dubai is impressive. This image also has the most famous landmark in the entire UAE…

Imgur

A more detailed look at the Burj Khalifa, the tallest man-made structure in the world.

Imgur

I honestly don’t know what this is; some kind of racetrack perhaps. It has smaller concentric tracks within it.

Imgur

Some agriculture imagery. As you can image, there’s not a lot of this in the desert. Center pivot irrigation, the style of farming here, makes for those circles which are fun to look at.

Imgur

A very wide image of a highway outside of Dubai. I like this because it shows they very carefully planted trees around the edges of the highway. It’s probably to keep sand from blowing on the road, but I’m guessing it also creates an illusion that you’re not actually driving through the desert.

Imgur

When planning neighborhoods and infrastructure, these UAE engineers really like geometric patterns, particularly concentric circles.

Imgur

More ornate geometric patterns, this time in the form of a highway interchange.

Imgur

I don’t know what this is. It appears to be some circular roads with some well placed trees extending outwards, but I can’t tell if this is supposed to be a neighborhood development. I think the engineers are just messing with me at this point.

Imgur

This looks like some kind of resort hotel.

Imgur

These are some very big yachts. They don’t appear very large until you compare them to the size of the cars parked nearby.

Imgur

I would occasionally find these compounds that are fenced off. I didn’t look up what they were.

Imgur

There was a lot of other stuff in the data set. I might return to it and try to find some more crazy stuff. If you’re into this kind of thing, then you should consider learning about geographic information systems (GIS).

Imagery Credit: Abu Dhabi Geospatial Portal

Things I Wish I had Known about AngularJS

Your first AngularJS app is going to suck.

And so will your second. And third. And probably every single one until you finally understand the framework at an intimate level. Angular is challenging; it will take weeks of development just to learn the basics and years of building non-trivial applications to actually become comfortable with it. There’s a joke that everyone loves to use Angular for new applications, but no one wants to maintain them. Because Angular is so new, I feel like a lot of Angular codebases are people’s first projects. And we’ve already established those are going to suck.

Over the past few years, I’ve been compiling a list of tips and tricks I’ve painfully learned when writing AngularJS code. My hope is that one day I’ll be able to hop in a time machine and slap my younger self with this before I unleash more terrible AngularJS code to humanity. Even if that never happens, you too can benefit from it!

Imgur Oh good, they found someone to maintain your first AngularJS codebase.

Keep controllers lightweight

The most common newbie mistake I run across is seeing controllers that contain thousands of lines of code. Most AngularJS tutorials start off by explaining how a controller allow you to put variables in scope and link them to your view, so the new developer begins thinking “okay, this must be where I put all my application logic”.

There’s a few problems with this approach:

  • Controllers are not as re-usable as you might think. They typically have only one place to live in the application’s architecture and they are fairly tightly coupled to that.
  • Controllers are slightly more difficult to unit test and they usually involve mocking out the $scope.
  • Controllers can be nested which means the executing environment might change; the downside is now your $scope is different and you could be looking at re-writing a lot of those aforementioned unit tests.

The solution is pretty simple: just use the service and factory AngularJS components and inject them into your controllers. They’re designed for this kind of thing. A well-written factory or service is very easy to unit test and is far more re-usable than controller code.

Here’s a good example. Let’s say you have a really basic web app that grabs a blog feed (as JSON), filters it down by date (passed via $routeParams), and renders it to a web page.

Bad version:

angular.module('blog-app', [])

.controller('BlogCtrl', function($scope, $http, $routeParams) {
  // Grab the start and end date from the route
  var start = $routeParams.start,
    end = $routeParams.end,
    range = moment().range(start, end);

  $http.get('/entries').then(function(entries) {
     entries.forEach(function(entry) {
       if (range.contains(entry.date)) {
         $scope.entries.push(entry);
       }
    });
  });
});

In this example, the logic for grabbing the entries and filtering them is tightly coupled to the controller, so we’re not really building reusable code. In addition, the code is tightly coupled to both a hard coded endpoint URL string and the route parameters. For small trivial apps, this isn’t a big deal, but as your web apps grow larger, this becomes awful to maintain. As a general rule of thumb, I never inject $http into controllers. Instead, bundle all of that interaction up into a factory:

Better version:

angular.module('blog-app', [])

.factory('EntryEndpoint', function($http) {
  return function(endpointURL) {
    this.getEntries = function(start, end) {
      var range = moment().range(start, end);
      var defer = $q.defer();
      var result = [];
      $http.get('/entries').then(function(entries) {
        entries = entries.map(function(entry) {
          return range.contains(entry.date);
        });
        defer.resolve(result);
      });
      return defer.promise;
    };
  };
})

.controller('BlogCtrl', function($scope, $routeParams, EntryEndpoint) {
  var start = $routeParams.start,
    end = $routeParams.end;

  var endpoint = new EntryEndpoint('/entries');
  endpoint.getEntries(start, end).then(function(entries) {
    $scope.entries = entries;
  });
});

This “better version” is a lot longer and appears more complicated, but the payoff is that we now have a clean separation between our components; the controller hands off the logic to factory which doesn’t care about a lot of the other things (like the route parameters or the current scope). We can also re-use this code in other controllers in other places.

Keep the logic out of your templates

One of the reasons I despise PHP is that it encourages you to mix your logic with your HTML code. This is basically just setting you up to write big monolithic and unmaintainable applications. AngularJS’s templating functionality, while being very good, can also lead you to load up your templates with a lot of logic that really doesn’t belong there. The ngIf directive in particular is a beast because if you rely on it too much, you might just have a huge chain of ngIf directives in your templates, which isn’t clean or fun to maintain. Angular also lets you manipulate the scope variables (or create new ones) from within the template code which is also something I’m not a fan of.

Here’s some template code that I’m used to seeing…

Bad version:

<ul>
  <li ng-repeat="entry in entries">
    <div ng-if="entry.attribute === 'valueOne'">   <!-- do something --> </div>
    <div ng-if="entry.attribute === 'valueTwo'">   <!-- do something else --> </div>
    <div ng-if="entry.attribute === 'valueThree'"> <!-- do yet another thing --> </div>
  </li>
</ul>

This is a really simple case, so it doesn’ts look too bad, but if you continue adding logic here, you’ll eventually get into nested ng-if statements or complicated boolean expressions that could otherwise be elegantly expressed in your controller code.

So how do you avoid junking up your templates with logic? I have a few methods I like:

  • If you find yourself using a lot of logic to build out a view stack as your application state changes, then you probably should be using ngRoutes to cleanly define how your view changes between application states.
  • Define your JavaScript objects to adhere to an interface and keep your view code from needing to know the implementation; this is just good development anyway.
  • Angular directives allow you to couple together your logic and view together (when it makes sense to) into a maintainable package. When you use directives with ngTransclude, you end up with a very powerful way to write clean code that is also highly flexible. More on that later.

Favor Factories over Services

This may be more of a personal preference than a hard rule, but I think Factories are underused when it compared to Services. The big difference between the two is that Factories can return a constructor definition which can be instantiated with new. We can, of course, pass parameters into the constructor, to custom tailor the object to the current situation, rather than just writing a giant service that tries to be one-size-fits-all.

This isn’t a 100% of the time rule (since Services are nice for simple interactions), but it’s really clean in the long run when you’re trying to build out your app’s domain objects.

Mixing AngularJS and jQuery

Here’s a fun piece of trivia: AngularJS includes a copy of jqLite and it will bind it to the global $ variable. If you include jQuery before you include the AngularJS library, it will use that instead. Because jqLite isn’t completely compatible with jQuery, I make sure that I include jQuery first so it doesn’t have any adverse effects on other plugins.

While on the topic of jQuery, it should be noted that AngularJS and jQuery aren’t incompatible libraries, however I really don’t like mixing jQuery DOM manipulation code with my AngularJS code. I see a ton of this in AngularJS code written by people who are new to the framework because they have exerience with jQuery and they used to solving all their problems with it. DOM manipulation within your Angular code tends to be bad since it couples your DOM view code to your logic.

There is one good exception, however: Angular directives give you access to the DOM element in the link function. Go crazy with DOM manipulation there. Other than that, I avoid it. If you absolutely must do some jQuery magic to manipulate the DOM, at least restrict it to controller code, which already has a tighter coupling to the view code anyway.

One module per file.

Prepare your hate mail! I am not a fan of splitting a module up into multiple files. Most novice AngularJS developers don’t even know this is possible, but it’s perfectly fine to have a file called first.js with:

angular.module('my-module', []).controller('MyController', function() {});

and another file called second.js with:

angular.module('my-module').controller('AnotherController', function() { });

A lot of people probably like this; it keeps your files small and normally that’s a good thing. The nastiness comes from the fact that you need to be absolutely sure that the first.js file is loaded before the second.js file, otherwise you’ll get a dreaded $injector:nomod error. You see, that first call to angular.module has two arguments, so we’re defining the module there. The second call just allows us to continue the module. If you’re including your .js files in a page, it’s not too bad to make sure this happens, but if you’re using something like grunt, you need to be sure your modules are being concat’d in the right order (the default is alphabetical). Some people get around this by creating “meta-modules” that just exist as dependency holders. This isn’t terrible, but I think it’s just going to bloat up your project structure. My personal preference is to keep things as one module per file.

Use ngRoutes

Someone once told me they didn’t use routes because it made their URLs look ugly. A small part of me died that day.

Routes are incredibly useful for building out your application states. I like to design my routes as early as possible in the application design since it’s much tougher to add them later. Even though the ngRoutes module isn’t part of the core AngularJS library, you should be using it.

Also, if you’re still hung up on ugly URLs, you can always switch it to HTML5 mode or opt for the Angular-UI-Route 3rd party library.

Don’t inject $routeParams

…into anything other than controllers. It’s really tempting to just inject that thing anywhere you need it, but injecting that provider into, say, a factory or a service, just tightly couples that component to your specific route.

Inject $routeParams into your controller code and have that pass the route variables to your individual objects in other ways (like in a Factory constructor, because those are a good idea, right?).

Don’t manipulate things in $rootScope

This is fairly obvious. The $rootScope is basically a global variable space that gets inherited for all child scopes. Putting things in the $rootScope isn’t too terrible (as long as you’re only doing it in one centralized place), but changing those variables will now change them for all scopes. You’re basically dealing with global variables at this point. Nested Controllers make things more complicated

Being able to nest controllers is pretty damn cool. I’m okay with it. In fact, you probably can’t write a modern web application without doing it. Just realize that doing this is going to increase the complexity of your application. If you’re nesting controllers four or five levels deep, then you probably need to think about your web app’s architecture.

Nested controllers tend to be coupled together, which is bad for all the obvious reasons.

Try to avoid using $scope.$parent

A newbie mistake that I see is trying to manipulate a variable in the current scope without realizing that things like ngRepeat create a new child scope. The less-than-ideal solution is to manipulate it with $scope.$parent. This is not an elegant solution, particularly if this scope is more deeply nested (then you get things like $scope.$parent.$parent.$parent).

My preferred solution is to wrap these variables in an outer object so that the child scope inherits a copy of that object which has the pointer to the actual variable. This is not the most intuitive thing to understand, but it’s far more maintainable than a family tree of parent scopes.

Directives are really cool

They are also incredibly complicated and difficult to learn, but if you can master them, you basically can have a deeper appreciation for how AngularJS works. One aspect of directives is particularly cool: ngTransclude. While directives are usually self-contained and tightly encapsulated components of your application, transclusion allows you to elegantly add on to the directive to tailor it to your specific use case. It’s too detailed to try to explain here, but it’s seriously a pretty cool way to build some sweet directives.

Conclusions

AngularJS is a beast. The learning curve on this library is insane. I feel that by learning it, however, you will become a better developer and you’ll curate your own lists like this. Hopefully you can pass your own experiences along to others.

Photo Credit: Landfill, by the Wisconsin Department of Natural Resources.

I'm not done talking about terrible project layouts

This is part two of a series where I wanted to write about how doing small things when setting up your project can have surprisingly positive effects.

This material was originally discussed at the NOVA-Python meetup by organizer, Ryan Day. His original focus was on developing good project layouts for Python + Flask web applications, but I think the material works well in a variety of environments.

Learning through Play

In high school, I wasn’t great at math. Listening to a math lecture was just boring. On top of that, my dad was a math teacher so I’d get math lectures at home as well. I think math is inherently a really bad subject to teach through the standard lecture format; it’s unfortunate that it’s still being taught that way. So while math lectures couldn’t keep my attention, I had a really dorky fascination with plugging things into the ol’ TI-83 graphing calculator and seeing what would come out. As the teacher was trying to drill into our heads what y=mx^2+b meant and how the quadratic formula could solve that, I thought it was far more interesting to just plug that into the graphing calculator and watch the points cascade over that small LCD screen. Then I’d tweak the variables around just to see what would happen.

Yeah, this is really nerdy. It’s no surprise I eventually decided to study computer science.

What I had no clue I was doing was actually learning in a much better way. Instead of just learning a formula, I was interacting with it; in a more whimsical version, I was playing with it. It’s only been recently discovered that this is not only a legitimate way to learn, it’s also really effective, particularly when it comes to software development (Sonmez).

Making your Project Friendly for Interactive Learning

This part’s incredibly easy, and yet not too obvious. When we join a project and check out the code for the first time, we tend to see a directory structure that has the following things:

  • The source code itself
  • Supporting documentation
  • Configuration files
  • Unit tests

What’s missing is a place for the developers to play around. Now, any developer can create this, but it’s a nice psychological boost to communicate that it’s okay to just play with the project and learn about it outside of running unit tests or wherever the program’s main entry point starts.

To make this work, just create an empty directory and name it whatever works. For Python projects, I like the name user-scripts. For Node.js projects, I like user-bin. Now toss a README.md file in that directory that tells the developer “Hey, this directory is for you. Go crazy. Write a bunch of weird one-off code that will help you learn about the project. Nothing in here will be saved to version control.”

Now the important part is to check that directory into your version control, and then immediately add it to your .gitignore.

app
    - scripts
    - styles
    - libs
    - docs
    - test
    - user-scripts

Now your new developers have a place where they can write code, play with your project’s internal libraries, and just get a feel for things through interaction.

Yeah, It’s not a big change and it’s not exactly profound or novel. Maybe your new developers are already doing this on their own, but in my experience a lot of new developers don’t have the instinct to try this out. There may be an implicit fear of breaking something or accidentally checking in something that will make them look dumb in front of the more experienced developers. They may feel that they shouldn’t be spending a lot of time writing code that isn’t going to make it into production. If you can break this perception, then you can really make joining a new project easier and more fun.

Bibliography

Sonmez, John. Z. Soft Skills: The Software Developer’s Life Manual. Manning Publications, 2015. 223. Print.

Designing project layouts that aren't terrible

Making the New Guy’s First Week Less Painful

The most awkward part of getting a new developer on a project is that first week (or so) where he or she needs to get familiar with the code, figure out how to run things, and generally get a feel for how the project is laid out.

This is always an anxious period of time for me (both as the new developer and as the one trying on bring someone onboard) because there’s not really a good way to do it.

I was recently at a meetup session of NOVA-Python, the Northern Virginia Python Users Group. The organizer, Ryan Day, had some good tips on building out a project structure that will make it easier for new developers on a project to jump in.

Two things in particular stood out as being particularly useful and I wanted to expand on them. This is the first of two entries.

The Config File as a Learning Experience

Maintaining configuration files kind of sucks. As developers, we try to not hardcode values that need to change and they inevitably wind up in some kind of external configuration. The problem is that now we have a config file that we check into version control and have to remember to change the values in each time we deploy code to a new environment.

If you’re like most developers, you probably populate a copy of the configuration file(s) with some sane values that work for 75% of the development team and hope that no one shows up with a different set up.

When buidling Flask apps, I typically have a config.py file that looks something like this:

DB_SERVER = 'localhost'
DB_PORT = 5432
DB_NAME = 'flaskdb'
DB_USER = 'austin'
DB_PASSWORD = 'p0stgres'

# more config ...

… and somewhere in my Flask app, I just read in that file:

app.config.from_pyfile('config.py')

I’m here to argue that checking things in like this is an anti-pattern. Here’s my better idea: create a config.sample.py file that doesn’t contain real values, but walks the new developer through the whole thing.

# The application requires a database connection
# See the db-setup.py script for setting one up.
# Change these values to match your development database.
# Need help setting up the database? Check out DATABASE.md.

DB_SERVER = 'db-hostname-here'
DB_PORT = 5432
DB_NAME = 'dev-db-name-here'
DB_USER = 'flaskapp'
DB_PASSWORD = 'change-this'

# more config ...

The idea is that the new developer should copy this into config.py and change the values while reading it over. Without having an actual config.py checked into version control, the new developer has to spend a few minutes reading over the configuration file.

Now to make this even more developer friendly, throw a message in the error that will be called in case they forget to set up their config.

try:
    app.config.from_pyfile('config.py')
except:
    print 'Error loading config.py. Did you remember to copy config.sample.py?'
    sys.exit(1)

Conclusion

So as end result, you have an extra step toward getting your application up and running. For small apps, this might be kind of annoying, but for larger apps that have a lot of configuration, this can help the new guy.