Python Basics#
the following cells have been obtained from the python introduction notebook of Jeffrey Kantor. These examples and urther can be fund under jckantor/CBE30338
variables and primitive data types#
#A variable stores a piece of data and gives it a name
answer = 42
#answer contained an integer because we gave it an integer!
is_it_thursday = True
is_it_wednesday = False
#these both are 'booleans' or true/false values
pi_approx = 3.1415
#This will be a floating point number, or a number containing digits after the decimal point
my_name = "Jacob"
#This is a string datatype, the name coming from a string of characters
#Data doesn't have to be a singular unit
#p.s., we can print all of these with a print command. For Example:
print(answer)
print(pi_approx)
42
3.1415
List and Dictionaries#
#What if we want to store many integers? We need a list!
prices = [10, 20, 30, 40, 50]
#This is a way to define a list in place. We can also make an empty list and add to it.
colors = []
colors.append("Green")
colors.append("Blue")
colors.append("Red")
print(colors)
#We can also add unlike data to a list
prices.append("Sixty")
#As an exercise, look up lists in python and find out how to add in the middle of a list!
print(prices)
#We can access a specific element of a list too:
print(colors[0])
print(colors[2])
#Notice here how the first element of the list is index 0, not 1!
#Languages like MATLAB are 1 indexed, be careful!
#In addition to lists, there are tuples
#Tuples behave very similarly to lists except that you can't change them
# after you make them
#An empty Tuple isn't very useful:
empty_tuple = ()
#Nor is a tuple with just one value:
one_tuple = ("first",)
#But tuples with many values are useful:
rosa_parks_info = ("Rosa", "Parks", 1913, "February", 4)
#You can access tuples just like lists
print(rosa_parks_info[0] + " " + rosa_parks_info[1])
# You cannot modify existing tuples, but you can make new tuples that extend
# the information.
# I expect Tuples to come up less than lists. So we'll just leave it at that.
['Green', 'Blue', 'Red']
[10, 20, 30, 40, 50, 'Sixty']
Green
Red
Rosa Parks
Conditions, Logical operators and If-Statements#
The word “if” is a keyword. When Python sees an if-statement, it will determine if the associated logical expression is true. If it is true, then the code in code block will be executed. If it is false, then the code in the if-statement will not be executed. The way to read this is “If logical expression is true then do code block.”
When there are several conditions to consider you can include elif-statements; if you want a condition that covers any other case, then you can use an else statement.
# To apply conditions in python the if statement is used.
# The statement is based on boolean functions yielding either true or false
#, these includes the following examples logical expressions.
# == (equal), != (does not equal), < (smaller than), <= (equal or smaller than)
# Individual expressions can be combined utilizing logical operators including
#, e.g. "and" / "or" a more detailed description can found under https://pythonnumericalmethods.berkeley.edu/notebooks/chapter01.05-Logial-Expressions-and-Operators.html
if 5 > 2:
print("Five is greater than two!") # please note that one as to indent the code block after a condition (4 spaces or 1 tab)
if 1 == 1:
print("One is equal to one")
if 5 != 2 and 5 > 2:
print("5 does not equal 2 and 5 is greater than two!" )
if 5 < 2:
print("Five is smaller than two!")
else:
print("Five is NOT smaller than two!")
Five is greater than two!
One is equal to one
5 does not equal 2 and 5 is greater than two!
Five is NOT smaller than two!
Getting started with GeoData#
the following part has been obtained from the official geopandas introduction notebook. You can find the original and further examples under this link https://geopandas.org/en/stable/getting_started/introduction.html
Introduction to GeoPandas#
This quick tutorial introduces the key concepts and basic features of GeoPandas to help you get started with your projects.
Concepts#
GeoPandas, as the name suggests, extends the popular data science library pandas by adding support for geospatial data. If you are not familiar with pandas, we recommend taking a quick look at its Getting started documentation before proceeding.
The core data structure in GeoPandas is the geopandas.GeoDataFrame, a subclass of pandas.DataFrame, that can store geometry columns and perform spatial operations. The geopandas.GeoSeries, a subclass of pandas.Series, handles the geometries. Therefore, your GeoDataFrame is a combination of pandas.Series, with traditional data (numerical, boolean, text etc.), and geopandas.GeoSeries, with geometries (points, polygons etc.). You can have as many columns with geometries as you wish; there’s no limit typical for desktop GIS software.
Each GeoSeries can contain any geometry type (you can even mix them within a single array) and has a GeoSeries.crs attribute, which stores information about the projection (CRS stands for Coordinate Reference System). Therefore, each GeoSeries in a GeoDataFrame can be in a different projection, allowing you to have, for example, multiple versions (different projections) of the same geometry.
Only one GeoSeries in a GeoDataFrame is considered the active geometry, which means that all geometric operations applied to a GeoDataFrame operate on this active column.
See more on data structures in the User Guide.
Let’s see how some of these concepts work in practice.
Reading and writing files#
First, we need to read some data.
Reading files#
Assuming you have a file containing both data and geometry (e.g. GeoPackage, GeoJSON, Shapefile), you can read it using geopandas.read_file(), which automatically detects the filetype and creates a GeoDataFrame. This tutorial uses the "nybb" dataset, a map of New York boroughs, which is part of the GeoPandas installation. Therefore, we use geopandas.datasets.get_path() to retrieve the path to the dataset.
import geopandas
path_to_data = geopandas.datasets.get_path("nybb")
gdf = geopandas.read_file(path_to_data)
gdf
/tmp/ipykernel_99/590640026.py:3: FutureWarning: The geopandas.dataset module is deprecated and will be removed in GeoPandas 1.0. You can get the original 'nybb' data from the geodatasets package.
from geodatasets import get_path
path_to_file = get_path('nybb')
path_to_data = geopandas.datasets.get_path("nybb")
| BoroCode | BoroName | Shape_Leng | Shape_Area | geometry | |
|---|---|---|---|---|---|
| 0 | 5 | Staten Island | 330470.010332 | 1.623820e+09 | MULTIPOLYGON (((970217.022 145643.332, 970227.... |
| 1 | 4 | Queens | 896344.047763 | 3.045213e+09 | MULTIPOLYGON (((1029606.077 156073.814, 102957... |
| 2 | 3 | Brooklyn | 741080.523166 | 1.937479e+09 | MULTIPOLYGON (((1021176.479 151374.797, 102100... |
| 3 | 1 | Manhattan | 359299.096471 | 6.364715e+08 | MULTIPOLYGON (((981219.056 188655.316, 980940.... |
| 4 | 2 | Bronx | 464392.991824 | 1.186925e+09 | MULTIPOLYGON (((1012821.806 229228.265, 101278... |
Reading Geopackage#
path_to_data = "please insert here"
gdf_gpkg = geopandas.read_file(path_to_data, driver = "GPKG")
gdf_gpkg
---------------------------------------------------------------------------
CPLE_OpenFailedError Traceback (most recent call last)
File fiona/ogrext.pyx:136, in fiona.ogrext.gdal_open_vector()
File fiona/_err.pyx:291, in fiona._err.exc_wrap_pointer()
CPLE_OpenFailedError: please insert here: No such file or directory
During handling of the above exception, another exception occurred:
DriverError Traceback (most recent call last)
Cell In[5], line 2
1 path_to_data = "please insert here"
----> 2 gdf_gpkg = geopandas.read_file(path_to_data, driver = "GPKG")
4 gdf_gpkg
File /usr/local/lib/python3.10/site-packages/geopandas/io/file.py:297, in _read_file(filename, bbox, mask, rows, engine, **kwargs)
294 else:
295 path_or_bytes = filename
--> 297 return _read_file_fiona(
298 path_or_bytes, from_bytes, bbox=bbox, mask=mask, rows=rows, **kwargs
299 )
301 else:
302 raise ValueError(f"unknown engine '{engine}'")
File /usr/local/lib/python3.10/site-packages/geopandas/io/file.py:338, in _read_file_fiona(path_or_bytes, from_bytes, bbox, mask, rows, where, **kwargs)
335 reader = fiona.open
337 with fiona_env():
--> 338 with reader(path_or_bytes, **kwargs) as features:
339 crs = features.crs_wkt
340 # attempt to get EPSG code
File /usr/local/lib/python3.10/site-packages/fiona/env.py:457, in ensure_env_with_credentials.<locals>.wrapper(*args, **kwds)
454 session = DummySession()
456 with env_ctor(session=session):
--> 457 return f(*args, **kwds)
File /usr/local/lib/python3.10/site-packages/fiona/__init__.py:292, in open(fp, mode, driver, schema, crs, encoding, layer, vfs, enabled_drivers, crs_wkt, allow_unsupported_drivers, **kwargs)
289 path = parse_path(fp)
291 if mode in ("a", "r"):
--> 292 colxn = Collection(
293 path,
294 mode,
295 driver=driver,
296 encoding=encoding,
297 layer=layer,
298 enabled_drivers=enabled_drivers,
299 allow_unsupported_drivers=allow_unsupported_drivers,
300 **kwargs
301 )
302 elif mode == "w":
303 colxn = Collection(
304 path,
305 mode,
(...)
314 **kwargs
315 )
File /usr/local/lib/python3.10/site-packages/fiona/collection.py:243, in Collection.__init__(self, path, mode, driver, schema, crs, encoding, layer, vsi, archive, enabled_drivers, crs_wkt, ignore_fields, ignore_geometry, include_fields, wkt_version, allow_unsupported_drivers, **kwargs)
241 if self.mode == "r":
242 self.session = Session()
--> 243 self.session.start(self, **kwargs)
244 elif self.mode in ("a", "w"):
245 self.session = WritingSession()
File fiona/ogrext.pyx:588, in fiona.ogrext.Session.start()
File fiona/ogrext.pyx:143, in fiona.ogrext.gdal_open_vector()
DriverError: please insert here: No such file or directory
Reading GeoJSONs#
path_to_data = "please insert here"
gdf_json = geopandas.read_file(path_to_data,driver = 'GeoJSON')
gdf_json
Reading Shapefile#
path_to_data = "please insert here"
gdf_shp = geopandas.read_file(path_to_data,driver = 'ESRI Shapefile')
gdf_shp
Writing files#
To write a GeoDataFrame back to file use GeoDataFrame.to_file(). The default file format is Shapefile, but you can specify your own with the driver keyword.
gdf.to_file("data\my_file.geojson", driver="GeoJSON")
writing to GeoPackage#
#in the case of geopackages one in addition has do define the layer
gdf.to_file("data\my_file.gpkg",layer='newyork', driver="GPKG")
See more on reading and writing data in the User Guide.
Simple accessors and methods#
Now we have our GeoDataFrame and can start working with its geometry.
Since there was only one geometry column in the New York Boroughs dataset, this column automatically becomes the active geometry and spatial methods used on the GeoDataFrame will be applied to the "geometry" column.
Measuring area#
To measure the area of each polygon (or MultiPolygon in this specific case), access the GeoDataFrame.area attribute, which returns a pandas.Series. Note that GeoDataFrame.area is just GeoSeries.area applied to the active geometry column.
But first, to make the results easier to read, set the names of the boroughs as the index:
gdf = gdf.set_index("BoroName")
gdf["area"] = gdf.area
gdf["area"]
Getting polygon boundary and centroid#
To get the boundary of each polygon (LineString), access the GeoDataFrame.boundary:
gdf['boundary'] = gdf.boundary
gdf['boundary']
Since we have saved boundary as a new column, we now have two geometry columns in the same GeoDataFrame.
We can also create new geometries, which could be, for example, a buffered version of the original one (i.e., GeoDataFrame.buffer(10)) or its centroid:
gdf['centroid'] = gdf.centroid
gdf['centroid']
Filtering geodataframes#
To get rid of unnecessary information or compare specific rows, we need to filter our dataframes further. To achive this the following cells will demonstrate how to extract certain rows/columns/cells using the GeoDataFrame.loc() fucntion. In addition we will take a look how we can apply logical conditions and geometry relations through a mask to our dataframe
#selecting a specific column
gdf.loc["Brooklyn"]
# selecting a specific row
gdf.loc[:,"geometry"]
#selecting a specific column and row
gdf.loc["Brooklyn","geometry"]
#selecting multiple rows
gdf.loc[["Brooklyn","Queens"]]
# filtering for a specific condition using masks
mask = gdf["BoroCode"]>=3
mask
gdf[mask]
queens = gdf.loc["Queens"]
gdf[~gdf.intersects(queens.centroid)] # the ~ negates a condition in pandas
Measuring distance#
We can also measure how far each centroid is from the first centroid location.
first_point = gdf['centroid'].iloc[0]
gdf['distance'] = gdf['centroid'].distance(first_point)
gdf['distance']
Note that geopandas.GeoDataFrame is a subclass of pandas.DataFrame, so we have all the pandas functionality available to use on the geospatial dataset — we can even perform data manipulations with the attributes and geometry information together.
For example, to calculate the average of the distances measured above, access the ‘distance’ column and call the mean() method on it:
gdf['distance'].mean()
Making maps#
GeoPandas can also plot maps, so we can check how the geometries appear in space. To plot the active geometry, call GeoDataFrame.plot(). To color code by another column, pass in that column as the first argument. In the example below, we plot the active geometry column and color code by the "area" column. We also want to show a legend (legend=True).
gdf.plot("area", legend=True)
You can also explore your data interactively using GeoDataFrame.explore(), which behaves in the same way plot() does but returns an interactive map instead.
gdf.explore("area", legend=False)
Switching the active geometry (GeoDataFrame.set_geometry) to centroids, we can plot the same data using point geometry.
gdf = gdf.set_geometry("centroid")
gdf.plot("area", legend=True)
And we can also layer both GeoSeries on top of each other. We just need to use one plot as an axis for the other.
ax = gdf["geometry"].plot()
gdf["centroid"].plot(ax=ax, color="black")
Now we set the active geometry back to the original GeoSeries.
gdf = gdf.set_geometry("geometry")
See more on mapping in the User Guide.
Geometry creation#
We can further work with the geometry and create new shapes based on those we already have.
Convex hull#
If we are interested in the convex hull of our polygons, we can access GeoDataFrame.convex_hull.
gdf["convex_hull"] = gdf.convex_hull
ax = gdf["convex_hull"].plot(alpha=.5) # saving the first plot as an axis and setting alpha (transparency) to 0.5
gdf["boundary"].plot(ax=ax, color="white", linewidth=.5) # passing the first plot and setting linewitdth to 0.5
Buffer#
In other cases, we may need to buffer the geometry using GeoDataFrame.buffer(). Geometry methods are automatically applied to the active geometry, but we can apply them directly to any GeoSeries as well. Let’s buffer the boroughs and their centroids and plot both on top of each other.
# buffering the active geometry by 10 000 feet (geometry is already in feet)
gdf["buffered"] = gdf.buffer(10000)
# buffering the centroid geometry by 10 000 feet (geometry is already in feet)
gdf["buffered_centroid"] = gdf["centroid"].buffer(10000)
ax = gdf["buffered"].plot(alpha=.5) # saving the first plot as an axis and setting alpha (transparency) to 0.5
gdf["buffered_centroid"].plot(ax=ax, color="red", alpha=.5) # passing the first plot as an axis to the second
gdf["boundary"].plot(ax=ax, color="white", linewidth=.5) # passing the first plot and setting linewitdth to 0.5
See more on geometry creation and manipulation in the User Guide.
Geometry relations#
We can also ask about the spatial relations of different geometries. Using the geometries above, we can check which of the buffered boroughs intersect the original geometry of Brooklyn, i.e., is within 10 000 feet from Brooklyn.
First, we get a polygon of Brooklyn.
brooklyn = gdf.loc["Brooklyn", "geometry"]
brooklyn
The polygon is a shapely geometry object, as any other geometry used in GeoPandas.
type(brooklyn)
Then we can check which of the geometries in gdf["buffered"] intersects it.
gdf["buffered"].intersects(brooklyn)
Only Bronx (on the north) is more than 10 000 feet away from Brooklyn. All the others are closer and intersect our polygon.
Alternatively, we can check which buffered centroids are entirely within the original boroughs polygons. In this case, both GeoSeries are aligned, and the check is performed for each row.
gdf["within"] = gdf["buffered_centroid"].within(gdf)
gdf["within"]
We can plot the results on the map to confirm the finding.
gdf = gdf.set_geometry("buffered_centroid")
ax = gdf.plot("within", legend=True, categorical=True, legend_kwds={'loc': "upper left"}) # using categorical plot and setting the position of the legend
gdf["boundary"].plot(ax=ax, color="black", linewidth=.5) # passing the first plot and setting linewitdth to 0.5
What next?#
With GeoPandas we can do much more than what has been introduced so far, from aggregations, to spatial joins, to geocoding, and much more.
Head over to the User Guide to learn more about the different features of GeoPandas, the Examples to see how they can be used, or to the API reference for the details.