How To Get Client IP Address – Amazon ELB

A while ago I had investigated how to get the Client IP address when using an Elastic Load Balancer (ELB) on Amazon EC2. On a physical network it’s very easy to get the Client’s IP address on the server-side. But the same code when hosted on Amazon EC2 via an ELB would yield the Private IP address of the EC2 Instance. Some changes might be necessary to get the “true” Client IP address

First, in order to get the Client IP address, the ELB’s Protocol must be set to route at the HTTP layer instead of TCP. If your ELB is already set to route at the TCP layer, you will have to schedule some downtime to create a new ELB that routes HTTP traffic instead and transfer your instances over to that new ELB.

Once that is done, you can access the Client IP address using the header HTTP_X_FORWARDED_FOR in the client request. To get the Private IP Address (of the instance that the request was routed to), you can use the header REMOTE_ADDR in the client request.

NOTE OF CAUTION: This solution does not work for ELB that routes HTTPS traffic (since it forwards at the TCP layer). The reason is because the HTTP traffic is encrypted using SSL which can be decrypted only at the endpoints. But currently the ELB cannot perform SSL acceleration and so it cannot get the Client IP address out. Read more on Load Balancing here.
UPDATE (10/15/2010): Amazon ELB has added support for HTTPS. So you should be able to do this on an ELB that routes HTTPS as well!

Importing Shapefile into PostgreSQL from an ASP.NET web application

I’ve noticed people looking for an answer to this quite often so I’m publishing the steps to  achieve this. Hopefully this will help organizations that want their GIS users to upload GIS data into the spatial database.

NOTE: If you want to simply import a large amount of shapefiles to your PostGIS database, you can simply write a BAT file that uses shp2pgsql and call that from an executable.
This article is to solve a very specific problem…importing a Shapefile into a PostGIS database, through a web application.

To be able to import the data using this process, you must have at least these 4 components of the shapefile:
SHP, SHX, DBF & PRJ.

Step 1) Upload the files using the traditional server control to upload the files
<FileUpload ID=”FUpl1″ runat=”server” />
Make sure you check for the validity of the files

Step 2) Get the projection of the shapefile from the PRJ  file (which is a text file with one line in it, usually). I’ll leave this to you to figure this part out….shouldn’t be too hard.
Get the SRID for the projection. If this is a custom projection, make sure the SRID for this custom projection is defined on the spatial database.

Step 3) Upload your files to your server using
FUpl1.PostedFile.SaveAs(SERVER_SIDE_PATH_TO_THE_FILE);

Step 4) Convert your shapefile into SQL code using the tools provided with PostGIS.
Right….easier said than done. Using Command.exe to run an executable in a web application directly isn’t really a bright idea, is it!
Here’s how you solve that dilemma.
Write a WCF service to do what you wanted to do here in your web application. Then install that WCF service and expose it to the web application. That way you can outsource the “dangerous” work to a semi web, semi windows application that will provide you the stability and security of a windows service but expose the same to the web application as a web service! That said…I leave it up to you to securely expose the WCF service to the web application ONLY.
Here’s a clue as to what can be done inside the WCF service to import the data:
C# Code:

        public string ConvertToSql(string ShapeFileName, string srid)
        {
            string result = "";
            string sqlfilename = ShapeFileName.ToLower().Replace(".shp",".sql");
            string pathToSqlFolder = WHERE_EVER_SQL_FILES_ARE_TO_BE_STORED;

            if (!pathToSqlFolder.EndsWith("\\"))
                pathToSqlFolder += "\\";
            if (!Directory.Exists(pathToSqlFolder))
            {
                return "ERROR: SQL files folder " + pathToSqlFolder + " does not exist.";
            }

            string SqlFilePath = pathToSqlFolder + sqlfilename;
            if (File.Exists(SqlFilePath))
               File.Delete(SqlFilePath);

            string pathToShapefilesFolder = WHERE_EVER_SHAPEFILE_IS_STORED;
            if (!pathToShapefilesFolder.EndsWith("\\"))
                pathToShapefilesFolder += "\\";
            string ShapefilePath = "";
            ShapefilePath = pathToShapefilesFolder + ShapeFileName;

            string args = @" -s {0} -c {1} {2} > {3}";
            //fill in the arguments & the rest
            args = String.Format(args, srid, ShapefilePath.Replace(".shp", ""), sqlfilename.Replace(".sql", ""), SqlFilePath);

            try
            {
                string pathToImporter = System.Configuration.ConfigurationSettings.AppSettings["PathToImportTool"];
                string strConverter = "shp2pgsql.exe ";
                strConverter = pathToImporter.EndsWith(@"\") ? pathToImporter + strConverter : pathToImporter + @"\" + strConverter;
                strConverter = strConverter + args;
                Process objProcess = new Process();
                ProcessStartInfo objPSI = new ProcessStartInfo(@"c:\windows\system32\cmd.exe", "/C " + strConverter);

                objPSI.UseShellExecute = true;
                objPSI.WindowStyle = ProcessWindowStyle.Normal;
                bool blnIsReady = false;

                objProcess.StartInfo = objPSI;
                objProcess.Start();
                objProcess.WaitForExit();
                blnIsReady = objProcess.HasExited;

                if (blnIsReady)
                    result = "SUCCESS: " + sqlfilename;
                else
                    result = "ERROR: Importing of shapefile did not succeed. No problem details available.";
            }
            catch (Exception ex)
            {
                result = "ERROR: Exception while importing shapefile to SQL."+ex.ToString();
            }
            return result;
        }

Alright, now that the SQL file has been created, we can proceed to the next step.

Step 5) Read the content of the SQL file, and execute it. Keep  in mind that the data will be imported into the spatial table in the same projection that the shapefile was in. If you want to reproject it, I suggest doing that by first importing it into a temporary table, reprojecting the data inserting it into the desired final destination.

There…you now have a Spatial table from your shapefile. ENJOY!!!

P.S:  Please feel free to propose any improvements, or if you were able to adapt this to solve a problem you were facing. I’d love to hear about that :-)

SharpMap.NET – Post 1

I’ve been fiddling around with SharpMap and PostGIS so that I could display some GIS Data in a web application. I searched around and found very meager resources out w.r.t SharpMap. So I decided to post my progress as I programmed my way around the component.

For my first post, I just displayed some GIS Data for the state of Louisiana.

ASPX Code:

<%@ Page Title="" Language="C#" MasterPageFile="~/Geo.Master" AutoEventWireup="true" CodeBehind="map.aspx.cs" Inherits="SharpMapOne.map" %>
<asp:Content ID="Content1" ContentPlaceHolderID="head" runat="server">
</asp:Content>
<asp:Content ID="Content2" ContentPlaceHolderID="ContentPlaceHolder1" runat="server">
        <div style="border: 1px solid #000000; padding: 10px;">
            <asp:Image ID="Image1" runat="server" />
        </div>
</asp:Content>

C# Code:

        protected void Page_Load(object sender, EventArgs e)
        {
            //Get the map object
            SharpMap.Map mp = new SharpMap.Map();
            mp.MinimumZoom = 100;
            mp.BackColor = System.Drawing.Color.Transparent;

            //create the vector layer
            SharpMap.Layers.VectorLayer vl = new SharpMap.Layers.VectorLayer("LA_HOUSE");
            vl.DataSource = new SharpMap.Data.Providers.PostGIS(ConfigurationSettings.AppSettings["GISConnString"], "la_uscon", "17796");
            vl.Style.Outline = System.Drawing.Pens.Red;
            vl.Style.EnableOutline = true;
            vl.Style.Line = new System.Drawing.Pen(System.Drawing.Color.Aquamarine);
            vl.Style.Fill = new SolidBrush(Color.Aquamarine);
            mp.Layers.Add(vl);

            //create the label acetate layer
            SharpMap.Layers.LabelLayer layASLabel = new SharpMap.Layers.LabelLayer("LA_HOUSE_DISTS");
            layASLabel.DataSource = vl.DataSource;
            layASLabel.LabelColumn = "cd111fp";
            layASLabel.Style.Font = new Font("Arial", 6, FontStyle.Bold);
            layASLabel.Style = new SharpMap.Styles.LabelStyle();
            layASLabel.Style.ForeColor = Color.Black;
            layASLabel.Style.Offset = new PointF(10, 0);
            layASLabel.TextRenderingHint = System.Drawing.Text.TextRenderingHint.AntiAlias;
            layASLabel.SmoothingMode = System.Drawing.Drawing2D.SmoothingMode.AntiAlias;
            layASLabel.Style.CollisionDetection = true;
            layASLabel.Style.CollisionBuffer = new SizeF(5, 5);
            layASLabel.MultipartGeometryBehaviour = SharpMap.Layers.LabelLayer.MultipartGeometryBehaviourEnum.Largest;
            mp.Layers.Add(layASLabel);

            mp.ZoomToExtents();
            System.Drawing.Image img = mp.GetMap();
            int ht = img.Height;
            int wd = img.Width;
            double ratio =  Convert.ToDouble( wd / ht);
            int new_ht = 400;
            int new_wd = -1;
            new_wd = Convert.ToInt32(new_ht * ratio);
            Image1.Height = new_ht;
            Image1.Width = new_wd;
            string imgID = SharpMap.Web.Caching.InsertIntoCache(5,img);

            Image1.ImageUrl ="mymap.aspx?ID=" + HttpUtility.UrlEncode(imgID);
        }

It results in this map.

Now now…I know it ain’t purdy, but its a map. We’ll pretty it up later and add functionality as well.

NOTE: Dont forget to add the Http Handler in your web.config that will handle the requests to mymap.aspx!
<httpHandlers>
…………
<add verb=”*” path=”mymap.aspx” type=”SharpMap.Web.HttpHandler,SharpMap”/>
</httpHandlers>

Point-In-Polygon in PostGIS

I was trying to get a PiP (point-in-polygon) done with all the GIS data that I had all of which was either Polygon or Multipolygon format.

I stored counties boundaries (Current County and Equivalent layer) I got from the Census Bureau in tables by state (for the sake of better organization). The data is in Geographic NAD 1983 (basically unprojected data).

You can see which county your lat/long falls into by using the following query:

select *
from la_counties
where st_contains(la_counties.the_geom,transform(GeometryFromText(‘POINT(-90.000 30.500)’,4326), 4269))=true and st_distance(la_counties.the_geom,transform(GeometryFromText(‘POINT(-90.000 30.500)’,4326), 4269))=0.00;

Executing the above query will return you one row with the results containing the county that the lat/long falls into. Great! Now we’ve got the PiP method down pat.

Okay. Now what if you have more layers that just counties, say ou have cities,  commercial districts, fire districts, and so on and so forth. You would have to query those spatial tables separately….right?
Well, yeah, if you choose to go the route of storing the data for each layer separately. But consider this route…..store all your layers in one spatial table (for each state/province). That way when you query that table with the same query above, you will get multiple rows…one from each layer!
Now you do not have to store all your layers in separate spatial tables each!

There is however a tiny issue when your GIS data is coming in from multiple sources….they dont quite follow a standard and align right next to one another…do they?
Now that is a conundrum for me to address another day!

I was trying to get a PiP (point-in-polygon) done with all the GIS data that I had all of which was either Polygon or Multi-polygon format. I stored counties boundaries (Current County and Equivalent layer) I got from the Census Bureau in tables by state (for the sake of better organization). The data is in Geographic NAD 1983 (basically unprojected data).

You can see which county your lat/long falls into by using the following query:

select *
from la_counties
where st_contains(la_counties.the_geom,transform(GeometryFromText(‘POINT(-90.000 30.500)’,4326), 4269))=true and st_distance(la_counties.the_geom,transform(GeometryFromText(‘POINT(-90.000 30.500)’,4326), 4269))=0.00;

Executing the above query will return you one row with the results containing the county that the lat/long falls into. Great! Now we’ve got the PiP method down pat.

Alright, now what if you have more layers that just counties, say ou have cities,  commercial districts, fire districts, and so on and so forth. You would have to query those spatial tables separately….right?Well, yeah, if you choose to go the route of storing the data for each layer separately. But consider this route…..store all your layers in one spatial table (for each state/province). That way when you query that table with the same query above, you will get multiple rows…one from each layer!

There you go! Now you do not have to store all your layers in separate spatial tables each.
There is however a tiny issue when your GIS data is coming in from multiple sources….they dont quite follow a standard and align right next to one another…do they? Now that is a conundrum for me to address another day!

PostGIS Tips

PostGIS is the spatial database attached to open source PostGres database. You can read all about it by clicking on the link.

It is a free way to host your GIS data in a geospatial database. If your organization is still storing its GIS data in the form of ESRI shapefiles, its time to move out of that mode! I know that established enterprise software out there are too expensive to invest in. PostGIS is on of the free ways to step into the direction of hosting GIS data and start moving in the right direction.

I looked around and found a really good step-by-step guide for beginners to help them get started with PostGIS. There are inbuilt functions within PostGIS that help in getting most of the rudimentary spatial querying done.

“Solving” reCaptcha 1.0

reCaptcha is an interesting captcha created by the School of Computer Science at Carnegie Mellon University. It was pretty challenging to solve this one.

Watch this…..with JavaScript enabled…

recaptcha with JavaScript enabled

recaptcha with JavaScript enabled

and now….with JavaScript disabled….

recaptcha with JavaScript disabled

recaptcha with JavaScript disabled

What happens is that when JavaScript is disabled, the reCaptcha is rendered in an IFRAME.
http://api.recaptcha.net/noscript?nocookie=1&k=AAAAAAUAAAAAAHD1aRKPPPPPanq65eHHYHYHYH_1

“Solving” Captchas

CAPTCHAs are used on websites to make sure it is a human and not a bot/rogue program on the page or entering the postal/site. There are sites that bypass the captcha using character resolution using neural networks. This is NOT one of those solutions. This solution shows how you can “capture” the captcha and display it on your site (or feed it to a service like Amazon’s Mechanical Turk, if you like ;) ).

The idea is to do the following:

1> GET the page’s source,
2> Save the form action of the FORM containing the captcha along with hidden fields (and their values).
3> Know which IMG tag would be the captcha, capture it.
4> Request that image & save it.
5> Have all the data for the form (assuming it is a for that the captcha is set up for), including all hidden fields.
6> Pass the image onto your site/service site where a human being will answer the CAPTCHA (remember this is an on-the-fly solution)
7> Get the word(s) in the captcha from your site/service site.
8> POST the entire form to the form action along with all the data and the captcha word(s) provided by the human in the field designated for it.
9> Collect the response and make sure the submission was successful (by looking for text that would be seen on success).

This way you can get service a page with a form containing a captcha!

But…what about reCaptcha?…..coming soon :D

Follow

Get every new post delivered to your Inbox.