Recently I started to notice that one of our sites gets strange HTTP GET requests from two IP ranges: 150.70.x.x and IP 62.24.x.x.
The short version of this goes as follows: we have web service when users can submit results via HTTP GET request. For example:
Each (next) user/request gets a new GUID, like:
Because GUIDs are unique, we should get each GUID only once. But this was not true for our web-service. We occasionally got duplicate requests. We started to investigate closer, and found that all double requests come from two IP ranges 150.70.x.x and IP 62.24.x.x, and all “second” requests follows the first after 30s – 5 min.
We did a Google search and found, that 150.70 IP range belongs to Trend Micro. There are even some statement from them:
Dear Site Owner,
To protect our customers from visiting a malicious or harmful site, web pages of the applicable URLs are downloaded and scanned by our servers. Thus, you may have noticed a few visits from our IPs. Please be assured that this poses no security risk to your web sites as our servers do not perform any action other than scanning the sites.
We then store the rating of the web site in our server cache so that our servers will no longer access those pages for analysis when a customer chooses to visit those web sites again. We have already asked our server owner to add the rating for the following domain(s);
The setting will take effect soon, please verify it again on your site.
Sorry for any inconvenience and please inform us of any other concerns you may have.
It seems, that Trend Micro is spying after their customers. Second range comes from Talk Talk UK’s ISP, who apparently spies after their customers too.
Here are some links with more reading:
Search Engine Spider and User Agent Identification Forum from webmasterworld.com
IP Address Inspector – projecthoneypot.org
Im Being Monitored/Watched?
P.S. Most probably they do not try to do HTTP POST request, I do not know. We probably should use POST too, but that’s another story.
P.S.S. Some encryption like SSL should be mandatory for all Internet traffic in future.
Disclaimer: this works for me and my clients today (Sep 2010). If Google or one on the web browsers change something, this may break all tracking, so use this on your own risk.
Below is working code fragments. In the process of making the code work, there was following problems / symptoms:
- Chrome and Safari refused to follow / ping image src at all;
- Firefox seemed to ignore / not follow googleadservices.com issued redirect 302 to googleads.g.doubleclick.net;
- It all seemed like some sort of Same origin policy;
- Firefox plug-in Firebug Net Panel showed strange question marks in Persist mode (seemed like request never completes). And on next click it showed – aborted.
Your HTML code:
<a href="http://some-lead-page-where-you-cannot-add-or-change-code" onclick="googletrack()">Some lead / sale / page view / other action</a>
Simple example code:
Example code with custom label passed as parameter:
Other fields such as value, etc. can be added as needed.
Tested on Internet Explorer 8.0.7600.16385, Mozilla Firefox 3.6.8, Google Chrome 6.0.472.53, Opera 10.61, Apple Safari 5.0.1.
Updated Sep 17, 2010.
What will make my website faster? How to optimize page load time? How to make web site appear to load faster for end user?
Updated Feb 9, 2011.
This is excerpt from article: How big HTML images / CSS / JS / png / etc. should be to fit network packet (MTU) efficiently
List of things that will make your website much faster:
- Server must support Keep-Alive. Otherwise change server / host / hosting company / etc. Do it NOW!
- Reduce HTTP requests (CSS sprites, combine CSS, combine JS, inline CSS, inline JS)
- Minifiy HTML, JS, CSS. Google Closure Tools, YUI Compressor, Minification
- Gzip text/html, css, js (IE6 does not un-gzip CSS and JS, if reference is not from HEAD)
- For IE 5 & 6 use gzip-only-text/html
- Check if (Content-Encoding: gzip) then (Vary: Accept-Encoding) to allow cache both versions in proxy servers
- CSS goes in head, JS goes at bottom – right before body closing tag
- Defer JS if possible. Defer allows to load js after onload
- Image formats: for photos – always use JPG, for everything else – PNG 8, for animations – GIF
- Compress images: for JPG use
jpegtran or jpegoptim Adobe Photoshop or Advanced JPEG Compressor (both non-free) or free GIMP, for PNG use OptiPNG or PNGOUT. I use non-free but the best one – PNGOUTWin
- Think connections! Waterfall. webpagetest.org (probably the best site for web page speed / optimization test), Zoompf (a little overkill, but can be useful), Pingdom Tools
- Try to serve assets in parallel, e.g, images.example.com, images2.example.com; subdomains can be on the same IP
- Use 2-3 max 4 sub-domains
- Optimize response headers, smaller, meaningful
- Avoid redirects (301, 302, META refresh, document.location)
- Remove ETAG
- Use Expires + cache-control
- 25k and greater files are not cached on iPhone
- Cache dynamic content PHP, ASPX, ruby etc.
- example.com?param is not cached by most proxies
- Serve static assets from cookie-less domain, like yimg.yahoo.com
- After onload via js can pre-cache images (if you know where visitor will go next)
- Can use double heads (if a lot of meta then put 2nd head at the bottom after closing BODY tag)
- Use CDN if can afford
And remember about:
Related tools / links:
More reading and watching:
While making website faster, do not forget about security:
Things that big guys (such as Google, Microsoft, Yahoo, Akamai) do. This requires custom software, customized Linux / BSD / Apache compilation, and of course deep knowledge in networking, OSI model, TCP, etc.:
P.S. This blog (wishmesh.com) is not optimized for anything (speed, size, etc.). People are lazy. They optimize only when there is reason to do it, and we like most humans are lazy, so this blog runs on default WordPress hosted on shared server.