Vacuum

Hacker News summary for Friday, 24 May 2013, 8:00 p.m. EST edition

An attempt to catch the top news of Hacker News on a Friday evening and summarize the top 10 stories; I won't make any decisions about what should be in the top 10, just note them here with a summary. It took me about 30 minutes to summarize 10 stories, without being selective about which ones were important.

1. Ben Howdle is "Too scared to write a line of code" (Medium). This reminds me of the Donald Knuth quote, "Premature optimization is the root of all evil".

2. Yahoo has "submitted a bid for the video site" Hulu (All Things D). They're not the only ones bidding, with Directv, Time Warner Cable, and several others in the running. 

3. Benjamin BenBen lets you visualize "how CSS rules are applied to a page" (css.benjaminbenben.com). If it's hanging, give Chrome a try.

4. B is "fast on the road and agile in the air" (Kickstarter). It's a four-wheeled remote control vehicle that doubles as a quadcopter when the going gets rough.

5. "If you already have a strong passphrase on your SSH private key, then converting it from the traditional private key format to PKCS#8 is roughly comparable to adding two extra keystrokes to your passphrase, for free." (Martin Kleppmann) Bonus: it's only four lines of shell to do this and verify that it works.

6. Plan 9 has some very tightly written, very strange C code; "it’s not every day that you come across code that makes you feel funny because of its unusual syntactic structure" (Spiros Eliopoulos). In this case it's nested if() clauses without indents that replace a more traditional use of &&.

7. "In December 2007, I left a cushy job to start a company. That company failed. This blog outlines that story." (Ryan Angilly) Today: the day our investors came to see the office.

8. vnc-over-gif "serves screen updates as animated gif over http." (Github) See also "gifsockets" which lets you do socket communication in a browser via an animated gif. [:dancing banana:]

9. "main;" is the shortest crashing C program, which will compile and then SIGSEGV. ( Jesper Öqvist)

10. New versions of jQuery (1.10 and 2.0.1) are out. (jQuery) What's new: "Our main goal with these two releases is to synchronize the features and behavior of the 1.x and 2.x lines, as we pledged a year ago when jQuery 2.0 was announced."

You are likely to be eaten by a grue.

The video, entitled "It is pitch dark", from nerdcore hip-hop grandmaster MC Frontalot.

The lyrics, in part:

You are likely to be eaten by a grue. If this predicament seems particularly cruel, consider whose fault it could be: not a torch or a match in your inventory.

What is a grue?

The grue is a sinister, lurking presence in the dark places of the earth. Its favorite diet is adventurers, but its insatiable appetite is tempered by its fear of light. No grue has ever been seen by the light of day, and few have survived its fearsome jaws to tell the tale.

More on "Grue" and Grue, The Philosophical Review © 1961 Duke University Press.

Mr. Grue believes that grass is grue, taken here to mean that it is green until the year 2000 and blue afterward.

Grue is the color of the y2k problem - green (providing riches) until the year 2000, and blue (providing sadness) afterwards. Noted here, as The Year 2000 Crisis: One software engineers' perspective":

This is not a new problem. There is a paradox, called "Goodman's Grue-Bleen paradox", which describes what we're facing. See The fall 1983 issue of the Bulletin of the Santayana Society, "on Grue and Bleen", by Angus Kerr-Lawson. See also Goodman's Paradox. In the paradox, an object is "grue" if it is green between a certain time, T, and blue after T. An object is "bleen" if it is blue before T, and green after T. The paradox is how to tell the difference between something which is green and something which is grue. At the moment, it is impossible to know. After T, it will be easy. Similarly, it is easy to tell the difference between something which is green and something which is bleen. But after T, it might not be. Further, any system built on distingushing between green and blue will work properly now with green, blue, and bleen objects, but will fail after time T.

Is there a relationship between the "grue" of Zork and the "grue" of the color paradox?

Awesome homebrew USB Lego printer printed "Hello World" from a Mac

The critic says "I like Lego City better."

whitehouse.gov: sending data to statse.webtrendslive.com

On the page:

<script type="text/javascript" src="/includes/webtrends.js"></script>

in that code:

function WebTrends(){
	var that=this;
	// begin: user modifiable
	this.dcsid="dcs0l9nq800000ctek411lue6_2c8b";
	this.domain="statse.webtrendslive.com";
	this.timezone=-5;
	this.fpcdom=".whitehouse.gov";
	this.onsitedoms="";
	this.downloadtypes="xls,doc,pdf,txt,csv,zip";
	this.rightclicktypes="xls,doc,pdf,txt,csv,zip";
	this.trackevents=true;
	this.enabled=true;
	this.i18n=false;
	this.fpc="WT_FPC";
	// end: user modifiable
	this.DCS={};
	this.WT={};
	this.DCSext={};
	this.images=[];
	this.index=0;
	this.exre=(function(){return(window.RegExp?new RegExp("dcs(uri)|(ref)|(aut)|(met)|(sta)|(sip)|(pro)
|(byt)|(dat)|(p3p)|(cfg)|(redirect)|(cip)","i"):"");})();
	this.re=(function(){return(window.RegExp?(that.i18n?{"%25":/\%/g}:{"%09":/\t/g,"%20":/ /g,"%23":/\#
/g,"%26":/\&/g,"%2B":/\+/g,"%3F":/\?/g,"%5C":/\\/g,"%22":/\"/g,"%7F":/\x7F/g,"%A0":/\xA0/g}):"");})(
);
}

in the privacy policy:

Browser information collected on the web site: We log IP addresses, which are the locations of computers or networks on the Internet, and analyze them in order to improve the value of our site. We also collect aggregate numbers of page hits in order to track the popularity of certain pages and improve the value of our site. We do not gather, request, record, require, collect or track any Internet users' Personal Information through these processes.

regime change: before and after on whitehouse.gov

$ for n in $(~/bin/range 1 60); do curl -L http://whitehouse.gov -o whitehouse.$n.html; sleep 60; done &
$ ls -ltr
total 1392
-rw-r--r--  1 emv  staff  33323 Jan 20 11:46 whitehouse.1.html
-rw-r--r--  1 emv  staff  33323 Jan 20 11:47 whitehouse.2.html
-rw-r--r--  1 emv  staff  33323 Jan 20 11:48 whitehouse.3.html
-rw-r--r--  1 emv  staff  33323 Jan 20 11:49 whitehouse.4.html
-rw-r--r--  1 emv  staff  33323 Jan 20 11:50 whitehouse.5.html
-rw-r--r--  1 emv  staff  33323 Jan 20 11:51 whitehouse.6.html
-rw-r--r--  1 emv  staff  33323 Jan 20 11:52 whitehouse.7.html
-rw-r--r--  1 emv  staff  33323 Jan 20 11:53 whitehouse.8.html
-rw-r--r--  1 emv  staff  33323 Jan 20 11:54 whitehouse.9.html
-rw-r--r--  1 emv  staff  33323 Jan 20 11:55 whitehouse.10.html
-rw-r--r--  1 emv  staff  33323 Jan 20 11:56 whitehouse.11.html
-rw-r--r--  1 emv  staff  33323 Jan 20 11:57 whitehouse.12.html
-rw-r--r--  1 emv  staff  33323 Jan 20 11:58 whitehouse.13.html
-rw-r--r--  1 emv  staff  33323 Jan 20 11:59 whitehouse.14.html
-rw-r--r--  1 emv  staff  33323 Jan 20 12:00 whitehouse.15.html
-rw-r--r--  1 emv  staff  10603 Jan 20 12:03 whitehouse.16.html
-rw-r--r--  1 emv  staff  48922 Jan 20 12:04 whitehouse.17.html
-rw-r--r--  1 emv  staff  48922 Jan 20 12:05 whitehouse.18.html
-rw-r--r--  1 emv  staff  48922 Jan 20 12:06 whitehouse.19.html

"crush collision", a new album from The MD5

No, not really, but wouldn't it be awesome?

This is the team that wrote the paper MD5 considered harmful today: Creating a rogue CA certificate which showed that the security model of HTTPS can be defeated using a combination of attacks on the MD5 hash algorithm plus non-random serial numbers in the popular RapidSSL and FreeSSL certificates.  They used a cluster of 200 PS3s for a weekend of number crunching.

The techno show Crush Collision is on Thursdays at 10pm Eastern on WCBN.

stuck on a postcard / postful / curl problem

FIXED, see the bottom.  12/31

rather than give up in total frustration, here's what I am trying to implement, using "curl" to drive the system.  This is from the "postful" API docs.

1. Upload the document.

To uploading a document, submit a POST to:

http://www.postful.com/service/upload

Be sure to include the Content-Type and Content-Length headers and the document itself as the body of the request.

POST /upload HTTP/1.0
Content-Type: application/octet-stream
Content-Length: 301456

... file content here ...

If the upload is successful, you will receive a response like the following:

<?xml version="1.0" encoding="UTF-8"?>
<upload>
 <id>290797321.waltershandy.2</id>
</upload>

What is critical in this response is the id assigned by the server to your document, in this example 290797321.waltershandy.2. You will use that id to reference the document in the next step.

I think it should work with

curl -v --user edward.vielmeti@gmail.com:t0ps3kr3t  --data-binary "upload=@m-28-shield.png" http://postful.com/service/upload

but, it doesn't.  ideas?  mac os if it matters, the

% curl --version
curl 7.16.3 (powerpc-apple-darwin9.0) libcurl/7.16.3 OpenSSL/0.9.7l zlib/1.2.3
Protocols: tftp ftp telnet dict ldap http file https ftps
Features: GSS-Negotiate IPv6 Largefile NTLM SSL libz 

update: typo fixed

update2: working now!  command line is

curl -k -v --user edward.vielmetti@gmail.com:s4x4s54  --data-binary "@m-28-shield.png" https://postful.com/service/upload

note three things: "-k" option; https, not http:; spell the user name right (duh).

next task: get the address side formatted right.

image processing on the mac with sips, ImageMagick - watching DTE restore power, slowly

As part of watching DTE slowly restore power to the area, I've been pulling down copies of their outage maps.  These are big PDF files, suitable for framing, that encode a bunch of data in them about where the power is out. 

Here's some tools I used to get details out of PDF into image formats, (hopefully) to get them back down to a spreadsheet full of numbers.

1.  Fetch the outage report automatically with curl:

DATESTAMP=$(date "+%Y-%m-%d-%H-%M")
curl -s -o outage.$DATESTAMP.pdf http://my.dteenergy.com/map/zipCodeOutageMap.pdf

This grabs the file and date/time stamps it appropriately.  I'm running in an every 15 minutes loop, which means it grabs the file that's updated every 30 minutes twice (a little redundancy, not too bad).

2.  Convert the report from PDF to PNG with sips:

sips -s format png outage.$DATESTAMP.pdf --out outage.$DATESTAMP.png

'sips' (the "scriptable image processing system") has a bunch of Applescript front end documentation for it, but there is also a command line lurking there as well.  I'm collecting bookmarks for tutorials on sips at delicious.  There's a lot that it can do, since it is mostly a front end for the core graphics library on the Mac which is quite powerful, but the doc is clearly not written with the intention that anyone uses it directly.

3.  Do fun stuff to a series of images with ImageMagick:

composite  -compose difference ~/Desktop/outage.2008-12-28-23-47.png outage.2008-12-29-13-28.png difference-progress.gif

ImageMagick has dozens, maybe hundreds, of transformations that you can use to take a set of images and turn them into something else.  It's powerful and also reasonably well documented, and I won't even try to summarize the ImageMagick command line options list which gives you plenty to play with.

4.  Stare at the results and try to make sense of them.

Here's the DTE progress map (middle) with before (left) and after (right) reflecting about 14 hours of restoration efforts.  Click through to Flickr for detail.  The large amounts of black in the middle photo represent locations where work progress has not resulted in a change in service that's visible, though there may in fact have been 1000s of lines restored.

5.  What I want is a tool that would work on a PNG and would give me the color value of a given pixel, or that would give me a histogram of all of the color values of all of the pixels in an image.  There is a histogram function in ImageMagick but it doesn't quite do the trick at first glance, and I'd be better off posting this than trying to do more myself!

The takeaway on this if you squint hard enough - 14 hours of effort, and not a lot of immediately visible progress across broad parts of Livingston, Washtenaw, and Downriver areas in DTE service restoration.

using "tidy" in a screen scraping pipeline

When the AATA mobile ride track is up, I have a script that runs periodically to decode the web pages and turn them into data.  The pages are formatted for humans, so I need to do screen scraping on them to reconstruct the original data values.

There are a lot of tools for screen scraping, with each of them appealing to a programmer who has a certain world view of the web and how things are put together.  The tool that I find myself recommending when someone who sounds like a programmer asks is Beautiful Soup (for python) or Rubyful Soup (for ruby), in part at least because the coder's attitude toward pragmatism is appropriate for the task at hand:

You didn't write that awful page. You're just trying to get some data out of it. Right now, you don't really care what HTML is supposed to look like.

Neither does this parser. 

Rubyful Soup's page says "no longer maintained, try hpricot instead"; I looked for hpricot and got some 404s.  Not sure really what's up with that.

As for what I really do, rather than what I recommend, well you have to understand that most of the code that I write that actually does anything useful is less than a page of text and mostly is full of regular expressions.   HTML in its unadorned state is mostly resistant to simple-minded regex parsing, but if you constrain the world enough you can get most data out of most pages by running the page through a preprocessor that normalizes it into something sane.

My favorite for that effort is the old school "HTML Tidy" application, which has a zillion options by now for taking your weird web pages and making them pass syntax checks. 

Thus the source of the main loop for the "mobile ride guide decoder":

curl -s -o $TMP/route.5.html http://mobile.theride.org/rideguide_m.asp?route=5
tidy -f /dev/null -q -w 0 -o $TMP/route.5.tidy.html $TMP/route.5.html
perl ./mobiletocsv.pl < $TMP/route.5.tidy.html >> route.5.log.csv

and the "mobiletocsv" perl script is very simple minded

#!perl

while(<>) {
   /AS OF (\d+:\d\d)/ && ($curtime = $1);
   $realtime = `date +%H:%M`;
   chop $realtime;
   /(\d+) min behind/ && ($late = $1);
   /(\d+) min ahead/ && ($late = -$1);
   /(\d+) on time/i && ($late = 0);
   /(\d\d\d) to (Ann Arbor|Ypsilanti)/ && ($busno = $1, $dest = $2);
   /(\d\d\d) (OutBound|Loop)/ && ($busno = $1, $dest = $2);
   /^@ (.*)<br>/ && ($curloc = $1);
   /^(.*) (\d+:\d\d)/ && ($newloc = $1, $timepoint = $2,
       print "$realtime,$curtime,$late,$busno,$dest,$curloc,$newloc,$timepoint\n
");
}

It's probably not resistant to aggressively malformed input, and I simply don't run it when the result is "please check your timetable for times". 

The first and second iterations that I wrote of this code tried to do it all without any tidy step, and it was hard especially because I had to try to parse syntax and parse data in the same step.  This version is compact enough that it might reasonably be the jumping off point for future enhancements. 

still waiting for the bus data to come back....

How to run Django on Google App Engine - results of the Ann Arbor Google App Engine hackathon

I'm at the Google App Engine hackathon in Ann Arbor.  My goal for the day is to get a Django instance running on top of appengine as much as possible, using as much of the existing code base that's out there that I can.  (This instead of say, hacking a wiki, which is what their sample app is - been there done that.)

Here's some background reading:

Using Django with Appengine - Shabda Raaj.  "Welcome to Django tutorial for integrating with Appengine. This is a port of the Django tutorial to use appengine instead of Pure Django. Like in the Django tutorial, we build a poll engine, where you can create polls and others can vote for them. Instead of the four part Django tutorial, we use only one tutorial. Also, as many parts of Django, most prominently its super Admin interface do not work, we will work around them. I will assume that you know python well. However I do not assume that you have previous experience with Django. Django concepts are explained."

app-engine-patch -

With app-engine-patch a major part of Django works on App Engine without any modifications. The most important change is that you have to use Google's Model class because the development model is too different from Django.

We also integrate a package called ragendja which provides functions that simplify Django and App Engine development.

to be continued....

Here's Guido van Rossum's keynote in September 2008 about Google App Engine and Django.