when I'm not skateboarding

welcome...to...big...data

2011-08-10T18:03:00.000-07:00

The following is excerpted from the hbase-user@apache.org mailing list...an instant classic!

=================================================================

Mongodb does an excellent job at single node scalability - they use mmap and many smart things and really kick ass ... ON A SINGLE NODE.

That single node must have raid (raid it going out of fashion btw), and you wont be able to scale without resorting to:

- replication (complex setup!)

- sharding

mongo claims to help on the last item, but it is still a risk point.

For really large data that must span multiple machines, there is no "clustered sql" type solution that isnt (a) borked in various ways (Oracle RAC I'm looking at you) or (b) stupid expensive (Oracle RAC, STILL looking at you)

Tools like HBase give you scalability at the cost of features (no automated secondary indexing, no query language).

Welcome... to... big... data.

-ryan

On Thu, Aug 11, 2011 at 12:44 AM, Edward Capriolo <> wrote:

> On Wed, Aug 10, 2011 at 4:26 PM, Li Pi <> wrote:

>

>> You'll have to build your own secondary indexes for now.

>>

>> On Wed, Aug 10, 2011 at 1:15 PM, Laurent Hatier 

>>

>> >wrote:

>>

>> > Yes, i have heard this index but is it available on hbase 0.90.3 ?

>> >

>> > 2011/8/10 Chris Tarnas <>

>> >

>> > > Hi Laurent,

>> > >

>> > > Without more details on your schema and how you are finding that 

>> > > number

>> > in

>> > > your table it is impossible to fully answer the question. I 

>> > > suspect

>> what

>> > you

>> > > are seeing is mongo's native support for secondary indexes. If 

>> > > you were

>> > to

>> > > add secondary indexes in HBase then retrieving that row should be 

>> > > on

>> the

>> > > order of 3-30ms. If that is you main query method then you could

>> > reorganize

>> > > your table to make that long number your row key, then you would 

>> > > get

>> even

>> > > faster reads.

>> > >

>> > > -chris

>> > >

>> > >

>> > > On Aug 10, 2011, at 10:02 AM, Laurent Hatier wrote:

>> > >

>> > > > Hi all,

>> > > >

>> > > > I would like to know why MongoDB is faster than HBase to select

>> items.

>> > > > I explain my case :

>> > > > I've inserted 4'000'000 lines into HBase and MongoDB and i must

>> > calculate

>> > > > the geolocation with the IP. I calculate a Long number with the 

>> > > > IP

>> and

>> > i

>> > > go

>> > > > to find it into the 4'000'000 lines.

>> > > > it's take 5 ms to select the right row with Mongo instead of 

>> > > > HBase

>> > takes

>> > > 5

>> > > > seconds.

>> > > > I think that the reason is the method : cur.limit(1) with 

>> > > > MongoDB but

>> > is

>> > > > there no function like this with HBase ?

>> > > >

>> > > > --

>> > > > Laurent HATIER

>> > > > Étudiant en 2e année du Cycle Ingénieur à l'EISTI

>> > >

>> > >

>> >

>> >

>> > --

>> > Laurent HATIER

>> > Étudiant en 2e année du Cycle Ingénieur à l'EISTI

>> >

>>

>

> http://www.xtranormal.com/watch/6995033/mongo-db-is-web-scale

>

HBase 0.89 Bulk Import capability

2011-01-23T13:08:00.000-08:00

I recently posted about my experience tuning HBase for mass inserts. When I learned that HBase actually supports a way to *avoid* the insert all together, I got really excited. As the Joshua Program says, "the only way to win the game is not to play"! What better way to speed up inserts than not to do them at all! You can't get much faster than zero. So how do you get data into HBase if you don't do inserts? Aha! You can actually generate HFileOutputFormat files, that essentiallly *are* database tables. And believe me, compared to running Puts from your reducer it is fast, and you don't have the "attack my database and maybe crash it" issue. To give you an idea of how fast it is, I had a job whose reducers ran for 45 minutes doing Puts. The bulk loader took 19 seconds to load the files, and it only took a couple minutes to generate the files from the reducer. So the whole bulk load process, end to end, was maybe 2 minutes.

The main "gotchas" along the way: The documentation isn't entirely clear about the fact that if you use HFileOutputFormat.configureIncrementalLoad, that your reducer is silently replaced by an internal HBase class that insures total ordering. Secondly, your mapper must output either Put or KeyValue. This is not a usable way to actually write mapreduce jobs (since you lose the ability to pass domain objects between map and reduce). Therefore, I found it best to alter my existing mapreduce jobs to produce a SequenceFileOutputFormat, in which I wrote Put from my reducer as the output class (since it doesn't interfere with your job logic/classes at all to change only the *final* output class to a Put). In fact, the class I wrote to generate the HFileOutputFormat will actually accept sequence files as input if they contain either a Put, or a single column value. I included support for a single column value just 'cause I had a couple jobs lying around that produced only a single column, and would have been a hassle to convert them to producing Puts. Then my generic bulk loader mapreduce job can be run on any of my output sequence files.

public class BulkLoader extends Configured implements Tool{


    public static class BulkLoaderMapper extends Mapper<WritableComparable, Writable, ImmutableBytesWritable, Put> {

        protected long version;
        protected String column;
        protected String family;

        @Override
        protected void setup(Mapper.Context context) throws IOException, InterruptedException {
            this.version = context.getConfiguration().getLong("version", System.currentTimeMillis());
            this.column = context.getConfiguration().get("column");
            this.family= context.getConfiguration().get("family");
        }


        private byte[] marshal(Writable o) throws IOException{
            ByteArrayOutputStream bos = new ByteArrayOutputStream();
            DataOutputStream dos = new DataOutputStream(bos);
            o.write(dos);
            return bos.toByteArray();
        }

        protected byte[] getHBaseRowKeyBytes(Writable k) throws IOException{
            return marshal(k);
        }

        protected byte[] getHBaseFamilyNameBytes(){
            return Bytes.toBytes(family);
        }

        protected byte[] getHBaseColumnNameBytes(){
            return Bytes.toBytes(column);
        }

        protected byte[] getHBaseValueBytes(Writable v) throws IOException{
            return marshal(v);
        }

        @Override
        protected void map(WritableComparable key, Writable val, Context context) throws IOException, InterruptedException {
            //KeyValue kv = new KeyValue(getHBaseRowKeyBytes(key), getHBaseFamilyNameBytes(), getHBaseColumnNameBytes(), version, getHBaseValueBytes());
            if(context.getConfiguration().get("input-format", "column").equals("put")){ //the serialized file actually is full of puts
                System.out.println(key.toString());
                Put p = (Put) val;
                //I am baffled as to why I cannot use getHBaseRowKeyBytes(key) here. It seems to not produce the correct key.
                //Therefore I m using p.getRow() to get the rowkey bytes.
                context.write(new ImmutableBytesWritable(p.getRow()), p);
            }else{
                Put put = new Put(getHBaseRowKeyBytes(key));
                put.add(getHBaseFamilyNameBytes(), getHBaseColumnNameBytes(), version, getHBaseValueBytes(val));
                context.write(new ImmutableBytesWritable(getHBaseRowKeyBytes(key)), put);
            }
        }
    }

    public static void main(String[] args) throws Exception {
        int exitCode = ToolRunner.run(new BulkLoader(), args);
        System.exit(exitCode);

    }

    @Override
    public int run(String[] otherArgs) throws Exception {

        Configuration config = (Configuration) getConf();//new Configuration();
        Job job = new Job(config);

        //job.setOutputKeyClass(ImmutableBytesWritable.class);
        //job.setOutputValueClass(Put.class);
        job.setMapOutputKeyClass(ImmutableBytesWritable.class);
        job.setMapOutputValueClass(Put.class);
        job.setMapperClass(BulkLoaderMapper.class);

        job.setJarByClass(BulkLoader.class);
        job.setInputFormatClass(SequenceFileInputFormat.class);
        System.out.println("sequencefile-input: " + config.get("sequencefile-input"));
        System.out.println("output-path: " + config.get("output-path"));
        System.out.println("bulk-load-table: " + config.get("bulk-load-table"));
        System.out.println("family: " + config.get("family"));
        System.out.println("column: " + config.get("column"));
        System.out.println("input-format: " + config.get("input-format", "column"));
        SequenceFileInputFormat.addInputPath(job, new Path(config.get("sequencefile-input")));
        job.setOutputFormatClass(HFileOutputFormat.class);
        Configuration hConfig = HBaseConfiguration.create(config);
        hConfig.setLong("version", System.currentTimeMillis());
        hConfig.set("hbase.zookeeper.quorum", config.get("zk", "zk1.x.com, zk2.x.xom"));
        job.setJobName("Bulk Loading table: " + hConfig.get("bulk-load-table","YOU HAVE NOT SET bulk-load-table PARAMETER"));
        HFileOutputFormat.setOutputPath(job, new Path(config.get("output-path")));
        HFileOutputFormat.configureIncrementalLoad(job, new HTable(hConfig, config.get("bulk-load-table")));


        job.waitForCompletion(true);

        return 0; //normal exit

    }
}

Finally, the 0.89 jar seemed to have some packaging problems. I had to create a "lib" directory (used by hadoop in job jars) and rejar the 0.89 jar with guava.jar and a perhaps a couple other missing dependencies. With that done I was able to run the bulkloader with the completebulkload option, using ant command:

Facebook announces websearch to rival Google's

2011-01-08T12:40:00.000-08:00

Admittedly I've made up the headline "Facebook announces websearch to rival Google's", but is it really such an outlandish proposition? It's certainly attention grabbing, and why not? Google's websearch has basically remained the same product for 10 years. Any engineer at Google would debate this, but I doubt anyone would debate that the engineers at facebook have achieved a stunning mastery over big data sets and management of the social network.

Based on the considerable "Big Data" and real-time data expertise at Facebook, would anyone really be surprised if such a headline surfaced? I'd find it quite a believable story: "Facebook announces intention to best Google at web search." It's hard to overstate the earthquake that such an announcement would send rumbling through silicon valley, and the entire world. In fact, what better way for Facebook to spend their $500 million in Goldman Sachs cash then acquiring promising engineers and technologies to directly attack Google's biggest Web cash cows: search and maps. Bloodying the nose of the competition by competing with their core value has always been Google's way: If an Operating System is of value, Google builds an OS. If mobile computing is of value, google builds a mobile OS, and it's own phone hardware. The day is going to come when a Google rival will compete with Google on Google's home turf: web search.

Minhashing is reaaally cool

2011-01-04T22:41:00.000-08:00

What is the Jaccard Coefficient? Answer: one of the most important concepts in information retrieval and theory of sets. The Jaccard Coeffieicnt is very simple. It is the measure of the fractional similarity of two sets. Let's take a simple example {A,B,C,D,E,F} vs. {B,E,F}. Fractional similarity of these sets is 3/6. Easily you can see that 3 elements are shared (the intersection is {B,E,F}), whereas the total number of items in both sets (the union) is 6 elements ({A,B,C,D,E,F}).

So without any mumbo-jumbo, we can see the Jaccard Coefficient (JC) is just the ratio of sameness (intersection) over the total amount of stuff present. Technically it's the cardinality (count) of elements in the intersection over cardinality (count) of elements in the union. So while the JC between {A,B,C,D,E,F} and {B,E,F} is 50%, we can see that the JC between {A,B,C,D,E,F,G,H,I} and {B,E,F} is only 33.33%. This illustrates the importance of the total amount of stuff present. The more irrelevant (non-intersection) stuff is present, the lower the Jaccard Coefficient.

Now pretend we have two bags. Bag 1 contains Set 1, and Bag 2 contains Set 2. I tell you the JC between these two sets is 95%. I reach into Bag 1 and I pull out the element "M". What is the probability that the element "M" is also in Bag 2? It's 95%. That's what the Jaccard Coefficient tells me. But now here is something quite amazing. What is the probability that the lowest symbol (lowest according to some ordering, like a hashcode) in Bag 1 is the *same* letter as the lowest letter in Bag 2. Again, it's 95%. Why? Some letter has to be the lowest one in a bag. Since only 5% of the letters in the other bag ought to be different, then there is a 95% chance that the lowest symbol in the second bag falls in the intersection of the sets, and is consequently the same.

Now it really gets interesting. If I group sets by a key, where the key is the lowest element of the set, then the probability that two sets get grouped by the same key is again the Jaccard Coefficient. That's cool. That's BADASS. Why? It's an almost shockingly simple way to create clusters of similar sets.

I find a lot of convoluted explanations of Minhashing, but you really don't need to know more than this guy's explanation. Here's a short piece of code, that given a sentence, splits the sentence into words, then builds N-grams out of the words (the N-grams are the set elements). Surrounding a word with ^ and $ is a well-known way for demarcating N-grams that are beginning and endings of words. A SHA-1 cryotographic hash function is a nice way to assign a random, but consistent ordering to any possible N-gram we might come up with. As suggested by this guy, rather than clustering on the *single* lowest element, we can tighten the clusters up by making the cluster key the numMins smallest elements, concatenated. I use this in a mapreduce job, where I write the minhash out as the key from the mapper, and the reducer collects all the sentences with the same minHash key, thereby clustering similar sentences. And lo and behold...it works.

      protected String minHash(String sentence, int nGramLength, int numMins) throws Exception{
            MessageDigest md = MessageDigest.getInstance("SHA-1");
            PriorityQueue queue = new PriorityQueue();
            for(String part :sentence.split("\\s")){
                for(String gram:getNGrams("^"+part+"$",nGramLength)){
                    md.reset();
                    queue.add(ByteBuffer.wrap(md.digest(gram.getBytes("UTF-8"))).getLong());
                }
            }            
            StringBuilder minHash = new StringBuilder();
            while(null != queue.peek() && numMins > 0){
                minHash.append(queue.remove());
                --numMins;
            }
            return minHash.toString();
        }.

Tuning HBase and Mapreduce for Mass Inserts

2010-12-26T16:49:00.000-08:00

For several months my colleagues and I have been earning our battle scars as we run progressively larger bulk inserts into HBase. I'm going to try to recall as many of the lessons learned as possible. At the time of this writing HBase 20.6 is the latest production release:

Use HBase 20.6. We had been running 20.3 and it suffered from a hideous deadlock bug, that would cause clients to simply stall and do nothing, until they were killed by the Hadoop task tracker. Even worse, because they were killed and restarted, this caused a vicious cycle of reducers stalling, restarting, stalling, etc. All the while threads and client connections were eaten up in HBase.
Use huge numbers of mappers, and huge numbers of reducers so that your data sets are broken into bite sized pieces. To do this, set mapred.tasktracker.[map|reduce].tasks.maximum (the max value that will be run by a single task tracker at once) to a value correlated to the number of cores on each box (we'll cal this "C") and the number of boxes "B". Then set set mapred.[map|reduce].tasks (the total number of mappers or reducers that will get run) equal to a*C*B. "a" is then a free parameter which controls how big your bite sized batches of data will be. As a rule of thumb, if you are reading data out of HBase via TableMapper, I try to make a*C*B between 1000 and 10000, but it all depends on how much processing you ned to do on each chunk.
This is an addendum to rule (2). All sorts of bad things happen when your mappers or reducers bite off too much data, and run for too long. Just a few items on the buffet of errors and warnings that you'll chase if your mappers or reducers try to handle too much data:

HBase clients leases expiring
Chance of OoME
Failure to report status for more than 600 seconds causes task to get shot in the head
Hbase client slows down on writes(annecdotal. I can't prove this, but I know that if you want to avoid ptential slowdowns, then following rule 2, in my experience, helps a lot)

Use fine grained status. Make calls to progress(), setStatus(msg), and use counters to track what your job is doing. Nothing is more annoying that not being able to tell what the hell your job is doing when it has been running for 5 hours. Rule 2 helps here too. The more mappers and reducers you have, the more continuous and fine grained will be your status reporting as each task completes with its data load into HBase.
Don't screw around too much with the default settings. Ryan Rawson has put out a good set of defaults in this PPT. You can turn these dials ad infinitum but I can pretty much promise you that there is no magic set of settings. This can be a massive time sync. Stick with Ryan's recommendations.
Watch out for problems with your hostnames. Let's say your hostname is "s1.x.com". If you execute "hostname" and all you get back is "s1" or "localhost" you are in for problems. Check your /etc/hosts and make sure that aliases like "s1" don't come before the fully qualified hostname.
Watch out for msiconfigured NICs. Problems with vanishing packets sent me chasing my tail for weeks. This was the classic buffet of errors, with everything seemingly leading to no resolution.
Disable the Write Ahead Log from your client code. Use an appropriately sized write buffer. There'sa point of diminishing returns here. Exceeding the size of the serverside buffers won't do you any good.
Don't try to get too smart. Most bulk HBase data-producing M/R jobs are going to do some processing, then write the data from the reducer. Since the reducers all receive keys in the same order, this causes all the reducers to attack the same HBase region simultaneously. We had this "great idea" that if we reversed the keys that we wrote out of our mapper, then un-reversed them in the reducer, that our reducers would be randomly writing to different region servers, not hitting a single region in lock step. Now, I have some theories on why this seemingly innocuous approach repeatably destroyed our entire Hbase database. I won't wax philosophical here, but one thing is certain. Any table created via batch inserts of randomized keys got totally hosed. Scans became dirt slow and compactions ran constantly, even days after the table was created. None of these problems made a whole lot of sense, which is why it took 3-4 weeks of debugging for us to back this "key randomizing" out of our code. The hosed tables, actually had to be dropped for the problem, and ensuing chaos to totally abate.
tail all your logs and scan them assiduously for errors and warnings. I use VNCserver with different desktops for tails of all the logs so that I never have to "re-setup" all my tailf commands when I log in and out of the server.
Disable speculative execution. You don't want two reducers trying to insert the exact same content into the database.

Black Friday Buys

2010-11-29T21:46:00.000-08:00

Picked up a wireless access point for 10 bucks on buy.com. Placed it at the highest point in the house, and set it up to pull it's own IP address from DHCP off my existing wireless access point, so it's basically a range extender. For a $10 "TRENDnet", I was really impressed at how easy it was to setup. I tried the same thing with a Linksys several months back. The linksys would work on and off and just periodically die, and it cost something like $50. Anyway, I'm posting from my living room and receiving a good signal, so definitely worth the $10.

NoSQL: Quite Possibly The Stupidest Catch Phrase Ever?

2010-11-27T19:13:00.000-08:00

A recent article on Gigaom is titled "Why cloud computing sells and NoSQL is fading". I'd like to parlay that discussion into one of "why NoSQL is quite possibly the stupidest catch phrase ever". Here's where I'm coming from. Clearly the "NoSQL" moniker arose from a desire to describe systems that avoided the scalability limits of relational databases. But from the get-go, the term is just...STUPID. First of all, SQL is a query language, and there are relational systems that don't use SQL. So, "NoSQL" isn't even a good way to say "non-relational". Secondly, defining systems in terms of what they *don't* do means that dozens of systems that shouldn't get grouped together, do get grouped together. Corn: NoSQL! Windbreakers: NoSQL!

Adding the confusion was this classic: Map Reduce: A Major Step Backwards. It's amazing that this was written by Michael Stonebreaker, but this just goes to show you that even a genius can write a ridiculous article when he's writing for a PR blog. Then there was the 180 degree follow up (ooops, how stupid do we look?): Reaffirming Our Commitment to Hadoop and Mapreduce.

Honestly, the entire NoSQL thing is, to me, a mark of short sited PR as well as a failure to recognize that SQL databases can and must peacefully co-exist with systems designed to tackle other aspects of scalability and data processing. When Hadoop and Mapreduce get thrown in with "NoSQL" it especially drives me nuts, because Hadoop and Mapreduce are not even data retrieval or management systems! C'mon folks, Hadoop is for data processing. And there are lots of reasons to take the output and put it in a relational database.

So, I was happy to see the Gigaom article. The NoSQL emperor has no clothes. It's a stupid catch phrase, and a clear indicator that the person using it is clueless.

SPIRE 2010 Los Cabos

2010-10-24T13:36:00.000-07:00

I recently attended the String Processing and Information Retrieval conference (SPIRE 2010). In terms of content, this conference far exceeded my expectations. Excellent presentations on the latest research. Stuff I'd *never* even heard of: Wavelet Trees for example. Very friendly presenters, who enthusiastically spent time with me between presentations to thoroughly teach some of the concepts.

Yahoo! Research presents Time Explorer

Microsoft Research Keynote

No complaints about the venue either. Met so many cool and interesting researchers in the field of IR (which is basically "search").

Java SimpleDateFormat: simply a waste of Time

2010-10-24T12:27:00.000-07:00

I've now sunk probably 8 hours into trying to make a minimally robust date parser that can handle dates in a variety of formats. Sadly, I'm afraid this is the best I've been able to do, given the fragility of the Java SimpleDateFormat class. I've composed a little framework in two pieces. The first piece is a properties file that defines the formats that the user will be allowed to enter dates in. The second piece is the code that jumps through an absurd series of hoops in order to work around bugs and shortcomings in SimpleDateFormat. If anyone knows a better way, that's been *tested*, please, for the love of GOD, post a comment here.

DateFormats.properties

format1=yyyy-MM-dd HH:mm:ss.SSS z
format2=yyyy-MM-dd HH:mm:ss.SSS
format3=yyyy-MM-dd HH:mm:ss
format4=yyyy-MM-dd HH:mm
format5=E, MMM dd, yyyy
format6=MMM dd yyyy
format7=MMM dd yyyy HH:mm
format8=MMM dd, yyyy
format9=MMM. dd, yyyy
format10=MM/dd/yy
format11=MM/dd/yyyy
format12=M/d/yy
format13=yyyy-MM-dd
format14=yyyy-MM-dd, E
format15=yyyy MMM. dd, HHh 1mmm ss's'
format16=yyyy/M/dd/HH:mm:ss
format17=yyyy-M-dd'T'HH:mm:ss

Notice in the formats above that all months and days are defined with two format chars. E.g. "MM" or "DD". Why not "M" or "D"?  SimpleDateFormat format strings are very rigid. We want the user to be able to enter 10/1/07 or 10/01/07. But I don't want to clutter the properties file with single and double format chars. For this reason, the code automatically pads any single digits with zero. So "1" in isolation is converted to "01". So if the user enters "Jul 7, 2001" the string is automatically zero-padded to "Jul 07, 2001". This allows the properties file to include only the double-digit variants of date numbers without having to account for every variation in which a digit could be either single or double digit. With regard to the code below, I haven't taken time to extract it as a library, so please forgive the unexplained parameters such as "id" and "paramName".This code convers the time provided by the user into the current Greenwich Mean Time.

Java Source Fragments

private static final Pattern ISOLATED_DIGIT = Pattern.compile("(^|\\D|\\s)(\\d)(\\D|$|\\s)"); //digit can be surrounded by non-digit but also begining or end of line or whitespace


public static Object parseDate(String dateString, String id, String paramName) {
        if(null == dateString){
            return dateString;
        }
        if(dateString.equalsIgnoreCase("NOW")){
            //default time stamp is Zulu time (time at prime meridian, Greenwich)
            long currentTime = System.currentTimeMillis();
            Timestamp ts = new java.sql.Timestamp(currentTime-TimeZone.getDefault().getOffset(currentTime));
            return ts;
        }else{
            dateString = zeroPadIsolatedDigits(dateString);
            if(log.isDebugEnabled())log.debug("input zero-padded to " + dateString);
            for(Map.Entry entry:DATE_FORMATS.entrySet()){
                String key = entry.getKey().toString();
                DateFormat dateFormat = new SimpleDateFormat(entry.getValue().toString());
                dateFormat.setLenient(false);
                try{
                    java.util.Date date = dateFormat.parse(dateString);
                    
                    //I have found bugs in the parse method, in which strings that *don't* actually match the format template
                    //get parsed without Exception, thus truncating trailing parts of the date (like the timezone). For this reason
                    //I am forced to regurgitate the date string, and compare it to the input, and if they don't match, skip forward in the loop
                    String regurgitated = dateFormat.format(date);//...and no, we can't just compare the regurgitated string b/c PST and get adjusted to PDT
                    if(regurgitated.length() != dateString.length()){ //compare their lengths to make sure nothing got truncated
                        if(log.isDebugEnabled())log.debug(dateString + " was parsed with format " + entry.getValue() + " but was apparently parsed incorrectly as " + regurgitated + " length difference was "+(regurgitated.length()-dateString.length()));
                        continue;
                    }else{
                        //the length of the regurgitated string matches the length of the input.
                        //We still need to eat the regurgitated string, and compare the resulting  ms since epoch to the Date we originally parsed to make sure we didn't
                        //encounter a SimpleDateFormat parsing bug.
                         //Example: 2010-9-23 12:22:23.000 PST gets regurgintated as 2010-09-23 13:22:23.000 PDT, which is different text, but same ms since epoch
                        //So the above illustrates why we cannot just compare the regurgitated text to the input dateString, because Java may decide on an equivalent but
                        //textually different representation from the input (PST is the same as PDT in the half of the year when daylight savings time isn't in effect).
                        java.util.Date reparsed = dateFormat.parse(regurgitated);
                        if(reparsed.getTime() != date.getTime()){
                            if(log.isDebugEnabled())log.debug(dateString+" produces different ms since epoch than  " +regurgitated );
                            continue;
                        }
                    }
                    if(log.isDebugEnabled())log.debug("handled date" + dateString +" using format template: " + entry.getValue().toString() + " which regurgitated " + dateString);
                    TimeZone timeZone = dateFormat.getTimeZone();
                    //if(log.isDebugEnabled())log.debug(timeZone);
                    //convert to GMT time and account for timezone
                    return new java.sql.Timestamp(date.getTime()-timeZone.getOffset(date.getTime()));
                }catch(ParseException pe){
                    if(log.isDebugEnabled()){
                        log.debug(pe.getMessage());
                        log.debug(dateString + "couldn't be parsed with format " + entry.getValue());
                    }
                }
            }
            throw Util.newApplicationException(id, "date "+ dateString+" could not be parsed.","DataFormatException", 130, paramName);
        }
    }

     private static String zeroPadIsolatedDigits(String s) {
        //Java date parsing will prepend leading zeros to islolated digits. For this reason, I prepend a zero to the isolated digits before parsing the string, so that the regurgitation step
        //produces a string identical to the input
        boolean done = false;
        //this is an obscure case in which the matching regions overlap, therefore m.find() only finds one of the overlapping instance.
        //For instance, 100-1-2. Find() won't find "-2 " because it overalps "-1-". Consequently we have to repeatedly call find on a new matcher after each replacement has been made
        while (!done) {
            Matcher m = ISOLATED_DIGIT.matcher(s);
            StringBuffer buf = new StringBuffer();
            if (m.find()) {
                if(log.isDebugEnabled())log.debug("found isolated digit:" + m.group(2));
                m.appendReplacement(buf, m.group(1) + "0" + m.group(2) + m.group(3)); //need to stuff back in the stuff before the isolated digit, then 0 (zero-pad), then the digit, then stuff after)
            }else {
                done = true;
            }
            m.appendTail(buf); //OK, al isolated digits have been replaced by leading-zero padded digits
            s = buf.toString(); //replace input with zero-padded input
        }
        return s;
    }

Data Storm

2010-10-23T13:03:00.000-07:00

This morning's project was installing a Davis Vantage Pro weather station. It's really an amazing piece of equipment. It logs windspeed and direction, solar radiation, and dozens of other variabls. It even has a rain collector. The data comes off the unit in real-time, and is transmitted to a nice LCD console that shows graphs, weather forecasts etc. I hope to log enough data on sun and wind in order to make a determination as to whether solar panels or wind power will be my best alterative energy source. I'm also interested in quantifying just how bizarre the microclimate of western san francisco actually is. How many days is it foggy? What time does the fog typically roll in or out in a given month? With a bit of free time, I hope to be able to load the data into nextdb to create a detailed data history.

I'm already getting some valuable siting data on windspeed. There are several interesting vertical-axis helical wind-power generators that might be appropriate. Hmmm.

Government "Black Ice" Becomes a Reality

2010-09-25T08:02:00.000-07:00

Keeping with the William Gibson theme, the future - whoops, I meant "the present"- look increasingly like something dreamed up by said sci fi author. This week we learned about STUXNET: Government Black Ice designed to attack industrial control facilities, and in particular the Bashir Nuclear Reactor. The sophisticated source code includes the ability to inject itself into programmable logic devices; hacking the actual hardware itself, relying on proprietary hardware secretes stolen from two Taiwanese manufacturers. Apparently too complex to have been written by a hacker in a dorm room, the media concludes that it must be "state sponsored". Maybe. Probably. But in a present that increasingly looks like the cyberscape described in Neuromancer we shouldn't rule out multinational corporations as the source. As pointed out in the media, STUXNET *is* a *weapon*. Cyberspace is a battlefield where corporations, states, and individuals can wage war on a practically equal footing, with the ability to cloak their identity. Imagine that; anonymous war with motives known only to the participants. Target: a nuclear reactor.

Awesome Op-ed in New York Times by William Gibson

2010-09-01T18:09:00.000-07:00

This a fantastic Op-ed by William Gibson on Google.

Neuromancer Vs. Matrix

2010-07-15T21:44:00.000-07:00

Neuromancer was written by William Gibson and published in 1984. I first read it high school back in 1993. Every few years I read it again, and each time I find something new to like.

In my most recent reading, I find myself noticing how much of the concept matter in The Matrix is based on Neuromancer. Gibson coined the term "The Matrix" and "cyberspace", and Gibson describes a Rasta "freeside" called "Zion" (albeit in orbit), and simstim memory sticks that jack in through a socket behind the ear. It's hard to imagine that Gibson wrote Neuromancer before there was a world wide web.

"The Matrix is ... a concentual hallucination. ..A graphic representation of data abstracted from the banks of every computer in the human system. Unthinkable complexity. Lines of light arranged in the nonspace of the mind, clusters and constellations of data. Like city lights, receding..."

This part reminds me of the "symbols" that fall down the screens in the movie The Matrix:

"And in the bloodlit dark behind his eyes, silver phosphenes boiling in from the edge of space, hypnagogic images jerking past like film compiled from random frames. Symbols, figures, faces, a blurred fragmented mandala of visual information".

Android Earcon

2010-07-07T22:55:00.000-07:00

Did you notice that Google's Android phones have a startup sound that's quite familiar...that is, if you are a fan of Blade Runner. The little "bootup jingle" is from the opening scene soundtrack by Vangelis in Blade Runner. Listen here, its at 1 minute 40 seconds.

Hadoop Summit

2010-07-07T16:04:00.000-07:00

Last week I attended the Hadoop Summi t. It was sold out ten days prior! That doesn't happen very often these days, in an environment where corporate travel is severely cut back. I even saw sizable contingents from Japan and Brazil. This is a very exciting time to be involved in the field of search and data processing. The science track was quite interesting, as was the talk by facebook about how they are crunching something like 90 TB per week, and are siting on 85 Peta Bytes. They are using hadoop mapreduce in near real-time. Processing latencies are getting close to 1 minute. That's nearly real-time availability of all the information you post on facebook, to everyone else (and every moving part) in the facebook system. Oh, and the lunch was great.

badassery with bits: counting 1 bits

2010-07-06T21:28:00.000-07:00

I was recently looking at some code to count the number of bits in an integer that are set to 1. Here is the method:

int = getARandomInt();
while(n!=0){
n &= n-1;
count++;
}
print("There are "+count+" bits turned on.");

Can you figure out why it works?

hint: negative one has every bit turned on because negating a number is to take it's two's complement (which flips every bit), and add 1. Watch what happens with the carry bit when you add n to -1.

Matrix Challenge

2010-06-13T21:53:00.000-07:00

Here is an interesting challenge posted on the website of a company called Rapleaf:

We wish to find, for an arbitrary N, an N x N matrix with the following property: each entry in the matrix is the smallest non-negative number which does not appear either above the entry or to its left.

Write a function that computes the entry in a particular row and column. That is, complete the following:
```
int entry_at(int row, int column) {
    // ...
}
```
Can you write the entry_at function to run in constant space and constant time?
Can you prove that your algorithm is correct?

For instance, here is a 6x6

I posted a solution before, which was wrong, because I only looked at a 6x6, and my solution failed for larger matrices. Thanks to whomever posted the comment pointing out that my solution was broken. So I rewrote it in Java, after looking at some larger matrices. Here is a printout of the 32x32 matrix produced by the program.

 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 
 1  0  3  2  5  4  7  6  9  8 11 10 13 12 15 14 17 16 19 18 21 20 23 22 25 24 27 26 29 28 31 30 
 2  3  0  1  6  7  4  5 10 11  8  9 14 15 12 13 18 19 16 17 22 23 20 21 26 27 24 25 30 31 28 29 
 3  2  1  0  7  6  5  4 11 10  9  8 15 14 13 12 19 18 17 16 23 22 21 20 27 26 25 24 31 30 29 28 
 4  5  6  7  0  1  2  3 12 13 14 15  8  9 10 11 20 21 22 23 16 17 18 19 28 29 30 31 24 25 26 27 
 5  4  7  6  1  0  3  2 13 12 15 14  9  8 11 10 21 20 23 22 17 16 19 18 29 28 31 30 25 24 27 26 
 6  7  4  5  2  3  0  1 14 15 12 13 10 11  8  9 22 23 20 21 18 19 16 17 30 31 28 29 26 27 24 25 
 7  6  5  4  3  2  1  0 15 14 13 12 11 10  9  8 23 22 21 20 19 18 17 16 31 30 29 28 27 26 25 24 
 8  9 10 11 12 13 14 15  0  1  2  3  4  5  6  7 24 25 26 27 28 29 30 31 16 17 18 19 20 21 22 23 
 9  8 11 10 13 12 15 14  1  0  3  2  5  4  7  6 25 24 27 26 29 28 31 30 17 16 19 18 21 20 23 22 
10 11  8  9 14 15 12 13  2  3  0  1  6  7  4  5 26 27 24 25 30 31 28 29 18 19 16 17 22 23 20 21 
11 10  9  8 15 14 13 12  3  2  1  0  7  6  5  4 27 26 25 24 31 30 29 28 19 18 17 16 23 22 21 20 
12 13 14 15  8  9 10 11  4  5  6  7  0  1  2  3 28 29 30 31 24 25 26 27 20 21 22 23 16 17 18 19 
13 12 15 14  9  8 11 10  5  4  7  6  1  0  3  2 29 28 31 30 25 24 27 26 21 20 23 22 17 16 19 18 
14 15 12 13 10 11  8  9  6  7  4  5  2  3  0  1 30 31 28 29 26 27 24 25 22 23 20 21 18 19 16 17 
15 14 13 12 11 10  9  8  7  6  5  4  3  2  1  0 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 
16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31  0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 
17 16 19 18 21 20 23 22 25 24 27 26 29 28 31 30  1  0  3  2  5  4  7  6  9  8 11 10 13 12 15 14 
18 19 16 17 22 23 20 21 26 27 24 25 30 31 28 29  2  3  0  1  6  7  4  5 10 11  8  9 14 15 12 13 
19 18 17 16 23 22 21 20 27 26 25 24 31 30 29 28  3  2  1  0  7  6  5  4 11 10  9  8 15 14 13 12 
20 21 22 23 16 17 18 19 28 29 30 31 24 25 26 27  4  5  6  7  0  1  2  3 12 13 14 15  8  9 10 11 
21 20 23 22 17 16 19 18 29 28 31 30 25 24 27 26  5  4  7  6  1  0  3  2 13 12 15 14  9  8 11 10 
22 23 20 21 18 19 16 17 30 31 28 29 26 27 24 25  6  7  4  5  2  3  0  1 14 15 12 13 10 11  8  9 
23 22 21 20 19 18 17 16 31 30 29 28 27 26 25 24  7  6  5  4  3  2  1  0 15 14 13 12 11 10  9  8 
24 25 26 27 28 29 30 31 16 17 18 19 20 21 22 23  8  9 10 11 12 13 14 15  0  1  2  3  4  5  6  7 
25 24 27 26 29 28 31 30 17 16 19 18 21 20 23 22  9  8 11 10 13 12 15 14  1  0  3  2  5  4  7  6 
26 27 24 25 30 31 28 29 18 19 16 17 22 23 20 21 10 11  8  9 14 15 12 13  2  3  0  1  6  7  4  5 
27 26 25 24 31 30 29 28 19 18 17 16 23 22 21 20 11 10  9  8 15 14 13 12  3  2  1  0  7  6  5  4 
28 29 30 31 24 25 26 27 20 21 22 23 16 17 18 19 12 13 14 15  8  9 10 11  4  5  6  7  0  1  2  3 
29 28 31 30 25 24 27 26 21 20 23 22 17 16 19 18 13 12 15 14  9  8 11 10  5  4  7  6  1  0  3  2 
30 31 28 29 26 27 24 25 22 23 20 21 18 19 16 17 14 15 12 13 10 11  8  9  6  7  4  5  2  3  0  1 
31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10  9  8  7  6  5  4  3  2  1  0

The valAboveDiag method takes advantage of the fact the matrix is symetric about the main diagonal. For values below the diagonal, the row and column numbers are simply transposed. To compute an arbitrary value in an NxN matrix, the method executes O(log(N)). However, if I always intended to print the entire matrix, I could have rewritten this to take advantage of values already stored in the matrix. In that case, adding a new value to the matrix would occur in constant time. The central idea behind the code below, is that the matrix is built up of matrices whose sizes are powers of 2 (e.g. 2x2,4x4,8x8, etc), organized in a pattern [[A,B][B,A]]. For a given cell, above the main diagonal, the cell is either in quandrant [0,0], [0,1], or [1,1]. Because of the pattern [[A,B][B,A]], this means that quadrant [0,0] is A, quadrant [0,1] is B, and [1,1] is A. Observing that B=A+mid, where "mid" is the "middle value" midway between 0 and the maximum value of the power-of-two block, we can only need to be able to produce A to find any other value in the power-of-two block. This process simply recurses until A is found to be the identity matrix (i.e. [[0,1],[1,0]]).

package random;

import java.util.Arrays;

/**
 *
 * @author geoffreyhendrey
 */
public class MadMatrix {
    int[][] a;

    public MadMatrix(int size){
        a = new int[size][size];
        for(int r=0;r<size;r++){
            for(int c=0;c<size;c++){
                a[r][c] = valAboveDiag(r,c);
            }
        }
    }

    @Override
    public String toString(){
        StringBuilder buf = new StringBuilder();
        for(int[] row:a){
            for(int column:row){
                buf.append(String.format("%2d ", column));
            }
            buf.append("\n");
        }
        return buf.toString();
    }


    private int valAboveDiag(int r, int c) {
        if(r>c){
            int tmp = c;
            c=r;
            r=tmp;
        }
        int order = getOrder(c);
        if(0 == order){
            if(r==c) return 0;
            return 1;
        }
        int mid = 1<<order;
        if(c >=mid && r<mid){
            return valAboveDiag(r, c-mid) + mid;
        }else{
            return valAboveDiag(r-mid, c-mid);
        }
    }

    private static int getOrder(int c){
        int order = 0;
        while(c > 1){
            c >>>=1;
            order++;
        }
        return order;
    }

    /**
     * @param args the command line arguments
     */
    public static void main(String[] args) {
        MadMatrix mm = new MadMatrix(32);
        System.out.println(mm);
    }

}

AES Encryption in Java

2010-04-04T18:59:00.000-07:00

I wrote this block of code a couple years ago, but it has served me well. I remember it being quite a challenge to get it to work properly. There was, and maybe still is a dearth of decent examples on using AES encryption from Java. The examples I found didn't deal with Cypher Block Coding (CBC). CBC is essential to decent encryption because it feeds the output of one cyphered block forward into the next, thus creating a cascade of randomness. All you need is an initial random phrase in the data you wish to encrypt. If you don't use CBC, you'll notice that blocks of your encrypted data can easily be recognized as separators in what you are encrypting. For example, let's say I encrypt the phrase "user:bob:password:abc". The substring "user:" and "password:" will cause an identifiable pattern in the encrypted data. If someone has access to snoop the wire, they can literally lift the encrypted username and password out of someone else's transmission, and paste it into their own. While CBC with a leading random phrase will make it impossible to identify such non-randomness, you still have to guard against someone injecting cruft into the encrypted data. This should be done by appending an MD5 hash of the unencrypted data, then encrypting the whole ball of wax.

    private static String encrypt(String encryptMe, byte[] cryptKey, SecretKey secretKey, byte[] iv){
        try {
            Cipher cipher = Cipher.getInstance("AES/CBC/PKCS5Padding", "SunJCE");
            cipher.init(Cipher.ENCRYPT_MODE, secretKey, new IvParameterSpec(iv));
            byte[] raw = encryptMe.toString().getBytes("ASCII");
            if(log.isDebugEnabled())log.debug("unencrypted bytes: " + raw.length);
            byte[] cipherText = new byte[cipher.getOutputSize(raw.length)];
            int ctLength = cipher.update(raw, 0, raw.length, cipherText, 0);
            ctLength += cipher.doFinal(cipherText, ctLength);
            if(log.isDebugEnabled())log.debug("ctLength: " + ctLength);
            if(log.isDebugEnabled())log.debug("cipherText Length: " + cipherText.length);
            //raw = cipher.doFinal(raw, 0, raw.length);
            byte[] copy = new byte[ctLength];
            System.arraycopy(cipherText, 0, copy, 0, ctLength);
            String encrypted = new String(new Base64().encode(copy), "US-ASCII");
            return encrypted;
        } catch (Exception ex) {
            throw new RuntimeException(ex.getMessage(), ex);
        }        
    }

what's in your history?

2010-04-02T20:22:00.000-07:00

[ghendrey@geoffunix hadoop]$ cut -f1 -d" " ~ghendrey/.bash_history | sort | uniq -c | sort -nr | head -n 20
    206 hadoop
     44
     27 ls
     23 cd
     18 su
     17 grep
     15 tac
     11 pwd
     10 exit
      8 env
      5 less
      4 /mnt/hadoop/hbase-0.20.3/bin/hbase
      4 man
      4 ifconfig
      3 telnet
      3 start-dfs.sh
      3 source
      3 mount
      3 ant
      2 vi

Webscale Computing and Hadoop

2010-02-23T19:11:00.000-08:00

I've been using Hadoop extensively for a month or so now at the office. I've been a big fan of Doug Cutting's Lucene technology, so when I heard that Doug Cutting was the guy behind, Hadoop, that pretty much pushed me over the edge and I started using Hadoop for building new types of database indexes from multi-gigabyte datasets.

Hadoop is an incredibly rich software stack consisting of three main pieces:

The Hadoop Distributed File System (HDFS)
A Map/Reduce infrastructure consisting of Daemons that layer services on top of HDFS
A set of APIs allowing a data processing task to be split into simple Map and Reduce phases

There's a fourth piece, which is the many related tools (HBase, Hive, Pig, etc) which are layered on top of Hadoop.

If one was to read the description of map/reduce one might conclude that it is pretty much nonsense. In fact, it sounds to trivial to even be called an algorithm. Put things into groups, operate on the groups. Big deal. It sounds pretty much like common sense. Until you work with Hadoop, you really cannot appreciate all of the benefits that the Hadoop stack brings. It's really the collective benefits of the entire stack that make for the game changing experience that programming with Hadoop really is.

With problems of processing huge datasets, the devil is in the details. Hadoop provides a framework that removes all of the annoying complexity associated with big data sets. In essence you write two simple methods and from that point on you don't care much whether you are operating on 1 byte or 1 TB. This is possible in large part because HDFS allows you to think of terrabytes worth of data as a simple URL such as hdfs://namenode/user/geoff/BigDataSet. HDFS takes care of replicating your data, and figuring out where the data blocks actually reside in the cluster. As an added bonus, Hadoop automatically deals with nuances such as the files being zipped. But wait, there's more. From the hadoop command line, you can run commands to cat and grep these URLs, again, acting as if they were simple local files.

For me, one of the most interesting side effects of running a hadoop cluster has been how it changes ones ability to interact with peers on a webscale computing project. You can now shoot your co-worker a URL to a 200GB file, and they can poke and prod at the contents by doing things like "hadoop fs -text hdfs://namenode/user/geoff/BigData".

I'll try to write more about this later, but for now suffice it to say that Hadoop is quite exciting to work with. I think the criticism by Michael Stonebreaker have totally missed the point of what a good implementation of a map/reduce framework can yield, probably because their criticism focused on the map/reduce algorithm, which in and of itself is trivial. And that's really the point. It's really all about the tools and the entire stack that make map/reduce simply a "part of this complete breakfast" when it comes to hadoop. So don't forget the toast, juice, and yummy Cheerios!

HATEOAS - Hypermedia As The Engine Of Application State

2010-02-21T12:24:00.000-08:00

Roy Fielding's PhD dissertation on Architectural Styles and the Design of Network-based Software Architectures is the seminal paper on the RESTful style of web architecture. Before discussing the paper, let's first discuss what it is, because it seems quite unusual in its approach as a PhD dissertation. Reading this paper in the 21st century, one might conclude that the paper is an "experience paper" (as defined here). However, as noted, "An experience paper that merely confirms that which is well known as opposed to that which is widely believed is of little value." However, in the early 1990's, the topic of the dissertation was probably not widely known. The irony now, is that as the term REST becomes more and more commonplace, and is applied to non-RESTful systems, the level of confusion over REST might be rising, not falling.

I was certainly confused for a long time about what constitutes a RESTful system. Isn't anything that has a URL RESTful? Definitely not, as the presence of resource identifier is just one of several properties that make a system RESTful. Fielding has recently written this article listing the properties he sees as essential to a RESTful system. After working quite a bit on the RESTful system for NextDB.net, it seems that me that the principle of Hypermedia As The Engine Of Application State (HATEOAS), identified on the original dissertation section 5.1.5, might be the most important principle of all. Incidentally, of all the properties outlines in the recent article, Fielding chose to title the article "REST APIs Must be Hypertext Driven". To my mind, that confirms my hunch that HATEOAS is perhaps the least understood principle of REST, and the principle in need of the most discussion and explanation.

The essence of HATEOAS is that the client needs to understand as little as possible. In essence, this is quite similar to the experience a human requires when he browses a website. In order to move from one page to the next, we must be given links that we can navigate. This is our common experience on web pages. Consider an alternative, and less user friendly, scenario in which the home page simply consisted of a set of instructions (an API, if you will). One instruction might read "To view articles about sports, visit a URL consisting of "http://sitename/topics?name=topic-name" and replace topic-name with 'SPORTS'". You would be forced to manually construct the URL and enter it into the browser. The notion of such a website seems absurd, and yet it describes the scenario common to RPC or "fake REST" APIs in which the client is responsible for constructing the state-transition links rather than being handed a hypermedia document that provides the available state transitions.

Quoting Fielding: "...all application state transitions must be driven by client selection of server-provided choices that are present in the received representations or implied by the user’s manipulation of those representations".

This is an interesting article that shows how non-HTML hypermedia can be used in keeping with HATEOAS. However, I think there are excellent reasons why Plain Old HTML (POH) is the best hypermedia choice for use with RESTful systems:

HTML was made for hypermedia (it has well known structures for encoding links)
Your mom knows what HTML is
By using POH for HATEOAS, your RESTful system's content can be indexed by search engines
You can easily view and traverse your system states using a web browser

POH for HATEOAS is the approach taken by NextDB's REST system. In the documentation, we do describe the explicit structure of several of the URL's, but we stick strongly to the principle outlined by Fielding that, beyond the entry point URL, or "bookmark" you shouldn't need to know the structure of any of the URLs. Using POH for HATEOAS, the proof of NextDBs' adherence to this principle is quite straightforward: I simply give you an entry point URL (a bookmark) to a table and allow you to navigate the links served in the POH. For example, here is the bookmark to an account named 'geoff' with a database named 'testchars' and a table named 'lines':

http://nextdb.net/nextdb/rest/geoff/testchars/lines/

From the URL above you can click on links to visit various pages of the content and sort the content. No knowledge of an API is required. Taken to its logical conclusion, I see no reason why the RESTful service should not be allowed to directly present styled content, when the consumer of the content is human. So that is exactly what we do in NextDB, by allowing the POH to include CSS. Here is an example:

http://nextdb.net/nextdb/rest/geoff/testchars/lines/style/newspaper-a

We're techically violating HATEOAS because we don't provide links to the styled content, rather the developer has to know about the available styles by reading our documentation. However, we'll soon correct this, as well as allowing the developer to pass in links to his own CSS for inclusion in the returned POH.

Finally, not only do we allow you to style the POH, but we also allow you to apply XSLT transformations to the POH in order to alter the structure (as opposed to style) of the POH. Fielding discusses "Processing Elements" or "Transformers of Data" in his thesis. I believe XSLT to be the POH analog for processing elements, in that they are well understood, easily encapsulated in a markup, and supported by a wide array of processors (including browser-side processing, although this tends to be less portable, which is why NextDB opted to perform the transformation on the server).

In summary, NextDB is a truly RESTful database. This is important not because it conforms to a buzzword, but because it has made NextDB so easy to use that even non-programmers are able to embed the POH hypermedia in their sites. Project Have Hope is a great example of one such site. The "catalogs" of sponsorable children and women in Africa is POH served out of NextDB.net. The HATEOAS architecture is the key to opening the database content to CMS engine-driven sites, as well as accessibility to web search engines.

Spam Bucket

2010-02-15T10:37:00.000-08:00

I'm curious to see how much spam accumulates in this database if I put a publicly writable database table on my blog

UPDATE: I have the answer: "A LOT"
"Salad Man", I commend you! "get three inches on your dick!" is a noble spam to leave on my blog, but I am dissapointed you did not include a video!!!

NextDB REST tables

2010-02-13T21:34:00.000-08:00

NextDB REST tables can be easily embedded anywhere on the web. For example:
<iframe src='http://www.nextdb.net/nextdb/rest/geoff/vids/sk8;~cols=PK,video_CONTENT_TYPE,video_FID/style/newspaper-a/pgsz/3' />

An update on JTIdy and XSLT

2010-02-12T21:52:00.000-08:00

A couple months back I blogged about JTidy I have an update to that story. If you plan to run your XHTML through an XSLT transformation, don't use tidy.setXHTML(true). The reason for this, I found after a lot of debugging. There are a "named entities", specifically ones such as â (and many others) that are declared in the XHTML DTD. And guess what? They are NOT valid in XML.

Quick refresher: XML has only five named entities that it supports.

So you're thinking, no biggie, I don't think my documents will ever have a â named entity in them so what do I care? Well, they can creep in my accident, if you are allowing users to save inputs into a database. For example, there are ISO-8859-1 (Latin-1) characters with no UTF-8 equivalent. One that I have repeatedly seeen over the years from European users is the "left quote". It doesn't even exist on an american keyboard, and when it gets posted to a system expecting UTF-8 it wreaks havoc, causing a run of several incorrectly decoded characters.

And so, you can get "inadvertant" named entities in your XHTML output due to this sort of character mangling where the UTF-8 byte stream interpretation gets borked. So now, instead of just getting some gibberish in your XSLT, the entire transformation crashes, and you get no output at all except a complaint that the input document had an unsupported character reference. Hence the solution: tell jtidy NOT to produce XHTML, but instead to produce plain old XML, like this: tidy.setXmlOut(true); When you do this JTidy doesn't put these named entities in anymore.

One other surprise I ran into is that XML actually *DOES* support numeric character entities. So you will still get numeric character entities in the XML output such as &#128, which although specifically forbidden by HTML still render properly in most browsers. So JTidy happily outputs these numeric character entities in its XML output. You are now safe to apply XSLT transformations on a valid XML input, however, just be aware that the XHTML output of your XSLT transformation is in violation of the aforementioned forbidden, but practically allowed, numeric entities.

Confused yet? Me too. Let's all speak Esperanto and adopt 7-bit ASCII as the only allowed character set.

Character Encoding Hell

2010-01-21T22:16:00.000-08:00

Over the years, one of the perennial hassles that web services programmers grapple with, is undoubtedly character encoding. One of the biggest contributing factors to the problem goes all the way back to the fact that the original definition of URL-encoding failed to specify how to deal with UTF-8 or other non-ascii encodings outside the reserved character set. The current spec states that if you want to send UTF-8 strings, for example "てすと (te-su-to) ", then you should percent encode each byte of the UTF-8 character sequence (%E3%81%A6).

Two excellent pages for test data, should you need to test some multi-byte UTF-8 characters are:

http://winrus.com/utf8-jap.htm and http://winrus.com/codes.htm

Sadly for AJAX applications, actually generating these percent encoded byte sequences is non-trivial, and doesn't seem to be readily available "off the shelf". I've had to resort to JS source such as this.Without such scripts, various attempts to use "off the shelf" JS methods produce wacky results, such as unicode representations of the strings (like \uXX\uXX) which is completely useless for transmission in a URL.

On the server, there are also problems in receiving and properly decoding the bytes. One of the biggest problems is that Java web servers don't do it in a standard way. For example, when I was working on Tomcat 5.x, the Servlet getParameter method would assume ISO-8859-1 (Latin 1) encoding, which would garble any properly UTF-8 percent encoded bytes. There is a well hidden setting to switch the default to UTF-8.

On the other hand, Jetty *does* assume the URL is UTF-8 percent encoded (love jetty!!). So without some config tweaks, don't expect servlet containers to *uniformly* deliver properly decoded UTF-8 strings.

My latest installment of UTF-8 Hell comes as I implement some more REST capabilities for Nextdb.net. Here is the content of a message I just posted to Paul Sandoz of Jersey fame (man, that guy must never sleep. he is *on the ball* on the Jersey mailing list). Mad props to Jersey. It is just killer. But I digress:

Hi Paul,

I got to the bottom of this by trying to unmarshal the string in three different ways. As I already mentioned the first way was just to call FormDataBodyPart.getValue().

toString(). This produced the improperly decoded String.

Then I tried two other ways, both of which correctly unmarshalled the bytes from the POST. As supporting information, here is the CURL line I was testing with, and an excerpt from the CURL trace, showing the proper bytes being posted.

curl --trace traced -F line=てすと http://localhost:8080/nextdb/rest/geoff/testchars/lines/rowid/PK/1

The 9 bytes highlighted below are the three japanese characters.

=> Send data, 148 bytes (0x94)
0000: 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d ----------------
0010: 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 63 61 --------------ca
0020: 33 32 31 65 31 30 39 66 36 37 0d 0a 43 6f 6e 74 321e109f67..Cont
0030: 65 6e 74 2d 44 69 73 70 6f 73 69 74 69 6f 6e 3a ent-Disposition:
0040: 20 66 6f 72 6d 2d 64 61 74 61 3b 20 6e 61 6d 65 form-data; name
0050: 3d 22 6c 69 6e 65 22 0d 0a 0d 0a e3 81 a6 e3 81 ="line".........
0060: 99 e3 81 a8 0d 0a 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d ......----------
0070: 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d ----------------
0080: 2d 2d 2d 2d 63 61 33 32 31 65 31 30 39 66 36 37 ----ca321e109f67
0090: 2d 2d 0d 0a

Both of the following two methods will properly unmarshal the correct String:

method 1: use InputStream to get raw bytes

                    InputStream is = theFormDataBodyPart. getValueAs(InputStream.class);
                    try {
                        byte[] bytes = Util.readInputStream(is, 1024 * 1024, 1024 * 1024 * 1024);
                        log.debug("this many bytes: " + bytes.length);
                        for(byte b:bytes){
                            log.debug(Integer.toHexString(0x00FF&b));
                        }
                        String s = new String(bytes, "UTF-8");
                        log.debug(s);
                        return s;
                    } catch (IOException ex) {
                        throw new RuntimeException(ex.getMessage(), ex);
                    }

Method 2) use theFormDataBodyPart.getValueAs(String.class)

Cheers,
geoff

Well said....

2010-01-04T21:07:00.001-08:00

The exchange between Churchill & Lady Astor: She said, "If you were my husband I'd give you poison." He said, "If you were my wife, I'd drink it."

A member of Parliament to Disraeli: "Sir, you will either die on the gallows or of some unspeakable disease." "That depends, Sir," said Disraeli, "whether I embrace your policies or your mistress."

"He had delusions of adequacy." - Walter Kerr

"He has all the virtues I dislike and none of the vices I admire." - Winston Churchill

"I have never killed a man, but I have read many obituaries with great pleasure." Clarence Darrow

"He has never been known to use a word that might send a reader to the dictionary." - William Faulkner (about Ernest Hemingway).

"Thank you for sending me a copy of your book; I'll waste no time reading it." - Moses Hadas

"I didn't attend the funeral, but I sent a nice letter saying I approved of it." - Mark Twain

"He has no enemies, but is intensely disliked by his friends." - Oscar Wilde

"I am enclosing two tickets to the first night of my new play; bring a friend.... if you have one." - George Bernard Shaw to Winston Churchill

"Cannot possibly attend first night, will attend second... if there is one." - Winston Churchill, in response.

"I feel so miserable without you; it's almost like having you here." - Stephen Bishop

"He is a self-made man and worships his creator." - John Bright

"I've just learned about his illness. Let's hope it's nothing trivial." - Irvin S. Cobb

"He is not only dull himself; he is the cause of dullness in others." - Samuel Johnson

"He is simply a shiver looking for a spine to run up." - Paul Keating

"In order to avoid being called a flirt, she always yielded easily." - Charles, Count Talleyrand

"He loves nature in spite of what it did to him." - Forrest Tucker

"Why do you sit there looking like an envelope without any address on it?" - Mark Twain

"His mother should have thrown him away and kept the stork." - Mae West

"Some cause happiness wherever they go; others, whenever they go." - Oscar Wilde

"He uses statistics as a drunken man uses lamp-posts... for support rather than illumination." - Andrew Lang (1844-1912)

"He has Van Gogh's ear for music." - Billy Wilder

"I've had a perfectly wonderful evening. But this wasn't it." - Groucho Marx

JavaOneRadio interview 2009

2009-12-16T23:18:00.000-08:00

Was just going through some photos from earlier this year. Here's a shot of the interview I did on JavaOneRadio about location based services and my work at deCarta.

Nerds!

2009-12-13T18:52:00.001-08:00

a==1 or 1==a

2009-12-12T22:31:00.000-08:00

I was in one of Dave Nagle's lectures at Carnegie Mellon in which he explained that he'd once spent an inordinate amount of time on a bug hunt. The cause: 'a=1' ... a typo. He meant to type 'a==1'. To insure he'd never suffer that pain again, he coded all his boolean comparisons with the literal on the left hand side, like this: '1==a'. This prevents the typo '1=a' because the compiler won't allow an assignment to a literal.

After going on the exact same bug hunt myself, many years ago, I kicked myself for not following his advice. I've coded all my Boolean expressions in the "literal on the left" style ever since, and the compiler has saved me many a bug hunt.

To that paradigm I now have a small addition to make. I've started coding my string equals comparisons in literal-left style as follows: "something".equals(a)

This has the nice effect warding off null pointer exceptions when a is null. I think this literal left style is better than (a+"").equals("something") which is another trick for warding off null values of a.

Keep the hot air flowing

2009-12-11T17:37:00.000-08:00

This is a funny satire of the climate change talks..."now powered exclusively by wind".

Keep it on the DL

2009-12-05T22:21:00.000-08:00

I came across what has got to be the most rarely used HTML element. Ever heard of <dl>? me neither. Stands for "Definition List". Inside a DL you put Definition Terms (DT) and inside a DT you put a Definition Description (DD). It actually is a very handy way to markup key/value pairs, such as "firstname: bob" or "Age:34". So if you've got some kind of Object and you want to markup its fields, give the DL a try.

JTidy and handling HTML embedded in database strings

2009-12-03T23:35:00.000-08:00

I've been dealing with an interesting issue. A customer's app needs to store documents created by end users. The document's are really just paragraphs created by the end user, using the YUI Rich Text/HTML editor. The user might also use the "plain text" editing mode, in which case they would actually enter tags like < h1> manually. With this freedom comes the inevitable possibility that they will enter invalid HTML. Since these documents will be stored in NextDB the user might also apply an XSLT transformation over the default HTML presentation of the result set.

The problem is that if the user enters invalid XHTML, like they just throw <blort&rt; into their document, then the XSLT will crash. The problem is made gnarlier by the fact that the documents themselves are likely to be fragments (and in fact they have to be fragments so that when they appear in the context of the default HTML presentation that they get treated as a valid portion of the overall document). In other words, if the user puts tags around their document, you have to strip those body tags off, so that the default HTML presentation doesn't have a nested body.

Enter JTidy. The JTidy library is all about handling crapped-up HTML, and fixing it on the fly. The journey to getting JTidy working was longer than I expected due to an unpleasant interaction with Maven. So, despite the fact that I had the most recent version of JTidy on my Netbeans's project's classpath, when I used Maven to startup Jetty using the mvn Jetty plugin, I would get runtime errors complaining that the method I was trying to call (tidy.setPrintBodyOnly(true);) didn't exist. So, like any good bughunt, the fub began. I knew that Tidy.class was somehow sneaking into my runtime classpath. The first place I looked was my local .m2 repository -- no joy. Finally occured to me that this must be an internal inclusion of an old Tidy jar by maven. When I ran 'grep -r Tidy.class' on my maven directory, I found that maven's 'uber jar' (maven-core-2.0.7-uber.jar) did in fact contain the older version of JTidy. Turns out if you look at the internal dependencies for Maven, you find that it depends on an old version of JTidy. So I unzipped the uber-jar, replaced all the JTidy classes with the latest and greatest, rezipped the jar, and ...badabing badaboom...problem solved. Bing bang boom, very good have a drink.

The actual coding took less than 2 minutes. Argghh talk about an 80/20 rule. Anyhow, JTidy does exactly what I want. Fix any crufted HTML that the user might enter, and then extract only the content of the body element (and even better, if the user doesn't include a body, JTidy fixes that first). So it all boils down to this (just to print the tidied content):

Tidy tidy = new Tidy();
tidy.setXHTML(true);
tidy.setPrintBodyOnly(true);
tidy.parse(new ByteArrayInputStream((val+"").toString().getBytes()), System.out);

Epilogue: maybe blogger.com should start using Jquery. Ironically, this rich text editor threw up on some tags I typed into this post, and I had to spend a few minutes cleaning up the mess!!!

Good news on C02 Emissions

2009-11-26T10:06:00.000-08:00

Obama's announcement, combined with China's recent pledge, is the first good news I've heard on climate change -- ever! Happy Thanksgiving!

Viterbi Maximum Liklihood Decoding

2009-11-25T08:15:00.000-08:00

Last night I was following up on a hunch that the Viterbi Maximum Liklihood Decoding algorithm could have application for spell checking. I remembered this algorithm from a class I took at Carnegie Mellon called Digital Communications. I went back to the course book, and started reviewing the matter. I hadn't remembered this course book as standing out from others, but now I can really appreciate how well written it is. It has very clear treatment of the subject matter with well broken-out examples, that build throughout the topic. Sklar is a great writer, and he clearly loves the subject matter. I like his pithy conclusions at the end of each chapter. He has a flair for understatement:

"In the last decade, coding emphasis has been in the area of convolutional codes since in almost every application, convolutional codes outperform block codes for the same implementation complexity of the encoded-decoder. For satellite communications channels, forward error correction techniques can easily reduce the required SNR for specified error performance by 5 to 6dB. This coding gain can translated directly into an equivalent reduction in required satellite effective radiated power (EIRP), which consequently reduced satellite weight and cost."

After reviewing the Viterbi algorithm, I was curious about who this Viterbi guy was, anway. I'd also seen a chart in the Sklar book, attributed to the "Linkabit" corporation. Turns out Viterbi founded Linkabit, then later Qualcomm, an eventually donated $50 million to USC where the Viterbi school of engineering bears his name. Interesting story.

Oh, and it turns out the Viterbi algorithm does have applications to spell checking.

Order-invariant Path Parameters with JSR-311 JAX-RS and Jersey

2009-11-22T13:20:00.000-08:00

When defining web service resource URLs, you may find yourself in the situation in which you want path parameters to be viable in any order. For example, shouldn't these two URL's both be accaptable, and point to the same resource?

.../person/firstname/geoff/lastname/hendrey
.../person/lastname/hendrey/firstname/geoff

I've found that the best way to do this is to creat a single resource class, in this case Person, and to define subresource locator methods for "firstname/{firstname}" and "lastname/{lastname}". The trick is to return "this" in your subresource locator method, which affectively turns the subresource locators into a means to setting instance fields on the Person instance.

So when we look in the "PersonResource" class, our subresource locator method for 'firstname' is:

@Path("/person")
public class PersonResource{
private String firstname;
private String lastname;

@Path("firstname/{firstname}") //subresource
public Person getSubResource(@PathParam("firstname") String fname){
this.firstname=fname; //set instance field on this resource
return this;
}

@Path("lastname/{lastname}") //subresource
public Person getSubResource(@PathParam("lastname") String lname){
this.lastname=lname; //set instance field on this resource
return this;
}
}
The locator method for lastname follows the same patter. And that's all there is to it! I have not found this pattern covered or described in any of the docs for Jersey, but I have found it indespensible.

Creating "alias" URLs for Jersey JAX-RS JSR-311 resources

2009-11-22T13:06:00.000-08:00

In NextDB.net REST we are supporting two syntaxes for accessing resources. Let's call it a "fully qualified" resource, and a "shortcut" to the resource. For example, to view a table with its fully qualified URL, you can enter a URL in the format:

http://nextdb.net/nextdb/rest/home/{account}/db/{db}/tbl/{tbl}

for example: http://nextdb.net/nextdb/rest/home/geoff/db/testchars/tbl/lines

The "shortcut" URL would be: http://nextdb.net/nextdb/rest/{account}/{db}/{tbl}

for example http://nextdb.net/nextdb/rest/geoff/testchars/lines

There's a simple "trick" you can apply using JSR-311 to make your resource appear at two different URLs. First, start with your "fully qualified resource". For example, here is snippet:

@Path("/home/{account}/db/{db}/tbl/{tbl}")
public class TableResource extends AbstractSelectableResource{
@PathParam("account") protected String account;
@PathParam("db") protected String db;
@PathParam("tbl") protected String table;
...

Now, to make an "alias" or "shortcut" URL, just define a dummy class that extends the resource you want to make a shortcut to. The purpose of this class is simply to "mount" the resource on an additional URL. Be sure to include all the path parameters and their names must be identical.

@Path("/{account}/{db}/{tbl}")
public class ShortcutToTableResource extends TableResource{
//this class is intentionally empty
}

Now your resource will be available on both URLs!

Jamwiki

2009-11-11T22:43:00.001-08:00

A few weeks ago I installed JamWiki to document the new NextDB REST features:

http://www.nextdb.net/wiki/en/REST

I wasted a lot of time on XWiki, which basically was bloated, didn't work well, and basically produces a constant stream of stack traces. JamWiki works really well, it's nice and clean.

New ideas for REST in NextDB.net

2009-11-08T09:46:00.000-08:00

I've got some more stuff to work on with regard to REST. Here's a
peek:

support for relationships in REST URLs (calling them 'links')

...account/db/link/FRIENDSHIP/from/uname/john/to/uname/tom

The path parameters "from" and "to" describe the tail and the head of
the relationship, respectively. In other words, we can draw a
relationship as an arrow from one table to another table. This makes
'from' and 'to' the natural path parameters.

The REST URL above is equivalent to:

NAME=FIND_FRIEND;
ROW f1 FROM FRIEND;
ROW f2 FROM FRIEND;
f1 RELATED f2 VIA FRIENDSHIP;
WHERE(){
f1.uname='john' AND f2.uname='tom'

}

In the REST URL, you don't have to specify any constraint on the 'to'
side of the relationship. The following URL would retrieve *all* of
john's friends.

..account/db/link/FRIENDSHIP/from/uname/john

is equivalent to:

NAME=FIND_FRIEND;
ROW f1 FROM FRIEND;
ROW f2 FROM FRIEND;
f1 RELATED f2 VIA FRIENDSHIP;
WHERE(TEXT name1){
f1.uname=${name1}

}

You'll also be able to execute your named queries directly via rest:

.../account/db/query/FIND_FRIEND/name1/john

So it all will play nicely with existing queries you have already
defined, and new ones you may define in the future.

I also intend to support cookies. So let's say you wanted to log a
user in via https, and then securely transmit his password with each
subsequent REST URL using regular HTTP (not https). First the user
logs in via a secure (httpS) rest access to a query you named LOGIN.
So the password below would not be in the clear, since the form on the
HTML page would GET to an HTTPS url. The URL is constructed
dynamically by the form on submission:

https://nextdb.net/...db/query/LOGIN/uname/geoff/pwd/xxxxxxx;set-cookie

the 'set-cookie' matrix parameter tells nextdb to return a set-cookie
HTTP header with a value equal to cypher('xxxxxxxx'). I.e., nextdb
will encrypt the value that was sent in your request's REST URL,
wherever it finds ;set-cookie, and tell the browser to pass a cookie
named 'pwd' (in this example) with a value that is the encrypted
version of whatever is in the REST URL for the password in the pwd/
{password} path parameter. The set-cookie matrix parameter can occur
at any path segment in the URL, and would be applied to whatever path
parameter is described by the path segment.

from that point on you can use regular rest URLs and nextdb will
substitute into your URL the encrypted cookie value. So you can use
URL's like this in your application, for later actions that require a
user's password:

.../db/link/ACCOUNT_DETAILS/from/pwd/;get-cookie

The get-cookie matrix parameter int the URL allows an HTML page to be
designed with static URLs, into which nextdb will dynamically
substitute the URL path parameters with values sent by the browser as
cookies. It's a bit of a mind bender but this allows a site be
designed with static URLs. The only dynamic URL ever generated by the
app is the single login query that sets the pwd/{password} path
parameter.

so when nextdb receives the static URL above, it will first see that
the URL incudes a get-cookie matric paramter. NextDB will then look
for a cookie named 'pwd' sent by the browser (or other http client),
decrypt the cookie value, and place the value into the URL where it
finds the get-cookie matrix parameter:

.../db/link/ACCOUNT_DETAILS/from/pwd/xxxxxxxx/

The URL above is then equivalent to this NextQuery expression:

NAME=ACCOUNT_DETAILS;
ROW r1 FROM USER;
ROW r2 FROM ACCOUNT;
r1 RELATED r2 VIA MY_ACCOUNT;
WHERE(){
r1.pwd='xxxxxxxx'

}

The net effect of this new style of REST url for queries and relationship traversal is to vastly reduce the amount of code an applicaiton has to write, and hopefully to make nextdb accessible to HTML/front-end developers.

Native REST support in NextDB.net

2009-11-08T09:32:00.000-08:00

I'm really stoked on using REST to access NextDB.net. I've setup a Wiki for NextDB.net and here is our REST wiki page: http://www.nextdb.net/wiki/en/REST

Beeing down with a bad cold the last few days, ironically this has given me
some time to work with Jen from Dartmouth Design on using the REST features
to build some features for Project Have Hope. She has graciously offered to
share the preliminary results. So far we've used nextdb for creating
profiles of children to sponsor. The entire page, including the paging
links, profile data, and paypal buttons, is created by applying an XSLT
transformation to the table served up by nextdb. The result is sourced in an
iframe and loaded into a Joomla module. This appears to be a very nice way
to integrate nextdb content with Joomla. I'm also very stoked to see NextDB
used on such a worthy project.

Here is the live integration site (if this is down it's just because the site has moved from integration staging to the live environment):

Check it out.

You can read about the XSLT approach here:

http://www.nextdb.net/wiki/en/REST#XSLT_(reformatting_your_data%27s_presentation

This is the XSLT that is being used by Project Have Hope.
http://nextdb.net/nextdb/rest/dd/phh/config_files/row/1/xslt.xslt

I've just learned XSLT so I think there are many things that can be done
smarter, and there is also a bit of cruft template in the XSLT that is never
used (for example the 'append-pad' template is never used) and needs to be
cleaned up. But that aside, it's an amazingly small amount of XSLT
considering this XSLT is the *only* code required to bring together this
dynamic page.

Three Views of REST resources

2009-08-03T23:28:00.000-07:00

whew. Just hacked most of the day away learning how to make a custom JAX-RS provider to generate an HTML view of a resource using an XSLT transformation on the XML.


@Provider
@Produces(MediaType.TEXT_HTML)
public class UsersProvider  implements MessageBodyWriter {
  @Override
  public void writeTo(Users users, Class type, Type genericType, Annotation[] annotations, MediaType mediaType, MultivaluedMap params, OutputStream os) throws IOException, WebApplicationException {
      try {          
          JAXBContext jc = new JSONJAXBContext(JSONConfiguration.natural().build(), new Class[]{users.getClass()});//JAXBContext.newInstance(users.getClass());
          DOMResult xml = new DOMResult();
          jc.createMarshaller().marshal(users, xml);
          StreamSource styleSheet = new StreamSource(getClass().getResourceAsStream("/stylesheet/Users.xsl"));
          Transformer transformer = TransformerFactory.newInstance().newTransformer(styleSheet);
          transformer.transform(new DOMSource(xml.getNode()), new StreamResult(os));
      }catch (Exception ex) {
          throw new RuntimeException(ex.getMessage(), ex);
      }
  }

  @Override
  public boolean isWriteable(Class type, Type genericType, Annotation[] annotations, MediaType mediaType) {
     return (Users.class.isAssignableFrom(type) && mediaType.isCompatible(MediaType.TEXT_HTML_TYPE));
  }

  @Override
  public long getSize(Users arg0, Class arg1, Type arg2, Annotation[] arg3, MediaType arg4) {
      return -1;
  }

}

In the end it's actually quite easy, but what threw me off was that JAX-RS has built-in support for the serialization of List<t> where T is a JAXB annotated class. The problem is that my custom provider, which applies a stylesheet to the XSL necessarily bypasses the built-in support for collections of JAXB annotated classes. Therefore, the method in my resource can't return a List. Instead, I created the Users (note the plural "s") wrapper class, and JAXB annoted it. Note incredibly annoyingly named accessor method, "getUser". Unfortunately, this had to be named in the singular, due to a quirk of JAXB. Naming it "getUser" produces
<users>
<user>...</user>
<user>...</user>

</users>

versus naming the accessor method in the plural produces (note redundant plural users):
<users>
<user&sgt;...</users>
<users>...</users>

</users>

so here is the JAXB annotated "wrapper" class for use instead of returning a raw List from the Resource method:

@XmlRootElement(name = "users")
public class Users {

private List users;

/**
* @return the users
*/
@XmlElement
public List getUser() {
return users;
}

/**
* @param users the users to set
*/
public void setUsers(List users) {
this.users = users;
}

}

then the Resource method just looks like this (note that it returns the Users wrapper class):


  @GET
  @Produces({MediaType.TEXT_HTML})
  @Path("/users/{user}")
  public Users  getUsersHTML(@PathParam("user") String username) throws Exception {
      Users users = new Users();
      if(username.equalsIgnoreCase("*")){
          users.setUsers(net.nextdb.SysAdminSearchAccounts.getUsers("%"));
      }else{
           users.setUsers(net.nextdb.SysAdminSearchAccounts.getUsers(username));
      }
      return users;
  }

JAX-RS and Jersey

2009-08-03T08:13:00.000-07:00

I spent the better part of the day messing with Jersey, an implementation of JAX-RS. I really like the built-in support for XML and JSON, as well as the JSONP support. The one thing that feels missing is some build-in support for creating lists of URIs. For example, if you have a User resource at Path .../user/JDoe, then plain old ".../user" ought naturally to return a list of URIs, one for each User. It would be nice if there was some support for that in the framework. Other than that, I really like JAX-RS and Jersey.

I'll probably wind up implementing my own providers for generating simple HTML pages and JSONP-HTML.

JSON, in retrospect

2009-07-21T23:22:00.000-07:00

In May of 2006, I sent this message to a friend. It's funny that the "see for yourself" page is no longer titled "see for yourself", probably because people actually did "see for themselves" and perhaps reached a conclusion similar to the one I reached below. In hindsight, JSON is wonderful and I still consider it to have different use cases than XML. I continue to heavily use both JSON and XML in my day-to-day tasks.

From: Geoff Hendrey
Sent: Wednesday, May 24, 2006 10:31 AM
To: Brent Hamby
Subject: Json questions

Hey dude,

I went to the json "see for yourself" page: http://www.json.org/example.html to compare the Json equivalent of XML.

I copied each of the examples into notepad and wrote down the size-on disk:

json size (bytes) XML size (bytes) (json size)/(XML size)
===================================================
604 630 95%
252 221 114%
630 656 96%
3554 10708 33%
898 1150 78%

AVG (json size)/(XML size)
=====================
83.2%

Based on an average size compaction to 83.2% of the XML size. Also, if you look at the 4th example they give (line 4 of the table above), they did a Json version of web.xml. Now web.xml is a notoriously bad XML design. It basically doesn't use ANY attributes, so you get pretty nasty tag bloat. So if we drop the 4th comparison, as being kinda silly, we get an average compaction to 95.4% of the original size. Is saving 4.5% really a big advantage?

I also saw several statements on the json web page that I would put in the "wacky" bin. This may just be personal preference, but I found the XML documents much easier to read than their json equivalents. The Json looks, to me, like "code", kinda like a snippet of PERL or something.

Anyway, it seems clear that being able to easily marshal an object representation of your data makes it much easier to develop software in any programming language. But it seems to me that json is munging this idea up with the wire format, which to me are orthogonal concerns.

JavaScript downloaded function executor

2009-06-20T19:07:00.000-07:00

Ran into a really annoying problem trying to use the YUI text editor in a jqueryUI modal dialog. I am using jquery to load a file "richEditor.html" that has my YUI editor on it. Problems:

required YUI script and link tags, that are needed by YUI, are defined on my page richEditor.html. I need to be move said tags from head to body, because jQuery.load function only loads the body of the HTML page.
said script and link tags *begin* downloading from yahoo when jQuery calls your load callback function. That's becuase richEditor.html itself has been downloaded by jQuery, but the said script and link tags referenced ON richEditor.html have only just begun asynchronously downloading. So if you need to access a function in one of the script tags that is downloading, you have no way to be sure the function has completed downloading.

As a solution, I wrote this JS Object that will simply try to invoke your function, and try again, after a configurable delay, up to maxTries. If the function fails to execute maxTries times, a fail handling function, provided by you, is executed.

Here is the Object:


function DownloadedFunctionCaller(funcToInvoke, waitMS, maxTries, _failHandler){
    var funcToInvoke = funcToInvoke;
    var waitMS = waitMS;
    var maxTries = maxTries;
    var failHandler = function(){
        alert("Unable to access the network. Please reload the page or try again.");
    }
    if(null != _failHandler){
        failHandler = _failHandler;
    }
    var that = this;
    var tries = 0;
    this.invoke = function(){
        try{
            funcToInvoke();
        }catch(err){
            if(++tries >= maxTries){
                failHandler();
                return;
            }
            setTimeout(that.invoke, waitMS);
        }
    }
}

and here is an example of using it to invoke a function named setEditor that is defined
in an external file, richEditor.html. The setEditor function will only work if all the YUI scripts are downloaded, hence the need to use the DonwloadedFunctionCaller



        var div = $("<div></div>");
        div.addClass("yui-skin-sam");
        div.load("richEditor.html", null, function(content){
                    var invokeMe = function(){
                        setEditor("Script loaded and executed.");
                    }
                    new DownloadedFunctionCaller(invokeMe, 500,10,function(){
                        alert("failed to download rich text editor")
                    }).invoke();
          });

If you click on this screengrab, to get a bette look at it, you will see that the text "script loaded and executed" has been set into the YUI editor's text area by the successful invocation of the setEditor method.

random lines of poetry

2009-05-23T09:51:00.000-07:00

I woke up early this morning around 6:30 thanks to Claritin-induced sleep deprivation. I started hacking on server code for automatically building "tables of contents" for very large database tables. I'm using a bunch of Yeats poems, and The Iliad as sample rows in the database. It makes debugging a lot more fun when the data is interesting. It's also a great way to read random lines of poetry. Here is a great stanza from the Iliad:

Generations of men are like the leaves.
In winter, winds blow them down to earth,
but then, when spring season comes again,
budding wood grows more. And so with men--
one generation grows, another dies away. (Iliad 6.181-5)

Did you know the phrase "another one bites the dust" is lifted from the Iliad?

"Grant that my sword may pierce the shirt of Hector about his heart, and that full many of his comrades may bite the dust as they fall dying round him."

PBWiki vulnerability ...this could be huge

2009-05-07T12:20:00.000-07:00

My sister seems to have uncovered a very interesting security vulnerability in PBWiki

She wrote it up here on http://www.dartmouthdesign.com/about/blog

Just by knowing the name of my wiki, she seems to have been able to take administrative control of my account. I even receive emails, like the one below, when she takes an action on my account (which I edited to remove the actual link in the email):



Hi,

 

An administrator, An administrator (geoff_hendrey@yahoo.com), has just updated hendreytravel to PBworks 2.0.

 

To access the wiki, you'll need to create a PBworks account by clicking here:
https://my.pbworks.com/verify.php? ... 
 

PBworks 2.0 has a brand new interface and improved features like folders, an easier editor and faster search.

 

Learn more about PBworks 2.0: http://pbworks.com/content/pbwiki-2-user-guide

 

Thanks,

The PBworks Team

 

PS: Having trouble? Contact the support team at http://pbworks.com/help.wiki?wiki=hendreytravel

Tesla rocks...

2009-04-11T18:37:00.000-07:00

Aside from the fawning interviewer, this is a great interview with the CEO of Tesla Motors.

http://www.techcrunch.com/2009/04/10/teslas-elon-musk-grows-a-pair-good-for-him/

It would be criminal if Tesla didn't receive the funding they need. To see a loser like GM sloshing in bailout funds is a sickening contrast.

Nextdb in 120 seconds

2009-03-31T20:27:00.000-07:00

Back in one piece...

2009-03-22T21:42:00.000-07:00

Spent the weekend near Tahoe in Truckie. We got 1.5 ft of snow between saturday night and sunday afternoon. Nevertheless Emily and I decided to drive back as soon as I-80 opened so we could get back Sunday night and not miss Monday at work. Tire chains are a real pain in the butt. Both my chains snapped. Fortunately I was able to pull to the side of the road in a relatively safe location (meaning semis were roaring by like 4 feet from me as I crawled under the car to disentagle the chains from the half axle - getting fairly mired in much at the same time). Tire chains are required at times on I-80, but they honsestly seem like very unsafe garbage. The road is littered with broken chains. My theory is they are designed to be a deterent to drivers, more than an actual functioning safety mechanism. Anyway, chain chaos aside, we had a very relaxing weekend watching snow fall on evergreens (I was tempted to say Snow Falling On Cedars, but I refrained).

Actually, I just checked the DOT report for I-80:

"Enter Highway Number(s)
You can also call 1-800.427.7623 for current highway conditions. Mobile

This highway information is the latest reported as of Sunday, March 22, 2009 at 21:50 .

I 80

[IN NORTHERN CALIFORNIA & THE SIERRA NEVADA]
WESTBOUND TRAFFIC IS BEING HELD AT CISCO GROVE (PLACER CO) - DUE TO AN
ACCIDENT - MOTORISTS ARE ADVISED TO USE AN ALTERNATE ROUTE

CHAINS ARE REQUIRED ON ALL VEHICLES EXCEPT 4-WHEEL-DRIVE VEHICLES WITH SNOW
TIRES ON ALL 4 WHEELS FROM 1 MI EAST OF BAXTER (PLACERCO) TO THE
DONNER LAKE INTERCHANGE (NEVADA CO) "

you know it's time to defrag when ...

2009-03-18T21:34:00.001-07:00

Need to improve the reviews widget

2009-03-17T22:38:00.000-07:00

Wow, people are actually using the reviews widget to leave reviews. I love that with nextdb I can just enter the web based admin to manually delete my test reviews (ok, ok, or to boost the number of stars in a posting! haha).

But seriously, I need to gut the thing and redo it with something like JQuery UI.

From Willow Tea Room, Glasgow

2009-03-17T21:19:00.000-07:00

Photos From Scotland

2009-03-16T20:01:00.000-07:00

In 2006 I took a trip to Scotland for an OGC meeting, and had the weekend to roam around. I visited the National Library and did a little geniological research. After a fair bit of cross-referenced parish records, pre-census, I believe I located the birth records for my great, great, great, grandfather and grandmother. Unfortunately the library closed right when I pieced it all together, but that's my next trip I guess. The microfilm will still be there!

One of the most fascinating aspects of Edinburgh and Glascow was there role in the Arts and Crafts and Art Nuveau movements. Phoebe Anna Traquair (P.A.T) was one of the female artists who figured prominently in this movement. If you ever have a chance to visit the Mansfield Traquair church, do not miss it.

You can get a better look, here: http://www.mansfieldtraquair.org.uk/nav/cag02.html

If that doesn't blow your hair back, get a look at this amazing painting by P.A.T.. Oh, wait, take a second look, it isn't a painting! It's SILK EMPROIDERY!!

It's almost incomprehensible that a few years ago, the Mansfield Traquair church was being used as a brick storage facility. Thankfully, it was recently rescued an restored. For the life of me I cannot understand why P.A.T. and John Duncan are not better known. Please read this blog to see some more breathtaking work by P.A.T.

I am so awestruck by John Duncan's St. Bride that I recently aquired a canvas reproduction of it from an obscure source on the web. Here is a shot of it gracing my living room.

Scotland has lots of badass wildcats, which came to America as stowaways, and became the big Main Coon cats.

Loch Ness... of course. On the way there we passed the spot where the Battle Of The Shirts (or lack thereof) took place. It turns out that fighting a clan battle in sweltering heat with armor is a good way to die of heat stroke. So it probably seemed like a reasonable idea is to call for a "time out" because of the heat, and take a dip in the loch. Unfortunately, it turned out to be a very bad idea to agree to fight the second half of the battle without armor because of the heat. The horrific mutual slaughter that ensued, fought with claymore broad swords and battle axes left just 12 survivors in total, and the Loch was litterally red with blood for days.
Edinburough....

Another Gem from William Morris

2009-03-15T20:41:00.000-07:00

For the Bed at Kelmscott

The wind's on the wold
And the night is a-cold,
And Thames runs chill
'Twixt mead and hill.
But kind and dear
Is the old house here
And my heart is warm
'Midst winter's harm.
Rest then and rest,
And think of the best
'Twixt summer and spring,
When all birds sing
In the town of the tree,
And ye in me
And scarce dare move,
Lest earth and its love
Should fade away
Ere the full of the day.
I am old and have seen
Many things that have been;
Both grief and peace
And wane and increase
No tale I tell
Of ill or well,
But this I say:
Night treadeth on day,
And for worst or best
Right good is rest.

ProPublica: Jouranlism in the public interest

2009-03-15T10:05:00.000-07:00

I'm replacing my daily reading of the New York Times Online Edition with that of Propublica.org

Basically, none of the major news outlets can be trusted. This is just a fact. Remember before we went into Iraq how the New York Times ran breathless stories on Sadam's vast underground bunker complex. It was all just a load of shit.

So now I'm going try getting my news from ProPublica, because their mission is to reinvigorate investigative journalism.

Developers are starting to catch on!

2009-03-13T07:37:00.000-07:00

Here is a link to a blog post written by a developer at SolutionSoft, about NextDB.

http://blog.solutionset.com/wpmu/2009/03/11/nextdb-a-front-end-database-solution/

It's a great post, and I think it illustrates how easy nextdb is to learn, and use. Nextb is getting preety close to exiting alpha and entering Beta. During the Beta we will have our "subscribe" buttons up on the site. We are targeting price points that are basically below the noise for even a personal account, so affording nextdb should not be a problem for anyone.

Brent is busy developing and refining a new set of utilities for adding forms to your AJAX apps. The forms will automatically configure themselves based on the content of your table or query, and will be CSS friendly.

I'm working on a notification system that will allow you to receive email when data is inserted into a table. You'll be able to request digests, as well as create email templates that receive data from the inserted row in order to create a personalized, pretty email.

ulimit -n .... the saga continues

2009-01-25T23:29:00.000-08:00

Man, I thought I had put the "ulimit -n issue" to bed once and for all with these configurations in CentOS. So, it was with great chagrin that I recently executed a "ulimit -n" and saw 1024.

Argghhh. Will this issue never end?

I've added "ulimit -n 100000" to my .bash_profile, and now I have confirmed that after logging into a new session, that the new ulimit of 100000 files is in effect.

I'm not happy with putting this in my .bash_profile, but the alternatives looked too dangerous and or flakey.

some recent tech photos

2009-01-21T18:15:00.000-08:00

At deCarta I am building a new map tile system for our servers. Krish installed it on a floppy drive! That was pretty funny.Brent and I went to the Cloud Connect conference. We tried to pay attention but we were too busy hacking :-)
Panel discussion at Cloud Connect was lively to say the least. I thought the dude from Salesforce.com was going to smash a chair on someone. But seriously, he is a very sharp dude. Would not want to debate him.

Test reviews Widget

2009-01-14T20:27:00.000-08:00

Please don't say "meme" with a straight face

2009-01-14T07:01:00.000-08:00

It's early, I'm grouchy, and I've seen the word "meme" one too many times. People: please, please don't use the word "meme" to describe internet trends, or anything else for that matter. It's just silly. It's like putting plastic spinny rims on an '83 celica. It's basically gheto-bling.

The Wanderings of Oisin

2009-01-08T23:59:00.000-08:00

I was trying to find an extant copy The Evergreen Periodical, when I came across this intersting read on the Celtic revival's influence on art nouveau, which in turn led me to The Wanderings of Oisin by Yeates.

"An old man stirs the fire to a blaze,
In the house of a child, of a friend, of a brother.
He has over-lingered his welcome; the days,
Grown desolate, whisper and sigh to each other;
He hears the storm in the chimney above,
And bends to the fire and shakes with the cold,
While his heart still dreams of battle and love,
And the cry of the hounds on the hills of old."

Hacking JS-Kit reviews widget

2009-01-08T17:21:00.000-08:00

I just installed the JS-Kit reviews system on my blog below. I was amazed to find the following security holes:

1) no CAPTCHA protection on posts. This means you can flood the reviews with bogus reviews and spam. Want to bring down the reviews system? Just write a 4-line script to execute this URL in a loop:
http://js-kit.com/comment.put?ref=http://notskateboarding.blogspot.com%2F&permalink=http%3A%2F%2Fmysite.com%2Fpermanent%2Flink%2Fto%2Fpage.html&rvy=1&js-CmtName=reviewer04&js-CmtCity=san%20franciso%2C%20USA&js-CmtEmail=geoff.hendrey%40gmail.com&js-Cmtsubmit=Submit%20review&js-CmtsubmitOrig=Submit%20review&js-CmtsubmitReply=Submit%20comment&js-Cmtcancel=Cancel&js-CmtText=BBBBBBBBBBBBBBBBBBBBBBBBBBB&js-CmtRating=8&tid=jst-1

2) Email span the person who posted a comment you don't like! This one is really amazing! Want to send someone 1000 emails anonymously? Same technique as above, just make it a comment instead of a posting, and the person who made the original posting gets 1000 emails!

test a reviews widget

2009-01-08T16:03:00.000-08:00

annoying CENTOS problems raising ulimit

2009-01-06T21:35:00.000-08:00

My last attempt to raise the open file count on a CentOS server apparently failed. After much dicking around, here is a set of steps that I believe will successfully raise the limit (don't be fooled by whatever you try. If you do a 'ulimit -n' and the returned value is 1024, you have not succeeded in raising the limit).

I did a bunch of reading, and found two postings, that combined seem to have succeeded in upping the max files to 65535.

Mainly I am writing this down so we can remember what the hell I did. These are the two useful posts:
http://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/ http://www.centos.org/modules/newbb/viewtopic.php?forum=1&topic_id=559&viewmode=flat

in summary:
1) edit /etc/sysctl.conf to add this line:
fs.file-max = 200000
2) sysctl -p
3)switch to /etc/security/limits.conf and add ther following lines

* soft nofiles 65535
* hard nofiles 65535
4)ulimit -n 65535

I think this worked this time because when I do 'ulimit -n' it returns 65535, whereas prior it always returned 1024. In our previous attempt we only did step 3.

The 200000 value is excessive, but who cares.
I am working on a widget for adding reviews to blogs and websites.

test review widget

2009-01-03T15:11:00.000-08:00

I am working on a widget for adding reviews to blogs and websites.

Tahoe Trip

2009-01-02T09:45:00.001-08:00

Happy Newton's Birthday

2008-12-24T09:29:00.000-08:00

George just pointed me to a great article in the NYTimes titled "The Ten Days of Newton".

On the tenth day of Newton,
My true love gave to me,
Ten drops of genius,
Nine silver co-oins,
Eight circling planets,
Seven shades of li-ight,
Six counterfeiters,
Cal-Cu-Lus!
Four telescopes,
Three Laws of Motion,
Two awful feuds,
And the discovery of gravity!

Happy Newton, everybody!

Histrator update

2008-12-23T18:18:00.000-08:00

I received this email from Hostrator:

"Dear Geoff,

We have some redirection issues, Note all your files are there intact, the issue occurred while we were upgrading the bandwidth for each user (was our surprise). Again your sites (files) are there and no need to worry about it, your site will be back online in a couple of hours hopefully."

I guess they resolved their issue, because my widget's HTML files are back online.

asscrack hosting, inc!

2008-12-22T23:46:00.000-08:00

Wow, notice that my signup widget on the right side of the page has been replaced by the hostrator home page? Either these guys are complete scumbags, and they trick you into using free hosting, then replace your own HTML files with their homepage, or they are so broken that they have genuinely lost the files that I hosted there. Either way, their homepage is appearing where my widget should be. Well, time to find another free host for my widget HTML files. Anyone know a *reliable* and *ethical* provider of HTML file hosting?

Encryption, digital signatures

2008-12-21T22:50:00.000-08:00

For a few weeks NextDB has had support for encryption via the CYPHER function, and sending email through the SENDEMAIL function. However, we didn't have support for providing application specific encryption keys, nor for digital signatures. Well, I spent today prototyping support for both of these two things. The CYPHER function now accepts a second argument, which is a 16-character string (128 bit privtate key). Digital signatures are accomplished through a new function called MD (Message Digest). A DECYPHER function allows you to decrypt whatever you encrypt. This creates a "round trip" model for your data, a lot like we do for our SURIDs. But you can put whatever applicaiton specific payloads you like inside the encrypted messages.

the impotus for this was being able to send out "confirmation" emails from NextQuery expressions, and not allow the content of the URL in the email to be tampered. The following is a NextQuery expression that I am using with a "5-star reviews" widget that I am developing. When the user posts a review, this query sends an email to the poster of the review, including a link to click to confirm (for sake of example, we are assuming the Reviews Widget has been places on 'mysite.com')

NAME=sendReviewConfirmationEmail;
ROW review FROM REVIEW;
WHERE(SURID pk){
review.PK = ${pk}
}
RETRIEVE SENDEMAIL(
'please confirm your review',
'click this link to confirm your review\n' ||
'http://mysite.com/reviewConfirm?'||CYPHER(review.title||':'||review.author||':'||MD(review.title||':'||review.author),'0123456789abcdef'),
review.authoremail,
'reviewAdmin@mysite.com'

)
into email.status;

The SENDEMAIL function accepts the subject, body, to, and from parameters. The interesting thing here is the third parameter, the body of the email, which includes a URL. Part of the URL (the part after the question mark) consists of a cyphered and digitally signed concatenation of the review's title, and the author's screenname.

Where things get really mindbending, is that the content of mysite.com/reviewConfirm is actually an HTML page with JavaScript embeded. The embedded JS can actually get access to parameters passed to the page, and thereby fulfil functions typcially handled on the serverside, even though they will run right there in the user's browser.

When the "reviewConfirm" JavaScript executes, it uses NextDB's JavaScript API to execute a query. Said query will use the DECYPHER function to decypher the URL's query parameter, and use the MD function to check the digital signature to be sure the confirmation email content has not been tampered. Said query returns a SURID with FOR UPDATE permission which allows the JS to change the value of the 'status' column of the row to 'approved'.

But what about replay attacks? That's the beauty of this technique. Because the CYPHER function's secret key can be different for each query, and because the content of the encrypted message can be used to confirm the presence of a row in the database, you can't "replay" a cyphered structure from your database against someone else's database.

Steal This Widget

2008-12-16T06:51:00.000-08:00

Widget (Take I)

2008-12-15T22:12:00.001-08:00

As you can see, I've added a rather oversized "signup" widget to my blog. I am experimenting with creating widgets using NextDB.net. This one will add your "user" information into the NextDB database when you signup.

Why?

User registration is really "square one" of building any website. If we can encapsulate user signup into a widget, then we can encapsulate all other site functions as well. Then, with site functions "widgetized" you can essentially turn your blog, or really any web page into a full-featured AJAX site.

I've learned a bit about how best to do this. The first thing I learned is that sites like blogger.com have a pretty friendly way for you to distribute your widget. It boils down to a simple button a viewer of your blog can press, and presto, they get the widget.

Second thing that took more messing around than it should was to find a webhost to host the HTML and image files for the widget. I intentionally didn't want to host any of the HTML or resources on nextdb.net, because I really wanted to behave as a nextdb user, who might be a graphic designer or someone in a dorm room, for whom a free webhost was their only option. I wound up creating an account at hostrator.com. But it is lame that they don't support SFTP or a better batch upload. But it's good enough. And I think that's sort of the point. With an AJAX widget it really doesn't matter who hosts the HTML file is hosted.

The Lapse of The Year

2008-12-14T09:56:00.000-08:00

SPRING I am too soft of heart
Much to speak ere I depart
Ask the summer tide to prove
The abundance of my love

SUMMER looked for long am I
Much shall change or ere I die
Prithee take it not amiss
Though I weary thee with bliss

Laden AUTUMN here I stand
Weak of heart and warn of hand
Speak the word that sets me free
Naught but rest seems good for me

Ah shall WINTER mend your case
Set your teeth the wind to face
Bear the snow down, tread the frost
All is gained when all is lost

-William Morris

NextDB.net Google Group

2008-12-13T17:27:00.000-08:00

NextDB.net now has a google group

http://groups.google.com/group/nextdb-user

We are hoping to get some good feedback from our user community. Small but growing.

St. Bride

2008-12-09T18:13:00.000-08:00

This is a painting titled "St. Bride" by the Scottish painter John Duncan, from 1913. At the moment, this is my favorite work of art. I was fortunate enough to spend some time reflecting on this painting at the National Gallery in Scotland. It's is surprising that this painting is not more notable, considering how interesting it is (not to mention beautiful).

The painting depicts two angels carrying Bridget to Bethlehem to swaddle baby Jesus. It's truly an amazing amalgamation of Christian and Celtic mythology; Briget becomes St. Bride and Christianity synthesizes the ancient beliefs.

"[The legend] ...goes further back than the days of the monkish chroniclers who first attempted to put the disguise of verbal Christian raiment on the most widely-loved and revered beings of the ancient Gaelic pantheon. Long before the maiden Brigida… made her fame as a 'daughter of God'… the Gaels worshipped a Brighde or Bride, goddess of women, of fire, of poetry… one whom the Druids held in honour as a torch bearer of the eternal light, a Daughter of the Morning."

This is a wonderfully rich posting on the "Celtic Twilight" movement. Apparently William Blake, JRR Tolkien, and John Duncan shared similar influences. The more I read about this painting, the more I understand why it resonates with me.

Thoughts and Pimps

2008-12-09T18:04:00.000-08:00

Wonderful tiled mosaic: "THOUGHT: written words; Spoken words". I took this photo near Rockefeller Center.

Early anthropological evidence of pimps (American Museum of Natural History)

This is just too easy!

2008-12-09T00:06:00.000-08:00

Wow, I am amazed how easy it is to wire up my new widget to NextDB.net. I mean, I really shouldn't be surprised, given we've been developing NextDB exactly to be this easy, but I'm still stoked. Here is a screenshot of the admin site for my hosted database in NextDB (just created one table for storing the widget data). Next to the admin site, on the right, is my shiny black widget.

Here is the whole 10 lines of code I literally cut-and-pasted out of the NextDB JS Docs into my widget's HTML.

BOOM, press the button on the widget, the data is inserted into the database, and I can even see the row using the admin tool. And there it is! There's my row that I inserted:

Database feeds

2008-12-08T23:11:00.000-08:00

I had an idea today that boils down to "RSS for Databases". I was inspired by the very geeky act of "server log watching". Basically, whenever you bring up a cool new database app, you wind up watching the logs just to get a blow-by-blow on what your users are doing. Now, obviously log watching is boring, and requires you to be logged in or ssh'd into the server. Why not apply the same "feeds" to the actual data in the database that we use to follow our favorite blogs or news sites? Obviously some filters would be required to avoid a torrent of data in the feed, but that's all do-able. I'm pretty excited about adding RSS Feeds as a native feature for NextDB database tables. I will probably work on it this weekend. At the moment I'm addicted to screwing with a new widget design in Adobe Fireworks.

Widgets!

2008-12-07T09:25:00.001-08:00

I am in the process of creating a series of widgets that can easily be re-skinned. These widgets will fulfill many basic site functions such as: user sign-up, login, user profile management, photo gallery, blog, forum, etc. Each widget can be easily re-skinned to suit the look and feel of your site, and they are pre-wired to communicate with NextDB.net. Here is an example of a sign-in widget.

Nice blog enrty in Wired

2008-12-06T16:35:00.000-08:00

Looks like Wired did a blog entry on NextDB.

It's funny, if you look in the comments, there are already haters. They don't even have a clue what NextDB does, or how our security model works. That's a good sign. Haters are scared of change.

Big week for NextDB.net

2008-12-06T08:54:00.000-08:00

On Thursday, Andres Ferrate, posted this article on Programmable Web titled "Mashups Get a Hosted Database With NextDB.net". We also got an entry on Programmable Web about our JavaScript API. Very quickly the wave of account signups poured in and it hasn't crested yet. I guess this a moment that every small startup has to cope with. Fortunately both Brent and I have been there before but it never gets old. You can't keep your eyes off the server logs, and it's almost a point of pride when you hit "Too many open files" and need to kick a "ulimit -n", which in fact happened yesterday as dozens of new databases were created on NextDB.net. Well, it's not exactly what I'd call being slashdotted, but it was a fun ride. Our challenge now, is how to effectively collect feedback from these Alpha users in order to improve the service based on their experience.

SF Park

2008-11-26T17:01:00.001-08:00

The new SF skatepark is pretty rad. It is growing on me. Here is some footy of parker I took on my point-and-shoot. This park is crowded. Note near collision with fat kid on bike (listen for the yell).

Read-End....

2008-11-26T16:53:00.000-08:00

It doesn't seem to bother anyone else in the office that when you place a page on the copy machine, and hit the big green "copy" button, that this message appears:

READ-END? WTF is READ-END? This copy machine can send emails, but they can't just make it copy a page when you hit the green button?

Steve's Feedback on our UI

2008-11-18T15:56:00.000-08:00

lots of issues with the "new user" concept. First time someone signs in have obvious links to getting started guide, creating a sample database, etc.

"import" has too much meaning to db guys. Make it say "create a sample database from template" WITH data.

"must match regex in" in tablename is not a good error message for invalid table names.

Suggested "nextdb for SQL database programmers".

Wants to know how to create the primary key.

Some kind of wizard to drive first time users.

tooltips on tabs with check box for "never show this again"

consider changing the query language parameter names and datatype names to 'human friendly' equivalents.

Steve would like to be able to set the default value.

wants to be able to use camel case in column names

relation name should be "relationship name". It also upper cases it.

show lines connecting related tables as an option.

thought he could not enter a query without having data.

did not like the interface for adding data.

wants a "data editor" tab along the top (loads table names along left hand side)

wants column types to be in parentheses.

Steve wants to be able to store HTML and edit it with a foldout editor. I want permalinks to the HTML and JS.

Great idea

trouble executing query with join

no way to relate rows

wants excel spreadsheet

wants a way to get his data out.

"You wouldn't want a mashup for your bank account."

2008-11-18T10:41:00.001-08:00

I am watching the opening panel discussion at Mashup Camp. Hart Rossman, CTO at SAIC just said "You wouldn't want a mashup for your bank account."

That really struck a chord because it captures the inadequacy with today's mashup security model. In fact, building a banking system on NextDB is sort of the logical extent of where the NextDB SURID technology can go.

Another interesting statement from one of the panelists:

"SLAs can't be that strong because you can't have really strong remedies."
I agree.

"you're not going to get a new CRM system out of a mashup."

I disagree. Once the database is secure and mashable, the entire application software development lifecycle can be handled by mashups.

The proverbial Map Mashup came up again as a proof point for time savings.

"when the users become the developers, they are doing what they want to do, not what IT wants to do"

The discussion briefly touched on whether or not Amazon Web Services (S3) constitutes a mashup. That's really a point that requires discussion. Once you realize that Amazon Web Services is not Mashable, due to its "web 1.0 security model", you find yourself somewhat dissapointed.

"where we run into a real challenge is in converting the mashup mentality into the largescale systems engineering skillset. You don't see a lot of folks puting a large ampount of skill and discipline into building the backend." paraphrasing what he said later: We're going to see things going in that direction though, and it will drive Mashups into the enterprise.

Mashup Camp 2008

2008-11-16T08:55:00.000-08:00

Tomorrow is Mashup Camp! I'm excited about this year's Mashup Camp because I think there is going to be a lot of activity around trying to figure out how to fuse mashups with Cloud Computing. And that, of course, is exactly where NextDB.net has a sweet spot. NextDB is fundamentally a mashable relational database in the cloud. What does that mean for the programmer? It means NextDB is the ONLY relational database that you can securely mashup, without have to write ANY serverside code.

So check this out. I added this "guest book" to my blog to keep track of folks I meet at Mashup Camp. This guestbook is a Mashup with NextDB.net, which makes it a nice little example of a database mashup between NextDB and blogger.com. Originally, Brent wrote this guestbook as an example for our talk at ajaxworld. I just went to his page, grabbed the html, js, and css and threw them into my blog. The only tricky part was that blogger.com's editor didn't like newline characters in the HTML and CSS. So I just saved them in a file without newlines, and pasted them back to my blog. Go ahead and sign in below, and watch the AJAX goodness! Here is a link to the JavaScript that the guestbook sources. Rather than use the nasty editor at blogger.com (and its associated issues with spaces), I just uploaded the js file to Project Path, which is a nifty collaboration site that allows file hosting.

Mashup Camp Guestbook

first name *
last name *
email
comments
	Enter the text above into the box below. Click image to get a new one if you can't read it.

json view:

NextDB At AjaxWorld 2008 West

2008-11-12T09:27:00.001-08:00

Brent and I recently presented about NextDB.net at AjaxWorld 2008 West, in San Jose CA.

I am going to pickup blogging a bit more regularly here. Anyway, the presentation went well. The message we hammered on was that NextDB has a fundamentally new security model that allows a hosted relational database to be exposed as a service. NextDB is the only relational database that is built from the ground up as a AJAX-programmable hosted service. As such, the round-trip security model is native; fundamentally "built in". This clearly differentiates NextDB from other pseudo-database or database services, because you can safely access it directly from JavaScript.

New York Family Vacation 2008

2008-11-10T18:39:00.000-08:00

Times Square

Radio City Music Hall Christmas Spectacular stage

Why detachment sucks

2006-10-31T20:04:00.000-08:00

Ahhh, yes, detachment. The idea that you must explicitely "detach" a persistent POJO from its persistence manager, and later explicitly attach it.

Detachment is another opaque feature of so-called Transparent Object Persistence.

Basically detachment is a work around for the fact that the JDO PersistenceManager is not itself serializable. So, when your Page, with your so-called POJO gets serialized, "oh ma god", you can't serialize your pojo. (that would be TOO SIMPLE, TOO POJO). NOOOO, you've got to implement a callback in which you detach your POJO before you serialize it.

The worst part of it is, this totally defeats the point of serializing the page. You see, in Wicket, they are serializing the Page and things on the page to keep the server from bloating with lots of concurrent clients. But wait a sec, if the PersistenceManager has a cache of POJO's, isn't that cache going to be full of those very POJO's your trying to serialize. Right, but you can't serialize the PersistenceManager.......OK, right, so how do you keep things trim. So you , um "detach" from the persistence manager, and serialize your detached POJO. In the mean time, you have allowed your PersistenceManager to get garbage collected. Had to, otherwise there was no point in serializing your POJO's. I mean, you can't keep the whole cache of 'em lying around in memory. (and even if you keep your PersistenceManager around, it's not a real cache. If you've serialized out your POJO, it ain't gonna be in the cache when you deserialize the POJO. NOPE, it the POJO will have been garbage collected from the PersistenceManager, because the PM is a pseudo cache, with WeakReferences, not a hard cache).

So, the architecture of detachment is fundamentally at odds with the notion of a cache. So when you re-attach, whoops, you have no cache. Been garbage collected. Nice knowing you. Bye bye cache.

the solution to this, is to have a persistence manager that IS serializable, which is what Shades has. And shades doesn't have a pseudo-cache that's subject to garbage collection EVEN when you STILL HAVE references to the PersistenceManager. Shades has a REAL CACHE, with HARD REFERENCES to clean values for all the loaded fields of your pojo, for dirty checking when you do updates. So you can serialize the POJO's with OR without also serializing the DatabaseSession (shades' version of a PersistenceManager, is called the DatabaseSession).

If there is a downside to the Shades way of doing business, its that you have to manage the cache explicitely. Shades has a simple method, "clear()" in the DatabaseSession. My take on it, is that it's one hell of a lot easier to clear the database session after the user hits the SAVE button, or just allow the DatabaseSession to be garbage collected, forget about it, and create a new one for your next sequence of CRUD operations, than it is to work with a pseudo cache.

some skate pics

2006-09-16T08:23:00.000-07:00

Smithgrind at the Crib Ramp

frontside noseblunt slide at 3rd and Army

immutable queries

2006-09-10T22:05:00.000-07:00

I got some feedback (see previous blog entry comments) that prompted me to implement "immutable queries". These are queries in which the SQL is 100% specified by the user.

String expr = "SELECT FNAME AS FNAME FROM STUDENT";
Query q = QueryFactory.newImmutableQuery(expr);
RecordCandidate cand = q.candidate(dict.getORM("STUDENT"));
cand.setFetchColumns("FNAME");
List students = new ArrayList();
sess.executeQuery(connection,q).populateList(students, Student.class);
System.out.println(students);

Of course it is still parameterizable, like this:

String expr="SELECT FNAME AS FNAME FROM STUDENT WHERE "+
"LNAME LIKE ${lname}";
Query q = QueryFactory.newImmutableQuery(expr);
RecordCandidate cand = q.candidate(dict.getORM("STUDENT"));
cand.setFetchColumns("FNAME");
List students = new ArrayList();
sess.setParameter("lname", "smith");
sess.executeQuery(connection,q).populateList(students, Student.class);
System.out.println(students);

I'll update the Shades download sometime next week.

sourceforge project

2006-09-06T23:55:00.000-07:00

I just created a project on sourceforge:
http://sourceforge.net/projects/shadesdb

check it out!

Questions from the Wicket world

2006-08-27T13:46:00.000-07:00

Here are some answers to questions that came up on the Wicket user list, about using Shades with the Wicket "library example"

1) how to do concise queries:

Usually the query is defined in the ORMDictionary. In
the example I showed, I defined the query on-the-fly.
Most of the time you would just grab a query from the
dicitonary. So it's this concise:

dbSess.setParameter("author", "william gibson");
dbSess.executeQuery(conn, dict.getQuery("q-by-auth"))

2) how to do arbitrarily complex joins:

There is no limit on how deep or complex the joins can
be with Shades. Here is an example where we setup a
query to find all books published by whatever
publisher publishes the works of William Gibson.
(again bear in mind that usually we would just grab
this query from the dictionary, I am only defining it
inline to be illustrative).

aBook.relatedTo(anAuthor,"book->author");
aBook.where("AUTHOR like 'william gibson');
aBook.relatedTo(aPublisher,"book<->publisher");
anotherBook.relatedTo(aPublisher, "book<->publisher");
query.setFetchGroups(anotherBook);
dbSess.excecuteQuery(conn, q);

3) performance

Ahhh, I am especially stoked on this. Shades uses
batch updates and the other tricks common to ORM's. I
learned most of these tricks doing JDOMax. I think the
biggest reason shades has good performance is because
the codebase is so tight. When it comes to defining
queries on the fly, shades will be better than any
language that compiles a query (for example JDOQL),
because Shades bypasses the parse phase, and the
translation of the AST from JDOQL to SQL. Shades does
query caching like any other ORM.

I'm a lover not a fighter

2006-08-26T15:15:00.000-07:00

I've been asked, rightly so, if Shades is 'better' than Hibernate or JDO implementations or EJB 3. The answer is no. It is not better. But it is different. It's also new, so it doubtless has too many bugs to even dream of playing in the same spaces as the excellent products like Kodo, Cocobase, and others, that implement a variety of standards.

What I want to say, is that I didn't write Shades for anyone else's benefit. I wrote Shades because software, for me, is fun to write. It relaxes my mind after a long day at work, and let's me play with puzzles and art all rolled into one. It's not a competition, especially since I don't get paid!

Queries

2006-08-26T14:20:00.000-07:00

Shades has an interesting way of doing queries. It's based on the idea of a RecordCandidate. I find it much easier to explain this using code than with a paragraph.

The following code returns all the Books, in a List;

ORMDictionary dict = MyORMDict.getInstance();
Query query = QueryFactory.newQuery(dict);
RecordCandidate aBook = query.candidate(dict.getORM("BOOK"), "aBook");
RecordSet rs = dbSession.executeQuery(jdbcConn, query);
List books = new ArrayList();
rs.populateList(books, Book.class);

There are several cool things:
1) The books got put into MY list (not a proxy List created by the data access framework).
2) There is no query language. Shades uses a new form of query-by-example. In the example
above the query is told what to retrieve by requesting a candidate.
3) I passed the jdbcConnection into the query (This means I control transactions using the JDBC transaction model).

Why did we request a RecordCandidate if we did didn't use it for anything. Well OK then, let's use it for something. In the query below we retrieve only the books authored by william gibson.

ORMDictionary dict = MyORMDict.getInstance();
Query query = QueryFactory.newQuery(dict);
RecordCandidate aBook = query.candidate(dict.getORM("BOOK"), "aBook");
RecordCandidate anAuthor = query.candidate(dict.getORM("AUTHOR"), "anAuthor");
anAuthor.where("NAME like 'william Gibson'");
aBook.relatedTo(anAuthor, "book->author");
RecordSet rs = dbSession.executeQuery(jdbcConn, query);
List books = new ArrayList();
rs.populateList(books, Book.class);

Shades let you bust into a query and insert your own SQL. You can see that on the line above that says "anAuthor.where...""
Dangerous? maybe, but fear not. shades encourages you to encapsulate your queries inside the ORMDictionary, and to parameterize them. Once yo've done that, there is no hint of SQL in the code, which subsequently looks like this:

ORMDictionary dict = MyORMDict.getInstance();
dbSession.setParameter("authorName", "william gibson");
RecordSet rs = dbSession.executeQuery(jdbcConn, dict.getQuery("query:book-by-auth"));
List books = new ArrayList();
rs.populateList(books, Book.class);

Here is another very cool thing about Shades queries. They, by default, return ALL the candidates that participate in the relationship graph. So in fact, you can retrieve the Book AND its author from the RecordSet, like this:

Book book = new Book();
Author author = new Author();
while(rs.hasNext()){
rs.populate(book, aBook);
rs.populate(author,anAuthor );
System.out.println(book.title +" was authored by " + author.name);
}

cool things about shades

2006-08-21T21:32:00.000-07:00

One of the most confusing aspects of persistence frameworks is persistant identity. Equals and hashcode typically must be implemented so that Object equalty amounts to a comparison between identity columns of the persistent instance. Since datastore identity is often implemented using autoincrementing primary keys, this leads to a framework dilema: to expose datastore identity in the pojo, or not. Some persistence frameworks force the pojo to hold onto the PK. Other frameworks use bytecode instrumentation to make it transparent.

The situation is arguably simpler for "Object Identity". It is assumed that "Object Identity", as opposed to datastore identity, means that a unique set of column values is mirrored in the pojo's fields. So the pojo naturally posseses as set of fields, that taken together uniquely identify the record in the datastore. Here we again find ourselves bitten by the false equivalence between a pojo in memory, and a record in the database. Remember, equals and hashcode must be overridden to use exactly the identity fields of the Object. This makes it extremely tricky to support "change of identity" during a transaction.

In implementing JDOMax I learned a lot about this issue. Change of identity is actually very common: changing a "User" Object's 'username' field, for example. Because most frameworks for transparent object persistence keep a cache, change of identity can have a devastating and unexpected effect on this cache. Because, equals and hashcode suddenly return different values at different times during the transaction, the cached pojos can appear to "dissappear" from cache.

So one of my biggest issues with ORMapping that claims to be transparent is that it ain't transparent. If I have to override equals and hashCode in a particular way, the mapping is not transparent.

Unlike any other O/R Mapping system that I know of, Shades does not impose any restrictions on how you have implement equals and hashcode. In fact, it doesn't care at all. Shades has a dynamic ORMapping system, as opposed to a static mapping. You can query a record from the database, and load it into a pojo whose equals and hashcode depends only on the 'lastname' field of the pojo. In the same transaction you can load a second record from the table into a pojo whose equals and hashcode depend only on "firstname". You can change the firstname or the lastname field of either pojo, during the same transaction. You can load a third record into yet another pojo whose equals and hashode depend on NONE of the fields of the object. Modifications to the objects are transparently tracked and flushed out to the database on a call to 'update'.

Shades provides a dynamic ORMapping system, in which an ORMapping can be created or chosen, at runtime, to perform I/O from table to pojo. This has an advantage of allowing the "identity" of the pojo to depend on different columns of the database in different situations. Anyone who has ever built an app using straight SQL knows that these "perspectives" on a table surface in the variety of different columns that are retrieved in different situations. Shades is designed to be fluid and adaptable to these common situations. In fact, I thought of the name "Shades" because a good data access framework should recognize the shades of grey that permeate data access programming. Perhaps most of all, shades tries to minimize the number of rules, states, and do's and don'ts.

Lot's of progress

2006-08-11T19:54:00.000-07:00

Well, I used to work for a guy who said writing software is a lot like baking waffles. The first waffle sticks to the iron and doesn't come out very well, but it primes the iron with just enough grease to make the second waffle perfect.

So it seems I have just baked my second waffle. I'm putting the finishing touches on Shades, a framework for ORMapping. My first waffle was JDOMax. Why the hell does the world need yet another ORMapping framework? Let me tell you:

Too much XML configuration
Transparent object persistence misidentifed as necessary
Relationships between records ARE NOT analogs to references between Objects.
Inheritance relationships are rarely natural in data models.
Transitive closure persistence unnecessary
Too many transactional states increases complexity beyond original problem

I learned all of the above the hard way! I'm not just pontificating on this crap. I spent roughly 3 years developing JDOMax and passing the Sun Test Compatibility Suite, so my first waffle was one mother of a waffle.

Looking at 1, "Too much XML configuration", your first reaction might be to think this is an implementation detail; just a consequence of bad XML design. I argue that this configuration complexity is a natural reflection of inherent complexity. Unfortunately, the inherent complexity lies not in the problem you are trying to solve, but in the act of object relational mapping itself. It seems that a static specification of mapping fails to easily convey *contextually* relevant information. In other words, a "mapping" is too static. What is really needed is an Object/Relational Interface, the implementation of which is *code*, and therefore can, in a few lines, create contextually relevant decisions that make ORMapping decisions dynamic and flexible rather than static and ridgid.

Shades has absolutely no XML nor annotation configuration. Rather, there's an interface called ORMapping that operated really a lot like a TableModel. A DefaultORMapping is usually extended by the program, and only a few methods need to be implemented. At first I was shocked by how easy it was but it's now apparent to me that O/R Mapping is better suited to programmetic configuration than document configuration. The reason XML configuration of O/R Mapping is so complex is because...well, because in THIS CASE it's way more complex than just writing a few lines of code.

Somewhere along the line we all just accepted that XML was the way to configure O/R mapping. We bought into a real big advantage of XML - that it's externalized, and can in theory be edited by this mythical "deployer" person. The problem is that this mythical deployer does not exist. In reality he's the programmer and it turns out, now that the wow cool factor has warn off, that the XML is harder to deal with than a few lines of code.

2. Transparent Object Persistence Misidentified as Necessary
This is a big deal. Transparent Object Persistence (TOP)is the idea that you operate on Objects with no concern for the fact that they may be persisted in a relational database. And even beyond that, TOP espouses that it is the Object AND it's references that are in the database. This just turns out to be wrong. THERE IS NO *INHERENT* MAPPING BETWEEN RELATIONSHIPS IN THE DATABASE AND REFERENCES IN THE JVM. But it is *possible* to relate object references to relationships in the datastore. The problem is that the mapping is not one-to-one, and again we are back to a solution that creates as least as many problems as it solves.

Quick example. Teacher Object has a collection of Student Objects. Each Student Object has a field named teacher. Database has an FK in STUDENT_TABLE that points at the PK of teacher table. So in Java land you remove a student from a teacher's collection. This must immediately null the teacher field in the student Object. This is unexpected. And making it work leads to a complex implementation of proxy collections. When the transaction is committed, the FK in the student is set to null. It's simply not at all clear that removing an element from a collection in memory is the right way to remove a relationship from the database. It's just not an intentional way to program. In other words, in order to "forget" about the relational database, we have to learn more than we ever bargained for, and understand subtleties, that even for experts can be mind bending. And the transparency turns out to be nothing near transparent. Consider a STUDENT_TABLE that places a unique constraint on STUDENT_LNAME. If you created a new Student in memory, and added it to a Teacher Object's collection, when would you find out this was invalid? You'd find out when you committed, and a nested datastore exception was thrown. What's so transparent about that. The transparency is a myth. And how would you handle the exception when you created 19 other Students in the transaction? Yes, yes, it is *possible* to handle using an array of nested exceptions, blah blah blah. BUT FOLKS, IT'S NOT EASY. YOU HAVE TO LEARN MORE TO MAKE IT WORK THAN TO SOLVE YOUR ORIGINAL PROBLEM!!!!!!!!!!!!

getting our of 'nam for som R&R

2006-08-02T16:21:00.000-07:00

A few weeks ago I read an article on TSS called Ted Neward explains ORM as "The Viet Nam of Computer Science.

I have to say, from personal experience, that I agree with him.I got interested in O/R several years ago, and wound up writing an implementaiton of JSR 12, Java Data Object called JDOMax www.jdomax.com. It took about 2.5 years to write the software and pass the Sun Test Compatibility Kit. While I was writing the software, I often got the feeling I was going down into the rabbit hole. And after some time, just passing the TCK became a goal in and of itself. When all was said and done, roughly 50,000 lines of code had been written and debugged, and was being used by a few hundred downloaders. Then I got married and haven't looked back at my hobby since.

My goal now is to produce some software that can be used to easily query , insert, and delete data from the relational database, with a small learning curve, and most of all a small internal codebase. Basically, I have to distill something that is vastly simpler than the fullblown JDO that I had implemented, if I am to have any hope of getting out of 'Nam for some R&R.

Over the last few years, there has been a lot of debate on "query by example" vs. "query language". Having written a JDOQL compiler using JavaCC, I knew that I was going to have dispatch with the query language. Too much complexity, and more code than I can maintain. The second thing I decided to do away with was transparent object persistence. Transitive closure persistence, simply isn't an essential feature, and it often confuses developers.

In short, I had to be willing to do many things differently, even if it meant joining the Viet Cong. I hoped that by freeing my mind of dogma I could produce a system that leveraged the best and most essential features of several approaches.

I began by writing lots of "fake programs". I wanted to be sure that no matter how I implemented the internals, that the software was easy to use, and produced tight code, with an absolute minimum of configuration.