Saturday, May 23, 2009

random lines of poetry


I woke up early this morning around 6:30 thanks to Claritin-induced sleep deprivation. I started hacking on server code for automatically building "tables of contents" for very large database tables. I'm using a bunch of Yeats poems, and The Iliad as sample rows in the database. It makes debugging a lot more fun when the data is interesting. It's also a great way to read random lines of poetry. Here is a great stanza from the Iliad:

Generations of men are like the leaves.
In winter, winds blow them down to earth,
but then, when spring season comes again,
budding wood grows more. And so with men--
one generation grows, another dies away. (Iliad 6.181-5)



Did you know the phrase "another one bites the dust" is lifted from the Iliad?

"
Grant that my sword may pierce the shirt of Hector about his heart, and that full many of his comrades may bite the dust as they fall dying round him."

Saturday, April 11, 2009

Tesla rocks...

Aside from the fawning interviewer, this is a great interview with the CEO of Tesla Motors.

http://www.techcrunch.com/2009/04/10/teslas-elon-musk-grows-a-pair-good-for-him/

It would be criminal if Tesla didn't receive the funding they need. To see a loser like GM sloshing in bailout funds is a sickening contrast.

Tuesday, March 31, 2009

Nextdb in 120 seconds

Sunday, March 22, 2009

Back in one piece...

Spent the weekend near Tahoe in Truckie. We got 1.5 ft of snow between saturday night and sunday afternoon. Nevertheless Emily and I decided to drive back as soon as I-80 opened so we could get back Sunday night and not miss Monday at work. Tire chains are a real pain in the butt. Both my chains snapped. Fortunately I was able to pull to the side of the road in a relatively safe location (meaning semis were roaring by like 4 feet from me as I crawled under the car to disentagle the chains from the half axle - getting fairly mired in much at the same time). Tire chains are required at times on I-80, but they honsestly seem like very unsafe garbage. The road is littered with broken chains. My theory is they are designed to be a deterent to drivers, more than an actual functioning safety mechanism. Anyway, chain chaos aside, we had a very relaxing weekend watching snow fall on evergreens (I was tempted to say Snow Falling On Cedars, but I refrained).

Actually, I just checked the DOT report for I-80:

"Enter Highway Number(s)
You can also call 1-800.427.7623 for current highway conditions. Mobile

Click It or Ticket

This highway information is the latest reported as of Sunday, March 22, 2009 at 21:50 .


I 80

[IN NORTHERN CALIFORNIA & THE SIERRA NEVADA]
WESTBOUND TRAFFIC IS BEING HELD AT CISCO GROVE (PLACER CO) - DUE TO AN
ACCIDENT - MOTORISTS ARE ADVISED TO USE AN ALTERNATE ROUTE

CHAINS ARE REQUIRED ON ALL VEHICLES EXCEPT 4-WHEEL-DRIVE VEHICLES WITH SNOW
TIRES ON ALL 4 WHEELS FROM 1 MI EAST OF BAXTER (PLACERCO) TO THE
DONNER LAKE INTERCHANGE (NEVADA CO) "

Wednesday, March 18, 2009

Tuesday, March 17, 2009

From Willow Tea Room, Glasgow

Monday, March 16, 2009

Photos From Scotland

In 2006 I took a trip to Scotland for an OGC meeting, and had the weekend to roam around. I visited the National Library and did a little geniological research. After a fair bit of cross-referenced parish records, pre-census, I believe I located the birth records for my great, great, great, grandfather and grandmother. Unfortunately the library closed right when I pieced it all together, but that's my next trip I guess. The microfilm will still be there!

One of the most fascinating aspects of Edinburgh and Glascow was there role in the Arts and Crafts and Art Nuveau movements. Phoebe Anna Traquair (P.A.T) was one of the female artists who figured prominently in this movement. If you ever have a chance to visit the Mansfield Traquair church, do not miss it.



You can get a better look, here: http://www.mansfieldtraquair.org.uk/nav/cag02.html

If that doesn't blow your hair back, get a look at this amazing painting by P.A.T.. Oh, wait, take a second look, it isn't a painting! It's SILK EMPROIDERY!!


It's almost incomprehensible that a few years ago, the Mansfield Traquair church was being used as a brick storage facility. Thankfully, it was recently rescued an restored. For the life of me I cannot understand why P.A.T. and John Duncan are not better known. Please read this blog to see some more breathtaking work by P.A.T.

I am so awestruck by John Duncan's St. Bride that I recently aquired a canvas reproduction of it from an obscure source on the web. Here is a shot of it gracing my living room.


Scotland has lots of badass wildcats, which came to America as stowaways, and became the big Main Coon cats.

Loch Ness... of course. On the way there we passed the spot where the Battle Of The Shirts (or lack thereof) took place. It turns out that fighting a clan battle in sweltering heat with armor is a good way to die of heat stroke. So it probably seemed like a reasonable idea is to call for a "time out" because of the heat, and take a dip in the loch. Unfortunately, it turned out to be a very bad idea to agree to fight the second half of the battle without armor because of the heat. The horrific mutual slaughter that ensued, fought with claymore broad swords and battle axes left just 12 survivors in total, and the Loch was litterally red with blood for days.
Edinburough....

Sunday, March 15, 2009

Another Gem from William Morris

For the Bed at Kelmscott

The wind's on the wold
And the night is a-cold,
And Thames runs chill
'Twixt mead and hill.
But kind and dear
Is the old house here
And my heart is warm
'Midst winter's harm.
Rest then and rest,
And think of the best
'Twixt summer and spring,
When all birds sing
In the town of the tree,
And ye in me
And scarce dare move,
Lest earth and its love
Should fade away
Ere the full of the day.
I am old and have seen
Many things that have been;
Both grief and peace
And wane and increase
No tale I tell
Of ill or well,
But this I say:
Night treadeth on day,
And for worst or best
Right good is rest.

ProPublica: Jouranlism in the public interest

I'm replacing my daily reading of the New York Times Online Edition with that of Propublica.org

Basically, none of the major news outlets can be trusted. This is just a fact. Remember before we went into Iraq how the New York Times ran breathless stories on Sadam's vast underground bunker complex. It was all just a load of shit.

So now I'm going try getting my news from ProPublica, because their mission is to reinvigorate investigative journalism.

Friday, March 13, 2009

Developers are starting to catch on!

Here is a link to a blog post written by a developer at SolutionSoft, about NextDB.


http://blog.solutionset.com/wpmu/2009/03/11/nextdb-a-front-end-database-solution/


It's a great post, and I think it illustrates how easy nextdb is to learn, and use. Nextb is getting preety close to exiting alpha and entering Beta. During the Beta we will have our "subscribe" buttons up on the site. We are targeting price points that are basically below the noise for even a personal account, so affording nextdb should not be a problem for anyone.

Brent is busy developing and refining a new set of utilities for adding forms to your AJAX apps. The forms will automatically configure themselves based on the content of your table or query, and will be CSS friendly.

I'm working on a notification system that will allow you to receive email when data is inserted into a table. You'll be able to request digests, as well as create email templates that receive data from the inserted row in order to create a personalized, pretty email.

Sunday, January 25, 2009

ulimit -n .... the saga continues

Man, I thought I had put the "ulimit -n issue" to bed once and for all with these configurations in CentOS. So, it was with great chagrin that I recently executed a "ulimit -n" and saw 1024.

Argghhh. Will this issue never end?

I've added "ulimit -n 100000" to my .bash_profile, and now I have confirmed that after logging into a new session, that the new ulimit of 100000 files is in effect.

I'm not happy with putting this in my .bash_profile, but the alternatives looked too dangerous and or flakey.

Wednesday, January 21, 2009

some recent tech photos

At deCarta I am building a new map tile system for our servers. Krish installed it on a floppy drive! That was pretty funny.Brent and I went to the Cloud Connect conference. We tried to pay attention but we were too busy hacking :-)
Panel discussion at Cloud Connect was lively to say the least. I thought the dude from Salesforce.com was going to smash a chair on someone. But seriously, he is a very sharp dude. Would not want to debate him.

Wednesday, January 14, 2009

Test reviews Widget


Please don't say "meme" with a straight face

It's early, I'm grouchy, and I've seen the word "meme" one too many times. People: please, please don't use the word "meme" to describe internet trends, or anything else for that matter. It's just silly. It's like putting plastic spinny rims on an '83 celica. It's basically gheto-bling.

Thursday, January 08, 2009

The Wanderings of Oisin

I was trying to find an extant copy The Evergreen Periodical, when I came across this intersting read on the Celtic revival's influence on art nouveau, which in turn led me to The Wanderings of Oisin by Yeates.

"An old man stirs the fire to a blaze,
In the house of a child, of a friend, of a brother.
He has over-lingered his welcome; the days,
Grown desolate, whisper and sigh to each other;
He hears the storm in the chimney above,
And bends to the fire and shakes with the cold,
While his heart still dreams of battle and love,
And the cry of the hounds on the hills of old.
"

Hacking JS-Kit reviews widget

I just installed the JS-Kit reviews system on my blog below. I was amazed to find the following security holes:

1) no CAPTCHA protection on posts. This means you can flood the reviews with bogus reviews and spam. Want to bring down the reviews system? Just write a 4-line script to execute this URL in a loop:
http://js-kit.com/comment.put?ref=http://notskateboarding.blogspot.com%2F&permalink=http%3A%2F%2Fmysite.com%2Fpermanent%2Flink%2Fto%2Fpage.html&rvy=1&js-CmtName=reviewer04&js-CmtCity=san%20franciso%2C%20USA&js-CmtEmail=geoff.hendrey%40gmail.com&js-Cmtsubmit=Submit%20review&js-CmtsubmitOrig=Submit%20review&js-CmtsubmitReply=Submit%20comment&js-Cmtcancel=Cancel&js-CmtText=BBBBBBBBBBBBBBBBBBBBBBBBBBB&js-CmtRating=8&tid=jst-1

2) Email span the person who posted a comment you don't like! This one is really amazing! Want to send someone 1000 emails anonymously? Same technique as above, just make it a comment instead of a posting, and the person who made the original posting gets 1000 emails!

test a reviews widget

Tuesday, January 06, 2009

annoying CENTOS problems raising ulimit

My last attempt to raise the open file count on a CentOS server apparently failed. After much dicking around, here is a set of steps that I believe will successfully raise the limit (don't be fooled by whatever you try. If you do a 'ulimit -n' and the returned value is 1024, you have not succeeded in raising the limit).

I did a bunch of reading, and found two postings, that combined seem to have succeeded in upping the max files to 65535.

Mainly I am writing this down so we can remember what the hell I did. These are the two useful posts:
http://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/ http://www.centos.org/modules/newbb/viewtopic.php?forum=1&topic_id=559&viewmode=flat

in summary:
1) edit /etc/sysctl.conf to add this line:
fs.file-max = 200000
2) sysctl -p
3)switch to /etc/security/limits.conf and add ther following lines

* soft nofiles 65535
* hard nofiles 65535
4)ulimit -n 65535

I think this worked this time because when I do 'ulimit -n' it returns 65535, whereas prior it always returned 1024. In our previous attempt we only did step 3.

The 200000 value is excessive, but who cares.
I am working on a widget for adding reviews to blogs and websites.

Saturday, January 03, 2009

test review widget

I am working on a widget for adding reviews to blogs and websites.

Friday, January 02, 2009

Tahoe Trip



Wednesday, December 24, 2008

Happy Newton's Birthday

George just pointed me to a great article in the NYTimes titled "The Ten Days of Newton".

On the tenth day of Newton,
My true love gave to me,
Ten drops of genius,
Nine silver co-oins,
Eight circling planets,
Seven shades of li-ight,
Six counterfeiters,
Cal-Cu-Lus!
Four telescopes,
Three Laws of Motion,
Two awful feuds,
And the discovery of gravity!

Happy Newton, everybody!

Tuesday, December 23, 2008

Histrator update

I received this email from Hostrator:

"Dear Geoff,

We have some redirection issues, Note all your files are there intact, the issue occurred while we were upgrading the bandwidth for each user (was our surprise). Again your sites (files) are there and no need to worry about it, your site will be back online in a couple of hours hopefully."

I guess they resolved their issue, because my widget's HTML files are back online.

Monday, December 22, 2008

asscrack hosting, inc!

Wow, notice that my signup widget on the right side of the page has been replaced by the hostrator home page? Either these guys are complete scumbags, and they trick you into using free hosting, then replace your own HTML files with their homepage, or they are so broken that they have genuinely lost the files that I hosted there. Either way, their homepage is appearing where my widget should be. Well, time to find another free host for my widget HTML files. Anyone know a *reliable* and *ethical* provider of HTML file hosting?

Sunday, December 21, 2008

Encryption, digital signatures

For a few weeks NextDB has had support for encryption via the CYPHER function, and sending email through the SENDEMAIL function. However, we didn't have support for providing application specific encryption keys, nor for digital signatures. Well, I spent today prototyping support for both of these two things. The CYPHER function now accepts a second argument, which is a 16-character string (128 bit privtate key). Digital signatures are accomplished through a new function called MD (Message Digest). A DECYPHER function allows you to decrypt whatever you encrypt. This creates a "round trip" model for your data, a lot like we do for our SURIDs. But you can put whatever applicaiton specific payloads you like inside the encrypted messages.

the impotus for this was being able to send out "confirmation" emails from NextQuery expressions, and not allow the content of the URL in the email to be tampered. The following is a NextQuery expression that I am using with a "5-star reviews" widget that I am developing. When the user posts a review, this query sends an email to the poster of the review, including a link to click to confirm (for sake of example, we are assuming the Reviews Widget has been places on 'mysite.com')

NAME=sendReviewConfirmationEmail;
ROW review FROM REVIEW;
WHERE(SURID pk){
review.PK = ${pk}
}
RETRIEVE SENDEMAIL(
'please confirm your review',
'click this link to confirm your review\n' ||
'http://mysite.com/reviewConfirm?'||CYPHER(review.title||':'||review.author||':'||MD(review.title||':'||review.author),'0123456789abcdef'),
review.authoremail,
'reviewAdmin@mysite.com'

)
into email.status;

The SENDEMAIL function accepts the subject, body, to, and from parameters. The interesting thing here is the third parameter, the body of the email, which includes a URL. Part of the URL (the part after the question mark) consists of a cyphered and digitally signed concatenation of the review's title, and the author's screenname.

Where things get really mindbending, is that the content of mysite.com/reviewConfirm is actually an HTML page with JavaScript embeded. The embedded JS can actually get access to parameters passed to the page, and thereby fulfil functions typcially handled on the serverside, even though they will run right there in the user's browser.

When the "reviewConfirm" JavaScript executes, it uses NextDB's JavaScript API to execute a query. Said query will use the DECYPHER function to decypher the URL's query parameter, and use the MD function to check the digital signature to be sure the confirmation email content has not been tampered. Said query returns a SURID with FOR UPDATE permission which allows the JS to change the value of the 'status' column of the row to 'approved'.

But what about replay attacks? That's the beauty of this technique. Because the CYPHER function's secret key can be different for each query, and because the content of the encrypted message can be used to confirm the presence of a row in the database, you can't "replay" a cyphered structure from your database against someone else's database.

Tuesday, December 16, 2008

Steal This Widget

Monday, December 15, 2008

Widget (Take I)

As you can see, I've added a rather oversized "signup" widget to my blog. I am experimenting with creating widgets using NextDB.net. This one will add your "user" information into the NextDB database when you signup.

Why?


User registration is really "square one" of building any website. If we can encapsulate user signup into a widget, then we can encapsulate all other site functions as well. Then, with site functions "widgetized" you can essentially turn your blog, or really any web page into a full-featured AJAX site.

I've learned a bit about how best to do this. The first thing I learned is that sites like blogger.com have a pretty friendly way for you to distribute your widget. It boils down to a simple button a viewer of your blog can press, and presto, they get the widget.

Second thing that took more messing around than it should was to find a webhost to host the HTML and image files for the widget. I intentionally didn't want to host any of the HTML or resources on nextdb.net, because I really wanted to behave as a nextdb user, who might be a graphic designer or someone in a dorm room, for whom a free webhost was their only option. I wound up creating an account at hostrator.com. But it is lame that they don't support SFTP or a better batch upload. But it's good enough. And I think that's sort of the point. With an AJAX widget it really doesn't matter who hosts the HTML file is hosted.

Sunday, December 14, 2008

The Lapse of The Year

SPRING I am too soft of heart
Much to speak ere I depart
Ask the summer tide to prove
The abundance of my love

SUMMER looked for long am I
Much shall change or ere I die
Prithee take it not amiss
Though I weary thee with bliss

Laden AUTUMN here I stand
Weak of heart and warn of hand
Speak the word that sets me free
Naught but rest seems good for me

Ah shall WINTER mend your case
Set your teeth the wind to face
Bear the snow down, tread the frost
All is gained when all is lost

-William Morris

Saturday, December 13, 2008

NextDB.net Google Group

NextDB.net now has a google group

http://groups.google.com/group/nextdb-user


We are hoping to get some good feedback from our user community. Small but growing.

Tuesday, December 09, 2008

St. Bride

This is a painting titled "St. Bride" by the Scottish painter John Duncan, from 1913. At the moment, this is my favorite work of art. I was fortunate enough to spend some time reflecting on this painting at the National Gallery in Scotland. It's is surprising that this painting is not more notable, considering how interesting it is (not to mention beautiful).


The painting depicts two angels carrying Bridget to Bethlehem to swaddle baby Jesus. It's truly an amazing amalgamation of Christian and Celtic mythology; Briget becomes St. Bride and Christianity synthesizes the ancient beliefs.

"[The legend] ...goes further back than the days of the monkish chroniclers who first attempted to put the disguise of verbal Christian raiment on the most widely-loved and revered beings of the ancient Gaelic pantheon. Long before the maiden Brigida… made her fame as a 'daughter of God'… the Gaels worshipped a Brighde or Bride, goddess of women, of fire, of poetry… one whom the Druids held in honour as a torch bearer of the eternal light, a Daughter of the Morning."

This is a wonderfully rich posting on the "Celtic Twilight" movement. Apparently William Blake, JRR Tolkien, and John Duncan shared similar influences. The more I read about this painting, the more I understand why it resonates with me.

Thoughts and Pimps

Wonderful tiled mosaic: "THOUGHT: written words; Spoken words". I took this photo near Rockefeller Center.


Early anthropological evidence of pimps (American Museum of Natural History)

This is just too easy!

Wow, I am amazed how easy it is to wire up my new widget to NextDB.net. I mean, I really shouldn't be surprised, given we've been developing NextDB exactly to be this easy, but I'm still stoked. Here is a screenshot of the admin site for my hosted database in NextDB (just created one table for storing the widget data). Next to the admin site, on the right, is my shiny black widget.


Here is the whole 10 lines of code I literally cut-and-pasted out of the NextDB JS Docs into my widget's HTML.


BOOM, press the button on the widget, the data is inserted into the database, and I can even see the row using the admin tool. And there it is! There's my row that I inserted:

Monday, December 08, 2008

Database feeds

I had an idea today that boils down to "RSS for Databases". I was inspired by the very geeky act of "server log watching". Basically, whenever you bring up a cool new database app, you wind up watching the logs just to get a blow-by-blow on what your users are doing. Now, obviously log watching is boring, and requires you to be logged in or ssh'd into the server. Why not apply the same "feeds" to the actual data in the database that we use to follow our favorite blogs or news sites? Obviously some filters would be required to avoid a torrent of data in the feed, but that's all do-able. I'm pretty excited about adding RSS Feeds as a native feature for NextDB database tables. I will probably work on it this weekend. At the moment I'm addicted to screwing with a new widget design in Adobe Fireworks.

Sunday, December 07, 2008

Widgets!

I am in the process of creating a series of widgets that can easily be re-skinned. These widgets will fulfill many basic site functions such as: user sign-up, login, user profile management, photo gallery, blog, forum, etc. Each widget can be easily re-skinned to suit the look and feel of your site, and they are pre-wired to communicate with NextDB.net. Here is an example of a sign-in widget.


Saturday, December 06, 2008

Nice blog enrty in Wired

Looks like Wired did a blog entry on NextDB.

It's funny, if you look in the comments, there are already haters. They don't even have a clue what NextDB does, or how our security model works. That's a good sign. Haters are scared of change.

Big week for NextDB.net

On Thursday, Andres Ferrate, posted this article on Programmable Web titled "Mashups Get a Hosted Database With NextDB.net". We also got an entry on Programmable Web about our JavaScript API. Very quickly the wave of account signups poured in and it hasn't crested yet. I guess this a moment that every small startup has to cope with. Fortunately both Brent and I have been there before but it never gets old. You can't keep your eyes off the server logs, and it's almost a point of pride when you hit "Too many open files" and need to kick a "ulimit -n", which in fact happened yesterday as dozens of new databases were created on NextDB.net. Well, it's not exactly what I'd call being slashdotted, but it was a fun ride. Our challenge now, is how to effectively collect feedback from these Alpha users in order to improve the service based on their experience.

Wednesday, November 26, 2008

SF Park

The new SF skatepark is pretty rad. It is growing on me. Here is some footy of parker I took on my point-and-shoot. This park is crowded. Note near collision with fat kid on bike (listen for the yell).

Read-End....

It doesn't seem to bother anyone else in the office that when you place a page on the copy machine, and hit the big green "copy" button, that this message appears:



READ-END? WTF is READ-END? This copy machine can send emails, but they can't just make it copy a page when you hit the green button?

Tuesday, November 18, 2008

Steve's Feedback on our UI

lots of issues with the "new user" concept. First time someone signs in have obvious links to getting started guide, creating a sample database, etc.

"import" has too much meaning to db guys. Make it say "create a sample database from template" WITH data.

"must match regex in" in tablename is not a good error message for invalid table names.

Suggested "nextdb for SQL database programmers".

Wants to know how to create the primary key.

Some kind of wizard to drive first time users.

tooltips on tabs with check box for "never show this again"

consider changing the query language parameter names and datatype names to 'human friendly' equivalents.

Steve would like to be able to set the default value.

wants to be able to use camel case in column names

relation name should be "relationship name". It also upper cases it.

show lines connecting related tables as an option.

thought he could not enter a query without having data.

did not like the interface for adding data.

wants a "data editor" tab along the top (loads table names along left hand side)

wants column types to be in parentheses.

Steve wants to be able to store HTML and edit it with a foldout editor. I want permalinks to the HTML and JS.

Great idea

trouble executing query with join

no way to relate rows

wants excel spreadsheet

wants a way to get his data out.

"You wouldn't want a mashup for your bank account."


I am watching the opening panel discussion at Mashup Camp. Hart Rossman, CTO at SAIC just said "You wouldn't want a mashup for your bank account."



That really struck a chord because it captures the inadequacy with today's mashup security model. In fact, building a banking system on NextDB is sort of the logical extent of where the NextDB SURID technology can go.

Another interesting statement from one of the panelists:

"SLAs can't be that strong because you can't have really strong remedies."
I agree.

"you're not going to get a new CRM system out of a mashup."

I disagree. Once the database is secure and mashable, the entire application software development lifecycle can be handled by mashups.

The proverbial Map Mashup came up again as a proof point for time savings.

"when the users become the developers, they are doing what they want to do, not what IT wants to do"

The discussion briefly touched on whether or not Amazon Web Services (S3) constitutes a mashup. That's really a point that requires discussion. Once you realize that Amazon Web Services is not Mashable, due to its "web 1.0 security model", you find yourself somewhat dissapointed.

"where we run into a real challenge is in converting the mashup mentality into the largescale systems engineering skillset. You don't see a lot of folks puting a large ampount of skill and discipline into building the backend." paraphrasing what he said later: We're going to see things going in that direction though, and it will drive Mashups into the enterprise.





Sunday, November 16, 2008

Mashup Camp 2008

Tomorrow is Mashup Camp! I'm excited about this year's Mashup Camp because I think there is going to be a lot of activity around trying to figure out how to fuse mashups with Cloud Computing. And that, of course, is exactly where NextDB.net has a sweet spot. NextDB is fundamentally a mashable relational database in the cloud. What does that mean for the programmer? It means NextDB is the ONLY relational database that you can securely mashup, without have to write ANY serverside code.

So check this out. I added this "guest book" to my blog to keep track of folks I meet at Mashup Camp. This guestbook is a Mashup with NextDB.net, which makes it a nice little example of a database mashup between NextDB and blogger.com. Originally, Brent wrote this guestbook as an example for our talk at ajaxworld. I just went to his page, grabbed the html, js, and css and threw them into my blog. The only tricky part was that blogger.com's editor didn't like newline characters in the HTML and CSS. So I just saved them in a file without newlines, and pasted them back to my blog. Go ahead and sign in below, and watch the AJAX goodness! Here is a link to the JavaScript that the guestbook sources. Rather than use the nasty editor at blogger.com (and its associated issues with spaces), I just uploaded the js file to Project Path, which is a nifty collaboration site that allows file hosting.

Mashup Camp Guestbook
first name *
last name *
email
comments


Enter the text above into the box below. Click image to get a new one if you can't read it.


json view:

Wednesday, November 12, 2008

NextDB At AjaxWorld 2008 West

Brent and I recently presented about NextDB.net at AjaxWorld 2008 West, in San Jose CA.

I am going to pickup blogging a bit more regularly here. Anyway, the presentation went well. The message we hammered on was that NextDB has a fundamentally new security model that allows a hosted relational database to be exposed as a service. NextDB is the only relational database that is built from the ground up as a AJAX-programmable hosted service. As such, the round-trip security model is native; fundamentally "built in". This clearly differentiates NextDB from other pseudo-database or database services, because you can safely access it directly from JavaScript.

Monday, November 10, 2008

New York Family Vacation 2008

Times Square

Radio City Music Hall Christmas Spectacular stage





Tuesday, October 31, 2006

Why detachment sucks

Ahhh, yes, detachment. The idea that you must explicitely "detach" a persistent POJO from its persistence manager, and later explicitly attach it.

Detachment is another opaque feature of so-called Transparent Object Persistence.

Basically detachment is a work around for the fact that the JDO PersistenceManager is not itself serializable. So, when your Page, with your so-called POJO gets serialized, "oh ma god", you can't serialize your pojo. (that would be TOO SIMPLE, TOO POJO). NOOOO, you've got to implement a callback in which you detach your POJO before you serialize it.

The worst part of it is, this totally defeats the point of serializing the page. You see, in Wicket, they are serializing the Page and things on the page to keep the server from bloating with lots of concurrent clients. But wait a sec, if the PersistenceManager has a cache of POJO's, isn't that cache going to be full of those very POJO's your trying to serialize. Right, but you can't serialize the PersistenceManager.......OK, right, so how do you keep things trim. So you , um "detach" from the persistence manager, and serialize your detached POJO. In the mean time, you have allowed your PersistenceManager to get garbage collected. Had to, otherwise there was no point in serializing your POJO's. I mean, you can't keep the whole cache of 'em lying around in memory. (and even if you keep your PersistenceManager around, it's not a real cache. If you've serialized out your POJO, it ain't gonna be in the cache when you deserialize the POJO. NOPE, it the POJO will have been garbage collected from the PersistenceManager, because the PM is a pseudo cache, with WeakReferences, not a hard cache).

So, the architecture of detachment is fundamentally at odds with the notion of a cache. So when you re-attach, whoops, you have no cache. Been garbage collected. Nice knowing you. Bye bye cache.

the solution to this, is to have a persistence manager that IS serializable, which is what Shades has. And shades doesn't have a pseudo-cache that's subject to garbage collection EVEN when you STILL HAVE references to the PersistenceManager. Shades has a REAL CACHE, with HARD REFERENCES to clean values for all the loaded fields of your pojo, for dirty checking when you do updates. So you can serialize the POJO's with OR without also serializing the DatabaseSession (shades' version of a PersistenceManager, is called the DatabaseSession).

If there is a downside to the Shades way of doing business, its that you have to manage the cache explicitely. Shades has a simple method, "clear()" in the DatabaseSession. My take on it, is that it's one hell of a lot easier to clear the database session after the user hits the SAVE button, or just allow the DatabaseSession to be garbage collected, forget about it, and create a new one for your next sequence of CRUD operations, than it is to work with a pseudo cache.

Saturday, September 16, 2006

some skate pics





Smithgrind at the Crib Ramp


frontside noseblunt slide at 3rd and Army

Sunday, September 10, 2006

immutable queries

I got some feedback (see previous blog entry comments) that prompted me to implement "immutable queries". These are queries in which the SQL is 100% specified by the user.

String expr = "SELECT FNAME AS FNAME FROM STUDENT";
Query q = QueryFactory.newImmutableQuery(expr);
RecordCandidate cand = q.candidate(dict.getORM("STUDENT"));
cand.setFetchColumns("FNAME");
List students = new ArrayList();
sess.executeQuery(connection,q).populateList(students, Student.class);
System.out.println(students);

Of course it is still parameterizable, like this:

String expr="SELECT FNAME AS FNAME FROM STUDENT WHERE "+
"LNAME LIKE ${lname}";
Query q = QueryFactory.newImmutableQuery(expr);
RecordCandidate cand = q.candidate(dict.getORM("STUDENT"));
cand.setFetchColumns("FNAME");
List students = new ArrayList();
sess.setParameter("lname", "smith");
sess.executeQuery(connection,q).populateList(students, Student.class);
System.out.println(students);

I'll update the Shades download sometime next week.

Wednesday, September 06, 2006

sourceforge project

I just created a project on sourceforge:
http://sourceforge.net/projects/shadesdb

check it out!

Sunday, August 27, 2006

Questions from the Wicket world

Here are some answers to questions that came up on the Wicket user list, about using Shades with the Wicket "library example"

1) how to do concise queries:

Usually the query is defined in the ORMDictionary. In
the example I showed, I defined the query on-the-fly.
Most of the time you would just grab a query from the
dicitonary. So it's this concise:

dbSess.setParameter("author", "william gibson");
dbSess.executeQuery(conn, dict.getQuery("q-by-auth"))

2) how to do arbitrarily complex joins:

There is no limit on how deep or complex the joins can
be with Shades. Here is an example where we setup a
query to find all books published by whatever
publisher publishes the works of William Gibson.
(again bear in mind that usually we would just grab
this query from the dictionary, I am only defining it
inline to be illustrative).

aBook.relatedTo(anAuthor,"book->author");
aBook.where("AUTHOR like 'william gibson');
aBook.relatedTo(aPublisher,"book<->publisher");
anotherBook.relatedTo(aPublisher, "book<->publisher");
query.setFetchGroups(anotherBook);
dbSess.excecuteQuery(conn, q);

3) performance

Ahhh, I am especially stoked on this. Shades uses
batch updates and the other tricks common to ORM's. I
learned most of these tricks doing JDOMax. I think the
biggest reason shades has good performance is because
the codebase is so tight. When it comes to defining
queries on the fly, shades will be better than any
language that compiles a query (for example JDOQL),
because Shades bypasses the parse phase, and the
translation of the AST from JDOQL to SQL. Shades does
query caching like any other ORM.

Saturday, August 26, 2006

I'm a lover not a fighter

I've been asked, rightly so, if Shades is 'better' than Hibernate or JDO implementations or EJB 3. The answer is no. It is not better. But it is different. It's also new, so it doubtless has too many bugs to even dream of playing in the same spaces as the excellent products like Kodo, Cocobase, and others, that implement a variety of standards.

What I want to say, is that I didn't write Shades for anyone else's benefit. I wrote Shades because software, for me, is fun to write. It relaxes my mind after a long day at work, and let's me play with puzzles and art all rolled into one. It's not a competition, especially since I don't get paid!

Queries

Shades has an interesting way of doing queries. It's based on the idea of a RecordCandidate. I find it much easier to explain this using code than with a paragraph.

The following code returns all the Books, in a List;

ORMDictionary dict = MyORMDict.getInstance();
Query query = QueryFactory.newQuery(dict);
RecordCandidate aBook = query.candidate(dict.getORM("BOOK"), "aBook");
RecordSet rs = dbSession.executeQuery(jdbcConn, query);
List books = new ArrayList();
rs.populateList(books, Book.class);

There are several cool things:
1) The books got put into MY list (not a proxy List created by the data access framework).
2) There is no query language. Shades uses a new form of query-by-example. In the example
above the query is told what to retrieve by requesting a candidate.
3) I passed the jdbcConnection into the query (This means I control transactions using the JDBC transaction model).

Why did we request a RecordCandidate if we did didn't use it for anything. Well OK then, let's use it for something. In the query below we retrieve only the books authored by william gibson.

ORMDictionary dict = MyORMDict.getInstance();
Query query = QueryFactory.newQuery(dict);
RecordCandidate aBook = query.candidate(dict.getORM("BOOK"), "aBook");
RecordCandidate anAuthor = query.candidate(dict.getORM("AUTHOR"), "anAuthor");
anAuthor.where("NAME like 'william Gibson'");
aBook.relatedTo(anAuthor, "book->author");
RecordSet rs = dbSession.executeQuery(jdbcConn, query);
List books = new ArrayList();
rs.populateList(books, Book.class);

Shades let you bust into a query and insert your own SQL. You can see that on the line above that says "anAuthor.where...""
Dangerous? maybe, but fear not. shades encourages you to encapsulate your queries inside the ORMDictionary, and to parameterize them. Once yo've done that, there is no hint of SQL in the code, which subsequently looks like this:

ORMDictionary dict = MyORMDict.getInstance();
dbSession.setParameter("authorName", "william gibson");
RecordSet rs = dbSession.executeQuery(jdbcConn, dict.getQuery("query:book-by-auth"));
List books = new ArrayList();
rs.populateList(books, Book.class);


Here is another very cool thing about Shades queries. They, by default, return ALL the candidates that participate in the relationship graph. So in fact, you can retrieve the Book AND its author from the RecordSet, like this:

Book book = new Book();
Author author = new Author();
while(rs.hasNext()){
rs.populate(book, aBook);
rs.populate(author,anAuthor );
System.out.println(book.title +" was authored by " + author.name);
}

Monday, August 21, 2006

cool things about shades

One of the most confusing aspects of persistence frameworks is persistant identity. Equals and hashcode typically must be implemented so that Object equalty amounts to a comparison between identity columns of the persistent instance. Since datastore identity is often implemented using autoincrementing primary keys, this leads to a framework dilema: to expose datastore identity in the pojo, or not. Some persistence frameworks force the pojo to hold onto the PK. Other frameworks use bytecode instrumentation to make it transparent.

The situation is arguably simpler for "Object Identity". It is assumed that "Object Identity", as opposed to datastore identity, means that a unique set of column values is mirrored in the pojo's fields. So the pojo naturally posseses as set of fields, that taken together uniquely identify the record in the datastore. Here we again find ourselves bitten by the false equivalence between a pojo in memory, and a record in the database. Remember, equals and hashcode must be overridden to use exactly the identity fields of the Object. This makes it extremely tricky to support "change of identity" during a transaction.

In implementing JDOMax I learned a lot about this issue. Change of identity is actually very common: changing a "User" Object's 'username' field, for example. Because most frameworks for transparent object persistence keep a cache, change of identity can have a devastating and unexpected effect on this cache. Because, equals and hashcode suddenly return different values at different times during the transaction, the cached pojos can appear to "dissappear" from cache.

So one of my biggest issues with ORMapping that claims to be transparent is that it ain't transparent. If I have to override equals and hashCode in a particular way, the mapping is not transparent.

Unlike any other O/R Mapping system that I know of, Shades does not impose any restrictions on how you have implement equals and hashcode. In fact, it doesn't care at all. Shades has a dynamic ORMapping system, as opposed to a static mapping. You can query a record from the database, and load it into a pojo whose equals and hashcode depends only on the 'lastname' field of the pojo. In the same transaction you can load a second record from the table into a pojo whose equals and hashcode depend only on "firstname". You can change the firstname or the lastname field of either pojo, during the same transaction. You can load a third record into yet another pojo whose equals and hashode depend on NONE of the fields of the object. Modifications to the objects are transparently tracked and flushed out to the database on a call to 'update'.

Shades provides a dynamic ORMapping system, in which an ORMapping can be created or chosen, at runtime, to perform I/O from table to pojo. This has an advantage of allowing the "identity" of the pojo to depend on different columns of the database in different situations. Anyone who has ever built an app using straight SQL knows that these "perspectives" on a table surface in the variety of different columns that are retrieved in different situations. Shades is designed to be fluid and adaptable to these common situations. In fact, I thought of the name "Shades" because a good data access framework should recognize the shades of grey that permeate data access programming. Perhaps most of all, shades tries to minimize the number of rules, states, and do's and don'ts.

Friday, August 11, 2006

Lot's of progress

Well, I used to work for a guy who said writing software is a lot like baking waffles. The first waffle sticks to the iron and doesn't come out very well, but it primes the iron with just enough grease to make the second waffle perfect.

So it seems I have just baked my second waffle. I'm putting the finishing touches on Shades, a framework for ORMapping. My first waffle was JDOMax. Why the hell does the world need yet another ORMapping framework? Let me tell you:

  1. Too much XML configuration
  2. Transparent object persistence misidentifed as necessary
  3. Relationships between records ARE NOT analogs to references between Objects.
  4. Inheritance relationships are rarely natural in data models.
  5. Transitive closure persistence unnecessary
  6. Too many transactional states increases complexity beyond original problem

I learned all of the above the hard way! I'm not just pontificating on this crap. I spent roughly 3 years developing JDOMax and passing the Sun Test Compatibility Suite, so my first waffle was one mother of a waffle.
Looking at 1, "Too much XML configuration", your first reaction might be to think this is an implementation detail; just a consequence of bad XML design. I argue that this configuration complexity is a natural reflection of inherent complexity. Unfortunately, the inherent complexity lies not in the problem you are trying to solve, but in the act of object relational mapping itself. It seems that a static specification of mapping fails to easily convey *contextually* relevant information. In other words, a "mapping" is too static. What is really needed is an Object/Relational Interface, the implementation of which is *code*, and therefore can, in a few lines, create contextually relevant decisions that make ORMapping decisions dynamic and flexible rather than static and ridgid.

Shades has absolutely no XML nor annotation configuration. Rather, there's an interface called ORMapping that operated really a lot like a TableModel. A DefaultORMapping is usually extended by the program, and only a few methods need to be implemented. At first I was shocked by how easy it was but it's now apparent to me that O/R Mapping is better suited to programmetic configuration than document configuration. The reason XML configuration of O/R Mapping is so complex is because...well, because in THIS CASE it's way more complex than just writing a few lines of code.

Somewhere along the line we all just accepted that XML was the way to configure O/R mapping. We bought into a real big advantage of XML - that it's externalized, and can in theory be edited by this mythical "deployer" person. The problem is that this mythical deployer does not exist. In reality he's the programmer and it turns out, now that the wow cool factor has warn off, that the XML is harder to deal with than a few lines of code.

2. Transparent Object Persistence Misidentified as Necessary
This is a big deal. Transparent Object Persistence (TOP)is the idea that you operate on Objects with no concern for the fact that they may be persisted in a relational database. And even beyond that, TOP espouses that it is the Object AND it's references that are in the database. This just turns out to be wrong. THERE IS NO *INHERENT* MAPPING BETWEEN RELATIONSHIPS IN THE DATABASE AND REFERENCES IN THE JVM. But it is *possible* to relate object references to relationships in the datastore. The problem is that the mapping is not one-to-one, and again we are back to a solution that creates as least as many problems as it solves.

Quick example. Teacher Object has a collection of Student Objects. Each Student Object has a field named teacher. Database has an FK in STUDENT_TABLE that points at the PK of teacher table. So in Java land you remove a student from a teacher's collection. This must immediately null the teacher field in the student Object. This is unexpected. And making it work leads to a complex implementation of proxy collections. When the transaction is committed, the FK in the student is set to null. It's simply not at all clear that removing an element from a collection in memory is the right way to remove a relationship from the database. It's just not an intentional way to program. In other words, in order to "forget" about the relational database, we have to learn more than we ever bargained for, and understand subtleties, that even for experts can be mind bending. And the transparency turns out to be nothing near transparent. Consider a STUDENT_TABLE that places a unique constraint on STUDENT_LNAME. If you created a new Student in memory, and added it to a Teacher Object's collection, when would you find out this was invalid? You'd find out when you committed, and a nested datastore exception was thrown. What's so transparent about that. The transparency is a myth. And how would you handle the exception when you created 19 other Students in the transaction? Yes, yes, it is *possible* to handle using an array of nested exceptions, blah blah blah. BUT FOLKS, IT'S NOT EASY. YOU HAVE TO LEARN MORE TO MAKE IT WORK THAN TO SOLVE YOUR ORIGINAL PROBLEM!!!!!!!!!!!!

Wednesday, August 02, 2006

getting our of 'nam for som R&R

A few weeks ago I read an article on TSS called Ted Neward explains ORM as "The Viet Nam of Computer Science.

I have to say, from personal experience, that I agree with him.I got interested in O/R several years ago, and wound up writing an implementaiton of JSR 12, Java Data Object called JDOMax www.jdomax.com. It took about 2.5 years to write the software and pass the Sun Test Compatibility Kit. While I was writing the software, I often got the feeling I was going down into the rabbit hole. And after some time, just passing the TCK became a goal in and of itself. When all was said and done, roughly 50,000 lines of code had been written and debugged, and was being used by a few hundred downloaders. Then I got married and haven't looked back at my hobby since.

My goal now is to produce some software that can be used to easily query , insert, and delete data from the relational database, with a small learning curve, and most of all a small internal codebase. Basically, I have to distill something that is vastly simpler than the fullblown JDO that I had implemented, if I am to have any hope of getting out of 'Nam for some R&R.

Over the last few years, there has been a lot of debate on "query by example" vs. "query language". Having written a JDOQL compiler using JavaCC, I knew that I was going to have dispatch with the query language. Too much complexity, and more code than I can maintain. The second thing I decided to do away with was transparent object persistence. Transitive closure persistence, simply isn't an essential feature, and it often confuses developers.

In short, I had to be willing to do many things differently, even if it meant joining the Viet Cong. I hoped that by freeing my mind of dogma I could produce a system that leveraged the best and most essential features of several approaches.

I began by writing lots of "fake programs". I wanted to be sure that no matter how I implemented the internals, that the software was easy to use, and produced tight code, with an absolute minimum of configuration.