Happy Newton, everybody!On the tenth day of Newton,
My true love gave to me,
Ten drops of genius,
Nine silver co-oins,
Eight circling planets,
Seven shades of li-ight,
Six counterfeiters,
Cal-Cu-Lus!
Four telescopes,
Three Laws of Motion,
Two awful feuds,
And the discovery of gravity!
Wednesday, December 24, 2008
Happy Newton's Birthday
George just pointed me to a great article in the NYTimes titled "The Ten Days of Newton".
Tuesday, December 23, 2008
Histrator update
I received this email from Hostrator:
"Dear Geoff,
We have some redirection issues, Note all your files are there intact, the issue occurred while we were upgrading the bandwidth for each user (was our surprise). Again your sites (files) are there and no need to worry about it, your site will be back online in a couple of hours hopefully."
I guess they resolved their issue, because my widget's HTML files are back online.
"Dear Geoff,
We have some redirection issues, Note all your files are there intact, the issue occurred while we were upgrading the bandwidth for each user (was our surprise). Again your sites (files) are there and no need to worry about it, your site will be back online in a couple of hours hopefully."
I guess they resolved their issue, because my widget's HTML files are back online.
Monday, December 22, 2008
asscrack hosting, inc!
Wow, notice that my signup widget on the right side of the page has been replaced by the hostrator home page? Either these guys are complete scumbags, and they trick you into using free hosting, then replace your own HTML files with their homepage, or they are so broken that they have genuinely lost the files that I hosted there. Either way, their homepage is appearing where my widget should be. Well, time to find another free host for my widget HTML files. Anyone know a *reliable* and *ethical* provider of HTML file hosting?
Sunday, December 21, 2008
Encryption, digital signatures
For a few weeks NextDB has had support for encryption via the CYPHER function, and sending email through the SENDEMAIL function. However, we didn't have support for providing application specific encryption keys, nor for digital signatures. Well, I spent today prototyping support for both of these two things. The CYPHER function now accepts a second argument, which is a 16-character string (128 bit privtate key). Digital signatures are accomplished through a new function called MD (Message Digest). A DECYPHER function allows you to decrypt whatever you encrypt. This creates a "round trip" model for your data, a lot like we do for our SURIDs. But you can put whatever applicaiton specific payloads you like inside the encrypted messages.
the impotus for this was being able to send out "confirmation" emails from NextQuery expressions, and not allow the content of the URL in the email to be tampered. The following is a NextQuery expression that I am using with a "5-star reviews" widget that I am developing. When the user posts a review, this query sends an email to the poster of the review, including a link to click to confirm (for sake of example, we are assuming the Reviews Widget has been places on 'mysite.com')
NAME=sendReviewConfirmationEmail;
ROW review FROM REVIEW;
WHERE(SURID pk){
review.PK = ${pk}
}
RETRIEVE SENDEMAIL(
'please confirm your review',
'click this link to confirm your review\n' ||
'http://mysite.com/reviewConfirm?'||CYPHER(review.title||':'||review.author||':'||MD(review.title||':'||review.author),'0123456789abcdef'),
review.authoremail,
'reviewAdmin@mysite.com'
)
into email.status;
The SENDEMAIL function accepts the subject, body, to, and from parameters. The interesting thing here is the third parameter, the body of the email, which includes a URL. Part of the URL (the part after the question mark) consists of a cyphered and digitally signed concatenation of the review's title, and the author's screenname.
Where things get really mindbending, is that the content of mysite.com/reviewConfirm is actually an HTML page with JavaScript embeded. The embedded JS can actually get access to parameters passed to the page, and thereby fulfil functions typcially handled on the serverside, even though they will run right there in the user's browser.
When the "reviewConfirm" JavaScript executes, it uses NextDB's JavaScript API to execute a query. Said query will use the DECYPHER function to decypher the URL's query parameter, and use the MD function to check the digital signature to be sure the confirmation email content has not been tampered. Said query returns a SURID with FOR UPDATE permission which allows the JS to change the value of the 'status' column of the row to 'approved'.
But what about replay attacks? That's the beauty of this technique. Because the CYPHER function's secret key can be different for each query, and because the content of the encrypted message can be used to confirm the presence of a row in the database, you can't "replay" a cyphered structure from your database against someone else's database.
the impotus for this was being able to send out "confirmation" emails from NextQuery expressions, and not allow the content of the URL in the email to be tampered. The following is a NextQuery expression that I am using with a "5-star reviews" widget that I am developing. When the user posts a review, this query sends an email to the poster of the review, including a link to click to confirm (for sake of example, we are assuming the Reviews Widget has been places on 'mysite.com')
NAME=sendReviewConfirmationEmail;
ROW review FROM REVIEW;
WHERE(SURID pk){
review.PK = ${pk}
}
RETRIEVE SENDEMAIL(
'please confirm your review',
'click this link to confirm your review\n' ||
'http://mysite.com/reviewConfirm?'||CYPHER(review.title||':'||review.author||':'||MD(review.title||':'||review.author),'0123456789abcdef'),
review.authoremail,
'reviewAdmin@mysite.com'
)
into email.status;
The SENDEMAIL function accepts the subject, body, to, and from parameters. The interesting thing here is the third parameter, the body of the email, which includes a URL. Part of the URL (the part after the question mark) consists of a cyphered and digitally signed concatenation of the review's title, and the author's screenname.
Where things get really mindbending, is that the content of mysite.com/reviewConfirm is actually an HTML page with JavaScript embeded. The embedded JS can actually get access to parameters passed to the page, and thereby fulfil functions typcially handled on the serverside, even though they will run right there in the user's browser.
When the "reviewConfirm" JavaScript executes, it uses NextDB's JavaScript API to execute a query. Said query will use the DECYPHER function to decypher the URL's query parameter, and use the MD function to check the digital signature to be sure the confirmation email content has not been tampered. Said query returns a SURID with FOR UPDATE permission which allows the JS to change the value of the 'status' column of the row to 'approved'.
But what about replay attacks? That's the beauty of this technique. Because the CYPHER function's secret key can be different for each query, and because the content of the encrypted message can be used to confirm the presence of a row in the database, you can't "replay" a cyphered structure from your database against someone else's database.
Tuesday, December 16, 2008
Monday, December 15, 2008
Widget (Take I)
As you can see, I've added a rather oversized "signup" widget to my blog. I am experimenting with creating widgets using NextDB.net. This one will add your "user" information into the NextDB database when you signup.
Why?
User registration is really "square one" of building any website. If we can encapsulate user signup into a widget, then we can encapsulate all other site functions as well. Then, with site functions "widgetized" you can essentially turn your blog, or really any web page into a full-featured AJAX site.
I've learned a bit about how best to do this. The first thing I learned is that sites like blogger.com have a pretty friendly way for you to distribute your widget. It boils down to a simple button a viewer of your blog can press, and presto, they get the widget.
Second thing that took more messing around than it should was to find a webhost to host the HTML and image files for the widget. I intentionally didn't want to host any of the HTML or resources on nextdb.net, because I really wanted to behave as a nextdb user, who might be a graphic designer or someone in a dorm room, for whom a free webhost was their only option. I wound up creating an account at hostrator.com. But it is lame that they don't support SFTP or a better batch upload. But it's good enough. And I think that's sort of the point. With an AJAX widget it really doesn't matter who hosts the HTML file is hosted.
Why?
User registration is really "square one" of building any website. If we can encapsulate user signup into a widget, then we can encapsulate all other site functions as well. Then, with site functions "widgetized" you can essentially turn your blog, or really any web page into a full-featured AJAX site.
I've learned a bit about how best to do this. The first thing I learned is that sites like blogger.com have a pretty friendly way for you to distribute your widget. It boils down to a simple button a viewer of your blog can press, and presto, they get the widget.
Second thing that took more messing around than it should was to find a webhost to host the HTML and image files for the widget. I intentionally didn't want to host any of the HTML or resources on nextdb.net, because I really wanted to behave as a nextdb user, who might be a graphic designer or someone in a dorm room, for whom a free webhost was their only option. I wound up creating an account at hostrator.com. But it is lame that they don't support SFTP or a better batch upload. But it's good enough. And I think that's sort of the point. With an AJAX widget it really doesn't matter who hosts the HTML file is hosted.
Sunday, December 14, 2008
The Lapse of The Year
SPRING I am too soft of heart
Much to speak ere I depart
Ask the summer tide to prove
The abundance of my love
SUMMER looked for long am I
Much shall change or ere I die
Prithee take it not amiss
Though I weary thee with bliss
Laden AUTUMN here I stand
Weak of heart and warn of hand
Speak the word that sets me free
Naught but rest seems good for me
Ah shall WINTER mend your case
Set your teeth the wind to face
Bear the snow down, tread the frost
All is gained when all is lost
-William Morris
Much to speak ere I depart
Ask the summer tide to prove
The abundance of my love
SUMMER looked for long am I
Much shall change or ere I die
Prithee take it not amiss
Though I weary thee with bliss
Laden AUTUMN here I stand
Weak of heart and warn of hand
Speak the word that sets me free
Naught but rest seems good for me
Ah shall WINTER mend your case
Set your teeth the wind to face
Bear the snow down, tread the frost
All is gained when all is lost
-William Morris
Saturday, December 13, 2008
NextDB.net Google Group
NextDB.net now has a google group
http://groups.google.com/group/nextdb-user
We are hoping to get some good feedback from our user community. Small but growing.
http://groups.google.com/group/nextdb-user
We are hoping to get some good feedback from our user community. Small but growing.
Tuesday, December 09, 2008
St. Bride
This is a painting titled "St. Bride" by the Scottish painter John Duncan, from 1913. At the moment, this is my favorite work of art. I was fortunate enough to spend some time reflecting on this painting at the National Gallery in Scotland. It's is surprising that this painting is not more notable, considering how interesting it is (not to mention beautiful).
The painting depicts two angels carrying Bridget to Bethlehem to swaddle baby Jesus. It's truly an amazing amalgamation of Christian and Celtic mythology; Briget becomes St. Bride and Christianity synthesizes the ancient beliefs.
"[The legend] ...goes further back than the days of the monkish chroniclers who first attempted to put the disguise of verbal Christian raiment on the most widely-loved and revered beings of the ancient Gaelic pantheon. Long before the maiden Brigida… made her fame as a 'daughter of God'… the Gaels worshipped a Brighde or Bride, goddess of women, of fire, of poetry… one whom the Druids held in honour as a torch bearer of the eternal light, a Daughter of the Morning."
This is a wonderfully rich posting on the "Celtic Twilight" movement. Apparently William Blake, JRR Tolkien, and John Duncan shared similar influences. The more I read about this painting, the more I understand why it resonates with me.
The painting depicts two angels carrying Bridget to Bethlehem to swaddle baby Jesus. It's truly an amazing amalgamation of Christian and Celtic mythology; Briget becomes St. Bride and Christianity synthesizes the ancient beliefs.
"[The legend] ...goes further back than the days of the monkish chroniclers who first attempted to put the disguise of verbal Christian raiment on the most widely-loved and revered beings of the ancient Gaelic pantheon. Long before the maiden Brigida… made her fame as a 'daughter of God'… the Gaels worshipped a Brighde or Bride, goddess of women, of fire, of poetry… one whom the Druids held in honour as a torch bearer of the eternal light, a Daughter of the Morning."
This is a wonderfully rich posting on the "Celtic Twilight" movement. Apparently William Blake, JRR Tolkien, and John Duncan shared similar influences. The more I read about this painting, the more I understand why it resonates with me.
Thoughts and Pimps
Wonderful tiled mosaic: "THOUGHT: written words; Spoken words". I took this photo near Rockefeller Center.
Early anthropological evidence of pimps (American Museum of Natural History)
Early anthropological evidence of pimps (American Museum of Natural History)
This is just too easy!
Wow, I am amazed how easy it is to wire up my new widget to NextDB.net. I mean, I really shouldn't be surprised, given we've been developing NextDB exactly to be this easy, but I'm still stoked. Here is a screenshot of the admin site for my hosted database in NextDB (just created one table for storing the widget data). Next to the admin site, on the right, is my shiny black widget.
Here is the whole 10 lines of code I literally cut-and-pasted out of the NextDB JS Docs into my widget's HTML.
BOOM, press the button on the widget, the data is inserted into the database, and I can even see the row using the admin tool. And there it is! There's my row that I inserted:
Here is the whole 10 lines of code I literally cut-and-pasted out of the NextDB JS Docs into my widget's HTML.
BOOM, press the button on the widget, the data is inserted into the database, and I can even see the row using the admin tool. And there it is! There's my row that I inserted:
Monday, December 08, 2008
Database feeds
I had an idea today that boils down to "RSS for Databases". I was inspired by the very geeky act of "server log watching". Basically, whenever you bring up a cool new database app, you wind up watching the logs just to get a blow-by-blow on what your users are doing. Now, obviously log watching is boring, and requires you to be logged in or ssh'd into the server. Why not apply the same "feeds" to the actual data in the database that we use to follow our favorite blogs or news sites? Obviously some filters would be required to avoid a torrent of data in the feed, but that's all do-able. I'm pretty excited about adding RSS Feeds as a native feature for NextDB database tables. I will probably work on it this weekend. At the moment I'm addicted to screwing with a new widget design in Adobe Fireworks.
Sunday, December 07, 2008
Widgets!
I am in the process of creating a series of widgets that can easily be re-skinned. These widgets will fulfill many basic site functions such as: user sign-up, login, user profile management, photo gallery, blog, forum, etc. Each widget can be easily re-skinned to suit the look and feel of your site, and they are pre-wired to communicate with NextDB.net. Here is an example of a sign-in widget.
Saturday, December 06, 2008
Nice blog enrty in Wired
Looks like Wired did a blog entry on NextDB.
It's funny, if you look in the comments, there are already haters. They don't even have a clue what NextDB does, or how our security model works. That's a good sign. Haters are scared of change.
It's funny, if you look in the comments, there are already haters. They don't even have a clue what NextDB does, or how our security model works. That's a good sign. Haters are scared of change.
Big week for NextDB.net
On Thursday, Andres Ferrate, posted this article on Programmable Web titled "Mashups Get a Hosted Database With NextDB.net". We also got an entry on Programmable Web about our JavaScript API. Very quickly the wave of account signups poured in and it hasn't crested yet. I guess this a moment that every small startup has to cope with. Fortunately both Brent and I have been there before but it never gets old. You can't keep your eyes off the server logs, and it's almost a point of pride when you hit "Too many open files" and need to kick a "ulimit -n", which in fact happened yesterday as dozens of new databases were created on NextDB.net. Well, it's not exactly what I'd call being slashdotted, but it was a fun ride. Our challenge now, is how to effectively collect feedback from these Alpha users in order to improve the service based on their experience.
Wednesday, November 26, 2008
SF Park
The new SF skatepark is pretty rad. It is growing on me. Here is some footy of parker I took on my point-and-shoot. This park is crowded. Note near collision with fat kid on bike (listen for the yell).
Read-End....
It doesn't seem to bother anyone else in the office that when you place a page on the copy machine, and hit the big green "copy" button, that this message appears:
READ-END? WTF is READ-END? This copy machine can send emails, but they can't just make it copy a page when you hit the green button?
READ-END? WTF is READ-END? This copy machine can send emails, but they can't just make it copy a page when you hit the green button?
Tuesday, November 18, 2008
Steve's Feedback on our UI
lots of issues with the "new user" concept. First time someone signs in have obvious links to getting started guide, creating a sample database, etc.
"import" has too much meaning to db guys. Make it say "create a sample database from template" WITH data.
"must match regex in" in tablename is not a good error message for invalid table names.
Suggested "nextdb for SQL database programmers".
Wants to know how to create the primary key.
Some kind of wizard to drive first time users.
tooltips on tabs with check box for "never show this again"
consider changing the query language parameter names and datatype names to 'human friendly' equivalents.
Steve would like to be able to set the default value.
wants to be able to use camel case in column names
relation name should be "relationship name". It also upper cases it.
show lines connecting related tables as an option.
thought he could not enter a query without having data.
did not like the interface for adding data.
wants a "data editor" tab along the top (loads table names along left hand side)
wants column types to be in parentheses.
Steve wants to be able to store HTML and edit it with a foldout editor. I want permalinks to the HTML and JS.
Great idea
trouble executing query with join
no way to relate rows
wants excel spreadsheet
wants a way to get his data out.
"import" has too much meaning to db guys. Make it say "create a sample database from template" WITH data.
"must match regex in" in tablename is not a good error message for invalid table names.
Suggested "nextdb for SQL database programmers".
Wants to know how to create the primary key.
Some kind of wizard to drive first time users.
tooltips on tabs with check box for "never show this again"
consider changing the query language parameter names and datatype names to 'human friendly' equivalents.
Steve would like to be able to set the default value.
wants to be able to use camel case in column names
relation name should be "relationship name". It also upper cases it.
show lines connecting related tables as an option.
thought he could not enter a query without having data.
did not like the interface for adding data.
wants a "data editor" tab along the top (loads table names along left hand side)
wants column types to be in parentheses.
Steve wants to be able to store HTML and edit it with a foldout editor. I want permalinks to the HTML and JS.
Great idea
trouble executing query with join
no way to relate rows
wants excel spreadsheet
wants a way to get his data out.
"You wouldn't want a mashup for your bank account."
I am watching the opening panel discussion at Mashup Camp. Hart Rossman, CTO at SAIC just said "You wouldn't want a mashup for your bank account."
That really struck a chord because it captures the inadequacy with today's mashup security model. In fact, building a banking system on NextDB is sort of the logical extent of where the NextDB SURID technology can go.
Another interesting statement from one of the panelists:
"SLAs can't be that strong because you can't have really strong remedies."
I agree.
"you're not going to get a new CRM system out of a mashup."
I disagree. Once the database is secure and mashable, the entire application software development lifecycle can be handled by mashups.
The proverbial Map Mashup came up again as a proof point for time savings.
"when the users become the developers, they are doing what they want to do, not what IT wants to do"
The discussion briefly touched on whether or not Amazon Web Services (S3) constitutes a mashup. That's really a point that requires discussion. Once you realize that Amazon Web Services is not Mashable, due to its "web 1.0 security model", you find yourself somewhat dissapointed.
"where we run into a real challenge is in converting the mashup mentality into the largescale systems engineering skillset. You don't see a lot of folks puting a large ampount of skill and discipline into building the backend." paraphrasing what he said later: We're going to see things going in that direction though, and it will drive Mashups into the enterprise.
Sunday, November 16, 2008
Mashup Camp 2008
Tomorrow is Mashup Camp! I'm excited about this year's Mashup Camp because I think there is going to be a lot of activity around trying to figure out how to fuse mashups with Cloud Computing. And that, of course, is exactly where NextDB.net has a sweet spot. NextDB is fundamentally a mashable relational database in the cloud. What does that mean for the programmer? It means NextDB is the ONLY relational database that you can securely mashup, without have to write ANY serverside code.
So check this out. I added this "guest book" to my blog to keep track of folks I meet at Mashup Camp. This guestbook is a Mashup with NextDB.net, which makes it a nice little example of a database mashup between NextDB and blogger.com. Originally, Brent wrote this guestbook as an example for our talk at ajaxworld. I just went to his page, grabbed the html, js, and css and threw them into my blog. The only tricky part was that blogger.com's editor didn't like newline characters in the HTML and CSS. So I just saved them in a file without newlines, and pasted them back to my blog. Go ahead and sign in below, and watch the AJAX goodness! Here is a link to the JavaScript that the guestbook sources. Rather than use the nasty editor at blogger.com (and its associated issues with spaces), I just uploaded the js file to Project Path, which is a nifty collaboration site that allows file hosting.
Mashup Camp Guestbook
So check this out. I added this "guest book" to my blog to keep track of folks I meet at Mashup Camp. This guestbook is a Mashup with NextDB.net, which makes it a nice little example of a database mashup between NextDB and blogger.com. Originally, Brent wrote this guestbook as an example for our talk at ajaxworld. I just went to his page, grabbed the html, js, and css and threw them into my blog. The only tricky part was that blogger.com's editor didn't like newline characters in the HTML and CSS. So I just saved them in a file without newlines, and pasted them back to my blog. Go ahead and sign in below, and watch the AJAX goodness! Here is a link to the JavaScript that the guestbook sources. Rather than use the nasty editor at blogger.com (and its associated issues with spaces), I just uploaded the js file to Project Path, which is a nifty collaboration site that allows file hosting.
Mashup Camp Guestbook
| json view: |
Wednesday, November 12, 2008
NextDB At AjaxWorld 2008 West
Brent and I recently presented about NextDB.net at AjaxWorld 2008 West, in San Jose CA.
I am going to pickup blogging a bit more regularly here. Anyway, the presentation went well. The message we hammered on was that NextDB has a fundamentally new security model that allows a hosted relational database to be exposed as a service. NextDB is the only relational database that is built from the ground up as a AJAX-programmable hosted service. As such, the round-trip security model is native; fundamentally "built in". This clearly differentiates NextDB from other pseudo-database or database services, because you can safely access it directly from JavaScript.
I am going to pickup blogging a bit more regularly here. Anyway, the presentation went well. The message we hammered on was that NextDB has a fundamentally new security model that allows a hosted relational database to be exposed as a service. NextDB is the only relational database that is built from the ground up as a AJAX-programmable hosted service. As such, the round-trip security model is native; fundamentally "built in". This clearly differentiates NextDB from other pseudo-database or database services, because you can safely access it directly from JavaScript.
Monday, November 10, 2008
Tuesday, October 31, 2006
Why detachment sucks
Ahhh, yes, detachment. The idea that you must explicitely "detach" a persistent POJO from its persistence manager, and later explicitly attach it.
Detachment is another opaque feature of so-called Transparent Object Persistence.
Basically detachment is a work around for the fact that the JDO PersistenceManager is not itself serializable. So, when your Page, with your so-called POJO gets serialized, "oh ma god", you can't serialize your pojo. (that would be TOO SIMPLE, TOO POJO). NOOOO, you've got to implement a callback in which you detach your POJO before you serialize it.
The worst part of it is, this totally defeats the point of serializing the page. You see, in Wicket, they are serializing the Page and things on the page to keep the server from bloating with lots of concurrent clients. But wait a sec, if the PersistenceManager has a cache of POJO's, isn't that cache going to be full of those very POJO's your trying to serialize. Right, but you can't serialize the PersistenceManager.......OK, right, so how do you keep things trim. So you , um "detach" from the persistence manager, and serialize your detached POJO. In the mean time, you have allowed your PersistenceManager to get garbage collected. Had to, otherwise there was no point in serializing your POJO's. I mean, you can't keep the whole cache of 'em lying around in memory. (and even if you keep your PersistenceManager around, it's not a real cache. If you've serialized out your POJO, it ain't gonna be in the cache when you deserialize the POJO. NOPE, it the POJO will have been garbage collected from the PersistenceManager, because the PM is a pseudo cache, with WeakReferences, not a hard cache).
So, the architecture of detachment is fundamentally at odds with the notion of a cache. So when you re-attach, whoops, you have no cache. Been garbage collected. Nice knowing you. Bye bye cache.
the solution to this, is to have a persistence manager that IS serializable, which is what Shades has. And shades doesn't have a pseudo-cache that's subject to garbage collection EVEN when you STILL HAVE references to the PersistenceManager. Shades has a REAL CACHE, with HARD REFERENCES to clean values for all the loaded fields of your pojo, for dirty checking when you do updates. So you can serialize the POJO's with OR without also serializing the DatabaseSession (shades' version of a PersistenceManager, is called the DatabaseSession).
If there is a downside to the Shades way of doing business, its that you have to manage the cache explicitely. Shades has a simple method, "clear()" in the DatabaseSession. My take on it, is that it's one hell of a lot easier to clear the database session after the user hits the SAVE button, or just allow the DatabaseSession to be garbage collected, forget about it, and create a new one for your next sequence of CRUD operations, than it is to work with a pseudo cache.
Detachment is another opaque feature of so-called Transparent Object Persistence.
Basically detachment is a work around for the fact that the JDO PersistenceManager is not itself serializable. So, when your Page, with your so-called POJO gets serialized, "oh ma god", you can't serialize your pojo. (that would be TOO SIMPLE, TOO POJO). NOOOO, you've got to implement a callback in which you detach your POJO before you serialize it.
The worst part of it is, this totally defeats the point of serializing the page. You see, in Wicket, they are serializing the Page and things on the page to keep the server from bloating with lots of concurrent clients. But wait a sec, if the PersistenceManager has a cache of POJO's, isn't that cache going to be full of those very POJO's your trying to serialize. Right, but you can't serialize the PersistenceManager.......OK, right, so how do you keep things trim. So you , um "detach" from the persistence manager, and serialize your detached POJO. In the mean time, you have allowed your PersistenceManager to get garbage collected. Had to, otherwise there was no point in serializing your POJO's. I mean, you can't keep the whole cache of 'em lying around in memory. (and even if you keep your PersistenceManager around, it's not a real cache. If you've serialized out your POJO, it ain't gonna be in the cache when you deserialize the POJO. NOPE, it the POJO will have been garbage collected from the PersistenceManager, because the PM is a pseudo cache, with WeakReferences, not a hard cache).
So, the architecture of detachment is fundamentally at odds with the notion of a cache. So when you re-attach, whoops, you have no cache. Been garbage collected. Nice knowing you. Bye bye cache.
the solution to this, is to have a persistence manager that IS serializable, which is what Shades has. And shades doesn't have a pseudo-cache that's subject to garbage collection EVEN when you STILL HAVE references to the PersistenceManager. Shades has a REAL CACHE, with HARD REFERENCES to clean values for all the loaded fields of your pojo, for dirty checking when you do updates. So you can serialize the POJO's with OR without also serializing the DatabaseSession (shades' version of a PersistenceManager, is called the DatabaseSession).
If there is a downside to the Shades way of doing business, its that you have to manage the cache explicitely. Shades has a simple method, "clear()" in the DatabaseSession. My take on it, is that it's one hell of a lot easier to clear the database session after the user hits the SAVE button, or just allow the DatabaseSession to be garbage collected, forget about it, and create a new one for your next sequence of CRUD operations, than it is to work with a pseudo cache.
Saturday, September 16, 2006
Sunday, September 10, 2006
immutable queries
I got some feedback (see previous blog entry comments) that prompted me to implement "immutable queries". These are queries in which the SQL is 100% specified by the user.
String expr = "SELECT FNAME AS FNAME FROM STUDENT";
Query q = QueryFactory.newImmutableQuery(expr);
RecordCandidate cand = q.candidate(dict.getORM("STUDENT"));
cand.setFetchColumns("FNAME");
List students = new ArrayList();
sess.executeQuery(connection,q).populateList(students, Student.class);
System.out.println(students);
Of course it is still parameterizable, like this:
String expr="SELECT FNAME AS FNAME FROM STUDENT WHERE "+
"LNAME LIKE ${lname}";
Query q = QueryFactory.newImmutableQuery(expr);
RecordCandidate cand = q.candidate(dict.getORM("STUDENT"));
cand.setFetchColumns("FNAME");
List students = new ArrayList();
sess.setParameter("lname", "smith");
sess.executeQuery(connection,q).populateList(students, Student.class);
System.out.println(students);
I'll update the Shades download sometime next week.
String expr = "SELECT FNAME AS FNAME FROM STUDENT";
Query q = QueryFactory.newImmutableQuery(expr);
RecordCandidate cand = q.candidate(dict.getORM("STUDENT"));
cand.setFetchColumns("FNAME");
List students = new ArrayList();
sess.executeQuery(connection,q).populateList(students, Student.class);
System.out.println(students);
Of course it is still parameterizable, like this:
String expr="SELECT FNAME AS FNAME FROM STUDENT WHERE "+
"LNAME LIKE ${lname}";
Query q = QueryFactory.newImmutableQuery(expr);
RecordCandidate cand = q.candidate(dict.getORM("STUDENT"));
cand.setFetchColumns("FNAME");
List students = new ArrayList();
sess.setParameter("lname", "smith");
sess.executeQuery(connection,q).populateList(students, Student.class);
System.out.println(students);
I'll update the Shades download sometime next week.
Wednesday, September 06, 2006
Sunday, August 27, 2006
Questions from the Wicket world
Here are some answers to questions that came up on the Wicket user list, about using Shades with the Wicket "library example"
1) how to do concise queries:
Usually the query is defined in the ORMDictionary. In
the example I showed, I defined the query on-the-fly.
Most of the time you would just grab a query from the
dicitonary. So it's this concise:
dbSess.setParameter("author", "william gibson");
dbSess.executeQuery(conn, dict.getQuery("q-by-auth"))
2) how to do arbitrarily complex joins:
There is no limit on how deep or complex the joins can
be with Shades. Here is an example where we setup a
query to find all books published by whatever
publisher publishes the works of William Gibson.
(again bear in mind that usually we would just grab
this query from the dictionary, I am only defining it
inline to be illustrative).
aBook.relatedTo(anAuthor,"book->author");
aBook.where("AUTHOR like 'william gibson');
aBook.relatedTo(aPublisher,"book<->publisher");
anotherBook.relatedTo(aPublisher, "book<->publisher");
query.setFetchGroups(anotherBook);
dbSess.excecuteQuery(conn, q);
3) performance
Ahhh, I am especially stoked on this. Shades uses
batch updates and the other tricks common to ORM's. I
learned most of these tricks doing JDOMax. I think the
biggest reason shades has good performance is because
the codebase is so tight. When it comes to defining
queries on the fly, shades will be better than any
language that compiles a query (for example JDOQL),
because Shades bypasses the parse phase, and the
translation of the AST from JDOQL to SQL. Shades does
query caching like any other ORM.
Saturday, August 26, 2006
I'm a lover not a fighter
I've been asked, rightly so, if Shades is 'better' than Hibernate or JDO implementations or EJB 3. The answer is no. It is not better. But it is different. It's also new, so it doubtless has too many bugs to even dream of playing in the same spaces as the excellent products like Kodo, Cocobase, and others, that implement a variety of standards.
What I want to say, is that I didn't write Shades for anyone else's benefit. I wrote Shades because software, for me, is fun to write. It relaxes my mind after a long day at work, and let's me play with puzzles and art all rolled into one. It's not a competition, especially since I don't get paid!
What I want to say, is that I didn't write Shades for anyone else's benefit. I wrote Shades because software, for me, is fun to write. It relaxes my mind after a long day at work, and let's me play with puzzles and art all rolled into one. It's not a competition, especially since I don't get paid!
Queries
Shades has an interesting way of doing queries. It's based on the idea of a RecordCandidate. I find it much easier to explain this using code than with a paragraph.
The following code returns all the Books, in a List;
ORMDictionary dict = MyORMDict.getInstance();
Query query = QueryFactory.newQuery(dict);
RecordCandidate aBook = query.candidate(dict.getORM("BOOK"), "aBook");
RecordSet rs = dbSession.executeQuery(jdbcConn, query);
List books = new ArrayList();
rs.populateList(books, Book.class);
There are several cool things:
1) The books got put into MY list (not a proxy List created by the data access framework).
2) There is no query language. Shades uses a new form of query-by-example. In the example
above the query is told what to retrieve by requesting a candidate.
3) I passed the jdbcConnection into the query (This means I control transactions using the JDBC transaction model).
Why did we request a RecordCandidate if we did didn't use it for anything. Well OK then, let's use it for something. In the query below we retrieve only the books authored by william gibson.
ORMDictionary dict = MyORMDict.getInstance();
Query query = QueryFactory.newQuery(dict);
RecordCandidate aBook = query.candidate(dict.getORM("BOOK"), "aBook");
RecordCandidate anAuthor = query.candidate(dict.getORM("AUTHOR"), "anAuthor");
anAuthor.where("NAME like 'william Gibson'");
aBook.relatedTo(anAuthor, "book->author");
RecordSet rs = dbSession.executeQuery(jdbcConn, query);
List books = new ArrayList();
rs.populateList(books, Book.class);
Shades let you bust into a query and insert your own SQL. You can see that on the line above that says "anAuthor.where...""
Dangerous? maybe, but fear not. shades encourages you to encapsulate your queries inside the ORMDictionary, and to parameterize them. Once yo've done that, there is no hint of SQL in the code, which subsequently looks like this:
ORMDictionary dict = MyORMDict.getInstance();
dbSession.setParameter("authorName", "william gibson");
RecordSet rs = dbSession.executeQuery(jdbcConn, dict.getQuery("query:book-by-auth"));
List books = new ArrayList();
rs.populateList(books, Book.class);
Here is another very cool thing about Shades queries. They, by default, return ALL the candidates that participate in the relationship graph. So in fact, you can retrieve the Book AND its author from the RecordSet, like this:
Book book = new Book();
Author author = new Author();
while(rs.hasNext()){
rs.populate(book, aBook);
rs.populate(author,anAuthor );
System.out.println(book.title +" was authored by " + author.name);
}
The following code returns all the Books, in a List;
ORMDictionary dict = MyORMDict.getInstance();
Query query = QueryFactory.newQuery(dict);
RecordCandidate aBook = query.candidate(dict.getORM("BOOK"), "aBook");
RecordSet rs = dbSession.executeQuery(jdbcConn, query);
List books = new ArrayList();
rs.populateList(books, Book.class);
There are several cool things:
1) The books got put into MY list (not a proxy List created by the data access framework).
2) There is no query language. Shades uses a new form of query-by-example. In the example
above the query is told what to retrieve by requesting a candidate.
3) I passed the jdbcConnection into the query (This means I control transactions using the JDBC transaction model).
Why did we request a RecordCandidate if we did didn't use it for anything. Well OK then, let's use it for something. In the query below we retrieve only the books authored by william gibson.
ORMDictionary dict = MyORMDict.getInstance();
Query query = QueryFactory.newQuery(dict);
RecordCandidate aBook = query.candidate(dict.getORM("BOOK"), "aBook");
RecordCandidate anAuthor = query.candidate(dict.getORM("AUTHOR"), "anAuthor");
anAuthor.where("NAME like 'william Gibson'");
aBook.relatedTo(anAuthor, "book->author");
RecordSet rs = dbSession.executeQuery(jdbcConn, query);
List books = new ArrayList();
rs.populateList(books, Book.class);
Shades let you bust into a query and insert your own SQL. You can see that on the line above that says "anAuthor.where...""
Dangerous? maybe, but fear not. shades encourages you to encapsulate your queries inside the ORMDictionary, and to parameterize them. Once yo've done that, there is no hint of SQL in the code, which subsequently looks like this:
ORMDictionary dict = MyORMDict.getInstance();
dbSession.setParameter("authorName", "william gibson");
RecordSet rs = dbSession.executeQuery(jdbcConn, dict.getQuery("query:book-by-auth"));
List books = new ArrayList();
rs.populateList(books, Book.class);
Here is another very cool thing about Shades queries. They, by default, return ALL the candidates that participate in the relationship graph. So in fact, you can retrieve the Book AND its author from the RecordSet, like this:
Book book = new Book();
Author author = new Author();
while(rs.hasNext()){
rs.populate(book, aBook);
rs.populate(author,anAuthor );
System.out.println(book.title +" was authored by " + author.name);
}
Monday, August 21, 2006
cool things about shades
One of the most confusing aspects of persistence frameworks is persistant identity. Equals and hashcode typically must be implemented so that Object equalty amounts to a comparison between identity columns of the persistent instance. Since datastore identity is often implemented using autoincrementing primary keys, this leads to a framework dilema: to expose datastore identity in the pojo, or not. Some persistence frameworks force the pojo to hold onto the PK. Other frameworks use bytecode instrumentation to make it transparent.
The situation is arguably simpler for "Object Identity". It is assumed that "Object Identity", as opposed to datastore identity, means that a unique set of column values is mirrored in the pojo's fields. So the pojo naturally posseses as set of fields, that taken together uniquely identify the record in the datastore. Here we again find ourselves bitten by the false equivalence between a pojo in memory, and a record in the database. Remember, equals and hashcode must be overridden to use exactly the identity fields of the Object. This makes it extremely tricky to support "change of identity" during a transaction.
In implementing JDOMax I learned a lot about this issue. Change of identity is actually very common: changing a "User" Object's 'username' field, for example. Because most frameworks for transparent object persistence keep a cache, change of identity can have a devastating and unexpected effect on this cache. Because, equals and hashcode suddenly return different values at different times during the transaction, the cached pojos can appear to "dissappear" from cache.
So one of my biggest issues with ORMapping that claims to be transparent is that it ain't transparent. If I have to override equals and hashCode in a particular way, the mapping is not transparent.
Unlike any other O/R Mapping system that I know of, Shades does not impose any restrictions on how you have implement equals and hashcode. In fact, it doesn't care at all. Shades has a dynamic ORMapping system, as opposed to a static mapping. You can query a record from the database, and load it into a pojo whose equals and hashcode depends only on the 'lastname' field of the pojo. In the same transaction you can load a second record from the table into a pojo whose equals and hashcode depend only on "firstname". You can change the firstname or the lastname field of either pojo, during the same transaction. You can load a third record into yet another pojo whose equals and hashode depend on NONE of the fields of the object. Modifications to the objects are transparently tracked and flushed out to the database on a call to 'update'.
Shades provides a dynamic ORMapping system, in which an ORMapping can be created or chosen, at runtime, to perform I/O from table to pojo. This has an advantage of allowing the "identity" of the pojo to depend on different columns of the database in different situations. Anyone who has ever built an app using straight SQL knows that these "perspectives" on a table surface in the variety of different columns that are retrieved in different situations. Shades is designed to be fluid and adaptable to these common situations. In fact, I thought of the name "Shades" because a good data access framework should recognize the shades of grey that permeate data access programming. Perhaps most of all, shades tries to minimize the number of rules, states, and do's and don'ts.
The situation is arguably simpler for "Object Identity". It is assumed that "Object Identity", as opposed to datastore identity, means that a unique set of column values is mirrored in the pojo's fields. So the pojo naturally posseses as set of fields, that taken together uniquely identify the record in the datastore. Here we again find ourselves bitten by the false equivalence between a pojo in memory, and a record in the database. Remember, equals and hashcode must be overridden to use exactly the identity fields of the Object. This makes it extremely tricky to support "change of identity" during a transaction.
In implementing JDOMax I learned a lot about this issue. Change of identity is actually very common: changing a "User" Object's 'username' field, for example. Because most frameworks for transparent object persistence keep a cache, change of identity can have a devastating and unexpected effect on this cache. Because, equals and hashcode suddenly return different values at different times during the transaction, the cached pojos can appear to "dissappear" from cache.
So one of my biggest issues with ORMapping that claims to be transparent is that it ain't transparent. If I have to override equals and hashCode in a particular way, the mapping is not transparent.
Unlike any other O/R Mapping system that I know of, Shades does not impose any restrictions on how you have implement equals and hashcode. In fact, it doesn't care at all. Shades has a dynamic ORMapping system, as opposed to a static mapping. You can query a record from the database, and load it into a pojo whose equals and hashcode depends only on the 'lastname' field of the pojo. In the same transaction you can load a second record from the table into a pojo whose equals and hashcode depend only on "firstname". You can change the firstname or the lastname field of either pojo, during the same transaction. You can load a third record into yet another pojo whose equals and hashode depend on NONE of the fields of the object. Modifications to the objects are transparently tracked and flushed out to the database on a call to 'update'.
Shades provides a dynamic ORMapping system, in which an ORMapping can be created or chosen, at runtime, to perform I/O from table to pojo. This has an advantage of allowing the "identity" of the pojo to depend on different columns of the database in different situations. Anyone who has ever built an app using straight SQL knows that these "perspectives" on a table surface in the variety of different columns that are retrieved in different situations. Shades is designed to be fluid and adaptable to these common situations. In fact, I thought of the name "Shades" because a good data access framework should recognize the shades of grey that permeate data access programming. Perhaps most of all, shades tries to minimize the number of rules, states, and do's and don'ts.
Friday, August 11, 2006
Lot's of progress
Well, I used to work for a guy who said writing software is a lot like baking waffles. The first waffle sticks to the iron and doesn't come out very well, but it primes the iron with just enough grease to make the second waffle perfect.
So it seems I have just baked my second waffle. I'm putting the finishing touches on Shades, a framework for ORMapping. My first waffle was JDOMax. Why the hell does the world need yet another ORMapping framework? Let me tell you:
Shades has absolutely no XML nor annotation configuration. Rather, there's an interface called ORMapping that operated really a lot like a TableModel. A DefaultORMapping is usually extended by the program, and only a few methods need to be implemented. At first I was shocked by how easy it was but it's now apparent to me that O/R Mapping is better suited to programmetic configuration than document configuration. The reason XML configuration of O/R Mapping is so complex is because...well, because in THIS CASE it's way more complex than just writing a few lines of code.
Somewhere along the line we all just accepted that XML was the way to configure O/R mapping. We bought into a real big advantage of XML - that it's externalized, and can in theory be edited by this mythical "deployer" person. The problem is that this mythical deployer does not exist. In reality he's the programmer and it turns out, now that the wow cool factor has warn off, that the XML is harder to deal with than a few lines of code.
2. Transparent Object Persistence Misidentified as Necessary
This is a big deal. Transparent Object Persistence (TOP)is the idea that you operate on Objects with no concern for the fact that they may be persisted in a relational database. And even beyond that, TOP espouses that it is the Object AND it's references that are in the database. This just turns out to be wrong. THERE IS NO *INHERENT* MAPPING BETWEEN RELATIONSHIPS IN THE DATABASE AND REFERENCES IN THE JVM. But it is *possible* to relate object references to relationships in the datastore. The problem is that the mapping is not one-to-one, and again we are back to a solution that creates as least as many problems as it solves.
Quick example. Teacher Object has a collection of Student Objects. Each Student Object has a field named teacher. Database has an FK in STUDENT_TABLE that points at the PK of teacher table. So in Java land you remove a student from a teacher's collection. This must immediately null the teacher field in the student Object. This is unexpected. And making it work leads to a complex implementation of proxy collections. When the transaction is committed, the FK in the student is set to null. It's simply not at all clear that removing an element from a collection in memory is the right way to remove a relationship from the database. It's just not an intentional way to program. In other words, in order to "forget" about the relational database, we have to learn more than we ever bargained for, and understand subtleties, that even for experts can be mind bending. And the transparency turns out to be nothing near transparent. Consider a STUDENT_TABLE that places a unique constraint on STUDENT_LNAME. If you created a new Student in memory, and added it to a Teacher Object's collection, when would you find out this was invalid? You'd find out when you committed, and a nested datastore exception was thrown. What's so transparent about that. The transparency is a myth. And how would you handle the exception when you created 19 other Students in the transaction? Yes, yes, it is *possible* to handle using an array of nested exceptions, blah blah blah. BUT FOLKS, IT'S NOT EASY. YOU HAVE TO LEARN MORE TO MAKE IT WORK THAN TO SOLVE YOUR ORIGINAL PROBLEM!!!!!!!!!!!!
So it seems I have just baked my second waffle. I'm putting the finishing touches on Shades, a framework for ORMapping. My first waffle was JDOMax. Why the hell does the world need yet another ORMapping framework? Let me tell you:
- Too much XML configuration
- Transparent object persistence misidentifed as necessary
- Relationships between records ARE NOT analogs to references between Objects.
- Inheritance relationships are rarely natural in data models.
- Transitive closure persistence unnecessary
- Too many transactional states increases complexity beyond original problem
Shades has absolutely no XML nor annotation configuration. Rather, there's an interface called ORMapping that operated really a lot like a TableModel. A DefaultORMapping is usually extended by the program, and only a few methods need to be implemented. At first I was shocked by how easy it was but it's now apparent to me that O/R Mapping is better suited to programmetic configuration than document configuration. The reason XML configuration of O/R Mapping is so complex is because...well, because in THIS CASE it's way more complex than just writing a few lines of code.
Somewhere along the line we all just accepted that XML was the way to configure O/R mapping. We bought into a real big advantage of XML - that it's externalized, and can in theory be edited by this mythical "deployer" person. The problem is that this mythical deployer does not exist. In reality he's the programmer and it turns out, now that the wow cool factor has warn off, that the XML is harder to deal with than a few lines of code.
2. Transparent Object Persistence Misidentified as Necessary
This is a big deal. Transparent Object Persistence (TOP)is the idea that you operate on Objects with no concern for the fact that they may be persisted in a relational database. And even beyond that, TOP espouses that it is the Object AND it's references that are in the database. This just turns out to be wrong. THERE IS NO *INHERENT* MAPPING BETWEEN RELATIONSHIPS IN THE DATABASE AND REFERENCES IN THE JVM. But it is *possible* to relate object references to relationships in the datastore. The problem is that the mapping is not one-to-one, and again we are back to a solution that creates as least as many problems as it solves.
Quick example. Teacher Object has a collection of Student Objects. Each Student Object has a field named teacher. Database has an FK in STUDENT_TABLE that points at the PK of teacher table. So in Java land you remove a student from a teacher's collection. This must immediately null the teacher field in the student Object. This is unexpected. And making it work leads to a complex implementation of proxy collections. When the transaction is committed, the FK in the student is set to null. It's simply not at all clear that removing an element from a collection in memory is the right way to remove a relationship from the database. It's just not an intentional way to program. In other words, in order to "forget" about the relational database, we have to learn more than we ever bargained for, and understand subtleties, that even for experts can be mind bending. And the transparency turns out to be nothing near transparent. Consider a STUDENT_TABLE that places a unique constraint on STUDENT_LNAME. If you created a new Student in memory, and added it to a Teacher Object's collection, when would you find out this was invalid? You'd find out when you committed, and a nested datastore exception was thrown. What's so transparent about that. The transparency is a myth. And how would you handle the exception when you created 19 other Students in the transaction? Yes, yes, it is *possible* to handle using an array of nested exceptions, blah blah blah. BUT FOLKS, IT'S NOT EASY. YOU HAVE TO LEARN MORE TO MAKE IT WORK THAN TO SOLVE YOUR ORIGINAL PROBLEM!!!!!!!!!!!!
Wednesday, August 02, 2006
getting our of 'nam for som R&R
A few weeks ago I read an article on TSS called Ted Neward explains ORM as "The Viet Nam of Computer Science.
I have to say, from personal experience, that I agree with him.I got interested in O/R several years ago, and wound up writing an implementaiton of JSR 12, Java Data Object called JDOMax www.jdomax.com. It took about 2.5 years to write the software and pass the Sun Test Compatibility Kit. While I was writing the software, I often got the feeling I was going down into the rabbit hole. And after some time, just passing the TCK became a goal in and of itself. When all was said and done, roughly 50,000 lines of code had been written and debugged, and was being used by a few hundred downloaders. Then I got married and haven't looked back at my hobby since.
My goal now is to produce some software that can be used to easily query , insert, and delete data from the relational database, with a small learning curve, and most of all a small internal codebase. Basically, I have to distill something that is vastly simpler than the fullblown JDO that I had implemented, if I am to have any hope of getting out of 'Nam for some R&R.
Over the last few years, there has been a lot of debate on "query by example" vs. "query language". Having written a JDOQL compiler using JavaCC, I knew that I was going to have dispatch with the query language. Too much complexity, and more code than I can maintain. The second thing I decided to do away with was transparent object persistence. Transitive closure persistence, simply isn't an essential feature, and it often confuses developers.
In short, I had to be willing to do many things differently, even if it meant joining the Viet Cong. I hoped that by freeing my mind of dogma I could produce a system that leveraged the best and most essential features of several approaches.
I began by writing lots of "fake programs". I wanted to be sure that no matter how I implemented the internals, that the software was easy to use, and produced tight code, with an absolute minimum of configuration.
I have to say, from personal experience, that I agree with him.I got interested in O/R several years ago, and wound up writing an implementaiton of JSR 12, Java Data Object called JDOMax www.jdomax.com. It took about 2.5 years to write the software and pass the Sun Test Compatibility Kit. While I was writing the software, I often got the feeling I was going down into the rabbit hole. And after some time, just passing the TCK became a goal in and of itself. When all was said and done, roughly 50,000 lines of code had been written and debugged, and was being used by a few hundred downloaders. Then I got married and haven't looked back at my hobby since.
My goal now is to produce some software that can be used to easily query , insert, and delete data from the relational database, with a small learning curve, and most of all a small internal codebase. Basically, I have to distill something that is vastly simpler than the fullblown JDO that I had implemented, if I am to have any hope of getting out of 'Nam for some R&R.
Over the last few years, there has been a lot of debate on "query by example" vs. "query language". Having written a JDOQL compiler using JavaCC, I knew that I was going to have dispatch with the query language. Too much complexity, and more code than I can maintain. The second thing I decided to do away with was transparent object persistence. Transitive closure persistence, simply isn't an essential feature, and it often confuses developers.
In short, I had to be willing to do many things differently, even if it meant joining the Viet Cong. I hoped that by freeing my mind of dogma I could produce a system that leveraged the best and most essential features of several approaches.
I began by writing lots of "fake programs". I wanted to be sure that no matter how I implemented the internals, that the software was easy to use, and produced tight code, with an absolute minimum of configuration.
Subscribe to:
Posts (Atom)
Posts
