A New JavaScript Wrapper Library for HTML5’s IndexedDB

What is IndexedDB?

One of the exciting new benefits of HTML5 technologies is the greatly expanded ability to work with local copies of data saved on a user’s machine directly from within a web application.  Since W3C has identified the Web SQL Database as obsolete, IndexedDB is quickly becoming the preferred solution for creating and managing off-line / local database solutions.  As of today however, the IndexedDB API is currently implemented by only two of the major browsers (Chrome and FF).  In addition, none of the mobile browser platforms currently provide support for IndexedDB.  Nonetheless, it appears that IndexedDB will be widely supported within the near future.  The current implementation status of this and other HTML5 technologies can always be checked here.

IndexedDB is not a relational database and could be more accurately referred to as a NoSQL ObjectStore.  The technology is described by W3C as “a database of records holding simple values and hierarchical objects. Each record consists of a key and some value. Moreover, the database maintains indexes over records it stores. An application developer directly uses an API to locate records either by their key or by using an index. A query language can be layered on this API. An indexed database can be implemented using a persistent B-tree data structure.” [1]

Does IndexedDB need a wrapper library?

According to W3C, IndexedDB was actually designed so a more user friendly or extended wrapper library could be “layered” on top of the API to add various features or functionality.  For example, the IndexedDB API provides no form of query language which can make complex data extractions challenging for anyone who is used to working with SQL.  These users may find a wrapper library that includes a query language layer very useful.  However, some developers who are more familiar with using object-relational mapping frameworks such as Microsoft’s Entity Framework or NHibernate may find that they actually prefer working directly with IndexedDB’s openCursor() or get() functions to extract data via an IndexedDB ObjectStore.

Since the IndexedDB API is a client-side database, data synchronization immediately stands out as an important wrapper feature candidate.  In use cases where web applications may offer clients off-line data entry, forms, record updates, or any other type of CRUD operations, data synchronization will be required to ensure that data changes made while disconnected from the internet are immediately updated to the server database once a connection becomes available.  Furthermore, data changes  from other users which may have occurred on the server must also be reconciled by the same client upon re-connection.

Writing cross-browser compatible code can also prove challenging when working with relatively new technologies.  For instance, Chrome and Firefox currently differ in the levels of their IndexedDB API implementations.  This becomes most obvious when writing JavaScript to open or create an IndexedDB database.  While Firefox is using the latest W3C specification’s onupgradeneeded event to determine if a database should be created or upgraded, Chrome is still using the older and now obsolete setVersion method.  Writing code that functions correctly on either browser can quickly become complicated.  For this and other such cases, a wrapper library for IndexedDB seems to make a lot of sense.

The ixDbEz.js wrapper library for IndexedDB

While working on a recent side project of my own, I realized that I was quickly building up a rather large collection of JavaScript for IndexedDB.  I decided to gather up all of this code and place it into one easy to use library called  ixDbEz.js (“IndexedDB Ez”).  The library is currently very simple in scope.  It is intended to accomplish the following two primary goals:

  1. Allow users to easily create, modify, and update IndexedDB databases with only a few lines of code.
  2. Provide data synchronization hooks for each of the ways that IndexedDB allows users to modify data within an IndexedDB database.

Goal#1 is very straightforward.  The primary goal of ixDbEz.js is to allow users to include the JavaScript library within a web application, and then quickly begin to create and use an IndexedDB database  in a very simplistic fashion.  The ixDbEz library exposes the primary functions required to create, upgrade, and modify data via the IndexedDB API.

Here is a quick JavaScript example of how to define and create or open an IndexedDB database using ixDbEz.js:

//The ixDbEx library is used like any other JavaScript library by including a script tag in the head section of your html…
<script src=”scripts/ixDbEz.js” type=”text/javascript“></script>

The following code can be used in a web page’s onload event or elsewhere to open, upgrade, or create an IndexedDB database on a client machine: 

//Define the local IndexedDB database structure in a function
//This code ONLY executes anytime:
// 1. The database does not exist.
// 2. The version number passed to StartDB is greater than the
// current version number.
// 3. The version number passed to StartDB is not a valid integer.

var ixdbDDL = (
function () {
     //Create an IndexedDB ObjectStore (a Table) and Indexes via ixDbEz
ixDbEz.createObjStore(“demoOS”, “keyField”, false);
ixDbEz.createIndex(“demoOS”, “ixField1”, “Field1”, true);
}
);

//Create or Open the local IndexedDB database via ixDbEz when the web application is accessed by the user
//Notice that the ixdbDDL variable function above is passed into the startDB command below.

ixDbConnection = ixDbEz.startDB(“demoDb”, 1, ixdbDDL);

How is this easier than using IndexedDB directly?

Although covering the nuts and bolts of the JavaScript contained within ixDbEz.js is beyond the scope of this article, the library’s code is very well documented for anyone who wants to take a closer look at or improve upon the ixDbEz.js code.  However, I think it is important to discuss a few details related to how the IndexedDB and ixDbEz internals work.  Specifically, gaining a greater understanding of IndexedDB database creation, upgrades, and connections (i.e. opening a database) are critical since these processes are slightly different than one might expect after working with other server-side databases.

The structure of an IndexedDB database can only be modified during what is called a “versionchange” transaction.   This means that the only time ObjectStores (aka tables) or indexes can be created or removed is during the “versionchange” transaction.  Basically, a “versionchange” transaction is automatically created by the IndexedDB API anytime a database is opened via the open method and one of the following two conditions occur:

  1. The requested database does not exist.
  2. The requested database’s version number is greater than the version number of the database on the local client’s machine.

During these situations, the latest version of the IndexedDB API requires that the onupgradeneeded event be fired.  Within this event, developers can create or make changes to ObjectStores (aka tables) or indexes.  This is, in fact, the only time that these structures can be updated.  As previously mentioned, writing cross-browser compatible code to support this process in both Chrome and Firefox can be challenging since Chrome does not yet fire the onupgradeneeded event.

The ixDbEz library uses a JavaScript variable to contain the current structure of any IndexedDB database.  This variable, shown in the example above as “ixdbDDL”,  is a complete JavaScript function which should always contain the commands required to build the entire IndexedDB database.  In addition, the ixDbEz.startDB command accepts three arguments which include the database name, version, and the “ixdbDDL” variable that contains the current IndexedDB database structure function.  When ixDbEz.startDB executes, the process first attempts to open the database.  In cases where the database does not exist or the version number indicates that an upgrade is required, ixDbEz.startDB behaves differently depending on which browser it is executing in.  Chrome triggers the older IndexedDB setVersion command, and Firefox responds by triggering the latest IndexedDB specification’s onupgradeneeded event.  However, no actual browser detection is used to facilitate this process.  So, when Chrome does eventually support the new onupgradeneeded event, ixDbEz will continue to work with no modifications required.  During the ixDbEz.startDB process, the library also defines a connection object and common error / process logging objects.  All other processes within the library take advantage of these common objects.

The current process of IndexedDB database creation, upgrades, and connections (i.e. opening a database) can quickly become complex.  The ixDbEz library achieves Goal #1 by allowing users to easily create, modify, and update IndexedDB databases with only a few lines of code.  Anyone can compare the example shown above with the ixDbEz.startDB source code to quickly understand how using the library saves time and is easier than using the IndexedDB API directly.

What are the data synchronization hooks in ixDbEx.js?

Data stored within any IndexedDB database can be quite volatile when compared to data stored within other server-side database technologies.  For instance, Chrome, Firefox, and most other browsers offer users complete control over the data that is stored on their own device.  When an IndexedDB database is created for the first time on a user’s device, the browser prompts the user requesting permission to store data on the user’s local machine.  Furthermore, browser settings allow the user to remove or “clean up” browser data at anytime with only a few clicks.  This means that developers cannot depend on these databases for critical data without a rock-solid data synchronization plan in place.

In cases where data is not entirely critical or seldom updated, local databases can be great for reducing server traffic and round-trips.  Local databases can also be used in combination with other HTML5 off-line features to offer users access to web applications even when they are not connected to the internet.  This functionality would be great for reference sites and other applications that may require users to capture information in remote locations where internet access is not available.  When data is important or changing frequently, web applications with local databases will require the ability synchronize data changes between various online / off-line users and the server database which would typically represent a master copy of the data.  In these scenarios, the data synchronization hooks within ixDbEz allow developers to quickly plug into the library and trigger their own custom data synchronization processes whenever data changes occur within an IndexedDB database.

All ixDbEz functions include two optional event style variables called onsuccess and onerror.  These variables can optionally be used to pass entire custom JavaScript functions into any ixDbEz wrapper function providing the perfect hook for any custom data synchronization, error handling, or other processes that should execute upon success or failure of an ixDbEz function call within a web application.

The following example demonstrates how a new record can be inserted into the “demoOS” ObjectStore (aka table) that we created in the previous example above:

//Define a new record using a JavaScript object.
// Notice that the primary key value, defined as “keyField” when “demoOS” was created above, is passed with the newRecord object. 

var newRecord = {};
newRecord.keyField = “ThePrimaryKey”;
newRecord.Field1 = “Field1_Value”;
newRecord.Field2 = “Field2_Value”;
newRecord.Field3 = “Field3_Value”;

//Add new record to the local database using the ixDbEz.js add command.
ixDbEz.add(“demoOS”, newRecord);

Creating a new data synchronization hook for this process would be as simple as defining a function and passing it through the onsuccess argument of the ixDbEz.js add function:

//Create a function and add it into a new variable named “myCustomHook”.
var myCustomHook = function () {
alert(“Data synchronization logic could go here!”);
};

//Add new record and data synchronization hook to the local database using the ixDbEz.js add command and onsuccess argument.
// Notice the JavaScript value of undefined is used in this example since the Key argument must is not needed.
ixDbEz.add(“demoOS”, newRecord, undefined, myCustomHook);

Conclusion

It is important to note that the current hooks provided in ixDbEz are just simply standard JavaScript callback functions that execute upon success or failure of the requested ixDbEz function.  They do not perform actual data synchronization between a client IndexedDB database and a server database.  Instead these “hooks” are intended to allow developers the ability to easily utilize any function within ixDbEz.js, and then to further extend that function as needed to achieve the requirements of their specific web application.  A future addition to the ixDbEz.js family will be the ixDbSync.js wrapper library which will actually perform client and server-side data synchronization for all ixDbEz.js function calls.

The ixDbEz wrapper library can save developers time and effort during the development and deployment of client side databases using the IndexedDB API.

Project Resources

References and Useful Links

  1. http://www.w3.org/TR/IndexedDB/
  2. http://www.caniuse.com/
  3. http://mobilehtml5.org/
  4. http://www.codeproject.com/Articles/325135/Getting-Started-with-IndexedDB
  5. http://html5-demos.appspot.com/static/html5storage/index.html#slide30
  6. http://www.jakemdrew.com/
  7. http://www.jakemdrew.com/Blog.aspx
Advertisements

22 thoughts on “A New JavaScript Wrapper Library for HTML5’s IndexedDB

  1. Hai,

    Am srividhya working in HTML5, the demo you have given is working fine in the url (http://ixdbdemo.jakemdrew.com), also works on webserver (for ex:http://localhost:8080/ixDbEz/sample.html). But it is not working from the file:// (for ex: file:///home/otc/Desktop/ixDbEz/sample.html).

    But in WebStorage, i can able to access the file:// and also http://. So please tell me is it possible to access indexedDB using the file://. If possible means let me know how to proceed

    thanks,
    N. srividhya

    • Hello,

      IndexedDB is at different stages of development and behaves differently across the various browsers that support it. I have not tested file:// url’s, but I did a little research it appears that this should work in Chrome 11 or greater. I also know from personal experience that there were a lot of features recently added to Firefox Aurora (the beta version of Firefox). If you have not tested your code on an Aurora browser, I would highly suggest downloading Aurora and running your code there to test http://www.mozilla.org/en-US/firefox/channel/. I was able to find another IndexedDB project on github that lists the same limitations in their project under the “Browser Support and Limitations” section here: https://github.com/superfeedr/indexeddb-backbonejs-adapter .

      I am currently working on version 2 of this project which has a lot of additions. I will do some testing at some point using file:// url’s and get back to you. I am pretty busy in graduate school right now, so it may be a few weeks.

      Thanks so much for using my project!

      Jake Drew

      • Hai jakemdrew ,
        thanks for the quick reply jake!!!
        I have tested this application in the chrome 18.0. It works fine both in the http:// and also file://. Actually we are developing the application for offline mode. So we need, the application should be works in offline mode (i.e the client download the form in their local machine and use it at any time in offline mode). So please tell me whether this is possible in the indexed DB method.

        thanks,
        srividhya

  2. hi Jake, this is a good demo and wrapper for indexedDB .
    However when i tried storing XML string in fields .. its not storing the value ..

    {“keyField”:”createOrder3″,”Field1″:” “,”Field2″:”sdsdfsdsdfdsf”,”Field3″:””}

    can we store XMLs eg.

    • I will have to look into this. If you need a quick fix right now, my suggestion would be to load the XML in an object as a property and save the object in the database.

  3. Hi Jake,
    How to retrieve the values, which is stored in the indexeddb database . Please give me the some samples or else tell me how to retrieve the data for the example which is posted by you.


    srividhya

    • Hello! I actually just released version 2 of ixDbEx on gitHub. Version 2 has a few new features that will help you out with this. Specifically, the ixDbEz.getCursor() function will allow you to get a single record or a range of records using the indexedDb IDBkeyRange object.

      Take a look at the gitHub homepage to get the latest version of the project and see several examples on how to get and manipulate data using getCursor() here: https://github.com/jakemdrew/ixDbEz

      • Hai Jake,
        Thanks for the quick reply!!!!!! I will see in the GitHub and i will come back to you if any doubts.

      • Hi!!!
        I want to place a “Edit” button like “Delete” button. If i click the Edit button the data will be populated to the the input box after editing the value , if i click the save Record button the value has to be updated in the indexeddb database. how to do this?

      • The web demo app http://ixdbdemo.jakemdrew.com/ has edit capability. If you go there and add a new record, you can then click on that record in the “ixDbEz (IndexedDB API) Data:” section and edit any of the field values. When you leave the field, the record updates. All the code for the edit is in the javascript of that page.

        You would use a put command to make the actual change in the database. Put replaces the record value of any existing record:

        ixDbEz.put(‘demoOS’, JSON.parse(this.textContent));

        One other note. You cannot update one field at a time. When you do the put, you must replace the entire record. IndexedDb is a little different than SQL or relational databases in that way.

  4. Hai,
    I have tested the sample application with “edit” and “delete” button. The data is saved in the firefox but not able to update by clicking the edit button and in the chrome browser am not able to save the data. I will send the code to you Pls check where the problem is occured? How will i send the code to you.

    • Both edit and delete functionality work fine on the demo app at the URL I provided you before: http://ixdbdemo.jakemdrew.com/ I have tested this functionality in both Firefox and Chrome with no problems. You are welcome to do the same for yourself. If you have older versions of Firefox or Chrome, you may want to upgrade to the latest version of your browser. I typically do not debug other people’s applications. However, if you have a URL where your website is posted, I will take a quick look at your site and see if anything jumps out at me..

      • Hai Jakemdrew,
        thanks for the quick reply!!!
        Yea for me also it is working fine if I have done the same like you.
        but in your app the data is updated where the json is displayed. But what i have done is the same the data is display like your json and have delete button to delete the row of data and am also having the edit button, if i click the edit button the data will populate in the corresponding field, after editing the data has to be updated in the same row, instead of you directly edited in the row where json value is displayed. So When populating time only the data is not saved properly. if directly am editing in the json value means it is working fine.

      • If you are using well formed JSON and converting it to and from a JavaScript object, then ixDbEz will work correctly for you. You must save the data as a JavaScript object in your objectstore (table). Use JSON.parse() to write the data into ixDbEz.put(), and use JSON.stringify() to read the data returned from a ixDbEz cursor. What you are describing does not sound like an ixDbEz error. It sounds like an error in your code.

        Also, IndexedDB is nothing like a relational database. It is an object store. You do not update one field at a time. If one field in the object is updated, you replace the entire object. If you want to learn more about how IndexedDB works, take a look at the W3C specs for it here.

        If you think this is a problem with the ixDbEz code, post a URL and I will take a quick look.

  5. Hai Jake,
    What you are telling is absolutely correct… i will check the code javascript, if not rectified, let you know

    • IndexedDB is at different stages of development and behaves differently across the various browsers that support it. I have not tested file:// url’s, but I did a little research it appears that this should work in Chrome 11 or greater. I also know from personal experience that there were a lot of features recently added to Firefox Aurora (the beta version of Firefox). If you have not tested your code on an Aurora browser, I would highly suggest downloading Aurora and running your code there to test http://www.mozilla.org/en-US/firefox/channel/. I was able to find another IndexedDB project on github that lists the same limitations in their project under the “Browser Support and Limitations” section here: https://github.com/superfeedr/indexeddb-backbonejs-adapter .

  6. hi,
    what about the storage size ? How much data we can store in IndexedDB? is there any way to increase what default web storage for each browser

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s