We stored 10+GB of images from a webcam in Lynne’s office in a stock BSS, but the the extremely large size of the data files makes it very difficult to use in a production environment given the amount of time it takes to check/reindex one after improper shutdown. It is very cool, though.
We like to know: using networked cameras as data sources, is it practical to store/retrieve images in a CygNet BSS? If so, can this be done using CStudio-based tools and applications?
We’re going to do a couple projects that look into what might have to be changed in the BSS architecture, and the practicality of using CStudio to display images associated with alarms, e.g. “There is a fire alarm on Well #124. Get all the images from all the cameras that can see it, in an N-minute bracket around the alarm time, and display them.” Create a functional demonstration application.
During/after the process, we’ll write up the findings in ways useful to product management, development, and possibly marketing.
Last year, for no good reason, I decided to take one of the network cameras lying around my office and stick the output into CygNet, on the theory that more data are better than fewer. The camera in question (http://yardcam.progarts.com) is a consumer-grade, low resolution (640×480), networked unit with the ability to capture still images (JPEG) and video (MP4). It can use email (SMTP) or FTP to send captured files to a server, in this case the research server.
Since last October, every time sometime outside my office moved, the camera sent about 4 images to the server, resulting in about 12GB (yes, GigaBytes) of data in a single BSS (RESEARCH.BSS ). So far as anyone is aware, this is the largest BSS in existence. Performance both for add/get activities is anecdotally acceptable, but it can take 8+ hours to dbcheck and reindex it.
Jeff has heard customers discussing the possible value of CygNet-based-and-accessible video/images, and has decided to create some formal projects that could be used in product management, development, and marketing.
What we’ve learned so far
- When the size of a BSS exceeds about 1GB, it takes an impractically long time to fix errors caused by improper shutdown.
- It’s next-to-impossible for a person to navigate a BSS with more than a few hundred items in it because there is no structured navigation beyond a 1-deep index (i.e. the ‘Application’ tag).
- BSS Explorer has a number of bugs that may or may not be related to large amounts of data. There was correlation, but no investigation of causality.
- Derek discovered some write-caching issues (NOTE: check this with D) in the bssc utility and fixed them.
- People love looking at pictures. I should have remembered this from my days building demos on top of OODBs, but people are fascinated by the ability to look back in time through the use of timeline-based media. It’s hard to imagine a better way to show off some of the capabilities of the product.
We would like to know if it is practical to store, maintain, and retrieve (very quickly) “large” amounts of multimedia data in a BSS. We hope to learn what changes (if any) would be required in the BSS architecture as well as some straightforward ways to access the media from CStudio applications.
Presumably, once some of this is demonstrably known, and assuming there is some customer interest, we would be in a position to determine the difficulty of bring a production-ready NewBSS product to market. At the very least, it would also make a very shiny demonstration platform for
Things we’d like to learn
How big is “big”?
How practical is it to use technologies like HTML 5/SilverLight/Flash in CStudio?
What are the performance limitations of a BSS?
Size of datastore
Number of records
Size of individual records
Is it better to store the media in a file system with just a reference of some sort in the BSS itself?
- Alternative storage capable of dealing with production volumes
- BSS Message set evaluation for possible extension
- Memory-based storage (e.g. Solid State Disks)
- Silverlight embedding
- Document management architecture or system
- Flash-based development for presentation
- Version 2 project stuff
- Image Source (camera) control/acquisition
- Ad hoc image acquisition (cell/vehicle-mounted cameras)
- Other analog (post-processed) data streams – Deluge data
Check the blog posts following this one for two documents describing the specifics of two projects in this topic.
Project: Investigate practical limits of BSS-based multimedia storage