tag:blogger.com,1999:blog-34368959.post5666030896636282784..comments2024-03-14T04:20:55.807-04:00Comments on Customer Experience Matrix: Looking for Differences in MPP Analytical DatabasesDavid Raabhttp://www.blogger.com/profile/03489754392712536104noreply@blogger.comBlogger11125tag:blogger.com,1999:blog-34368959.post-52132625752653189152008-10-21T13:45:00.000-04:002008-10-21T13:45:00.000-04:00Hi Andy,Thanks for your question. I've addressed ...Hi Andy,<BR/><BR/>Thanks for your question. I've addressed that topic in a some detail in a recent DM Review column, which you can find in my archive at <BR/>http://archive.raabassociatesinc.com/2008/07/analytical-database-options/<BR/><BR/>Basically, the MPP systems use a row-oriented structure and rely on partitioning to spread the work of querying across multiple processors and memory stores. The columnar systems use a column-oriented structure that lets them reduce the work of querying by only reading the columns of data needed for a particular query. The one other salient point is that the columnar systems generally have not been deployed at the 100TB scales common to the MPP systems. (Sybase IQ is a major exception--it has proven itself at that level.) Of course, the vendors swear scalability is no problem, but I always want to see some production installations first.David Raabhttps://www.blogger.com/profile/03489754392712536104noreply@blogger.comtag:blogger.com,1999:blog-34368959.post-35186163431283013562008-10-21T13:17:00.000-04:002008-10-21T13:17:00.000-04:00This is a really useful posting. How are these too...This is a really useful posting. <BR/>How are these tools different from vendors like Vertica and ParAccel that boast the use of column based DBMSAndyhttps://www.blogger.com/profile/12473808042419593748noreply@blogger.comtag:blogger.com,1999:blog-34368959.post-20089188178164210642008-10-07T10:57:00.000-04:002008-10-07T10:57:00.000-04:00http://www.asterdata.com/blog/index.php/2008/10/06...http://www.asterdata.com/blog/index.php/2008/10/06/aster-ncluster-30-aligning-product-with-vision/<BR/> <BR/>http://www.asterdata.com/blog/index.php/2008/10/06/growing-your-business-with-frontline-data-warehouses/<BR/> <BR/>http://www.asterdata.com/blog/index.php/2008/10/06/conversation-with-lenin-gali-director-of-bi-for-sharethis/Leah McLeanhttps://www.blogger.com/profile/06982379692798678133noreply@blogger.comtag:blogger.com,1999:blog-34368959.post-31992762832524371382008-08-29T15:49:00.000-04:002008-08-29T15:49:00.000-04:00David:Two other points (apologies for multiple pos...David:<BR/>Two other points (apologies for multiple posts): Kognitio's database is WX2, not WS2...and to clarify, WX2 has been repeatedly and significantly upgraded during its time on the market. Yes, the database is the "most mature" on the market today (to use your words), but it's also in its sixth iteration, with significant investment in time and money being made to get WX2 to the point where it is today. It's so stable, in fact, that several companies are using WX2 in a BC/DR implementation. You may be less likely to do that with databases where the code is not as proven over time.<BR/><BR/>Thanks, again...and have a great weekend.Unknownhttps://www.blogger.com/profile/15751623554905507353noreply@blogger.comtag:blogger.com,1999:blog-34368959.post-6216153209698686522008-08-29T12:33:00.000-04:002008-08-29T12:33:00.000-04:00I did receive a clarification from Kognitio about ...I did receive a clarification from Kognitio about how they WX2 avoid excessive inter-node traffic. The one-sentence answer is that when data is loaded from disk to memory, the system redistributes it so related data is in the same node’s memory.<BR/><BR/>The long answer explains how they do this efficiently. One critical point is that WS2 converts multi-step SQL statements into a single stream of machine language. This extracts only the necessary data and determines how it should be organized, so the volume of data is minimized and the data makes just one trip through the network before reaching the correct node. It also lets the system process all intermediate results in memory, rather than writing th results of each step to disk as a conventional SQL engine might. Another critical point is that the system can store some data permanently in memory, so it does not need to reload it from disk for each new query. This data is selected by the database administrator, not automatically, so there is some user skill involved. The system does provide some help via reports on past results. <BR/><BR/>The Kognitio Web site offers several white papers with some additional information. Start with the one written by Bloor Research, whose work I greatly respect. But even that paper doesn’t make the critical point about data being redistributed when loaded into memory--you read it here first.David Raabhttps://www.blogger.com/profile/03489754392712536104noreply@blogger.comtag:blogger.com,1999:blog-34368959.post-9968552054005088202008-08-29T11:05:00.000-04:002008-08-29T11:05:00.000-04:00David - Thanks for writing about Aster. From our e...David - Thanks for writing about Aster. From our experience, intelligent partitioning is actually far superior to round-robin (also known as 'random') scattering of data across nodes. With random scattering, the system cannot exploit the benefit of optimal organization of data on different nodes and needs to pull data from all nodes. This actually forces systems that do random scattering to keep data in memory, because pulling all data from disks is infeasible. Since memory per node is limited, in-memory systems require a lot of hardware (e.g. 100 nodes for 5 TB) increasing total cost significantly. Aster goes beyond simple hash partitioning and have patent-pending partitioning algorithms for optimally placing data across nodes (POD Partitioning). In addition, Aster nodes optimize query performance by combining compression and aggressive buffering to optimally shuffle data segments at query run-time (POD Transport). With Aster, one could set up a 5 TB system with 3-4 nodes. With 100 nodes, the capacity would be 150-200 TB, giving far superior price/performance results.Asterhttps://www.blogger.com/profile/13507510447126768115noreply@blogger.comtag:blogger.com,1999:blog-34368959.post-9497042478351085892008-08-29T07:31:00.000-04:002008-08-29T07:31:00.000-04:00I shall ask them to clarify...I shall ask them to clarify...David Raabhttps://www.blogger.com/profile/03489754392712536104noreply@blogger.comtag:blogger.com,1999:blog-34368959.post-29358072916091148142008-08-28T23:53:00.000-04:002008-08-28T23:53:00.000-04:00Agreed -- I don't see the connection between loadi...Agreed -- I don't see the connection between loading data into memory and reducing inter-node traffic either :)Daniel Abadihttps://www.blogger.com/profile/16753133043157018521noreply@blogger.comtag:blogger.com,1999:blog-34368959.post-74733846082389664402008-08-28T22:11:00.000-04:002008-08-28T22:11:00.000-04:00Sorry to have been obscure. Most MPP databases pl...Sorry to have been obscure. Most MPP databases place related data on the same node. For example, if most of your queries look at data by customer, you might put all data for the same customer on the same node. This means that most query processing can occur without moving data from one node to another, which improves performance because such movement could easily become a bottleneck. This allocation process is what I referred to as "intelligent partitioning", and, as I said, is a very common strategy. Kognitio doesn't bother with this, apparently because loading data into memory before the query is executed allows it to reduce the volume of inter-node traffic. Quite honestly, I don't see why that is so, but I'll take their word for it.David Raabhttps://www.blogger.com/profile/03489754392712536104noreply@blogger.comtag:blogger.com,1999:blog-34368959.post-14452307693842337652008-08-28T21:22:00.000-04:002008-08-28T21:22:00.000-04:00David,You made a statement I didn't quite understa...David,<BR/><BR/>You made a statement I didn't quite understand:<BR/><BR/>"It can do this [distribute incoming data in a round-robin fashion] without creating excessive inter-node traffic because it loads data into memory during query execution"<BR/><BR/>Can you explain this in a little more detail?Daniel Abadihttps://www.blogger.com/profile/16753133043157018521noreply@blogger.comtag:blogger.com,1999:blog-34368959.post-51681947546789919932008-08-28T16:25:00.000-04:002008-08-28T16:25:00.000-04:00David:One quick point, if I may: Kognitio's offer...David:<BR/>One quick point, if I may: Kognitio's offering is called Data Warehousing as a Service (DaaS). In addition, please understand that Kognitio allows its WX2 database to be deployed across multiple locations at no additional charge, since its licensing is based on the amount of data being analyzed, not seats or sites.<BR/><BR/>Thanks.Unknownhttps://www.blogger.com/profile/15751623554905507353noreply@blogger.com