Since nearly the first time I logged in to an Oracle database, I remember finding issues with documentation and the occasional error in documentation. Its understandable...usually this was due to a change or a new feature that was introduced and the documentation just wasn't updated. The real problem was in me...I was judging Oracle's documentation vs perfection...I should have lowered my expectations and appreciated what it was instead of being upset for what it wasn't.
While evaluating and designing the architecture for an HP Vertica database for a client, I gained a new appreciation for Oracle's documentation. I expected to find everything I needed to do a perfect Vertica cluster install across 2 data centers in an active/active configuration for DR. When the documentation failed and I resorted to gGoogle, I mostly found people with the same questions I had and no solutions.
Soooo...I thought I'd make a few notes on what I learned and landed on. I am by no means a Vertica expert, but I've definitely learned a lot about and I've had the opportunity to stand on the shoulders of a few giants recently.
Our requirement is to store 10TB of actual data, we don't know how well it will compress...so we're ignoring compression for capacity planning purposes. How much physical storage do you need for that much data? Vertica licensing is based on data capacity, but that's not the amount of capacity used...its the amount of data ingested. Vertica makes "projections" that (in Oracle terms) I think of as self-created materialized views and aggregates that can later be used for query re-write. Vertica will learn from your queries what it needs to do in the future to improve performance, it'll create projections and these projections use storage. Since there's columnar compression in Vertica by default, these projections are stored efficiently...and they aren't counted toward your licensed total. I've heard stories that companies have had so many (200+) that the performance of importing data was hampered...these physically stored objects are updated as data is loaded. Since projections will take up storage you have to account for that in the early design, but it completely depends on your dataset and access patterns. Estimates based on other companies I've spoken with are between 0% (everything is deduped) and 50% (their ETL is done in Vertica, so less deduplication), so lets say 35%.
Also, you're strongly recommended to use local storage (raid 1+0...mirrored and striped), and the storage is replicated in multiple nodes for protection. They call this concept k-safety. The idea is that you can lose "k" nodes, and the database would still continue to run normally. We would run K+1 (the default).
In order to do a rebalance (needed when you add or remove a node), the documentation suggests you have 40% capacity free.
Also, Vertica expects you to isolate your "catalog" metadata from your actual data, so you need to set up one mirrored raid group with ~150GB for catalog...and OS, etc. They give an example architecture using HP hardware with servers that have 24 slots for drives. 2 of them are used for mirroring the OS/Catalog, leaving 22 for your actual data. Knowing SSD's are the future for storage, the systems we worked on are Cisco UCS C-series with 24 slots filled with 100% SSD's. From the feedback from Vertica, this will help with rebuild times, but not so much with normal processing performance, since so much of Vertica is done in memory. There's a huge price increase in $/GB between 400 and 800GB drives.
So...if you have 6 nodes with 22 slots, each populated with 400GB SSD's, you have 52,800GB. Half that for raid 1+0=26,400. If you have an HA architecture, you'd expect to half that again (3 nodes in datacenter 1, 3 nodes in datacenter 2)...which brings you to 13,200GB. Since you have to keep at least 40% free for a rebalance operation, that brings you down to 7,920GB. We have to account for projections...we said the would be 35% of our dataset...which brings us to 5,148GB. All the data in Vertica is copied to 2 nodes, so half the storage again....2,574GB.
Hmmm...2.5TB of storage is less than our 10TB requirement. I'll show you how we changed the design to double capacity in my next post.
While evaluating and designing the architecture for an HP Vertica database for a client, I gained a new appreciation for Oracle's documentation. I expected to find everything I needed to do a perfect Vertica cluster install across 2 data centers in an active/active configuration for DR. When the documentation failed and I resorted to gGoogle, I mostly found people with the same questions I had and no solutions.
Soooo...I thought I'd make a few notes on what I learned and landed on. I am by no means a Vertica expert, but I've definitely learned a lot about and I've had the opportunity to stand on the shoulders of a few giants recently.
Our requirement is to store 10TB of actual data, we don't know how well it will compress...so we're ignoring compression for capacity planning purposes. How much physical storage do you need for that much data? Vertica licensing is based on data capacity, but that's not the amount of capacity used...its the amount of data ingested. Vertica makes "projections" that (in Oracle terms) I think of as self-created materialized views and aggregates that can later be used for query re-write. Vertica will learn from your queries what it needs to do in the future to improve performance, it'll create projections and these projections use storage. Since there's columnar compression in Vertica by default, these projections are stored efficiently...and they aren't counted toward your licensed total. I've heard stories that companies have had so many (200+) that the performance of importing data was hampered...these physically stored objects are updated as data is loaded. Since projections will take up storage you have to account for that in the early design, but it completely depends on your dataset and access patterns. Estimates based on other companies I've spoken with are between 0% (everything is deduped) and 50% (their ETL is done in Vertica, so less deduplication), so lets say 35%.
Also, you're strongly recommended to use local storage (raid 1+0...mirrored and striped), and the storage is replicated in multiple nodes for protection. They call this concept k-safety. The idea is that you can lose "k" nodes, and the database would still continue to run normally. We would run K+1 (the default).
In order to do a rebalance (needed when you add or remove a node), the documentation suggests you have 40% capacity free.
Also, Vertica expects you to isolate your "catalog" metadata from your actual data, so you need to set up one mirrored raid group with ~150GB for catalog...and OS, etc. They give an example architecture using HP hardware with servers that have 24 slots for drives. 2 of them are used for mirroring the OS/Catalog, leaving 22 for your actual data. Knowing SSD's are the future for storage, the systems we worked on are Cisco UCS C-series with 24 slots filled with 100% SSD's. From the feedback from Vertica, this will help with rebuild times, but not so much with normal processing performance, since so much of Vertica is done in memory. There's a huge price increase in $/GB between 400 and 800GB drives.
So...if you have 6 nodes with 22 slots, each populated with 400GB SSD's, you have 52,800GB. Half that for raid 1+0=26,400. If you have an HA architecture, you'd expect to half that again (3 nodes in datacenter 1, 3 nodes in datacenter 2)...which brings you to 13,200GB. Since you have to keep at least 40% free for a rebalance operation, that brings you down to 7,920GB. We have to account for projections...we said the would be 35% of our dataset...which brings us to 5,148GB. All the data in Vertica is copied to 2 nodes, so half the storage again....2,574GB.
Hmmm...2.5TB of storage is less than our 10TB requirement. I'll show you how we changed the design to double capacity in my next post.
No comments:
Post a Comment