Effect of MBPS vs Latency

I do *a lot* of performance testing on high performance storage arrays for multiple vendors.  Usually if I'm involved, the client is expecting to put mission critical databases on their new expensive storage, and they need to know it performs well enough to meet their needs. 


So...parsing that out..."meet their needs"...means different things to different people.  Most businesses are cyclical, so the performance they need today is likely not the performance they need at their peek.  For example...Amazon does much more business the day after Thanksgiving than they do in a random day in May.  If you gather the usage stats being used in May and size it appropriately, you're going to get a call in a few months when performance is exposed. 


Before I talk about latency, let me just say AWR does a great job of keeping performance data, if you have your data kept long enough...preferably at least 2 business cycles so you can do comparisons and projections.


This statement will keep AWR data for 3 years, capturing it at an aggressive 15 minute interval:


execute dbms_workload_repository.modify_snapshot_settings (interval => 15,retention => 1576800);


...at that point, see my other post re:gathering IOPS and Throughput requirements.


Anyway, I often have discussions with people who don't understand the effect of latency on OLTP databases.  This is a overly-focused serial example, but its enough to make the point.  Think about this...let's say you have a normal 8K block Oracle database using Netapp or EMC NFS on an active-active 10Gb network.  Let's say your amazing all-SSD storage array is capable of flowing 10Gb between multiple paths.   So...the time to move 8K over a 10Gb pipe is...


Throughput...
10Gb/s=10485760Kb/s
(8Kb/s)/(10485760Kb/s)=0.000000762939453125 seconds to copy 8Kb over the 10Gb pipe.


Latency...by the time it passes through your FC network, gets processed by the storage array, gets retrieved from disk, and makes it back to your server can easily exceed a few ms...but for fun let's say we're getting an 8ms response time.  That's .008 seconds.


.008/0.000000762939453125=10,485.76...


...so the effect of latency on your block is 10,485X greater than the effect of throughput.  If your throughput magically got faster but your latency stayed the same...performance wouldn't really improve very much.  If you went from 8ms to 5ms, on the other hand, this would have a huge effect on your database performance.


There's a lot that can affect latency...usually the features in use on the storage array play a big part.  CPU utilization on the storage array can become too high.  This is ultra complicated for the storage array guys to diagnose.  On EMC VMAX3's for example, CPU is allocated to "pools" for different features.  So...even though you may not use eNAS, by default, you allocate a lot of your VMAX CPU cores to it.  When your FC traffic pegs its cores and latency tanks...the administrator may think to look at the CPU utilization and not see an issue...there's free CPU available...just not in the pool used for the FC front end cores, so it creates a bottleneck.  Awesome performance improvements are possible by working closely with your storage vendor to reduce latency during testing...about 6 months ago I worked with a team that achieved improvements by over 50% from the standard VMAX3 as delivered by adjusting those allocations.


All this to say...Latency is very important for common OLTP databases.  Don't ignore throughput, but don't focus on it.

The last secret tweak for improving datapump performance (Part 2)

In my previous post, I mentioned some of the common datapump performance tweaks we see.  In this one, I want to talk about one that's never mentioned, and it might be the best of all.


5.  ADOP - The last tweak...I don't think I've seen any blog posts or Oracle documentation about this as it applies to datapump...is ADOP-Auto degree of parallelization.  This can be a *huge*...by 4X or more...improvement on imports, which is typically where most of your datapump time is spent.  This has been around since 11.2, but until recently (12.1,12.2) its been a little difficult to control how parallel things would run at.  To enable it, you simply set:


parallel_degree_policy=auto
parallel_min_time_threshold=60


(which means, if the optimizer thinks this statement will take more than 60 seconds, it will consider parallelizing it)


This is nice because quick queries will run without the overhead of parallelism, and long running queries might find value in parallelism.


Today in 12c, we have the parallel_degree_level parameter, but in 11.2 we could tweak the parallelism by adjusting max_pmbps in sys.resource_io_calibrate$.  From my testing, 200=~parallel 2 or 3.  A SMALLER value increases the amount of parallelism (50=~parallel 20.)  Effectively this gives us the same effect as the new 12c feature...which is to make Oracle make rational decisions on how parallel the auto parallelism should be. 


Datapump is a logical copy (as opposed to a physical backup/restore) so it can't copy the original indexes, it has to rebuild them.  If you have a typical import with 10,000 tables and indexes, the last 100 are big, the last 10 are huge.  Datapump's parallelism will rip through the small objects very quickly, with one process per object.  When the time comes to rebuild the indexes on the huge tables, datapump will again assign one process per index rebuild.  When the create index statement is analyzed by the optimizer, it will create it in parallel based on the algorithm derived from max_pmbps  (even though the create index statement may be parallel 2 or noparallel).  This will save many hours on a large datapump import.  Its crazy to see a serialized create index statement with 50 busy parallel slaves...but that's what can happen.


When its done, the indexes all have the original parallel spec they started with.  Nothing is any different than it would have been if you hadn't used ADOP (other than it was done much, much faster.)


One word of caution:  You have to watch your system resources and parallel limits.  If you have datapump running at parallel 50, that means potentially you'll be rebuilding 50 indexes simultaneously.  If they're each "large" and the optimizer thinks they'll take over [parallel_min_time_threshold] seconds to rebuild, each of them could be built parallel and you could have hundreds (datapump 50 * adop 50) of parallel processes.  This is a wonderful thing if your system can handle it.  Depending on your parallel limit parameters, ADOP may queue the statements until you have enough parallel processes...to prevent the system from overloading.  IMHO, that's also a wonderful thing, but it may be unexpected.  The truly unfortunate situation is when you have them too high.  You'll use up your server resources and inefficiently use CPU...and may even swap if you run low on RAM.  So test!


...but that's what dry runs are for.  I hope this last tweak helps you.  I've seen it make miraculous differences meeting otherwise impossible SLA's for datapump export/imports.  Two other posts you may want to read are:
1. Gwen Shapira has the best post on it, IMHO)
2. Kerry Osborne has a nice post on the 12c changes.




Previous post -> The last secret tweak for improving datapump performance (Part 1)
This post -> The last secret tweak for improving datapump performance (Part 2)

The last secret tweak for improving datapump performance (Part 1)

There are 10,000 blog posts and oracle docs on the internets (thank you, Mr Bush) for improving datapump performance.  This is one of the features used very frequently in Oracle shops around the world.  And DBA's are under huge stress to meet impossible downtime SLA's for their export/import. I think they all miss the best tweak (ADOP)...but they basically summarize to one simple fact...outside of parallelism, if you have a well-tuned database on fast storage, there's not a lot more you can do to improve performance more than a few percentage.  The obvious improvements are:




1. Parallelize! To quote Vizzini, "I do not think it means what you think it means."




This will create multiple processes and each will take one object and work with it, each one serialized (usually).  This is great, and if all your objects are equally sized, this is perfect...but that's not typically reality.  Usually you have a few objects that are much bigger than the rest and each of them by default will only get a single process.  This limit really hurts during imports, when you need to rebuild large indexes...more on this later.

Check that its working as expected by hitting control-c and typing in status.  Ideally, you should see all parallel slaves working.  Check not just that they exist, but that they're working (verify you used %U in your dump filename if they aren't.)  ie: DUMPFILE=exaexp%U.dmp PARALLEL=100




2. If you're importing into a new database, you have the flexibility to make some temporary changes to tweak things.  Verify Disk Asynchronous IO is set to true (DISK_ASYNCH_IO=true) and disable all the block verification (DB_BLOCK_CHECK=FALSE, DB_BLOCK_CHECKSUM=FALSE)  These aren't "game changers" but they'll give you 10-20% improvements, depending on how you had them set previously.




3. Memory Settings - Datapump parallelization uses some of the streams API's, and so the streams pool is used.  Make sure you have enough memory for the shared_pool, streams_pool and the db_cache_size parameters.  Consult your gv$streams_pool_advice, gv$shared_pool_advice, gv$db_cache_advice and gv$sga_target_advice  views.  I like to tune it so as the delta in ESTD_DB_TIME_FACTOR from one row to the row below it approaches zero, the corresponding size of the pool is close to 1.  (Any more than that is a waste, any less than that is lost performance.) 


Sometimes you'll see a huge dropoff and its more clear than this example...but you get the idea.  If you're importing into a new database, you'll need to run the import dry run and then check these views to make sure this is tuned well.

SGA_SIZE	SGA_SIZE_FACTOR	ESTD_DB_TIME	ESTD_DB_TIME_FACTOR	Time Factor Diff
158720	0.5	12162648	1.2045
178560	0.5625	11421479	1.1311	0.0734
198400	0.625	10937800	1.0832	0.0479
218240	0.6875	10607608	1.0505	0.0327
238080	0.75	10604578	1.0502	0.0003
257920	0.8125	10375361	1.0275	0.0227
277760	0.875	10222886	1.0124	0.0151
297600	0.9375	10101716	1.0004	0.012
317440	1	10097674	1	0.0004
337280	1.0625	10017905	0.9921	0.0079
357120	1.125	9955300	0.9859	0.0062
376960	1.1875	9908850	0.9813	0.0046
396800	1.25	9905821	0.981	0.0003
416640	1.3125	9870479	0.9775	0.0035
436480	1.375	9844225	0.9749	0.0026
456320	1.4375	9826049	0.9731	0.0018
476160	1.5	9821001	0.9726	0.0005

4. Something often missed when importing into a new database, size your redo logs to be relatively huge.  The redo logs will work like a cache and cycle around.  Eventually if you're adding data extremely fast, the last log will fill and can't switch until the next log is cleared.  "Huge" is relative to the size, speed of your database and hardware.  While you're running your import, select * from v$log and make sure you see at least one "inactive" logs in front of the current log.


The best, virtually unused tweak is in the next post....







This post:  The last secret tweak for improving datapump performance (Part 1)
Next post: The last secret tweak for improving datapump performance (Part 2)

Vertica Architecture (Part 2)

This is continued from my previous post.

In review, I spoke with some very smart and experienced Vertica consultants regarding the DR architecture, and found the most obvious solutions all had huge drawbacks.

1. Dual-Load: Double your license costs(?), there's also the potential to have the two clusters out of synch, which means you need to put logic in your loads to handle the possibility that a load succeeds in datacenter 1 and fails in datacenter 2.
2. Periodic Incremental Backups:Need identical standby system (aka, half the capacity and performance of your hardware because the standby is typically idle)
3. Replication solutions provided by storage vendors: The recommended design uses local storage, not storage arrays, so this is difficult to implement, in addition to the expense and the potential of replicating media failures.

At first, here's what we did instead:

Initially (aka don't do this), we set up 2 failgroups, 3 nodes in datacenter 1 and 3 in datacenter 2. Failgroups in Vertica are intended for use where you could have known dependencies that are transparent to Vertica...for example, a server rack.  Both failgroups are in the same cluster, and so data that's entered into nodes 1,2 or 3 get replicated automatically by Vertica to the other failgroup's nodes 4, 5 and 6.

We were trying to protect ourselves from the possibility of a complete datacenter failure, or a WAN failure.  The WAN is a 10Gb, low latency dark fiber link with a ring design, so highly available.  Although the network is HA, the occasional "blip" happens, where a very brief outage causes a disconnection.  Clusters don't like disconnections.

We were very proud of this design until we tested it...it completely failed.  It made sense...although logically we had all the data we needed in a single failgroup, if we simulated a network outage we'd see all 6 nodes go down.  This is actually an intentional outcome, and a good thing.  If you've worked with clusters before...you know its much better to have the cluster go down than to have it stay up in a split brain scenario and corrupt all your data.  If the cluster stays up and becomes out of synch, you have to fix whatever the initial issue was, and you compound the problem with the need to restore all your data.

So...intentionally, if you have half your nodes go down, Vertica causes the whole cluster to go down, even if you have all the data you need to stay up in the surviving nodes.  Oracle RAC uses a disk voting mechanism to decide which part of the cluster stays up, but there's no such mechanism in Vertica.

We were back to the 3 original options...all with their drawbacks.  While pouring over the documentation looking for an out-of-the-box solution, I noticed Vertica 8 introduced a new type of node called an Execute node.  Again...very little documentation on this, but I was told this was a more official way to deal with huge ingest problems like they had at Facebook (35TB/hr).  Instead of using Ephimeral nodes (nodes in transition between being up and being down) like they did, you could create execute nodes that only store the catalog...they store no other data, but only exist for the purpose of ingestion.

Upon testing, we also found Execute nodes "count" as a node in the cluster...so instead of having 6 nodes-3 nodes in DC1 and 3 in DC2, we'd add a 7th node in a cloud (we chose Oracle's cloud.)  Its a great use case for a cloud server because it has almost no outgoing data, almost no CPU utilization (only enough to maintain the catalog) and the only IO is for the catalog.  So now, if DC1 went down, we had a quorum of 4 surviving nodes (4,5,6,7)...if DC2 went down, we still have 4 surviving nodes (1,2,3,7).  If all the nodes stayed up, but the WAN between DC1 and DC2 stopped functioning, Vertica would kill one of the failgroups and continue to function...so no risk of a split brain.

We're continuing to test, but at this point, its performed perfectly.  This has effectively doubled our performance and capacity because we have a 6 node cluster instead of two 3 node clusters.  Its all real time, and there's no complex dual load logic to program in our application.

Next, I'll talk about Vertica backups.

Vertica Capacity Architecture

Since nearly the first time I logged in to an Oracle database, I remember finding issues with documentation and the occasional error in documentation.  Its understandable...usually this was due to a change or a new feature that was introduced and the documentation just wasn't updated.  The real problem was in me...I was judging Oracle's documentation vs perfection...I should have lowered my expectations and appreciated what it was instead of being upset for what it wasn't.

While evaluating and designing the architecture for an HP Vertica database for a client, I gained a new appreciation for Oracle's documentation.  I expected to find everything I needed to do a perfect Vertica cluster install across 2 data centers in an active/active configuration for DR.  When the documentation failed and I resorted to gGoogle, I mostly found people with the same questions I had and no solutions.

Soooo...I thought I'd make a few notes on what I learned and landed on. I am by no means a Vertica expert, but I've definitely learned a lot about and I've had the opportunity to stand on the shoulders of a few giants recently.

Our requirement is to store 10TB of actual data, we don't know how well it will compress...so we're ignoring compression for capacity planning purposes.  How much physical storage do you need for that much data?  Vertica licensing is based on data capacity, but that's not the amount of capacity used...its the amount of data ingested.  Vertica makes "projections" that (in Oracle terms) I think of as self-created materialized views and aggregates that can later be used for query re-write.  Vertica will learn from your queries what it needs to do in the future to improve performance, it'll create projections and these projections use storage.  Since there's columnar compression in Vertica by default, these projections are stored efficiently...and they aren't counted toward your licensed total.  I've heard stories that companies have had so many (200+) that the performance of importing data was hampered...these physically stored objects are updated as data is loaded.  Since projections will take up storage you have to account for that in the early design, but it completely depends on your dataset and access patterns.  Estimates based on other companies I've spoken with are between 0% (everything is deduped) and 50% (their ETL is done in Vertica, so less deduplication), so lets say 35%.

Also, you're strongly recommended to use local storage (raid 1+0...mirrored and striped), and the storage is replicated in multiple nodes for protection.  They call this concept k-safety. The idea is that you can lose "k" nodes, and the database would still continue to run normally.  We would run K+1 (the default).

In order to do a rebalance (needed when you add or remove a node), the documentation suggests you have 40% capacity free.

Also, Vertica expects you to isolate your "catalog" metadata from your actual data, so you need to set up one mirrored raid group with ~150GB for catalog...and OS, etc.  They give an example architecture using HP hardware with servers that have 24 slots for drives.  2 of them are used for mirroring the OS/Catalog, leaving 22 for your actual data.  Knowing SSD's are the future for storage, the systems we worked on are Cisco UCS C-series with 24 slots filled with 100% SSD's.  From the feedback from Vertica, this will help with rebuild times, but not so much with normal processing performance, since so much of Vertica is done in memory.  There's a huge price increase in $/GB between 400 and 800GB drives.

So...if you have 6 nodes with 22 slots, each populated with 400GB SSD's, you have 52,800GB. Half that for raid 1+0=26,400.  If you have an HA architecture, you'd expect to half that again (3 nodes in datacenter 1, 3 nodes in datacenter 2)...which brings you to 13,200GB.  Since you have to keep at least 40% free for a rebalance operation, that brings you down to 7,920GB.  We have to account for projections...we said the would be 35% of our dataset...which brings us to 5,148GB.  All the data in Vertica is copied to 2 nodes, so half the storage again....2,574GB.

Hmmm...2.5TB of storage is less than our 10TB requirement.  I'll show you how we changed the design to double capacity in my next post.

UDEV updated

In my previous post, I wrote about automating the creation of your Oracle RAC cluster.  One of the more complicated parts to that is using shared VMDK's and configuring UDEV to create aliases so all your RAC nodes have the same name for the same ASM disk, no matter what order udev finds the devices.

Assuming you used the VM create script in the previous post, the prerequisites of:

1.  "disk.EnableUUID"="true"; 
2. Data on SCSI adapter 1, Redo on SCSI adapter 2, and FRA on SCSI adapter 3
3. Node equivalence

...should already be met, and this should just work.  This needs to be executed as root, and should be tested in a sandbox environment until you're confident.  I've used it for years, but like everything on the internet, no warranties or promises implied or otherwise.  It may be found to connect to a secret government computer and play Global Thermonuclear War.

#! /bin/sh
###################################
# Name: udev_rules.sh
# Date: 5/9/2012
# Purpose:  This script will create all the udev rules necessary to support
#        Oracle ASM for RH 5 or RH6.  It will name the aliased devices
#        appropriately for the different failgroups, based on the contoller
#        they're assigned to.
# Revisions:
#   5/8/2012  - JAB: Created
#   5/10/2012 - JAB: Will now modify the existing rules to allow the addition of a
#          single new disk.
#   1/8/2013  - JAB: assorted RH6 related issues corrected.
###################################
data_disk=0
redo_disk=0
arch_disk=0
release_test=`lsb_release -r | awk 'BEGIN {FS=" "}{print $2}' | awk 'BEGIN {FS="."}{print $1}'`
echo "Detected RH release ${release_test}"

if [ -f "/etc/udev/rules.d/99-oracle-asmdevices.rules" ]; then
  echo -e "Detected a pre-existing asm rules file.  Analyzing...\c"
  for y in {1..50}
  do
    found_data_disk=`cat /etc/udev/rules.d/99-oracle-asmdevices.rules|grep asm-data-disk${y}`
    found_redo_disk=`cat /etc/udev/rules.d/99-oracle-asmdevices.rules|grep asm-redo-disk${y}`
    found_arch_disk=`cat /etc/udev/rules.d/99-oracle-asmdevices.rules|grep asm-arch-disk${y}`
    if [ -n "${found_data_disk}" ]; then
      let "data_disk++"
    fi
    if [ -n "${found_redo_disk}" ]; then
      let "redo_disk++"
    fi
    if [ -n "${found_arch_disk}" ]; then
      let "arch_disk++"
    fi
    echo -e ".\c"
  done
  echo "complete."
  echo "Existing rules file contains:"
  echo " ASM Data Disks: ${data_disk}"
  echo " ASM Redo Disks: ${redo_disk}"
  echo " ASM Arch Disks: ${arch_disk}"
  new_file="false"
else
  echo "Detected no pre-existing asm udev rules file.  Building..."
  new_file="true"
fi

for x in {a..z}
do
  if [ -n "`ls /dev/sd* | grep sd${x}1 `" ] ; then
    asm_test1=`file -s /dev/sd${x}1 |grep "/dev/sd${x}1: data" `
    asm_test2=`file -s /dev/sd${x}1 |grep "Oracle ASM" `
    #echo "Testing for sd${x}1 complete."
    if [[ -n "${asm_test1}" || -n "${asm_test2}" ]] ; then
      # ie: scsi_device:1:0:1:0
      if [ "${release_test}" = "5" ]; then
        controller=`ls /sys/block/sd${x}/device|grep scsi_device | awk 'BEGIN {FS=":"}{print $2}'`
        result=`/sbin/scsi_id -g -u -s /block/sd${x}`
      elif [ "${release_test}" = "6" ]; then
        controller=`ls /sys/block/sd${x}/device/scsi_device | awk 'BEGIN {FS=":"}{print $1}'`
        result=`/sbin/scsi_id -g -u -d /dev/sd${x}`
      fi
      if [ "${controller}" = "3" ]; then
        if [ -f "/etc/udev/rules.d/99-oracle-asmdevices.rules" ]; then
          found_uuid=`cat /etc/udev/rules.d/99-oracle-asmdevices.rules|grep $result`
        else
          found_uuid=
        fi
        if [[ -z "${found_uuid}" || "${new_file}" = "true" ]]; then
          echo "Detected a new data disk.  Adding rule to /etc/udev/rules.d/99-oracle-asmdevices.rules"
          let "data_disk++"
          if [ "${release_test}" = "5" ]; then
            echo "KERNEL==\"sd?1\", BUS==\"scsi\", PROGRAM==\"/sbin/scsi_id -g -u -s /block/\$parent\", RESULT==\"${result}\", NAME=\"asm-data-disk${data_disk}\", OWNER=\"oracle\", GROUP=\"dba\", MODE=\"0660\"" >> /etc/udev/rules.d/99-oracle-asmdevices.rules
          elif [ "${release_test}" = "6" ]; then
            echo "KERNEL==\"sd?1\", BUS==\"scsi\", PROGRAM==\"/sbin/scsi_id -g -u -d /dev/\$parent\", RESULT==\"${result}\", NAME=\"asm-data-disk${data_disk}\", OWNER=\"oracle\", GROUP=\"dba\", MODE=\"0660\"" >> /etc/udev/rules.d/99-oracle-asmdevices.rules
          fi
        fi
      elif [ "${controller}" = "4" ]; then
        if [ -f "/etc/udev/rules.d/99-oracle-asmdevices.rules" ]; then
          found_uuid=`cat /etc/udev/rules.d/99-oracle-asmdevices.rules|grep $result`
        else
          found_uuid=
        fi
        if [[ -z "${found_uuid}" || "${new_file}" = "true" ]]; then
          echo "Detected a new Redo disk.  Adding rule to /etc/udev/rules.d/99-oracle-asmdevices.rules"
          let "redo_disk++"
          if [ "${release_test}" = "5" ]; then
            echo "KERNEL==\"sd?1\", BUS==\"scsi\", PROGRAM==\"/sbin/scsi_id -g -u -s /block/\$parent\", RESULT==\"${result}\", NAME=\"asm-redo-disk${redo_disk}\", OWNER=\"oracle\", GROUP=\"dba\", MODE=\"0660\"" >> /etc/udev/rules.d/99-oracle-asmdevices.rules
          elif [ "${release_test}" = "6" ]; then
            echo "KERNEL==\"sd?1\", BUS==\"scsi\", PROGRAM==\"/sbin/scsi_id -g -u -d /dev/\$parent\", RESULT==\"${result}\", NAME=\"asm-redo-disk${redo_disk}\", OWNER=\"oracle\", GROUP=\"dba\", MODE=\"0660\"" >> /etc/udev/rules.d/99-oracle-asmdevices.rules
          fi
        fi
      elif [ "${controller}" = "5" ]; then
        if [ -f "/etc/udev/rules.d/99-oracle-asmdevices.rules" ]; then
          found_uuid=`cat /etc/udev/rules.d/99-oracle-asmdevices.rules|grep $result`
        else
          found_uuid=
        fi
        if [[ -z "${found_uuid}" || "${new_file}" = "true" ]]; then
          echo "Detected a new Arch disk.  Adding rule to /etc/udev/rules.d/99-oracle-asmdevices.rules"
          let "arch_disk++"
          if [ "${release_test}" = "5" ]; then
            echo "KERNEL==\"sd?1\", BUS==\"scsi\", PROGRAM==\"/sbin/scsi_id -g -u -s /block/\$parent\", RESULT==\"${result}\", NAME=\"asm-arch-disk${arch_disk}\", OWNER=\"oracle\", GROUP=\"dba\", MODE=\"0660\"" >> /etc/udev/rules.d/99-oracle-asmdevices.rules
          elif [ "${release_test}" = "6" ]; then
            echo "KERNEL==\"sd?1\", BUS==\"scsi\", PROGRAM==\"/sbin/scsi_id -g -u -d /dev/\$parent\", RESULT==\"${result}\", NAME=\"asm-arch-disk${arch_disk}\", OWNER=\"oracle\", GROUP=\"dba\", MODE=\"0660\"" >> /etc/udev/rules.d/99-oracle-asmdevices.rules
          fi
        fi
      fi
    else
      echo "/dev/sd${x} is not an asm disk."
    fi
  fi
done
echo "Complete."

echo "To see the ASM UDEV rules: cat /etc/udev/rules.d/99-oracle-asmdevices.rules"

Updated VMWare 6 Oracle RAC vm create script

I don't know if its just me, but it seems like everything I need to do has an impossible timeline.  When I make VM's for testing, its SOOOO tedious to use a point-and-click interface like virtual center. In the past for large proof of concepts I've had to create literally hundreds of VM's running RAC. To make things faster and to keep my sanity, I wrote a series of scripts to make setting up an Oracle environment nearly automated.

The steps are:
1. Create the RAC node vm's (See this post below)
2. Install the OS via kickstart (or just boot from the OS install ISO and manually configure the OS)
3. Install the Oracle pre-install rpm
        - If you aren't running Oracle Enterprise Linux, or if you want to customize the UID/GID etc, click < HERE >
4. Configure udev/passwordless SSH for root/Oracle
        - For the udev part, this is automated via my script here
5. For Oracle Virtual Machine, Oracle Database Appliance and Exadata, the RAC Pack wrote a wonderful script called OneCommand.  I find this to be a *huge* timesaver, so I extract that from the OVM OVF (found at edelivery.oracle.com) and modify the params.ini and netconfig.ini files.

That's it...if you have everything prepped, depending on your hardware you can go from nothing to running a new 4 node RAC environment in about 20 minutes, with very little "human" time.

A friend of mine integrated these steps into his company's internal cloud...filling in a few variables on a web page (server names, cpu, memory, storage sizes, db name, etc) he can click a button, go to lunch and when he comes back he can log in to his new cluster.

A few years ago I posted my VMWare Oracle RAC node create script, here's an updated version for VSphere 6 and PowerCLI 6.3 Release 1.  It has the changes needed for shared VMDK's and a way to disable the change block tracking that occasionally finds its way into the VMX's.  If you're setting up a flex cluster, you'll need to uncomment the asm_network lines.

Just modify the parameters in the top section.  Hopefully this saves you some time!

#################################################################
## This script works with VMWare 6's change in eager zeroing
# Modify variables below
#################################################################

connect-viserver -Server VC06.poc.local -Protocol https

$vmName1 = "strora01"
$vmName2 = "strora02"
$vmName3 = "strora03"
$vmName4 = "strora04"
$rac_vm_cpu = 50
$rac_vm_ram_mb = (16GB/1MB)
$rac_vm_ram_mb_rez = (8GB/1MB)
# below is the network label in vmware for the vlan
$public_network_name = "VLAN100(pub)"
$private_network_name = "VLAN101(priv)"
#$asm_network_name = "VLAN102(asm)"
#$backup_network_name = "VLAN103(backup)"
$osstore = "os_binaries"
$osstore_size_MB = (50GB/1MB)
$orastore = "os_binaries"
$orastore_size_KB = (100GB/1KB)
$datastore1 = "ora_data1"
$datastore2 = "ora_data2"
$datastore3 = "ora_data3"
$datastore4 = "ora_data4"
$datastore_size_KB = (400GB/1KB)
$recostore1 = "ora_redo1"
$recostore2 = "ora_redo2"
$recostore3 = "ora_redo3"
$recostore4 = "ora_redo4"
$recostore_size_KB = (16GB/1KB)
$archstore1 = "ora_fra1"
$archstore2 = "ora_fra2"
$archstore3 = "ora_fra3"
$archstore4 = "ora_fra4"
$archstore_size_KB = (25GB/1KB)    
$vm_hostname = "ucsblade135.poc.local"            
$OS_CDROM = "[os_binaries]OS_IMAGE/V77197-01.iso"

#################################################################
#  End variable section
#  No edits should be necessary below this line
#################################################################
 
$VM1 = new-vm `
        -Host "$vm_hostname" `
        -Name $vmName1 `
        -Datastore (get-datastore "$osstore") `
        -GuestID rhel6_64Guest `
        -MemoryMB 4096 `
        -DiskMB $osstore_size_MB `
        -NetworkName "$public_network_name" `
        -DiskStorageFormat "Thin"

$vm2 = new-vm `
        -Host "$vm_hostname" `
        -Name $vmName2 `
        -Datastore (get-datastore "$osstore") `
        -GuestID rhel6_64Guest `
        -MemoryMB 4096 `
        -DiskMB $osstore_size_MB `
        -NetworkName "$public_network_name" `
        -DiskStorageFormat "Thin"

$VM3 = new-vm `
        -Host "$vm_hostname" `
        -Name $vmName3 `
        -Datastore (get-datastore "$osstore") `
        -GuestID rhel6_64Guest `
        -MemoryMB 4096 `
        -DiskMB $osstore_size_MB `
        -NetworkName "$public_network_name" `
        -DiskStorageFormat "Thin"

$VM4 = new-vm `
        -Host "$vm_hostname" `
        -Name $vmName4 `
        -Datastore (get-datastore "$osstore") `
        -GuestID rhel6_64Guest `
        -MemoryMB 4096 `
        -DiskMB $osstore_size_MB `
        -NetworkName "$public_network_name" `
        -DiskStorageFormat "Thin"

Function Change-Memory {
      Param (
            $VM,
            $MemoryMB
      )
      Process {
            $VMs = Get-VM $VM
            Foreach ($Machine in $VMs) {
                  $VMId = $Machine.Id

                  $VMSpec = New-Object VMware.Vim.VirtualMachineConfigSpec
                  $VMSpec.memoryMB = $MemoryMB
                  $RawVM = Get-View -Id $VMId
                  $RawVM.ReconfigVM_Task($VMSpec)
            }
      }
}

Change-Memory -MemoryMB $rac_vm_ram_mb -VM $VM1
Change-Memory -MemoryMB $rac_vm_ram_mb -VM $VM2
Change-Memory -MemoryMB $rac_vm_ram_mb -VM $VM3
Change-Memory -MemoryMB $rac_vm_ram_mb -VM $VM4

Set-VM -vm(get-vm $VM1) -NumCpu $rac_vm_cpu -RunAsync -Version v11 -Confirm:$false
Set-VM -vm(get-vm $vm2) -NumCpu $rac_vm_cpu -RunAsync -Version v11 -Confirm:$false
Set-VM -vm(get-vm $VM3) -NumCpu $rac_vm_cpu -RunAsync -Version v11 -Confirm:$false
Set-VM -vm(get-vm $VM4) -NumCpu $rac_vm_cpu -RunAsync -Version v11 -Confirm:$false

Get-VM $VM1 | Get-VMResourceConfiguration | Set-VMResourceConfiguration -MemReservationMB $rac_vm_ram_mb_rez
Get-VM $vm2 | Get-VMResourceConfiguration | Set-VMResourceConfiguration -MemReservationMB $rac_vm_ram_mb_rez
Get-VM $VM3 | Get-VMResourceConfiguration | Set-VMResourceConfiguration -MemReservationMB $rac_vm_ram_mb_rez
Get-VM $VM4 | Get-VMResourceConfiguration | Set-VMResourceConfiguration -MemReservationMB $rac_vm_ram_mb_rez

New-NetworkAdapter -VM $vm1 -NetworkName "$private_network_name" -StartConnected -Type vmxnet3 -Confirm:$false
New-NetworkAdapter -VM $vm2 -NetworkName "$private_network_name" -StartConnected -Type vmxnet3 -Confirm:$false
New-NetworkAdapter -VM $vm3 -NetworkName "$private_network_name" -StartConnected -Type vmxnet3 -Confirm:$false
New-NetworkAdapter -VM $vm4 -NetworkName "$private_network_name" -StartConnected -Type vmxnet3 -Confirm:$false

Function Enable-MemHotAdd($vm){
    $vmview = Get-vm $vm | Get-View
    $vmConfigSpec = New-Object VMware.Vim.VirtualMachineConfigSpec

    $extra = New-Object VMware.Vim.optionvalue
    $extra.Key="mem.hotadd"
    $extra.Value="true"
    $vmConfigSpec.extraconfig += $extra

    $vmview.ReconfigVM($vmConfigSpec)
}

enable-memhotadd $vm1
enable-memhotadd $vm2
enable-memhotadd $vm3
enable-memhotadd $vm4

Function Enable-vCpuHotAdd($vm){
    $vmview = Get-vm $vm | Get-View
    $vmConfigSpec = New-Object VMware.Vim.VirtualMachineConfigSpec

    $extra = New-Object VMware.Vim.optionvalue
    $extra.Key="vcpu.hotadd"
    $extra.Value="true"
    $vmConfigSpec.extraconfig += $extra

    $vmview.ReconfigVM($vmConfigSpec)
}

enable-vCpuHotAdd $vm1
enable-vCpuHotAdd $vm2
enable-vCpuHotAdd $vm3
enable-vCpuHotAdd $vm4

New-HardDisk -vm($VM1) -CapacityKB $orastore_size_KB -StorageFormat Thin -datastore "$orastore"
New-HardDisk -vm($vm2) -CapacityKB $orastore_size_KB -StorageFormat Thin -datastore "$orastore"
New-HardDisk -vm($VM3) -CapacityKB $orastore_size_KB -StorageFormat Thin -datastore "$orastore"
New-HardDisk -vm($VM4) -CapacityKB $orastore_size_KB -StorageFormat Thin -datastore "$orastore"

$New_Disk1 = New-HardDisk -vm($VM1) -CapacityKB $datastore_size_KB -StorageFormat EagerZeroedThick -datastore "$datastore1"
$New_Disk2 = new-harddisk -vm($vm2) -diskpath ($New_Disk1 | %{$_.Filename})
$New_Disk3 = new-harddisk -vm($vm3) -diskpath ($New_Disk1 | %{$_.Filename})
$New_Disk4 = new-harddisk -vm($vm4) -diskpath ($New_Disk1 | %{$_.Filename})

$New_SCSI_1_1 = $New_Disk1 | New-ScsiController -Type ParaVirtual -Confirm:$false
$New_SCSI_2_1 = $New_Disk2 | New-ScsiController -Type ParaVirtual -Confirm:$false
$New_SCSI_3_1 = $New_Disk3 | New-ScsiController -Type ParaVirtual -Confirm:$false
$New_SCSI_4_1 = $New_Disk4 | New-ScsiController -Type ParaVirtual -Confirm:$false

$New_Disk1 = New-HardDisk -vm($VM1) -CapacityKB $datastore_size_KB -StorageFormat EagerZeroedThick -datastore "$datastore2"
$New_Disk2 = new-harddisk -vm($vm2) -diskpath ($New_Disk1 | %{$_.Filename})
$New_Disk3 = new-harddisk -vm($vm3) -diskpath ($New_Disk1 | %{$_.Filename})
$New_Disk4 = new-harddisk -vm($vm4) -diskpath ($New_Disk1 | %{$_.Filename})

set-harddisk -Confirm:$false -harddisk $New_Disk1 -controller $New_SCSI_1_1
set-harddisk -Confirm:$false -harddisk $New_Disk2 -controller $New_SCSI_2_1
set-harddisk -Confirm:$false -harddisk $New_Disk3 -controller $New_SCSI_3_1
set-harddisk -Confirm:$false -harddisk $New_Disk4 -controller $New_SCSI_4_1

$New_Disk1 = New-HardDisk -vm($VM1) -CapacityKB $datastore_size_KB -StorageFormat EagerZeroedThick -datastore "$datastore3"
$New_Disk2 = new-harddisk -vm($vm2) -diskpath ($New_Disk1 | %{$_.Filename})
$New_Disk3 = new-harddisk -vm($vm3) -diskpath ($New_Disk1 | %{$_.Filename})
$New_Disk4 = new-harddisk -vm($vm4) -diskpath ($New_Disk1 | %{$_.Filename})

set-harddisk -Confirm:$false -harddisk $New_Disk1 -controller $New_SCSI_1_1
set-harddisk -Confirm:$false -harddisk $New_Disk2 -controller $New_SCSI_2_1
set-harddisk -Confirm:$false -harddisk $New_Disk3 -controller $New_SCSI_3_1
set-harddisk -Confirm:$false -harddisk $New_Disk4 -controller $New_SCSI_4_1

$New_Disk1 = New-HardDisk -vm($VM1) -CapacityKB $datastore_size_KB -StorageFormat EagerZeroedThick -datastore "$datastore4"
$New_Disk2 = new-harddisk -vm($vm2) -diskpath ($New_Disk1 | %{$_.Filename})
$New_Disk3 = new-harddisk -vm($vm3) -diskpath ($New_Disk1 | %{$_.Filename})
$New_Disk4 = new-harddisk -vm($vm4) -diskpath ($New_Disk1 | %{$_.Filename})

set-harddisk -Confirm:$false -harddisk $New_Disk1 -controller $New_SCSI_1_1
set-harddisk -Confirm:$false -harddisk $New_Disk2 -controller $New_SCSI_2_1
set-harddisk -Confirm:$false -harddisk $New_Disk3 -controller $New_SCSI_3_1
set-harddisk -Confirm:$false -harddisk $New_Disk4 -controller $New_SCSI_4_1

###################################

$New_Disk1 = New-HardDisk -vm($VM1) -CapacityKB $recostore_size_KB -StorageFormat EagerZeroedThick -datastore "$recostore1"
$New_Disk2 = new-harddisk -vm($vm2) -diskpath ($New_Disk1 | %{$_.Filename})
$New_Disk3 = new-harddisk -vm($vm3) -diskpath ($New_Disk1 | %{$_.Filename})
$New_Disk4 = new-harddisk -vm($vm4) -diskpath ($New_Disk1 | %{$_.Filename})

$New_SCSI_1_2 = $New_Disk1 | New-ScsiController -Type ParaVirtual -Confirm:$false
$New_SCSI_2_2 = $New_Disk2 | New-ScsiController -Type ParaVirtual -Confirm:$false
$New_SCSI_3_2 = $New_Disk3 | New-ScsiController -Type ParaVirtual -Confirm:$false
$New_SCSI_4_2 = $New_Disk4 | New-ScsiController -Type ParaVirtual -Confirm:$false

$New_Disk1 = New-HardDisk -vm($VM1) -CapacityKB $recostore_size_KB -StorageFormat EagerZeroedThick -datastore "$recostore2"
$New_Disk2 = new-harddisk -vm($vm2) -diskpath ($New_Disk1 | %{$_.Filename})
$New_Disk3 = new-harddisk -vm($vm3) -diskpath ($New_Disk1 | %{$_.Filename})
$New_Disk4 = new-harddisk -vm($vm4) -diskpath ($New_Disk1 | %{$_.Filename})

set-harddisk -Confirm:$false -harddisk $New_Disk1 -controller $New_SCSI_1_2
set-harddisk -Confirm:$false -harddisk $New_Disk2 -controller $New_SCSI_2_2
set-harddisk -Confirm:$false -harddisk $New_Disk3 -controller $New_SCSI_3_2
set-harddisk -Confirm:$false -harddisk $New_Disk4 -controller $New_SCSI_4_2

$New_Disk1 = New-HardDisk -vm($VM1) -CapacityKB $recostore_size_KB -StorageFormat EagerZeroedThick -datastore "$recostore3"
$New_Disk2 = new-harddisk -vm($vm2) -diskpath ($New_Disk1 | %{$_.Filename})
$New_Disk3 = new-harddisk -vm($vm3) -diskpath ($New_Disk1 | %{$_.Filename})
$New_Disk4 = new-harddisk -vm($vm4) -diskpath ($New_Disk1 | %{$_.Filename})

set-harddisk -Confirm:$false -harddisk $New_Disk1 -controller $New_SCSI_1_2
set-harddisk -Confirm:$false -harddisk $New_Disk2 -controller $New_SCSI_2_2
set-harddisk -Confirm:$false -harddisk $New_Disk3 -controller $New_SCSI_3_2
set-harddisk -Confirm:$false -harddisk $New_Disk4 -controller $New_SCSI_4_2

$New_Disk1 = New-HardDisk -vm($VM1) -CapacityKB $recostore_size_KB -StorageFormat EagerZeroedThick -datastore "$recostore4"
$New_Disk2 = new-harddisk -vm($vm2) -diskpath ($New_Disk1 | %{$_.Filename})
$New_Disk3 = new-harddisk -vm($vm3) -diskpath ($New_Disk1 | %{$_.Filename})
$New_Disk4 = new-harddisk -vm($vm4) -diskpath ($New_Disk1 | %{$_.Filename})

set-harddisk -Confirm:$false -harddisk $New_Disk1 -controller $New_SCSI_1_2
set-harddisk -Confirm:$false -harddisk $New_Disk2 -controller $New_SCSI_2_2
set-harddisk -Confirm:$false -harddisk $New_Disk3 -controller $New_SCSI_3_2
set-harddisk -Confirm:$false -harddisk $New_Disk4 -controller $New_SCSI_4_2


#######################


$New_Disk1 = New-HardDisk -vm($VM1) -CapacityKB $archstore_size_KB -StorageFormat EagerZeroedThick -datastore "$archstore1"
$New_Disk2 = new-harddisk -vm($vm2) -diskpath ($New_Disk1 | %{$_.Filename})
$New_Disk3 = new-harddisk -vm($vm3) -diskpath ($New_Disk1 | %{$_.Filename})
$New_Disk4 = new-harddisk -vm($vm4) -diskpath ($New_Disk1 | %{$_.Filename})

$New_SCSI_1_3 = $New_Disk1 | New-ScsiController -Type ParaVirtual -Confirm:$false
$New_SCSI_2_3 = $New_Disk2 | New-ScsiController -Type ParaVirtual -Confirm:$false
$New_SCSI_3_3 = $New_Disk3 | New-ScsiController -Type ParaVirtual -Confirm:$false
$New_SCSI_4_3 = $New_Disk4 | New-ScsiController -Type ParaVirtual -Confirm:$false

$New_Disk1 = New-HardDisk -vm($VM1) -CapacityKB $archstore_size_KB -StorageFormat EagerZeroedThick -datastore "$archstore2"
$New_Disk2 = new-harddisk -vm($vm2) -diskpath ($New_Disk1 | %{$_.Filename})
$New_Disk3 = new-harddisk -vm($vm3) -diskpath ($New_Disk1 | %{$_.Filename})
$New_Disk4 = new-harddisk -vm($vm4) -diskpath ($New_Disk1 | %{$_.Filename})

set-harddisk -Confirm:$false -harddisk $New_Disk1 -controller $New_SCSI_1_3
set-harddisk -Confirm:$false -harddisk $New_Disk2 -controller $New_SCSI_2_3
set-harddisk -Confirm:$false -harddisk $New_Disk3 -controller $New_SCSI_3_3
set-harddisk -Confirm:$false -harddisk $New_Disk4 -controller $New_SCSI_4_3

$New_Disk1 = New-HardDisk -vm($VM1) -CapacityKB $archstore_size_KB -StorageFormat EagerZeroedThick -datastore "$archstore3"
$New_Disk2 = new-harddisk -vm($vm2) -diskpath ($New_Disk1 | %{$_.Filename})
$New_Disk3 = new-harddisk -vm($vm3) -diskpath ($New_Disk1 | %{$_.Filename})
$New_Disk4 = new-harddisk -vm($vm4) -diskpath ($New_Disk1 | %{$_.Filename})

set-harddisk -Confirm:$false -harddisk $New_Disk1 -controller $New_SCSI_1_3
set-harddisk -Confirm:$false -harddisk $New_Disk2 -controller $New_SCSI_2_3
set-harddisk -Confirm:$false -harddisk $New_Disk3 -controller $New_SCSI_3_3
set-harddisk -Confirm:$false -harddisk $New_Disk4 -controller $New_SCSI_4_3

$New_Disk1 = New-HardDisk -vm($VM1) -CapacityKB $archstore_size_KB -StorageFormat EagerZeroedThick -datastore "$archstore4"
$New_Disk2 = new-harddisk -vm($vm2) -diskpath ($New_Disk1 | %{$_.Filename})
$New_Disk3 = new-harddisk -vm($vm3) -diskpath ($New_Disk1 | %{$_.Filename})
$New_Disk4 = new-harddisk -vm($vm4) -diskpath ($New_Disk1 | %{$_.Filename})

set-harddisk -Confirm:$false -harddisk $New_Disk1 -controller $New_SCSI_1_3
set-harddisk -Confirm:$false -harddisk $New_Disk2 -controller $New_SCSI_2_3
set-harddisk -Confirm:$false -harddisk $New_Disk3 -controller $New_SCSI_3_3
set-harddisk -Confirm:$false -harddisk $New_Disk4 -controller $New_SCSI_4_3

$ExtraOptions = @{
    # per VMware, SAP and Oracle VMware Best Practices
    "disk.EnableUUID"="true";
    "ctkEnabled"="false";
    "scsi0:0.ctkEnabled"="false";
    "scsi0:1.ctkEnabled"="false";
    "scsi1:0.ctkEnabled"="false";
    "scsi1:1.ctkEnabled"="false";
    "scsi1:2.ctkEnabled"="false";
    "scsi1:3.ctkEnabled"="false";
    "scsi2:0.ctkEnabled"="false";
    "scsi2:1.ctkEnabled"="false";
    "scsi2:2.ctkEnabled"="false";
    "scsi2:3.ctkEnabled"="false";
    "scsi3:0.ctkEnabled"="false";
    "scsi3:1.ctkEnabled"="false";
    "scsi3:2.ctkEnabled"="false";
    "scsi3:3.ctkEnabled"="false";
    "ethernet2.coalescingScheme"="disabled";
    "sched.mem.pshare.enable"="false";
    "numa.vcpu.preferHT"="true";
    "tools.syncTime" = "False";

    # per VMware's Hardening Guide - Enterprise Level
    "isolation.tools.diskShrink.disable"="true";
    "isolation.tools.diskWiper.disable"="true";
    "isolation.tools.copy.disable"="true";
    "isolation.tools.paste.disable"="true";
    "isolation.tools.setGUIOptions.enable"="false";
    "isolation.device.connectable.disable"="true";
    "isolation.device.edit.disable"="true";
    "vmci0.unrestricted"="false";
    "log.keepOld"="10";
    "log.rotateSize"="1000000";
    "tools.setInfo.sizeLimit"="1048576";
    "guest.command.enabled"="false";
    "tools.guestlib.enableHostInfo"="false"
}
$vmConfigSpec = New-Object VMware.Vim.VirtualMachineConfigSpec;
Foreach ($Option in $ExtraOptions.GetEnumerator()) {
    $OptionValue = New-Object VMware.Vim.optionvalue
    $OptionValue.Key = $Option.Key
    $OptionValue.Value = $Option.Value
    $vmConfigSpec.extraconfig += $OptionValue
}

$vmview=get-vm $vmName1 | get-view
$vmview.ReconfigVM_Task($vmConfigSpec)
$vmview=get-vm $vmName2 | get-view
$vmview.ReconfigVM_Task($vmConfigSpec)
$vmview=get-vm $vmName3 | get-view
$vmview.ReconfigVM_Task($vmConfigSpec)
$vmview=get-vm $vmName4 | get-view
$vmview.ReconfigVM_Task($vmConfigSpec)

new-CDDRIVE -VM $VM1  -isopath "$OS_CDROM" -startConnected
new-CDDRIVE -VM $VM2  -isopath "$OS_CDROM" -startConnected
new-CDDRIVE -VM $VM3  -isopath "$OS_CDROM" -startConnected
new-CDDRIVE -VM $VM4  -isopath "$OS_CDROM" -startConnected

function Set-MultiWriter{
  param($VM, $DiskName)
  echo $VM
  $TempVM = $VM | Get-View
  $Devicy=$TempVM.Config.Hardware.Device
  foreach ($device in $Devicy) {
    if($device -is [VMware.Vim.VirtualDisk] -and $device.deviceInfo.Label -eq $DiskName) {
      $diskDevice = $device
      $diskDeviceBaking = $device.backing
      break
    }
  }

  $spec = New-Object VMware.Vim.VirtualMachineConfigSpec
  $spec.deviceChange = New-Object VMware.Vim.VirtualDeviceConfigSpec
  $spec.deviceChange[0].operation = 'edit'
  $spec.deviceChange[0].device = New-Object VMware.Vim.VirtualDisk
  $spec.deviceChange[0].device = $diskDevice
  $spec.DeviceChange[0].device.backing = New-Object VMware.Vim.VirtualDiskFlatVer2BackingInfo
  $spec.DeviceChange[0].device.backing = $diskDeviceBaking
  $spec.DeviceChange[0].device.backing.sharing = "sharingMultiWriter"
  Write-Host "`nEnabling Multiwriter flag on on VMDK:" $diskName "for VM:" $vmname
  $task = $TempVM.ReconfigVM_Task($spec)
  $task1 = Get-Task -Id ("Task-$($task.value)")
  $task1 | Wait-Task
}
Set-MultiWriter $VM1 "Hard disk 3"
Set-MultiWriter $VM1 "Hard disk 4"
Set-MultiWriter $VM1 "Hard disk 5"
Set-MultiWriter $VM1 "Hard disk 6"
Set-MultiWriter $VM1 "Hard disk 7"
Set-MultiWriter $VM1 "Hard disk 8"
Set-MultiWriter $VM1 "Hard disk 9"
Set-MultiWriter $VM1 "Hard disk 10"
Set-MultiWriter $VM1 "Hard disk 11"
Set-MultiWriter $VM1 "Hard disk 12"
Set-MultiWriter $VM1 "Hard disk 13"
Set-MultiWriter $VM1 "Hard disk 14"
Set-MultiWriter $VM2 "Hard disk 3"
Set-MultiWriter $VM2 "Hard disk 4"
Set-MultiWriter $VM2 "Hard disk 5"
Set-MultiWriter $VM2 "Hard disk 6"
Set-MultiWriter $VM2 "Hard disk 7"
Set-MultiWriter $VM2 "Hard disk 8"
Set-MultiWriter $VM2 "Hard disk 9"
Set-MultiWriter $VM2 "Hard disk 10"
Set-MultiWriter $VM2 "Hard disk 11"
Set-MultiWriter $VM2 "Hard disk 12"
Set-MultiWriter $VM2 "Hard disk 13"
Set-MultiWriter $VM2 "Hard disk 14"
Set-MultiWriter $VM3 "Hard disk 3"
Set-MultiWriter $VM3 "Hard disk 4"
Set-MultiWriter $VM3 "Hard disk 5"
Set-MultiWriter $VM3 "Hard disk 6"
Set-MultiWriter $VM3 "Hard disk 7"
Set-MultiWriter $VM3 "Hard disk 8"
Set-MultiWriter $VM3 "Hard disk 9"
Set-MultiWriter $VM3 "Hard disk 10"
Set-MultiWriter $VM3 "Hard disk 11"
Set-MultiWriter $VM3 "Hard disk 12"
Set-MultiWriter $VM3 "Hard disk 13"
Set-MultiWriter $VM3 "Hard disk 14"
Set-MultiWriter $VM4 "Hard disk 3"
Set-MultiWriter $VM4 "Hard disk 4"
Set-MultiWriter $VM4 "Hard disk 5"
Set-MultiWriter $VM4 "Hard disk 6"
Set-MultiWriter $VM4 "Hard disk 7"
Set-MultiWriter $VM4 "Hard disk 8"
Set-MultiWriter $VM4 "Hard disk 9"
Set-MultiWriter $VM4 "Hard disk 10"
Set-MultiWriter $VM4 "Hard disk 11"
Set-MultiWriter $VM4 "Hard disk 12"
Set-MultiWriter $VM4 "Hard disk 13"
Set-MultiWriter $VM4 "Hard disk 14"