OCI I/O-Performance with E-Business Suite (Part 3/3) - LVM

Wikipedia gives a good explanation on the purpose of Linux LVM:

In Linux, Logical Volume Manager (LVM) is a device mapper framework that provides logical volume management for the Linux kernel. LVM is used for the following purposes:

• Creating single logical volumes of multiple physical volumes or entire hard disks (somewhat similar to RAID 0, but more similar to JBOD), allowing for dynamic volume resizing.

• Managing large hard disk farms by allowing disks to be added and replaced without downtime or service disruption, in combination with hot swapping.

• On small systems (like a desktop), instead of having to estimate at installation time how big a partition might need to be, LVM allows filesystems to be easily resized as needed.

• Performing consistent backups by taking snapshots of the logical volumes.

• Encrypting multiple physical partitions with one password.

• LVM can be considered as a thin software layer on top of the hard disks and partitions, which creates an abstraction of continuity and ease-of-use for managing hard drive replacement, repartitioning and backup.

For our purpose we are mainly interested in the first aspect of LVM.

Unfortunately, E-Business Suite instances provisioned through Oracle Cloud Manager (be it new installations or be it the restore of a backup) "on Compute" only use a single "Block Volume" to store all the data and the Oracle-Home of the Database (the whole structure beginning with /u01). Fortunately, this can be changed with the following sequence of steps outlined below:

1. In the Oracle Cloud Infrastructure Console create 9 new Block Volumes (lvm1-lvm9) with 500 GB each and attach them to the database compute instance.

2. Run the iscsi-attach commands shown in OCI console for each of the volumes (making them available in the linux vm).

3. Convert the LUNs into LVM physical volumes using:

   pvcreate /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk

4. Extend the volume group to use those disks:

   vgextend ebs_vg /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk

5. Create a new logical volume:

   lvcreate -i9 -I4M -l100%FREE -n ebs_lv3 ebs_vg

   Note: I have made good experience using a stripe size of 4 Megabyte, however this may depend on your payload and DB Block size.

6. Verify with lvdisplay -m that the new logical volume is available and appropriately striped.

The easiest approach is to do an "offline" copy of all the data to the new logical volume. For this I did:

1. Stop everything and unmount /u01

2. Move everything to the new volume:

   dd if=/dev/dm-0 of=/dev/dm-1 bs=128M status=progress oflag=direct iflag=direct

   Double-check to take the appropriate source and target. If you want to resize to a smaller target in the same process you can also use filesystem copy (while preserving all links and permissions). The copy worked with appx. 600MByte/s so took a bit more than 2 hours for our database.

3. Change /etc/fstab to mount ebs_lv2 instead of ebs_lv and remount /u01

4. Bring the database up again.

It is also possible to do an online-migration using an approach as described here:

1. Instead of step (5) of the lvcreation do a lvconvert to create a raid1 between the existing LUN and the new LUNs:

   lvconvert --type mirror -m 1 --mirrorlog core --stripes 8 /dev/ebs_vg/ebs_lv /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj

2. If this gives an error like "Number of extents requested (1165823) needs to be divisible by 8." first extend the logical volume by 1-7 extends to have something dividable.

3. Then break the mirror:

   lvconvert -m0 /dev/ebs_vg/ebs_lv /dev/sdb

4. Reduce the VG by the old LUN:

   vgreduce /dev/ebs_vg /dev/sdb

5. Remove the PV:

   pvremove /dev/sdb

6. Detach the old 4.5TB Volume using the iscsi Commands, then using the OCI Cloud Console. Finally destroy the old/large volume.

Measuring Performance gives us an I/O-Performance that is by a factor of 7.7 (out of the theoretical maximum of 9) above the performance with a single drive. The throughput is a bit more than factor 3 higher than with a single drive:

Max IOPS = 178.571

Max MBPS = 1.409

This poor increase in throughput is due to the fact that we have hit the networking speed of our 12 OCPU VM: With "only" 12 GBps of network bandwidth we can reach no more than appx. 1400 MByte per second.

To check if this scales up even further with the added Network Bandwidth, change the number of OCPUs from 12 to 20, which gives us:

max_iops = 213.391 (theoretical max. by Storage: 225K)

max_mbps = 2.115 (theoretical max. by Storage 4.320; by Network 2.500 MByte/S)

This is again close to the physical limit of the network interface regarding the throughput, and as we see, the IOPS went a bit higher as well.

Overall, the performance was more or less the same compared to ASM. Note that for ASM I chose to use only 7 LUNs (since that was enough to contain all the data; then in addition, I had a LUN with 1 TB for the Oracle Home/...) while in the case with LVM everything went to the same logical volume requiring 9 LUNs (and thus providing an even better performance).

Using multiple Block Storage devices behind a single LVM logical volume with striping provides a convenient and reliable way to store the contents of an Oracle Database. In Oracle Cloud Infrastructure, more than a single block volume can easily be used to store the data. By striping of the data across multiple volumes, one can get a much better I/O performance than by using a single block volume, as provided by oracle "out of the box". For a database with multiple terabytes of storage, this provides a massive improvement in performance without additional OCI costs.

There are only two major drawbacks of this approach:

• Unfortunately, so far, it is not possible to perform a fast-clone using E-Business Suite Cloud manager. There is an Enhancement Request planning to add support for a similar functionality.

• Creating a backup and restoring this backup (e.g. for Cloning production to test) using Cloud Manager Tooling is possible. However, the target will also just have a single block volume. This leads to the testing environment being (slightly, hopefully this has only a performance impact) different from production.

To round up, I would recommend going for the LVM approach compared to ASM for the time being - especially if you do not use ASM for all of your "other" Oracle Databases in use.

The numbers are so excellent using this approach, that for now I do not give an "LVM Cache" on top of DenseIO instances with locally attached NVMe storage a try (as I had initially thought of). According to the documentation, this would at least not help much for an 8 OCPU instance much, given that DenseIO would then also "only" provide 250.000 IOPS. And it adds a lot of complexity and additional costs, especially since no "DenseIO E3 shape" is currently available.