Introduction
Previous Exadata models used 4 Flash "assemblies" which contained 4 individual "Flash Modules" (known as "FMOD's") on
each Flash card, for a total of 16 Flash Modules/Flashdisks seen by the OS. In contrast, the newer Exadata Storage
Servers used in X5-2, X4-8 (when equipped with X5-L storage nodes), X5-8, X6-2 and X6-8 and Supercluster (with
Exadata X5-2 storage) use 4 Flash F160 (X5) or Flash F320 (X6) NVMe (Non-Volatile Memory Express) PCI-E Flash cards
which have their NAND flash configured as a single module. Since these cards do not use multiple FMOD's they appear to
the OS as 1 Flash disk per Flash card, or total of 4 Flash disks per server instead of 16 Flash disks per server seen in
the previous models of Flash cards. Therefore the cell configuration layer also sees 1 physical disk and creates 1 lun on
the whole card, total of 4 disks per Server, rather than 16 per server seen in the previous
models of Exadata Storage Servers.
More information: Why does the Exadata X5 and X6 storage nodes have only 4 Flashdisks per node? (Doc ID 2139076.1)
Exadata X4 (800 GB * 4 Flash Cache Cards)
===========================================
Oracle Exadata x4 - "Sun Flash Accelerator F80 PCIe Card"
1. Each Falsh Card divides into 4 regions (Per Exadata Storage Server)
186.2 GB * 4 - First Card (FD_00_celladm01 to FD_03_celladm01)
186.2 GB * 4 - Second Card (FD_04_celladm01 to FD_07_celladm01)
186.2 GB * 4 - Third Card (FD_08_celladm01 to FD_11_celladm01)
186.2 GB * 4 - Fourth Card (FD_12_celladm01 to FD_15_celladm01)
CellCLI> list flashcache
celladm01_FLASHCACHE normal
CellCLI> list flashcache detail
name: celladm01_FLASHCACHE
cellDisk:
FD_15_celladm01,
FD_04_celladm01,
FD_11_celladm01,
FD_00_celladm01,
FD_07_celladm01,
FD_14_celladm01,
FD_06_celladm01,
FD_12_celladm01,
FD_05_celladm01,
FD_09_celladm01,
FD_10_celladm01,
FD_08_celladm01,
FD_02_celladm01,
FD_13_celladm01,
FD_01_celladm01,
FD_03_celladm01
creationTime: 2016-12-13T19:14:45-05:00
degradedCelldisks:
effectiveCacheSize: 2.908935546875T
id: 86ecd6a4-501f-4a8d-8c8a-eea9e5b58f77
size: 2.908935546875T
status: normal
CellCLI>
CellCLI> list lun detail
name: 1_0
cellDisk: FD_00_celladm01
deviceName: /dev/sdi
diskType: FlashDisk
id: 1_0
isSystemLun: FALSE
lunSize: 186.26451539993286G
physicalDrives: FLASH_1_0
status: normal
name: 1_1
cellDisk: FD_01_celladm01
deviceName: /dev/sdj
diskType: FlashDisk
id: 1_1
isSystemLun: FALSE
lunSize: 186.26451539993286G
physicalDrives: FLASH_1_1
status: normal
name: 1_2
cellDisk: FD_02_celladm01
deviceName: /dev/sdk
diskType: FlashDisk
id: 1_2
isSystemLun: FALSE
lunSize: 186.26451539993286G
physicalDrives: FLASH_1_2
status: normal
name: 1_3
cellDisk: FD_03_celladm01
deviceName: /dev/sdl
diskType: FlashDisk
id: 1_3
isSystemLun: FALSE
lunSize: 186.26451539993286G
physicalDrives: FLASH_1_3
status: normal
name: 2_0
cellDisk: FD_04_celladm01
deviceName: /dev/sdm
diskType: FlashDisk
id: 2_0
isSystemLun: FALSE
lunSize: 186.26451539993286G
physicalDrives: FLASH_2_0
status: normal
name: 2_1
cellDisk: FD_05_celladm01
deviceName: /dev/sdn
diskType: FlashDisk
id: 2_1
isSystemLun: FALSE
lunSize: 186.26451539993286G
physicalDrives: FLASH_2_1
status: normal
name: 2_2
cellDisk: FD_06_celladm01
deviceName: /dev/sdo
diskType: FlashDisk
id: 2_2
isSystemLun: FALSE
lunSize: 186.26451539993286G
physicalDrives: FLASH_2_2
status: normal
name: 2_3
cellDisk: FD_07_celladm01
deviceName: /dev/sdp
diskType: FlashDisk
id: 2_3
isSystemLun: FALSE
lunSize: 186.26451539993286G
physicalDrives: FLASH_2_3
status: normal
name: 4_0
cellDisk: FD_08_celladm01
deviceName: /dev/sde
diskType: FlashDisk
id: 4_0
isSystemLun: FALSE
lunSize: 186.26451539993286G
physicalDrives: FLASH_4_0
status: normal
name: 4_1
cellDisk: FD_09_celladm01
deviceName: /dev/sdf
diskType: FlashDisk
id: 4_1
isSystemLun: FALSE
lunSize: 186.26451539993286G
physicalDrives: FLASH_4_1
status: normal
name: 4_2
cellDisk: FD_10_celladm01
deviceName: /dev/sdg
diskType: FlashDisk
id: 4_2
isSystemLun: FALSE
lunSize: 186.26451539993286G
physicalDrives: FLASH_4_2
status: normal
name: 4_3
cellDisk: FD_11_celladm01
deviceName: /dev/sdh
diskType: FlashDisk
id: 4_3
isSystemLun: FALSE
lunSize: 186.26451539993286G
physicalDrives: FLASH_4_3
status: normal
name: 5_0
cellDisk: FD_12_celladm01
deviceName: /dev/sda
diskType: FlashDisk
id: 5_0
isSystemLun: FALSE
lunSize: 186.26451539993286G
physicalDrives: FLASH_5_0
status: normal
name: 5_1
cellDisk: FD_13_celladm01
deviceName: /dev/sdb
diskType: FlashDisk
id: 5_1
isSystemLun: FALSE
lunSize: 186.26451539993286G
physicalDrives: FLASH_5_1
status: normal
name: 5_2
cellDisk: FD_14_celladm01
deviceName: /dev/sdc
diskType: FlashDisk
id: 5_2
isSystemLun: FALSE
lunSize: 186.26451539993286G
physicalDrives: FLASH_5_2
status: normal
name: 5_3
cellDisk: FD_15_celladm01
deviceName: /dev/sdd
diskType: FlashDisk
id: 5_3
isSystemLun: FALSE
lunSize: 186.26451539993286G
physicalDrives: FLASH_5_3
status: normal
CellCLI>
CellCLI> list physicaldisk
FLASH_1_0 11000152396 normal
FLASH_1_1 11000152536 normal
FLASH_1_2 11000152472 normal
FLASH_1_3 11000151778 normal
FLASH_2_0 11000153100 normal
FLASH_2_1 11000152864 normal
FLASH_2_2 11000152013 normal
FLASH_2_3 11000152389 normal
FLASH_4_0 11000151113 normal
FLASH_4_1 11000151026 normal
FLASH_4_2 11000151119 normal
FLASH_4_3 11000151104 normal
FLASH_5_0 11000151785 normal
FLASH_5_1 11000152298 normal
FLASH_5_2 11000131661 normal
FLASH_5_3 11000131306 normal
Note: I have truncated physical disk information for clarity
CellCLI> list physicaldisk detail
name: FLASH_1_0
deviceName: /dev/sdi
diskType: FlashDisk
flashLifeLeft: 100
luns: 1_0
makeModel: "Sun Flash Accelerator F80 PCIe Card"
physicalFirmware: UIO6
physicalInsertTime: 2016-12-13T12:45:11-05:00
physicalSize: 186.26451539993286G
slotNumber: "PCI Slot: 1; FDOM: 0"
status: normal
name: FLASH_1_1
deviceName: /dev/sdj
diskType: FlashDisk
flashLifeLeft: 100
luns: 1_1
makeModel: "Sun Flash Accelerator F80 PCIe Card"
physicalFirmware: UIO6
physicalInsertTime: 2016-12-13T12:45:11-05:00
physicalSize: 186.26451539993286G
slotNumber: "PCI Slot: 1; FDOM: 1"
status: normal
name: FLASH_1_2
deviceName: /dev/sdk
diskType: FlashDisk
flashLifeLeft: 100
luns: 1_2
makeModel: "Sun Flash Accelerator F80 PCIe Card"
physicalFirmware: UIO6
physicalInsertTime: 2016-12-13T12:45:11-05:00
physicalSize: 186.26451539993286G
slotNumber: "PCI Slot: 1; FDOM: 2"
status: normal
name: FLASH_1_3
deviceName: /dev/sdl
diskType: FlashDisk
flashLifeLeft: 100
luns: 1_3
makeModel: "Sun Flash Accelerator F80 PCIe Card"
physicalFirmware: UIO6
physicalInsertTime: 2016-12-13T12:45:11-05:00
physicalSize: 186.26451539993286G
slotNumber: "PCI Slot: 1; FDOM: 3"
status: normal
name: FLASH_2_0
deviceName: /dev/sdm
diskType: FlashDisk
flashLifeLeft: 100
luns: 2_0
makeModel: "Sun Flash Accelerator F80 PCIe Card"
physicalFirmware: UIO6
physicalInsertTime: 2016-12-13T12:45:11-05:00
physicalSize: 186.26451539993286G
slotNumber: "PCI Slot: 2; FDOM: 0"
status: normal
name: FLASH_2_1
deviceName: /dev/sdn
diskType: FlashDisk
flashLifeLeft: 100
luns: 2_1
makeModel: "Sun Flash Accelerator F80 PCIe Card"
physicalFirmware: UIO6
physicalInsertTime: 2016-12-13T12:45:11-05:00
physicalSize: 186.26451539993286G
slotNumber: "PCI Slot: 2; FDOM: 1"
status: normal
name: FLASH_2_2
deviceName: /dev/sdo
diskType: FlashDisk
flashLifeLeft: 99
luns: 2_2
makeModel: "Sun Flash Accelerator F80 PCIe Card"
physicalFirmware: UIO6
physicalInsertTime: 2016-12-13T12:45:11-05:00
physicalSize: 186.26451539993286G
slotNumber: "PCI Slot: 2; FDOM: 2"
status: normal
name: FLASH_2_3
deviceName: /dev/sdp
diskType: FlashDisk
flashLifeLeft: 100
luns: 2_3
makeModel: "Sun Flash Accelerator F80 PCIe Card"
physicalFirmware: UIO6
physicalInsertTime: 2016-12-13T12:45:11-05:00
physicalSize: 186.26451539993286G
slotNumber: "PCI Slot: 2; FDOM: 3"
status: normal
name: FLASH_4_0
deviceName: /dev/sde
diskType: FlashDisk
flashLifeLeft: 100
luns: 4_0
makeModel: "Sun Flash Accelerator F80 PCIe Card"
physicalFirmware: UIO6
physicalInsertTime: 2016-12-13T12:45:11-05:00
physicalSize: 186.26451539993286G
slotNumber: "PCI Slot: 4; FDOM: 0"
status: normal
name: FLASH_4_1
deviceName: /dev/sdf
diskType: FlashDisk
flashLifeLeft: 100
luns: 4_1
makeModel: "Sun Flash Accelerator F80 PCIe Card"
physicalFirmware: UIO6
physicalInsertTime: 2016-12-13T12:45:11-05:00
physicalSize: 186.26451539993286G
slotNumber: "PCI Slot: 4; FDOM: 1"
status: normal
name: FLASH_4_2
deviceName: /dev/sdg
diskType: FlashDisk
flashLifeLeft: 100
luns: 4_2
makeModel: "Sun Flash Accelerator F80 PCIe Card"
physicalFirmware: UIO6
physicalInsertTime: 2016-12-13T12:45:11-05:00
physicalSize: 186.26451539993286G
slotNumber: "PCI Slot: 4; FDOM: 2"
status: normal
name: FLASH_4_3
deviceName: /dev/sdh
diskType: FlashDisk
flashLifeLeft: 100
luns: 4_3
makeModel: "Sun Flash Accelerator F80 PCIe Card"
physicalFirmware: UIO6
physicalInsertTime: 2016-12-13T12:45:11-05:00
physicalSize: 186.26451539993286G
slotNumber: "PCI Slot: 4; FDOM: 3"
status: normal
name: FLASH_5_0
deviceName: /dev/sda
diskType: FlashDisk
flashLifeLeft: 100
luns: 5_0
makeModel: "Sun Flash Accelerator F80 PCIe Card"
physicalFirmware: UIO6
physicalInsertTime: 2016-12-13T12:45:11-05:00
physicalSize: 186.26451539993286G
slotNumber: "PCI Slot: 5; FDOM: 0"
status: normal
name: FLASH_5_1
deviceName: /dev/sdb
diskType: FlashDisk
flashLifeLeft: 100
luns: 5_1
makeModel: "Sun Flash Accelerator F80 PCIe Card"
physicalFirmware: UIO6
physicalInsertTime: 2016-12-13T12:45:11-05:00
physicalSize: 186.26451539993286G
slotNumber: "PCI Slot: 5; FDOM: 1"
status: normal
name: FLASH_5_2
deviceName: /dev/sdc
diskType: FlashDisk
flashLifeLeft: 100
luns: 5_2
makeModel: "Sun Flash Accelerator F80 PCIe Card"
physicalFirmware: UIO6
physicalInsertTime: 2016-12-13T12:45:11-05:00
physicalSize: 186.26451539993286G
slotNumber: "PCI Slot: 5; FDOM: 2"
status: normal
name: FLASH_5_3
deviceName: /dev/sdd
diskType: FlashDisk
flashLifeLeft: 100
luns: 5_3
makeModel: "Sun Flash Accelerator F80 PCIe Card"
physicalFirmware: UIO6
physicalInsertTime: 2016-12-13T12:45:11-05:00
physicalSize: 186.26451539993286G
slotNumber: "PCI Slot: 5; FDOM: 3"
status: normal
CellCLI> help
Exadata X6 (3 TB * 4 Flash Cache Cards)
========================================
Oracle Exadata x6 - "Oracle Flash Accelerator F320 PCIe Card"
1. Each Flash Card divides into 1 region - Per Storage Server
2.9 TB * 1 - First Card (FLASH_1_1)
2.9 TB * 1 - Second Card (FLASH_2_1)
2.9 TB * 1 - Third Card (FLASH_4_1)
2.9 TB * 1 - Fourth Card (FLASH_5_1)
CellCLI> list flashcache
celadm01_FLASHCACHE normal
CellCLI> list flashcache detail
name: celadm01_FLASHCACHE
cellDisk: FD_02_celadm01,FD_01_celadm01,FD_00_celadm01,FD_03_celadm01
creationTime: 2016-08-02T17:26:28-04:00
degradedCelldisks:
effectiveCacheSize: 11.64312744140625T
id: f37efd89-6030-455d-9300-f48657344bf6
size: 11.64312744140625T
status: normal
CellCLI>
CellCLI> list physicaldisk
FLASH_1_1 S2T7NAAH301716 normal
FLASH_2_1 S2T7NAAH301662 normal
FLASH_4_1 S2T7NAAH301680 normal
FLASH_5_1 S2T7NAAH301723 normal
CellCLI> list physicaldisk detail
name: FLASH_1_1
deviceName: /dev/nvme3n1
diskType: FlashDisk
luns: 1_1
makeModel: "Oracle Flash Accelerator F320 PCIe Card"
physicalFirmware: KPYAGR3Q
physicalInsertTime: 2016-07-25T12:58:28-04:00
physicalSize: 2.910957656800747T
slotNumber: "PCI Slot: 1; FDOM: 1"
status: normal
name: FLASH_2_1
deviceName: /dev/nvme2n1
diskType: FlashDisk
luns: 2_1
makeModel: "Oracle Flash Accelerator F320 PCIe Card"
physicalFirmware: KPYAGR3Q
physicalInsertTime: 2016-07-25T12:58:28-04:00
physicalSize: 2.910957656800747T
slotNumber: "PCI Slot: 2; FDOM: 1"
status: normal
name: FLASH_4_1
deviceName: /dev/nvme0n1
diskType: FlashDisk
luns: 4_1
makeModel: "Oracle Flash Accelerator F320 PCIe Card"
physicalFirmware: KPYAGR3Q
physicalInsertTime: 2016-07-25T12:58:28-04:00
physicalSize: 2.910957656800747T
slotNumber: "PCI Slot: 4; FDOM: 1"
status: normal
name: FLASH_5_1
deviceName: /dev/nvme1n1
diskType: FlashDisk
luns: 5_1
makeModel: "Oracle Flash Accelerator F320 PCIe Card"
physicalFirmware: KPYAGR3Q
physicalInsertTime: 2016-07-25T12:58:28-04:00
physicalSize: 2.910957656800747T
slotNumber: "PCI Slot: 5; FDOM: 1"
status: normal
CellCLI>
Thursday, July 6, 2017
Flash Cache Cards in Oracle Exadata x4 and Oracle Exadata x6
Subscribe to:
Post Comments (Atom)
Nice Explanation...
ReplyDeleteThanks Raj Kumar
Delete