(no subject)
Jan. 22nd, 2009 21:23![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
nepilsonishttp://www.opensolaris.org/jive/thread.jspa?threadID=88453&tstart=0#326654
хорошее письмо о том, какой RAID лучше — средствами RAID-контроллера или дискового массива, или программный, обеспечиваемый zfs.
Общий смысл — «смотря для чего», но написано интересно.
Там же ссылка на интересное исследование http://pages.cs.wisc.edu/~krioukov/Krioukov-ParityLost.pdf о том, что разные реализации одного и того же уровня RAID далеко не во всех ситуациях эквивалентны.
Так как это было опубликовано в списке рассылки, то «на всякий случай» скопирую это письмо сюда целиком:
Re: [zfs-discuss] ZFS vs ZFS + HW raid? Which is best?
Posted: Jan 13, 2009 6:16 PM in response to: kebabber
>>>>> "ok" == Orvar Korvar writes:
ok> Nobody really knows for sure.
ok> I will tell people that ZFS + HW raid is good enough, and I
ok> will not recommend against HW raid anymore.
jesus, ok fine if you threaten to let ignorant Windows morons strut around like arrogant experts, voicing some baseless feel-good opinion and then burdening you with proving a negative, then I'll take the bait and do your homework for you.
What people "don't really know for sure" is what hardware RAID you mean. There are worlds of difference between NetApp RAID and Dell PERC RAID, differences in their ability to avoid data loss in common failure scenarios, not just in some bulleted features or admin tool GUI's that the Windows admins understand. For example among dirt-cheap RAID5's, whether or not there's a battery-backed write cache influences how likely they are to corrupt the filesystem on top, while the Windows admins probably think it's for "performance issues" or something. And there are other things. NetApp has most of them, so if you read a few of the papers they publish bragging about their features you'll get an idea how widely the robustness of RAIDs can vary:
http://pages.cs.wisc.edu/~krioukov/Krioukov-ParityLost.pdf
Advantages of ZFS-on-some-other-RAID:
* probably better performance for RAID6 than for raidz2. It is not because of "checksum overhead". It's because RAID6 stripes use less seek bandwidth than ZFS which mostly does full-stripe writes like RAID3.
* It's performant to use a filesystem other than ZFS on a hardware RAID, which maybe protects your investment somewhat.
* better availability, sometimes.
It's important to distinguish between availability and data loss. Availability means when a disk goes bad, applications don't notice. Data loss is about, AFTER the system panics, the sysadmin notices, unplugs bad drives, replaces things, reboots, is the data still there or not? And is *ALL* the data there, satisfying ACID rules for databases and MTA's, or just most of the data there? because it is supposed to be ALL there. Even when you have to intervene, that is not license to break all the promises and descend into chaos.
Some (probably most) hardware RAID has better availability, is better at handling failing or disconnected drives than ZFS. But ZFS on JBOD is probably better at avoiding data loss than ZFS on HWRAID.
* better exception handling, sometimes. Really fantastically good hardware RAID can handle the case of two mirrored disks, both with unreadable sectors, but none of the unreadable sectors are in common. Partial disk failures are the common case, not the exception, while the lower-quality RAID implementations most of us are stuck with treat disks as either good or bad, as a whole unit.
I expect most hardware RAIDs don't have much grace in this department either, though. It seems really difficult to do it well because disks lock up when you touch their bad blocks, so to gracefully extract information from a partially-failed disk, you have to load special firmware into them and/or keep a list of poison blocks which you must never accidentally read.
My general procedure for recovering hardware RAIDs is, (0) shut down the RAID and use as JBOD, (1) identify the bad disks and buy blank disks to replace them, (2) copy bad disks onto blank disks using dd_rescue or dd conv=noerror,sync, and (3) run the RAID recovery tool. There are a lot of bad things about this procedure! It denies the RAID layer access to the disk's reports about which sector is bad, which leads to parity pollution, and silent corruption that slips through the RAID layer, discussed below. It can also not work, if the RAID is identifying disks like ZFS sometimes does, by devid/serialno rather than by a disk label or port number.
but RAID layers read the report "bad sector" as in fact a report "bad disk!" so with the very cheap RAID's I've used this 0,1,2,3 corruption-prone procedure which works around the bad exception handling saves more data than the supported procedure.
* novolatile cache. Some cheap hardware RAID gives you a battery-backed write cache that can be more expensive to get the ZFS slog way.
Advantages to ZFS-on-JBOD:
* unified admin tool. There is one less layer, so you can administer everything with zpool. With hardware RAID you will have to use both zpool and the hardware RAID's configurator.
* most hardware RAID has no way to deal with silent corruption. It can deal with latent sector errors, when the drive explicitly reports a failed read, but it has no way to deal with drives that silently return incorrect data. There are a huge number of patterns in which this happens in real life according to Netapp, and they have names for them like "torn writes" and "misdirected writes" and so on. RAID5's read-modify-write behavior can magnify this type of error through "parity pollution."
http://www.usenix.org/event/fast08/tech/full_papers/bairavasundaram/bairavasundaram.pdf
I think there are some tricks in Solaris for Veritas and SVM mirrors, which applications like Oracle can use if they do checksumming above the filesystem level. Oracle can say, "no, I don't like that block. Can you roll the dice again? What else might be at that lseek offset?" See DKIODMR under dkio(7I). I'm not sure in exactly what circumstance the tricks work, but am sure they will NOT work on hardware RAID-on-a-card. Also the tricks are not used by ZFS, so if you insert ZFS between your RAID and your application, the DKIODMR-style checksum-redundancy is blocked off. The idea is that you should use ZFS checksums and ZFS redundancy instead, but you can't do that by feeding ZFS a single LUN.
* hardware RAID ties your data to a RAID implementation. This is more constraining than tying it to ZFS because the Nexenta and Opensolaris licenses allow you to archive the ZFS software implementation and share it with your friends. This has bitten several of my friends particularly with "RAID-on-a-card" products. It is really uncool to lose your entire dataset to a bad controller, and then be unable to obtain the same controller with the same software revision, because the hardware has gone through so many steppings, and the software isn't freely redistributable, archiveable, or even clear that it exists at all. Sometimes the configurator tool is very awkward to run. There may even be a remedial configurator in the firmware and a full-featured configurator which is difficult to run just when you need it most, when recovering from a failure.
Some setups are even worse, are unclear about where they are storing the metadata---on the diskset or in the controller? so you could lose the whole dataset because of a few bits in some $5 lithium battery RTC chip. Being a good sysadmin in this environment requires levels of distrust that are really unhealthy.
The more expensive implementations hold your data hostage to a support contract, so while you are paying to buy the RAID you are in fact more like renting a place to put your data, and they reserve the right to raise your rent. Without the contract you don't just lose their attention---you cannot get basic things like older software, or manuals for the software you have now, and they threaten to sue you for software license violation if you try to sell the hardware to someone else.
* there are bugs either in ZFS or in the overall system that make ZFS much more prone to corruption when it's not managing a layer of redundancy.
In particular, we know ZFS does not work well when underlying storage reports writes are committed to disk when they're not, and this problem seems to be rampant:
* some SATA disks do it, but the people who know which ones aren't willing to tell. They only say "a Major vendor."
* Sun's sun4v hypervisor does it for virtualized disk access on the T2000.
* Sun's iSCSI target does it (iscsitadm. not sure yet about Comstar.)
* We think many PeeCee virtualization platforms like virtualbox and vmware and stuff might do it.
* Linux does it if you are using disk through LVM2.
The problem is rampant, seems to be more dangerous to ZFS than other filesystems, and progress in tracking it down and fixing it is glacial. Giving ZFS a mirror or a raidz seems to improve its survival.
To work around this with hardware RAID, you need to make a zpool that's a mirror of two hardware RAIDsets. This wastes a lot of disk. If you had that much disk with ZFS JBOD, you could make much better use of it as a filesystem-level backup, like backup with rsync to a non-ZFS filesystem or to a separate zpool with 'zfs send | zfs recv'. You really need this type of backup with zfs because of the lack of fsck and the huge number of panics and assertions.
HTH.
To my reckoning the consensus best practice is usually JBOD right now. When I defend ZFS-over-RAID5 it is mostly because I think the poor availability during failures and the corruption bugs discussed in the last point need to be tracked down and squashed.
Here's my list of papers that have been mentioned on this list, so you can catch up. You can also dump all the papers on the annoying Windows admins, and when they say "I really think hardware RAID has fewer `issues' because I just think so. It's your job to prove why not," then you can answer "well have you read the papers? No?
Then take my word for it." If they doubt the papers' authority, then cite the price the people writing the papers charge for their services---the Windows admins should at least understand that.
http://www.usenix.org/event/fast08/tech/full_papers/bairavasundaram/bairavasundaram.pdf
http://www.cs.wisc.edu/adsl/Publications/latent-sigmetrics07.ps
http://labs.google.com/papers/disk_failures.html
http://pages.cs.wisc.edu/~krioukov/Krioukov-ParityLost.pdf
http://www.nber.org/sys-admin/linux-nas-raid.html
_______________________________________________
zfs-discuss mailing list
zfs-discuss at opensolaris dot org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
хорошее письмо о том, какой RAID лучше — средствами RAID-контроллера или дискового массива, или программный, обеспечиваемый zfs.
Общий смысл — «смотря для чего», но написано интересно.
Там же ссылка на интересное исследование http://pages.cs.wisc.edu/~krioukov/Krioukov-ParityLost.pdf о том, что разные реализации одного и того же уровня RAID далеко не во всех ситуациях эквивалентны.
Так как это было опубликовано в списке рассылки, то «на всякий случай» скопирую это письмо сюда целиком:
Re: [zfs-discuss] ZFS vs ZFS + HW raid? Which is best?
Posted: Jan 13, 2009 6:16 PM in response to: kebabber
>>>>> "ok" == Orvar Korvar writes:
ok> Nobody really knows for sure.
ok> I will tell people that ZFS + HW raid is good enough, and I
ok> will not recommend against HW raid anymore.
jesus, ok fine if you threaten to let ignorant Windows morons strut around like arrogant experts, voicing some baseless feel-good opinion and then burdening you with proving a negative, then I'll take the bait and do your homework for you.
What people "don't really know for sure" is what hardware RAID you mean. There are worlds of difference between NetApp RAID and Dell PERC RAID, differences in their ability to avoid data loss in common failure scenarios, not just in some bulleted features or admin tool GUI's that the Windows admins understand. For example among dirt-cheap RAID5's, whether or not there's a battery-backed write cache influences how likely they are to corrupt the filesystem on top, while the Windows admins probably think it's for "performance issues" or something. And there are other things. NetApp has most of them, so if you read a few of the papers they publish bragging about their features you'll get an idea how widely the robustness of RAIDs can vary:
http://pages.cs.wisc.edu/~krioukov/Krioukov-ParityLost.pdf
Advantages of ZFS-on-some-other-RAID:
* probably better performance for RAID6 than for raidz2. It is not because of "checksum overhead". It's because RAID6 stripes use less seek bandwidth than ZFS which mostly does full-stripe writes like RAID3.
* It's performant to use a filesystem other than ZFS on a hardware RAID, which maybe protects your investment somewhat.
* better availability, sometimes.
It's important to distinguish between availability and data loss. Availability means when a disk goes bad, applications don't notice. Data loss is about, AFTER the system panics, the sysadmin notices, unplugs bad drives, replaces things, reboots, is the data still there or not? And is *ALL* the data there, satisfying ACID rules for databases and MTA's, or just most of the data there? because it is supposed to be ALL there. Even when you have to intervene, that is not license to break all the promises and descend into chaos.
Some (probably most) hardware RAID has better availability, is better at handling failing or disconnected drives than ZFS. But ZFS on JBOD is probably better at avoiding data loss than ZFS on HWRAID.
* better exception handling, sometimes. Really fantastically good hardware RAID can handle the case of two mirrored disks, both with unreadable sectors, but none of the unreadable sectors are in common. Partial disk failures are the common case, not the exception, while the lower-quality RAID implementations most of us are stuck with treat disks as either good or bad, as a whole unit.
I expect most hardware RAIDs don't have much grace in this department either, though. It seems really difficult to do it well because disks lock up when you touch their bad blocks, so to gracefully extract information from a partially-failed disk, you have to load special firmware into them and/or keep a list of poison blocks which you must never accidentally read.
My general procedure for recovering hardware RAIDs is, (0) shut down the RAID and use as JBOD, (1) identify the bad disks and buy blank disks to replace them, (2) copy bad disks onto blank disks using dd_rescue or dd conv=noerror,sync, and (3) run the RAID recovery tool. There are a lot of bad things about this procedure! It denies the RAID layer access to the disk's reports about which sector is bad, which leads to parity pollution, and silent corruption that slips through the RAID layer, discussed below. It can also not work, if the RAID is identifying disks like ZFS sometimes does, by devid/serialno rather than by a disk label or port number.
but RAID layers read the report "bad sector" as in fact a report "bad disk!" so with the very cheap RAID's I've used this 0,1,2,3 corruption-prone procedure which works around the bad exception handling saves more data than the supported procedure.
* novolatile cache. Some cheap hardware RAID gives you a battery-backed write cache that can be more expensive to get the ZFS slog way.
Advantages to ZFS-on-JBOD:
* unified admin tool. There is one less layer, so you can administer everything with zpool. With hardware RAID you will have to use both zpool and the hardware RAID's configurator.
* most hardware RAID has no way to deal with silent corruption. It can deal with latent sector errors, when the drive explicitly reports a failed read, but it has no way to deal with drives that silently return incorrect data. There are a huge number of patterns in which this happens in real life according to Netapp, and they have names for them like "torn writes" and "misdirected writes" and so on. RAID5's read-modify-write behavior can magnify this type of error through "parity pollution."
http://www.usenix.org/event/fast08/tech/full_papers/bairavasundaram/bairavasundaram.pdf
I think there are some tricks in Solaris for Veritas and SVM mirrors, which applications like Oracle can use if they do checksumming above the filesystem level. Oracle can say, "no, I don't like that block. Can you roll the dice again? What else might be at that lseek offset?" See DKIODMR under dkio(7I). I'm not sure in exactly what circumstance the tricks work, but am sure they will NOT work on hardware RAID-on-a-card. Also the tricks are not used by ZFS, so if you insert ZFS between your RAID and your application, the DKIODMR-style checksum-redundancy is blocked off. The idea is that you should use ZFS checksums and ZFS redundancy instead, but you can't do that by feeding ZFS a single LUN.
* hardware RAID ties your data to a RAID implementation. This is more constraining than tying it to ZFS because the Nexenta and Opensolaris licenses allow you to archive the ZFS software implementation and share it with your friends. This has bitten several of my friends particularly with "RAID-on-a-card" products. It is really uncool to lose your entire dataset to a bad controller, and then be unable to obtain the same controller with the same software revision, because the hardware has gone through so many steppings, and the software isn't freely redistributable, archiveable, or even clear that it exists at all. Sometimes the configurator tool is very awkward to run. There may even be a remedial configurator in the firmware and a full-featured configurator which is difficult to run just when you need it most, when recovering from a failure.
Some setups are even worse, are unclear about where they are storing the metadata---on the diskset or in the controller? so you could lose the whole dataset because of a few bits in some $5 lithium battery RTC chip. Being a good sysadmin in this environment requires levels of distrust that are really unhealthy.
The more expensive implementations hold your data hostage to a support contract, so while you are paying to buy the RAID you are in fact more like renting a place to put your data, and they reserve the right to raise your rent. Without the contract you don't just lose their attention---you cannot get basic things like older software, or manuals for the software you have now, and they threaten to sue you for software license violation if you try to sell the hardware to someone else.
* there are bugs either in ZFS or in the overall system that make ZFS much more prone to corruption when it's not managing a layer of redundancy.
In particular, we know ZFS does not work well when underlying storage reports writes are committed to disk when they're not, and this problem seems to be rampant:
* some SATA disks do it, but the people who know which ones aren't willing to tell. They only say "a Major vendor."
* Sun's sun4v hypervisor does it for virtualized disk access on the T2000.
* Sun's iSCSI target does it (iscsitadm. not sure yet about Comstar.)
* We think many PeeCee virtualization platforms like virtualbox and vmware and stuff might do it.
* Linux does it if you are using disk through LVM2.
The problem is rampant, seems to be more dangerous to ZFS than other filesystems, and progress in tracking it down and fixing it is glacial. Giving ZFS a mirror or a raidz seems to improve its survival.
To work around this with hardware RAID, you need to make a zpool that's a mirror of two hardware RAIDsets. This wastes a lot of disk. If you had that much disk with ZFS JBOD, you could make much better use of it as a filesystem-level backup, like backup with rsync to a non-ZFS filesystem or to a separate zpool with 'zfs send | zfs recv'. You really need this type of backup with zfs because of the lack of fsck and the huge number of panics and assertions.
HTH.
To my reckoning the consensus best practice is usually JBOD right now. When I defend ZFS-over-RAID5 it is mostly because I think the poor availability during failures and the corruption bugs discussed in the last point need to be tracked down and squashed.
Here's my list of papers that have been mentioned on this list, so you can catch up. You can also dump all the papers on the annoying Windows admins, and when they say "I really think hardware RAID has fewer `issues' because I just think so. It's your job to prove why not," then you can answer "well have you read the papers? No?
Then take my word for it." If they doubt the papers' authority, then cite the price the people writing the papers charge for their services---the Windows admins should at least understand that.
http://www.usenix.org/event/fast08/tech/full_papers/bairavasundaram/bairavasundaram.pdf
http://www.cs.wisc.edu/adsl/Publications/latent-sigmetrics07.ps
http://labs.google.com/papers/disk_failures.html
http://pages.cs.wisc.edu/~krioukov/Krioukov-ParityLost.pdf
http://www.nber.org/sys-admin/linux-nas-raid.html
_______________________________________________
zfs-discuss mailing list
zfs-discuss at opensolaris dot org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
![[identity profile]](https://www.dreamwidth.org/img/silk/identity/openid.png){kind=small}
Во, раз ты такой умный, ...
Date: 2009-01-23 07:57 (UTC)То есть приоритеты такие:
1. Надёжность.
2. Простота установки и, особенно, поддержки в рабочем состоянии
3. Быстродействие.
Re: Во, раз ты такой умный, ...
Date: 2009-01-23 15:18 (UTC)То есть твой выбор — это райд на мамке.
Но я это за решение, прямо скажем, не признаю — не гибкие, а главное — что будет, если мамка помрёт? Правильно — «50/50», как говорится, что данные твои уйдут в «страну доброй охоты». А в лучшем случае тебе придётся искать точно такую же мамку.
В общем, в контексте хранилища для домашних данных я сам последние год или полтора очень медленно и неохотно думаю о том, что вместо моего большого, мощного и тихого™ компа надо ставить ещё один — NAS. И ставить на него то ли линукс с софтверным райдом5, то ли фрю с zfs.
С другой стороны, важно, сколько у тебя данных. Если ты спокойно укладываешься в объём одного харда, то используй зеркало мазерборда совершенно спокойно: диск с половиной зеркала ты сможешь спокойно унести, куда угодно, и всё будет ок. Только рекомендую перед покупкой мамки или этого отдельного райд-контроллера сначала расспросить на форуме, или как-то у продавца заставить подтвердить, что этот райд можно диагностировать и чинить из операционки, а не только из его «биоса».
А то в теории-то всех их можно, а вот практика, как всегда, отстаёт. Часто программ для этого тупо нет, или они есть, но они не работают.
Re: Во, раз ты такой умный, ...
Date: 2009-01-23 15:55 (UTC)Кроме того, в нас надо тащить кабель. А это неприятно.
Ну ладно, в общем, асус пэ5ку, похоже, неотвратим.
Re: Во, раз ты такой умный, ...
Date: 2009-01-23 16:21 (UTC)Кстате, ты ведь в курсе, что все райды ценой примерно «до 500 усд», на самом деле нихрена не аппаратные: поддержка райда делается на уровне «биос понимает, где брать загрузчик, а драйвер в ядре делает всё остальное, используя остальной компьютер». То есть ты выбираешь между софтверным райдом, который точит сан или линукс сообщество на протяжении нескольких лет, и софтверным райдом, написанный за 2 недели индусами для кетайцiв, и используемый, кроме тебя, ещё 20 тысячами покупателей именно этого мазера, которые таки сподобились райд включить.
В общем, на мой взгляд, это дистиллированный случай сговора производителей: одни стали делать недо-райды, чтобы за их упоминание получить бонус у покупателей, а другие тут же (начиная с вин XP) выкинули поддержку программных нормальных райдов из ОС — «ибо нех, юзайте говно на мазерах, ил покупайте серверные версии». Потому что в Win NT и Win 2000 поддержка нормального софтового райда была.
Зато что делать, если оно «таки да»?
Софтверные райды гораздо гибче настраиваемы — в домашних условиях это может быть важно. Например, включить в одит райд 4 диска разных размеров, при этом «торчащие» куски использовать без райда, или в райде другого уровня.
Опять же, «если что», его можно и перестраивать всячески — менять уровень райда, количество дисков и размеры областей, причём в обе стороны. Такой функциональностью даже дисковые массивы с ценами «от 30 000 USD за ящик без дисков» не многие обладают, а дома, повторюсь, ситуация «зачем тебе перестраивать, просто купи ещё одну полку дисков» решается гораздо труднее, чем там, где бюджеты с 5–7 нулями.
В общем, пока ёмкости одного диска хватает — используй любой встроенный райд в зеркале, и всё будет более-менее нормально. Если зеркала не достаточно — то дешевле ~500 USD только за контроллер внутрикомпьютерных вариантов для винды мне не известно.