Самотестирование диска с помощью smartctl
Устанавливаем необходимый пакет smartmontools
Существуют два вида тестов — foreground (когда диск в ответ на все команды имеет статус CHECK CONDITION) и background — когда диск остается работоспособным.
Тесты бывают short и long.
Short — ограничен по времени и тестирует только часть диска.
Long — аналогичен финальному тестированию на производстве, не ограничен по времени и тестирует весь диск.
Запускаем “длинный” тест, который можно запускать прямо во время работы системы:
Узнать сколько времени займет тот или иной тест можно командой:
После того как он закончится смотрим результат с помощью команды smartctl -l selftest /dev/sdb:
Также можно протестировать только часть диска. Команда:
протестирует сектора с 10 по 20 включительно.
Кроме того, можно одной командой протестировать несколько диапазонов:
Запустить foreground тест можно, указав опцию -C:
Короткий foreground тест, который принудительно обновит значения параметров SMART можно запустить так:
Утилиты sg_verify и sg_reasign
Утилита sg_verify предназначена для работы с дисками, поддерживающими полный набор команд SCSI. В настоящее время — это диски с интерфейсом SAS (но не SATA). Обнаружить нечитающийся сектор можно, например, с помощью команды dd:
Затем, когда dd сообщит об ошибке, с помощью sg_verify можно удостовериться, что проблема именно в том секторе:
Зетем можно проверить, что размер GLIST не очень велик и там есть место:
Теперь можно выполнить переназначкение сектора и проверить, что размер GLIST изменился:
Размер GLIST изменился на 1 — чего и следовало ожидать. В результате в переназначаемом секторе будет записан заданный производителем паттерн (либо — если сектор все-же удалось прочесть — его содержимое не изменится).
Также утилита sg_reassign может выполнить переназначение группы секторов.
Для дисков с интерфейсами SAS и SATA наборы команд для работы с переназначаемыми секторами различны. Диск с интерфейсом SAS (неважно подключен он к SATA или SAS контроллеру) способен безусловно переназначить группу секторов в резервную область независимо от того, читаемы сектора или нет. Использование sg_reasign для диска SATA скорее всего никакого эффекта не даст.
Переназначение секторов для дисков с интерфейсом SATA
Для переназначения секторов на дисках с интерфейсом SATA можно просто дать команду на перезапись этого сектора, либо воспользоваться hdparm:
Принципиальная разница между дисками SAS и SATA в том, что случае SAS можно принудительно переназначить даже хороший сектор, а в случае SATA переназначение при записи принимает контроллер диска. При этом,этот вариант годится для обоих интерфейсов:
или для группы секторов:
Аналог MHDD под Linux
С удивлением обнаружил наличие под Linux некоторого аналога (по крайней мере визуально) популярной программы под DOS — MHDD.
Речь идет об утилите, входящей в состав SystemRescueCd — whdd.
How to Fix/Repair Badblocks in Linux
The badblocks in a storage device are the portions of the device that are not readable for some reason. These badblocks can be recovered if we are able to reach the exact location of that block.
The SMART technology built into many storage devices (such as many ATA-3, later ATA, IDE and SCSI-3 hard drives) monitors the reliability of hard drive and can predict drive failures. The SMART stands for Self-Monitoring, Analysis and Reporting Technology.
This tutorial describes the actions that can be taken when smartmontools detect and report some badblocks on hard disk running linux.
About Smartmontools
The smartmontools package provides two utilities: smartrd and smartctl.
smartd is the deamon that polls the ATA and SCSI devices every 30 minutes (this value can be changed) and logs the SMART errors and changes in SMART attributes using SYSLOG interface.
The smartctl performs SMART tasks and can be used to print SMART self-tests and error logs among other tasks such as support for polling TapeAlert messages from SCSI tape drives. The usage of this command will be clear as we proceed through this article. This article proceeds through some examples of disk failure for different types of filesystem.
smartctl command
The smartctl command reports a badblock at Logical Block Address LBA = 0x016561e9 which in decimal number system is 23421417.
The LBA counts sectors in units of 512 bytes starting at zero. The value of «Current_Pending_Sector» attribute in «smartctl -A» command confirms the bad sector.
Fix badblocks
Step 1:
Locate the partition on which the bad block resides. The fdisk command can be used to view the sectors of the hard disk partitions.
Here we can see that the LBA 23421417 lies in the third partition, i.e. /dev/hda3. The offset value of the sector is 23421417 — 5269320 = 18152097 sectors in the partition /dev/hda3.
Now we need to check the type of filesystem of the partition. This can be checked from /etc/fstab file.
Step 2:
Now we need to find the block size of the filesystem using tune2fs command
This reports the block size to be 4096 bytes.
Step 3:
Find the filesystem block that contains this problematic LBA. We use the following formula:
b = File System block number
B = File system block size in bytes
L = LBA of bad sector
S = Starting sector of partition as shown by fdisk -lu
and (int) denotes the integer part.
For our example, L=23421417, S=5269320, and B=4096.
b = (int)18152097*512/4096 = (int)2269012.125
Step 4:
Use debugfs to locate the inode stored in this block, and hence the file that is stored at that location.
Here, in this case, the block is not in use. So the rest of this step can be skipped and we can jump directly to next step. Otherwise, if the block is in use, as reported by the following output:
In this case, the problematic file is: /data/S1/R/H/714197568-714203359/H-R-714202192-16.gwf
In case of ext3 filesystem, this block can be the part of journal itself. The inode will be very small and debugfs will not be able to report any filename.
In this case, we can remove the journal with tune2fs command:
Now, we repeat the step 4, and if the problem is not reported anymore, we can rebuild the journal:
Step 5:
This step will destroy the data on that block by writing zeroes on it. The bad block will be recovered but the data of the file will be lost. If you are sure, you can proceed with the following step:
Now we can again check the «smartctl -A» output to verify that everything is back to normal.
Here you can see that the value of «Current_Pending_Sector» is zero.
Smart Error on Ext2/ext3
This email from smartd shows the first sign of trouble. As talked about in the previous example, we run «smartctl -a /dev/hda» to confirm the problem:
The LBA reported is 0x021d9f44 (base 16) = 35495748 (base 10)
Here, 3 sectors are unreadable. Using the following bash script, we can check the sectors around that area.
35495734
1+0 records in
1+0 records out
35495735
dd: reading `/dev/hda’: Input/output error
0+0 records in
0+0 records out
35495751
dd: reading `/dev/hda’: Input/output error
0+0 records in
0+0 records out
35495752
1+0 records in
1+0 records out
This shows that 17 sectors 35495735-35495751 are unreadable.
The filesystem blocks that contain this area are:
L=35495735 to 35495751
S=5269320
B=4096
so b=3778301 to 3778303
To identify files at these locations, we run debugfs:
We can use md5sum to confirm our file:
So, we force the disk to reallocate the bad blocks:
Now we can check if the bad block are creating no trouble with the value of «Current_Pending_Sector» attribute (smartctl -A command):
Unassigned sectors
In the above examples, we have not considered the case when the bad blocks are not assigned to any file. This will be clear when we run debugfs command to find the file corresponding to a particular block. If this is the case, then we can first create a file that is large enough to fill the remaining filesystem (again, dd command is the rescue).
This command will run until there is no space left on the filesystem. And now we can proceed through rest of the steps.
BadBlock on ReiserFS
In this example, the filesystem used is ReiserFS. So, some of the commands used will be different from the above case.
The SMART error log indicates the bad block address to be 58656333. The partition table indicates that the block is in a partition with ReiserFS filesystem, starting at block address 54781650.
Get the block size of filesystem
Get the block number
More information about the block
The problem has occurred looks like a hardware problem.
Here, we see that reading the block fails. But we now know from this output that it is unused block.
At this point, we can try to write the bad block and see if the drive remaps the bad block. If it cannot remap the block, use badblock option (-B) with reiserfs utils to handle this block correctly.
bread: Cannot read the block (484335): (Input/output error).
At least we have the correct bad block.
Find the affected file
Run badblock -n to provoke reallocation
If everything happens as expected, debugreiserfs -1 484335 /dev/hda3 reports no errors. Otherwise:
Use dd command to write zeroes on the particular area
Badblock Repair LVM
This example considers the badblock to be on an LVM volume:
An error is reported and the bad block is found to be at LBA 37383668 with the following command:
sfdisk can help to find the physical partition of the bad block:
The bad block is in /dev/hdb3 partition, which is an LVM based partition. The offset of this block is: (37383668 — 1188810) = 36194858
The physical partition used by LVM is divided into PE (Physical Extent). The ‘pvdisplay’ command gives the size of PE of the LVM partition:
# part=/dev/hdb3 ; pvdisplay -c $part | awk -F: ‘
4096
To get its size in LBA block size (512 bytes or 0.5 KB), we multiply this number by 2 : 4096 * 2 = 8192 blocks for each PE.
Now we search the PE in which the bad block is residing: physical partition’s bad block number / sizeof(PE)
36194858 / 8192 = 4418.3176
Now we need to find the logical partition corresponding to PE number 4418.
Hence, the PE 4418 is in /dev/WDC80Go/ext1 logical partition.
Size of logical block of filesystem on /dev/WDC80Go/ext1 is
The logical partition starts on PE 3072:
(# PE’s start of partition * sizeof(PE)) + parttion offset[pe_start] = (3072 * 8192) + 384 = 25166208
There are 512 blocks of physical partition, so the bad block number is:
(36194858 — 25166208) / (sizeof(fs block) / 512) = 11028650 / (4096 / 512) = 1378581.25
You can verify if this is the actual bad block with dd command:
If the command issues some error, then the calculation for the bad block is correct. Otherwise, recheck the calculations to find the correct block. Once you have found the correct block, resolve the issue with dd command as explained in all above examples:
Conclusion
All the examples given in this article concentrate on finding the correct bad block and the partition. Once you have found the bad block, all you need to do is to run dd command to write zeroes on the block. The different examples provide the methods of finding the bad block location in different filesystems and in different scenarios.