The short answer: When an array ejects a drive that comes back after a reboot and reads normally, a timeout mismatch is usually to blame. The Linux kernel gives up waiting after thirty seconds, while a desktop drive keeps retrying a problem sector for more than two minutes — and in the meantime the array declares it dead. That is why drives built for NAS use have error recovery limiting switched on (ERC at Seagate, TLER at Western Digital) and let themselves be interrupted after seven seconds. On a drive that cannot do this, the timeout is raised on the system side instead. Before you configure anything, though, you have to know whether the drive dropped out because of the waiting, or because it really is failing.
What actually happened in that array
The Linux RAID wiki describes the correct sequence like this: a drive hits a sector it cannot read on the first attempt, „tries harder" for a while, then gives up and returns an error to the system. The RAID code recalculates what the data should have looked like and writes it back to the drive. A hiccup like this is normal operation and, as long as the drive is not dying, it resolves itself — a read failure on one sector does not mean the same spot cannot be written to.
It falls apart the moment the drive does not return the error in time. The kernel waits thirty seconds and then gives up. The RAID code recalculates the data and wants to write it back, but the drive is still trying to read and does not respond — so the array concludes it is dead and ejects it from the volume. The wiki puts it bluntly: this is how a single read error on drives like these can kill an entire array.
Yet the drive is behaving exactly as designed. A lone drive in a computer has nowhere else to get the data from, so trying to read for as long as possible is the right thing to do. In a redundant array it is the other way round: the data lives elsewhere too, and a drive that holds a request for two minutes is worse for the array than a drive that quickly admits failure. The very same piece of hardware is therefore „fine„ in a PC and „faulty" in a NAS.
How to recognise that this is exactly what is going on
A timeout mismatch has several fairly distinctive signs:
- the drive vanished from the array all at once, not after weeks of gradually worsening performance,
- after a reboot, or after being pulled out of the bay and pushed back in, it reports back and can be read from normally,
- SMART on the ejected drive does not look like SMART on a drive that is dying (the article how to tell that an HDD is dying explains how to distinguish the two),
- the array contains desktop drives, or at least one such drive that somebody added when expanding it.
One question is decisive, though: can the drive limit its recovery time at all? smartctl -l scterc /dev/sda will give you the answer. If the output shows that the drive does not support SCT Error Recovery Control, you have your cause on the table.
The combination with SMR technology is particularly nasty. The wiki states that SMR drives stall for more than ten minutes while they reshuffle data, so no sensible timeout setting will save them. Whether your model belongs to this group is addressed by the CMR and SMR overview.
Which drives have the limit and which do not
Here it pays not to rely on what is „common knowledge", but on what the manufacturer has written in the documentation.
| Drive series | What the manufacturer says about the limit | Source |
|---|---|---|
| WD Red, WD Gold, Ultrastar | TLER is enabled, recovery is limited to seven seconds and cannot be disabled | WD support, TLER on RAID edition internal drives |
| WD Red Plus | The product brief describes NASware tuning for RAID NAS, but the word TLER does not appear in it — on a specific unit the state needs to be verified with smartctl | Product brief WD Red Plus |
| WD Red Pro | The same: the datasheet talks about NASware and stays silent about TLER — verify on the unit | Data sheet WD Red Pro |
| Seagate IronWolf | The specification table has a dedicated „Error Recovery Control" row with the value Yes for every model | Datasheet IronWolf |
| Seagate IronWolf Pro | According to the datasheet, the AgileArray firmware enables dual-plane balancing and time-limited error recovery | Datasheet IronWolf Pro |
| Cheap desktop series | Most modern cheap desktop drives do not allow error recovery to be configured | Linux RAID wiki, Timeout Mismatch |
Mind what the table does not say. The fact that a series has the limit does not yet mean that your particular unit has it set to a sensible value — the wiki warns that even older drives supporting ERC often have unsuitable settings. The reliable source of truth is the output from the drive sitting in your bay. When choosing new units, the comparison of WD Red, IronWolf and Toshiba N300 will help.
What can be configured
On a Linux server with mdadm there are essentially two levers, and which one you use depends on what the drive is capable of.
The drive supports ERC. You set the recovery time to seven seconds:
smartctl -l scterc,70,70 /dev/sdaThe values are in tenths of a second — this is the most common source of confusion, because 70 looks like seventy seconds. The smartctl manual states it precisely: the READTIME and WRITETIME arguments are in tenths of a second, a value of 0 disables the feature, values below 65 are probably not supported, and for RAID 70,70 is the typical setting. Twenty seconds would therefore be scterc,200,200.
By default the setting is volatile. Newer smartctl supports the ,p option, after which the values survive a power cycle — but it requires a drive conforming to ATA ACS-4 or newer. For all other drives, what the wiki recommends applies: run the setting at every boot, because cheaper units forget everything when the system is switched off.
The drive does not support ERC. Then the timeout is raised on the kernel side, so that it is longer than what the drive allows itself:
echo 180 > /sys/block/sdX/device/timeoutThree minutes is enough even for a desktop drive that stalls for more than two. The value in /sys does not survive a reboot, so this belongs in a startup script as well.
And one trap the wiki flags separately: on drives that do not support SCT at all, the attempt to set ERC does fail, but returns a zero exit code. A script that goes purely by the exit code therefore evaluates such a drive as being fine. So read the value back after every change you make.
Synology and QNAP: what the administrator has to hand
On off-the-shelf NAS boxes you do not play with timeouts — you work with what the manufacturer’s system offers.
On Synology, a deactivated drive puts the entire storage pool into a degraded state. The pool stays accessible, because no data has been lost. Reactivating a deactivated drive is simple according to the manufacturer’s KB: pull it out of the bay, wait a few seconds and push it back in. Before you order a new drive, go through the official procedure for telling a faulty drive from a faulty bay — this is exactly the situation in which hardware gets bought for nothing. What the individual DSM messages mean is covered by the article Storage Pool Degraded or Volume Crashed.
On QNAP, the RAID group status changes to Error when drives are disconnected, and volumes as well as storage pools drop into an Inactive state. The QTS documentation describes recovery through Storage & Snapshots and the Recover RAID function, with one condition that cannot be worked around: every drive has to go back into its original position. It also warns that this recovery only helps with temporarily disconnected drives, not with an actual drive failure. QNAP RAID Group Degraded or Volume Read-Only follows on from this.
When it is no longer a timeout but a defect
A timeout mismatch is only one of the explanations. Do not put an ejected drive back into the array blindly and do not start a rebuild until you know why it dropped out. If an unreadable spot really is behind it, the rebuild reads the entire surface of the other drives, and that is exactly the load under which a second drive gives out. The mechanics of that failure are covered by why a rebuild kills RAID arrays.
Leave the array alone if the drive clicks, disappears from the list even without load, drops out repeatedly at the same spot, or if more than one drive is already reporting a problem. Synology has its own diagnostic procedure for a growing number of bad sectors and it is worth going through it before anything gets started. When it is safer to get the data out of the array is described by how to get data out of a NAS and data recovery from a RAID array.
Frequently asked questions
Why does the drive come back after a reboot if it is „faulty"?
Because it does not have to be faulty. The array ejected it for not answering in time, not for being unable to read. That does not mean everything is fine, though — next time it will happen again.
Can I set ERC on a drive that the output says does not support it?
No. What remains is to raise the kernel timeout to 180 seconds and accept that on a read error the array will wait for minutes. The permanent solution is to replace it with a unit that has the limit from the factory.
Does this apply to SMR drives as well?
For SMR it is not enough. According to the wiki, SMR drives stall for more than ten minutes, and a script changing the timeout will do nothing about that.
How do I find out whether the problem is in the drive or in the bay?
Move the drive to a different position and watch whether the error travels with the drive or stays with the bay. Synology has its own guide for this and it is sensible to go through it before you order a replacement.
Can I set ERC manually on an off-the-shelf NAS?
The commands above assume shell access on a Linux server. On an off-the-shelf NAS box it is more practical to choose drives with the limit enabled from the factory and to stick to the manufacturer’s diagnostics.
Sources
- Linux RAID wiki, Timeout Mismatch (archived page, last edited 19 May 2022) — https://archive.kernel.org/oldwiki/raid.wiki.kernel.org/index.php/Timeout_Mismatch.html
- smartctl(8), the
-l sctercoption — https://manpages.debian.org/bookworm/smartmontools/smartctl.8.en.html - Western Digital, TLER on RAID edition internal drives cannot be disabled — https://support-en.wd.com/app/answers/detailweb/a_id/3562/~/tler-on-raid-edition-internal-drives-cannot-be-disabled
- Seagate, IronWolf datasheet — https://www.seagate.com/www-content/datasheets/pdfs/ironwolf-18tb-DS1904-20-2111US-en_US.pdf
- Seagate, IronWolf Pro datasheet — https://www.seagate.com/files/www-content/datasheets/pdfs/ironwolf-pro-20tb-DS1914-19-2112US-en_US.pdf
- Western Digital, product brief WD Red Plus — https://documents.westerndigital.com/content/dam/doc-library/en_us/assets/public/western-digital/product/internal-drives/wd-red-plus-hdd/product-brief-western-digital-wd-red-plus-hdd.pdf
- Western Digital, data sheet WD Red Pro — https://documents.westerndigital.com/content/dam/doc-library/en_us/assets/public/western-digital/product/internal-drives/wd-red-pro-hdd/data-sheet-wd-red-pro-hdd.pdf
- Synology, managing drives in Storage Manager — https://kb.synology.com/en-ro/DSM/help/DSM/StorageManager/disk
- Synology, repairing a storage pool — https://kb.synology.com/en-us/DSM/help/DSM/StorageManager/storage_pool_repair?version=7
- Synology, how to tell a NAS problem from a drive problem — https://kb.synology.com/en-us/DSM/tutorial/How_do_I_know_whether_the_problem_is_with_my_Synology_NAS_or_drive_when_drive_errors_occur
- Synology, bad sector warnings — https://kb.synology.com/en-global/DSM/tutorial/How_to_diagnose_drives_health_status_when_receiving_bad_sector_warning
- QNAP, Recovering a RAID group with an Error status (QTS 5.2.x) — https://docs.qnap.com/operating-system/qts/5.2.x/en-us/recovering-a-raid-group-with-an-error-status-7E19C750.html
Is a drive dropping out of your array and you do not know whether it is faulty or merely slow? We will go through the logs and SMART output with you before a rebuild starts — as part of IT management for companies or as a one-off IT audit. If the array has already gone down, data recovery from RAID and NAS follows on. Get in touch via contact, by phone +420 775 556 063 or by e-mail info@ithope.cz. For now, do not pull the drives out of their bays and above all do not swap them around — the order is crucial during recovery.
Where to find us
ITHOPE is a company based in Brno, Czechia: we do data recovery from disks, RAID and NAS, PC and laptop service and IT for business, all in our own lab at Viniční 3587/216, 615 00 Brno.
Phone +420 775 556 063 (24/7 line). You can bring the device in person, or send it by courier (Zásilkovna, PPL).