TS-1685 keeps shutting down or rebooting
I installed my QNAP TS-1685 a week ago. I installed 10 of HGST 10TB Deskstar HDDs and made a volume using RAID 6. All seemed to be fine until I started sync from aws S3 using hybrid backup sync. Everytime after sync started, the system shuts down or reboot. I updated firmware from 4.3.3.xxxx to 4.3.4.0675. After first update it said that the firmware in the flash and the HDDs don't match so I manually updated firmware again. Since NAS turned of during sync, the system said that file system is not clean, so I deleted volume and re created. After last firmware update, it seemed the issue was resolved, it was much more stable than before, however it rebooted again. All the HDD SMART info was good. I synced about couple hundreds GB of video files, some of them are several GB and others are less than 1GB. Sync job was set 1 files at a time. When sync started RAID group synchronizing was being performed. I couldn't wait for 60+ hours to finish synchronizing. I contacted QNAP support with this issue and they suggested that I use NAS after RAID sync. So I finished RAID sync but somehow auto scheduled RAID scrub was started and system shut down again. Because system shut down during scrub, it always try to scrub from scratch and shut down randomly after around 1 hour after booting. And I also see the "The file system is not clean" warnings. Sometimes it stuck in the booting. Then I found a reddit post that there is some problem with firmware. So I decided to updated firmware without HDDs since RAID scrub seems to be a problem. But now it won't boot at all.
QNAP support is not very helpful so I post this here.
Any advice would be much appreciated.
My first stupid question -
do you have a UPS on your system. If you have unstable power, and you have momentary power loss, the QNAP (and anything else) will shut down.
I never use RAID scrubbing. This was done because of an old firmware issue that QNAP had a long time ago. It really doesn't apply anymore, and dramatically slows down the system, so I always disable RAID scrubbing. Once the drives are stable (after optimization) - you should be fine. And if you created a RAID (like a RAID 5 or RAID 6) - even if you have a drive failure, you should have no issues.
Certainly - you could have a DEFECTIVE power supply in the QNAP, and this would cause the issue of you losing power to your QNAP. I have installed over 100 TS-1685 systems without issue, but this does not mean that there is not a defective power supply in your system. Of course, I would
#1 - make sure you have a UPS on your system - preferably one that can communicate with your QNAP via the USB port, in case of power loss - like a Cyberpower or APC UPS
#2 - make sure you have good AC power cables going to the QNAP
#3 - make sure that the AC outlet strip that you are using to power the QNAP is good, and that the plugs are not intermittent. Many cheap AC outlet strips (and wall outlets) are loose and defective. You should check this (make sure that you don't lose power when you wiggle the cable in the outlet.
But if all of these things are ok, and you are using a UPS, and the problem still exists, then YES, you probably have a defective power supply. If you contact QNAP, you have a WARANTEE, and they will replace the QNAP for you (you keep your drives, and insert them in the new QNAP). If you simply cannot get your system to boot, because you lost power during a firmware update (or downgrade) - then
#1 - make sure all your power is good for the QNAP, so this doesn't happen again, and
#2 - call QNAP. In the US, the support # is (909) 595 2782. They can manually download the firmware to your desktop, and SSH into your QNAP via terminal, to manually update the firmware.
Are you in the US ?
Rescue 1, Inc.
Tentatively chipping in...
Scrubbing will push the system hard because it has to read every sector on every HDD in order to verify integrity. So like Bob says that could be highlighting a weak PSU.
You could also have a HDD issue if the machine chucks the towel in after about the same uptime. It depends how bad it is, but I've seen storage go nuts if a drive is seriously acting out of turn. Doesn't matter if it's ZFS, hardware RAID, MD - this can cause horrible problems. Some systems are just better at this than others at this.
Could also be thermal or memory, but both seem unlikely - especially if your system uses ECC.
If you can get in to the QNAP via SSH try running the
NB: I have no idea if dmesg is installed on a QNAP, but it is a Linux system, so I expect so.
Some more info here...
As for scrubbing, I'd schedule this at least monthly as a weekend task. Thats my view, feel free to ignore.
The 1685 is connected UPS, when it shut down other devices were fine I can say UPS is fine. I deleted my RAID 6 volume last Friday night and the NAS was up and running during whole weekend without any issue. So it is unlikely the PSU is a faulty one. Yesterday I updated latest firmware, 4.3.4.0695 and this morning the RAID sync was done without issue. So I synced 4 video files from AWS S3 and it also went well. But when I rename one of the video files, it suddenly turned off again. I could rename the file after reboot, but I don't think I can rely on this machine.
I am a software engineer working at a VR company located in Silicon Valley, but our production team is in LA so I went to LA and installed the system. Now that the system became unstable, I came to LA again yesterday and have been trying to fix the issue. But I think I am stuck again.
I just found out that one specific file was the problem. If I try to rename it, the system goes down. Is it a sign of HDD fault?
Hi Sungwoo -
I have heard of all kinds of problems in my day - but this the first time in my life, that I ever heard of the power supply shutting off, when you rename a file ! That's a new one on me - and while I believe you - I find it hard to believe that this is actually happening. You can have EVERY DRIVE FAIL in the QNAP (and every other system) and the QNAP (and every other system) will continue to run. At that point, you would have lost your data, but the QNAP (and every other system) would not SHUT DOWN the power.
All of this makes no sense to me. I have never heard of a problem like this before - on a QNAP, or any other system.
Corrupt files cause systems to lock up and "beach ball" - but they don't cause the power supply to fail, and shut off !!!!!
Sorry, but I don't have a better answer for you on this one.
If you want me to remote in and take a look, I will be happy to do so.
Rescue 1, Inc.
reply from Bob Zelin again - I just re read your original post.
You write -
All seemed to be fine until I started sync from aws S3 using hybrid backup sync. Everytime after sync started, the system shuts down or reboot. I updated firmware from 4.3.3.xxxx to 4.3.4.0675.
REPLY - you are doing a cloud sync using MyQnapCloud or Hybrid Backup Sync. This is when you are having these issues. Your "corrupt" file is probably because you have special characters in the file name (or file names).
In a Linux system, you cant have \ / # $ . (that's a period) in the file names. So sungwoo.kang.mov will cause issues. Even with that, this will not "shut off" the power supply.
You write -
After first update it said that the firmware in the flash and the HDDs don't match so I manually updated firmware again.
REPLY - again, this does not make sense, or I am mis reading what you wrote. The firmware for the SATA hard drives has got nothing to do with the firmware of the QNAP. With that said, you MUST be using approved hard drives for your QNAP, that are compatible. I recently had a client that bought expensive enterprise SAS drives, and they did not work with the QNAP.
You write -
Since NAS turned of during sync, the system said that file system is not clean, so I deleted volume and re created.
REPLY - the ONLY time that you get the error message "file system is not clean" is if you LOSE power to your system.
This means that it did not REBOOT - it LOST POWER, and then you get this error message "file system is not clean".
No corrupt file or defective drive is going to SHUT DOWN THE POWER from the QNAP. I am not saying that you don't have a bad power supply in the QNAP, but a bad file does not turn off the power.
After last firmware update, it seemed the issue was resolved, it was much more stable than before, however it rebooted again. All the HDD SMART info was good. I synced about couple hundreds GB of video files, some of them are several GB and others are less than 1GB. Sync job was set 1 files at a time. When sync started RAID group synchronizing was being performed. I couldn't wait for 60+ hours to finish synchronizing. I contacted QNAP support with this issue and they suggested that I use NAS after RAID sync. So I finished RAID sync but somehow auto scheduled RAID scrub was started and system shut down again.
REPLY - RAID Scrub is auto enabled on all new firmware builds. To go to the Storage and Snapshots icon, click on the Gear icon, and disable RAID SCRUB. RAID SCRUB will dramatically slow down your system when it is doing this process - but it certainly won't SHUT DOWN your QNAP. Only a bad power supply (or bad power source) will do this.
Because system shut down during scrub, it always try to scrub from scratch and shut down randomly after around 1 hour after booting. And I also see the "The file system is not clean" warnings.
REPLY - again, this happens when you LOSE POWER, not when you reboot. You are losing power.
Sometimes it stuck in the booting. Then I found a reddit post that there is some problem with firmware. So I decided to updated firmware without HDDs since RAID scrub seems to be a problem. But now it won't boot at all.
this was from your first post. I have seen and replied to your comments on your last post as well.
A corrupt file will not cause the QNAP (or anything else) to power off. Only a bad power supply, or a bad power source will do this. I know that you have already spoken with QNAP support in California - didn't they ask for your system log dump on this system ? You CANNOT use special characters in your file names. When I build systems, I always (on Mac's) disable DS_Store with defaults write com.apple.desktopservices DSDontWriteNetworkStores true, so the Mac does not generate ./ in front of folder names, which will halt a large file transfer from a drive to the QNAP. If you have lots of $ # \ / . in your file names, this will cause problems syncing from S3 to the QNAP.
I am here to respond to any other comments that you may have, and you can also contact me.
Rescue 1, Inc.
Thanks for your comment Bob!
Since S3 bucket uses linux also, it has strict file name restriction, too. So sync from S3 to NAS should be fine.
Actually, a QNAP support guy connected my laptop through Teamview and look into my NAS. And we reproduced the shut down. All we did was launch a Filestation and go to the folder that has 4 video files and refresh the Filestation. After I turned it on again, RAID sync started. The support team suggested that after sync finished, connect the NAS to the direct power source.
I did the test this morning and it worked well. I found that when the file station making the thumbnails of videos, CPU utilization goes 99%. That means high power consumption so I assumed UPS is faulty. So I tested UPS with 2 laptops running full load test, but it could handle up to 380W which is more than the QNAP NAS and switch's power consumption.
Anyway, since it's working well and I need to comback home today, I decided to let our editors use QNAP without UPS until we get another UPS. If you feel I miss something or have some other advice, please let me know.