When the web breaks

AWS errorsIf you weren’t aware of it already, there was a problem on the internet yesterday for a couple of hours. A little problem in terms of describing it: Amazon Web Services (AWS) was “having problems”. However, it was really a big problem for any app or service relying on the S3 services. One of those services was us. Our downloads were unavailable for a period of time. Slack, Trello, Time Inc., and others were affected. Some even experienced some home automation problems because they relied on the IFFFT website.

This encouraged discussion about the dangers of relying on the cloud so much. Sure the promises sound all well and good, but what if the problem had not been solved in a matter of hours? How would you have gotten to your data? What if your DEVONthink databases only lived on the servers that went down and Amazon couldn’t bring them back? And what if that data was critical to your business, your academic pursuits, or even information vital in saving someone’s life?

This is part of why we don’t support storing everything solely in the cloud and fully advocate and support local resources. DEVONthink databases are stored locally*. If you are indexing files, they should be locally available, not existing only in the cloud. And we strongly suggest a primary backup strategy with local media.

But: “Hard drives are old-fashioned. It’s the 21st century! Everything is in the cloud!” First, no, not everything is in the cloud. And secondly, always keep this question in mind: “What would happen if someone pulled the plug on the cloud right now?” Yesterday proved that it’s a question worth asking.

* Here is our oft-repeated warning: You should never put your databases in any cloud-synced folder, including the Documents or Desktop folders if you are using the disk management feature in macOS 10.12 Sierra. ​Never. If you have, you should ​relocate them or you could irreparably damage them. If you use Sierra’s iCloud features the safest location is a folder in your home directory, like ​~/Databases. If you have them in an unsafe location, quit DEVONthink and move your databases to a new folder in your home directory.

11 Responses to “When the web breaks”

  1. David Parker says:

    Thanks, Jim. Sound advice that we should all take note of.


  2. Stefanie says:

    Thanks Jim for advice.

    For private people it makes therefore sense to have your own cloud running. That is one reason I am very passionate about the own cloud because people are just relying on their own setup and responsibility.



  3. Peter Terlingen says:

    The Cloud has only value when its always accessible.
    There are many regions in the world where the coverage of networks is bad.
    Travelling in the Netherlads, Belgium, France, Spain and Italy I prefer to rely on my harddisk.

  4. Jim Neumann says:

    @David: Thanks. It’s not talked about nearly enough nowadays.

    @Stefanie: That can certainly be a good (and private) alternative!

    @Peter: Agreed! There is a phenomenon I call “the myth of network persistence”, the idea that the cloud and network is ever present no matter where you are. I can travel less than 15 minutes from a city of about 75,000 people (not huge, but not tiny) and be off-the-grid.

  5. Greg says:

    I have my databases in my Documents folder and they are backed-up to the Cloud using Crashplan. Is this what you mean by having the databases in a Cloud-synced folder because I interpret it differently.

  6. Philip says:

    Dear Jim,

    I have my database(s) stored locally on my laptop. As a “backup”, these are synched via WebDAV to my NAS device, which will be backed up – in the near future – to a second NAS. Of course, TimeMachine backups are also regularly performed. If this may appear quite paranoid (or dumb) in your eyes – which of these would you stop doing first? (The reason for the WebDAV “copy” is that I can also add items using DevonThinkToGo…



  7. Robin says:

    Thanks, Jim, for this brief treatment of a vital topic. Could you perhaps point us all to the relevant sections of DT(PO)’s documentation or even do a mini tutorial series on the various options – there are several best solutions, depending on individual requirements (e.g. do I use DTTG or not)?
    Best Regards

  8. Jim Neumann says:

    @Greg: The Documents folder is okay as long as you’re no using the disk management feature of Sierra.

    @Philip: Doesn’t sound paranoid or dumb at all. It sounds like you have a good strategy to me.

    @Robin: Thanks for the comment. A tutorial on what specifically?

  9. Robin says:

    @Jim: Thanks for responding. I guess I’m thinking less of a tutorial and more of an overview discussion of the merits of different solutions for cloud or non-cloud storage and/or syncing of DT – taking the issue of your post into account. That would include the do’s and don’t’s but also point to the relevant sections of the manual/help files to provide the how-to’s as well (hence “tutorial”. It might cover different use-scenarios, such as with or without DT To Go, need to access from multiple (macOS) devices or only from one, and so on, and treat the related but different issues of syncing and backing up. Does that make sense?

  10. Barry says:

    Your point is well taken Jim. Perhaps people forget that in reality their data, when stored in the “cloud” is actually sitting on someone else’s server which is always a physical location and subject to all the risks of any physical location. The cloud provides the links so you can have access to that location where your data is stored, and of course if that access is compromised then it has the same effect as the servers being down. The links we call the cloud also rely on physical infrastructure. So there are two major risks to information in the cloud (ignoring for the moment the hacking risk) one being the failure of the server(s) and the other being the links to access the server(s). So keeping local backups is still very good advice.
    Having lost my data severaI times over 20 years through hard disk failure, I now use a combination approach. All my documents are synced to dropbox and I have at least two computers ( a laptop and a desktop) synced to that dropbox ensuring there are always 3 physical locations where my critical data is stored. Then I run full hard drive backups on each Mac periodically to capture not only the documents (giving me a 4th copy) but all of my apps, OSX and settings for a quick and full recovery for when my computer hard drive fails or I upgrade the computer. On top of that irreplaceable archive files are burned to DVD and stored in a safe.
    Sometimes you have to lose everything to understand the pain of trying to reproduce or replace it.

  11. Jim Neumann says:

    @Barry: Well said, and definitely sage advice. It is always heartbreaking when I get a support Ticket form someone and mention backups and they say, “I’ve been meaning to start doing backups but I need help NOW!” /cry