[Anthony Aufdenkampe]

@neilh, do a hard reset of your browser cache our URL to get TSV working on Chrome. That is definitely the issue there. We’ll be adding “cache-busting” code in our next update.

[Anthony Aufdenkampe]

We started the proxy forwarding about 45 minutes ago, so all data should now be rerouted from our old servers to the new servers on AWS.

Let us know how things look!

[Anthony Aufdenkampe]

@mbarney and others,

After doing some research, we decided that shutting down our old servers for 30 minutes would not likely force the DNS on devices to refresh.

Our plan is to set up proxy forwarding on our LimnoTech server to instantly forward all data to the AWS servers. We plan to do that today, and will also set up a forwarding log to keep track of which devices are still sending to LimnoTech. We’ll leave this up for weeks or maybe months, or until nearly all devices make the switch. It may be that all devices need to be rebooted for that to happen. We’ll see.

Next week, we’ll sync the data over from the old servers to the new servers on AWS. That will fill in the data gaps that you’ve all been seeing.

[Anthony Aufdenkampe]

@mbarny, that is very helpful to know that you have had success with power-cycling the stations. We’re seeing a few other sites switch over on their own in the last 24 hours.

[Anthony Aufdenkampe]

@mbarney, wouldn’t that have been great if the Hologram pause worked! Thanks for trying it out!

Have you seen any of your sites start reporting again in the last 24 hours? The couple that I’ve been tracking have not, which is disappointing. I definitely recognize how painful it would be to manually power cycle these stations.

We’ll likely try to shut down our production server this afternoon unless we see that some stations are starting to send data to the new server on their own.

[Anthony Aufdenkampe]

Fortunately, we deployed to AWS region US-east-2 (Ohio) and it was the US-east-1 (Virgina) data center that went down, so our release yesterday was not affected by the AWS outage!

Unfortunately, we’ve seen two issues with caching mess up our otherwise smooth release:

1. Domain Name Service (DNS) caches are causing some devices to send data to our old production servers (see details at https://www.envirodiy.org/topic/status-update-on-mmw/#post-16150).
2. Browser caches are causing sparkline plots to again fail for some users, because your web browser still has old JavaScript in it that is pointing to a database that no longer exists.

The solution to #2 is to do a hard reset/delete/remove of your browser cache for monitormywatershed.org and related websites. This will cause your browser to fetch the latest JavaScript from our web servers, which will render the sparkline plots correctly.

Our first task for the next round of development is to add some “cache-busting” code into the app, so errors like #2 don’t happen again (See our GitHub issue #529).

Also, we won’t see #1 again, because now that we are on AWS, we will no longer need to change IP addresses when issuing a new release of the Monitor My Watershed web application.

 

[Anthony Aufdenkampe]

Hi All,

This is all related to a temporary data routing issue when we switched from LimnoTech servers to Amazon Web Services (AWS) servers yesterday with our MonitorMW v0.12 release. The data are not lost but rather still getting routed to our old servers because the Domain Name System (DNS) update that points our URLs to our new IP addresses is taking a long time to propagate down to the local networks and devices. We expect this to sort itself out in a couple of days. Then we’ll sync over the lost data.

See this post for more details: https://www.envirodiy.org/topic/status-update-on-mmw/#post-16150

 

[Anthony Aufdenkampe]

@mbarney, @neilh20, thanks for pinging us.

As @heather mentioned, we released MonitorMW v0.12 yesterday at that time, which has some major under-the-hood improvements to substantially improve reliability, including now being hosted by AWS (zone: us-east-2; Ohio). For details read our v0.12.0: Update to Python 3.8 & Django 2.2; Migrate to AWS release notes on GitHub.

Unfortunately, to migrate to AWS we needed to change the Domain Name System (DNS) records for our URLs, which in turn needed to propagate to all internet name servers. That propagation took a surprisingly long time, especially for two in your regions: Verizon in Brooklyn NY (this could cover Michigan), and Corporate West in San Jose CA (which covers the entire US West). We reissued the DNS change today a little after 1 ET, which did seem to resolve those persistent holdout name servers. However, on top of that, devices and networks have their own DNS caches that can sometimes persist for quite a while.

Fortunately, all your missing data is still showing up in our old database on LimnoTech servers, so we will be able to sync that to new AWS servers in coming days.

We’re hoping that the DNS caches flush on their own in the coming day or two. Meanwhile, we are looking into a couple of options in case that doesn’t happen:

• Temporarily shut down our old production server, to try to force devices to clear their caches.
  • We’ll likely do this late Thu or early Fri of this week, for about 20-30 minutes, during which time data will be unfortunately lost.
• Ask you all to power cycle your devices, early next week, if data is still flowing to our old servers.
• Set up forwarding from our old servers to the new AWS servers, to get data instantaneously logging in the correct database.
  • This is not a long-term solution because it includes a potential fail point that we’ve been trying to move away from for a while!

@mbarney, the behavior you saw with your new station makes perfect sense given the issues we’re seeing.

@neilh20, thanks for your patience on your suggestions for reliable data delivery approaches. Now that we’ve migrated to AWS and upgraded the software stack, we’re poised to finally start carefully considering your suggestions.

[Anthony Aufdenkampe]

We think we’ve got it resolved (thanks @ptomasula!).

The main MMW site is back up and data looks to be flowing to the database server again.

We’re rebuilding the tsa_catalog now, so there might be some short term glitches with interactions to TSA capabilities and sparkline plots.

Unfortunately, some data was lost last night, before were able to address any of this.

[Anthony Aufdenkampe]

Check out this repo: https://github.com/EnviroDIY/Trigger

This example sketch shows how it would be used in combination with the ModularSensors library: https://github.com/EnviroDIY/Trigger/blob/master/examples/simple_logging/simple_logging.ino

It’s old, so some components are probably outdated, but it’s generally what you’re looking for.

 

[Author]

Posts