corruption on upstream TCP data and retries/failure of HTTPS and SSH

  • 6
  • Problem
  • Updated 4 years ago
  • Solved
Archived and Closed

This conversation is no longer open for comments or replies and is no longer visible to community members.

MODEM firmware is rev  UT_2.2.3.0.13
data corruption with or without a router
have tested with 3 different router
windows 7 and 8, and 3 flavors of LINUX
HTTPS and SSH will catch the corruption and retry,  may or may not succeed with the file upload.
FTP will normally succeed but the uploaded files are corrupted, even though the TCP layer thinks the packets are good.
example corrupted images can be seen at the link below. these were uploaded with FTP
http://vast.net/bill/junk/
Exede, you have a serious problem with your software!

 
Photo of slowBill

slowBill

  • 54 Posts
  • 21 Reply Likes

Posted 4 years ago

  • 6
Photo of ExedeKarmin

ExedeKarmin

  • 384 Posts
  • 40 Reply Likes
Hello Slowbill,

Sounds like you need to speak with my Technical Department.  Please send me an email to exedelistens@viasat.com with your account and contact information.
Photo of slowBill

slowBill

  • 54 Posts
  • 21 Reply Likes
I added the original files to http://vast.net/bill/junk/ and a txt file showing the bit errors in one of the 2 meg files. somewhat of a pattern to the corruption. bit zero reset to 0 then 8 bytes later bit zero set to 1.
here is a frightening thought, what if this is caused by bad RAM in a piece of equipment used for beam 329, and more frightening what if that equipment is 22,000 miles above the earth? 
bit errors below,
decimal address, Octal good data Octal corrupt data 
cmp -l ./IMG_1753org.JPG ./IMG_1753viasat.JPG 
 dec       Oct Oct    bits changed
 135627  45  44     bit 0 reset
 135635 326 327     bit 0 set
 183807  71  70     bit 0 reset
 183815 216 217     bit 0 set
 507160 131 130     bit 0 reset
 507168 120 121     bit 0 set
 630312 335 334     bit 0 reset
 630944 246 247     bit 0 set
 832740 127  26     bit 0 reset, bit 6 reset
 832748  34 135     bit 0 set, bit 6 set
 869760  14  15     bit 0 set
 870376 335 334     bit 0 reset
1123757 373 372     bit 0 reset
1123765 132 133     bit 0 set
1136273  41  40     bit 0 reset
1136281 236 237     bit 0 set
1140253 267 266     bit 0 reset
1140261 336 337     bit 0 set
1180073 357 356     bit 0 reset
1180081 306 307     bit 0 set
1191241 233 232     bit 0 reset
1191249 100 101     bit 0 set
1375194  60  61     bit 0 set
1375858 257 256     bit 0 reset
1698501 223  23     bit 7 reset
1698509 142 342     bit 7 set
1727557 357 356     bit 0 reset
1727565  56  57     bit 0 set
1802017 155 154     bit 0 reset
1802025 262 263     bit 0 set
1809717 175  74     bit 0 reset, bit 6 reset
1809725 250 351     bit 0 set
1811613 257 256     bit 0 reset
1812709  14  15     bit 0 reset
1996558  47  46     bit 0 reset
1996566  14  15     bit 0 set
 

  
(Edited)
Photo of A. Everett Neuman

A. Everett Neuman

  • 430 Posts
  • 303 Reply Likes
I agree, if the beer can took a hit. The fix would be a few days in coming.

Lab has the truck, but the fuel cost are about 12 million dollars. And then of course, what size crescent wrench to bring???

Sure would like to see the schematic for the can in the sky. JEP seems to have a lot of details stashed away and has looked into the inter workings more than most.
Photo of slowBill

slowBill

  • 54 Posts
  • 21 Reply Likes
you'er a funny guy Everett. 
on a different note, it seems like I have been seeing some complaints like
"I have had no problems for x number of years, now I use up my Gigs in x number of hours"
I cant help but wonder if these people are being impacted by corruption causing automatic updates to endlessly error out and retry, thereby using many times more data than normal.
rev  UT_2.2.3.0.13 beam 329
Photo of A. Everett Neuman

A. Everett Neuman

  • 430 Posts
  • 303 Reply Likes
The subject of lost data has been a popular one. As to now the only answer I have seen is to give the customer some free added Gig's. Of course, many people do not post back if their issues where fixed or not, so no firm answer if it was fixable or not.
Photo of JEP

JEP

  • 987 Posts
  • 718 Reply Likes
Bill - ViaSat-1 functions as a "bent pipe".  It is strictly RF in and RF out.  It doesn't know anything about ones and zeroes.  The received signal is amplified, hetrodyned to a new frequency, boosted up to transmit power levels and away it goes.  This keeps things as simple as possible on hard to service equipment and also the "bent pipe" mentality does not lock it into any specific protocol, packet size, etc.  This allows channels to be used for almost any application as long as it doesn't exceed the bandwidth of the channel.  I'm just guessing, but I suspect ViaSat-1 also has spare receivers and transmitters in case something fails. 
Photo of Dorothy Allen

Dorothy Allen

  • 1 Post
  • 1 Reply Like
A quick hello to slowBill and a thank you for posting here about your experience.  A lot of people can realize when a problem exists but have difficulty describing exactly what it is nor do we have any idea of what's causing it much less how to fix it.  Please continue to talk about your internet experiences.  Thanks.
Photo of Old Labs

Old Labs

  • 3699 Posts
  • 3717 Reply Likes
Dorothy,

If experiencing issues that sound like ones you may be having, describe them as best you can. There are enough of us who can make the transition/correlation to/from geek-speak to everyday English.

The more people that report an issue the less likely the issue can be dismissed, and it will receive a higher priority,  but the more detail the better.
(Edited)
Photo of Goatreich Sturmkaiser

Goatreich Sturmkaiser

  • 12 Posts
  • 2 Reply Likes
Same boat as me, I'm a webmaster and for the past 2 weeks I haven't been able to work on my websites at all, Everytime I FTP in to my site to upload, about 1/8th of my files end up corrupted and need reupload....  

And surprise surprise, I'm over my data cap again, and I'm left with a plethora of unfinished projects, broken pages I can't make public, I'm losing money, and everyday Exede continues to suck a fat one, I fall further behind on what I need to do. 
(Edited)
Photo of slowBill

slowBill

  • 54 Posts
  • 21 Reply Likes
hello Goatreich Sturmkaiser, do you have SSH access to your server? if so, you might try uploading your files with SFTP which uses the encrypted SSH connection as a workaround. the transfers may fail, but when they do go trough there will be no corruption of the file. 
Photo of Goatreich Sturmkaiser

Goatreich Sturmkaiser

  • 12 Posts
  • 2 Reply Likes
4th night in a row trying to download a 795GB game patch on my PC.... 9 tries, 2 fails, 7 CRC failures, after the 4 attempt the download is throttled to 52kps...  Tried updating an Xbox One game, 1.7GB's, Downloaded on the first attempt, max speed 700kbs, installed, and game does not load do to file error, need to deleted the entire 28GB game, reinstall, and apply all updates again...  So essentially I cannot play this game anymore until YOU fix YOUR services.

Didn't even attempt to upload my work files, as the past few nights have given me nothing but img and html file corruptions. Can't fix all the broken pages I've already uploaded before noticing the problem. 

Stayed up like a Vampire.... AGAIN.... to accomplish nothing....

Thanks Exede
(Edited)
Photo of A. Everett Neuman

A. Everett Neuman

  • 430 Posts
  • 303 Reply Likes
Hi Goatreich,
I have a question, if you have contacted Exede through exedelistens@viasat.com 
did they mention a way to roll back the firmware?

Sure would seem to be the best test to determine if the problem in in the newest update.

I know Lab has been on top of this, but not sure if he has heard of a rollback possibility.
Photo of Goatreich Sturmkaiser

Goatreich Sturmkaiser

  • 12 Posts
  • 2 Reply Likes
Last night LNFZ speed seemed to only be active on some pages/files and not others. 

For example I was on Youtube, most videos would autoload at 144p and would play for about 5 seconds before freezing, buffering for 2-3 minutes, play 5 seconds, buffer again. This is exactly what happens when I try to watch Youtube during the day while over my GB's......  Then I would find one magic video here or there that autoloaded at 720 and played flawlessly without interruption.

My average page load time while over my allotted GB's is 12 seconds (UGH!), LNFZ would speed up 90% of the pages I view, but 10% are still being restricted to slow speed, taking 12s to load.

------------------------------------------------------------------------

Here's where things get really confusing.... I download a weekly 4hr long radio show in MP3 format, on average it's a 150mb file. I try downloading from the offcial website, and it starts downloading at my restricted speed of 30kbs.... so I look for the file elsewhere, downloading from various file hosts, and even torrents... all of them restricted to 30kbs. All the sources I tried are hosting the same file as the official site with the exact same file name. Then I find one torrent where the filename was altered from the original, suddenly it's downloading at 2MBS.

So now I have my radio show to listen to today while working, but I'm noticing the ongoing corruption issues while listening. There are odd echoes here and there, and on occasions there are places where the audio jumps to a different part of the show for a second or two. 

For example, you're listening to a comedy sketch at the opening of the show, and then suddenly 2 seconds of a phone conversation (that happens in 2 hours later) cuts in for no reason, then you're thrown back into sketch.

------------------------------------------------------------------------------

And as far as I can tell from the maps I saw here, I should be on Beam 321
Photo of A. Everett Neuman

A. Everett Neuman

  • 430 Posts
  • 303 Reply Likes
Hi Goatreich,
"2 seconds of a phone conversation" Now that is strange.

With the reports of the latest update being related to this issue, I wounder if there is a filtering problem with Exede Voice and Exede Internet in the big can in the sky or the servers on the ground???

You mix zero's and one's 0010110 with Hello Mommy! And you have the next hit show for HBO.

More shielding on the coax Scotty! and full speed ahead.
Photo of Goatreich Sturmkaiser

Goatreich Sturmkaiser

  • 12 Posts
  • 2 Reply Likes
It's not 2 seconds of a random phone conversation... it's a phone conversation that's a part of the show.  Let's say the the phone conversation takes place 2 hours deep into the show, but 10 minutes into listening to the show from the start, 2 second long blips of that conversation cut in on what's currently happening "music, comedy bit, dj talking etcetera.

it happened about 25-30 times in 4 hours, where 2 seconds of something from a different part of the shows timeline, just randomly inserts itselfs out of place.
Photo of Old Labs

Old Labs

  • 3699 Posts
  • 3717 Reply Likes
Goatreich,

You've indicated above you're on beam 321 and also have the suspect modem software. From the Wildblue World forum, 321 has been identified as one of the problem beams. Your gateway is Lovelock NV, and Accelenet server is likely Salt Lake but you can verify the latter using tracert as noted below (same as mine except for beam)

slowBill indicates he received a message stating the problem has been escalated to engineering. Hopefully relief is on the way sooner rather than later.  
(Edited)
Photo of Old Labs

Old Labs

  • 3699 Posts
  • 3717 Reply Likes
Same issues being reported at these links (as well as others and all involving the latest software update):

https://community.exede.com/exede/topics/is-exede-injecting-random-data-into-compressed-or-encrypted...

https://community.exede.com/exede/topics/im-paying-for-a-service-im-not-receiving-and-im-no-longer-hesitant-to-involve-the-law

http://www.wildblueworld.com/forum/showthread.php?9164-Corruption-when-using-Router

I may (or may not) have worked around it with some MTU tweaking as described in the 1st thread, and suspect not for reasons mentioned there since I didn't tweak everything yet it seems to have abated on everything.

In addition to modem software version, including beam number, gateway and Acelenet server also may help to further isolate from my perspective (but Viasat probably knows the latter two simply from the beam number) .
(Edited)
Photo of Old Labs

Old Labs

  • 3699 Posts
  • 3717 Reply Likes
slowBill,

Sorry I missed this one, but surprise, surprise, surprise...

I'm on 329 also and it should be -  Lovelock NV Gateway, Salt Lake City Accelenet Server - as is JEP.

There's a way to tell but I have to dig into the way back stacks for it.

More commonality for the issue - my guess is that the modem update was rolled out first and once rolled out to everyone on a beam - some server changes were then rolled out to take advantage of that modem update and that's where the problem lies (or perhaps both).

Still persisting here regardless of MTU tweaking (although it does appear to lessen the frequency of occurrence). I'm guessing that while Everett has the modem update his server(s) haven't been updated. I had the update for over a week before the errors started.

P.S. Just no hit it again with the Exede Internet Image and header on this page showing SSL Connection Error.
(Edited)
Photo of Old Labs

Old Labs

  • 3699 Posts
  • 3717 Reply Likes
slowBill (or anyone else for that matter),

Acccelenet server can be determined by doing a tracert www.google.com at the command prompt. The first result containing a recognizable city is usually it (typically the 5th or 6th line) - in my case and likely yours Salt Lake.

See the following for clarification:
 
http://www.wildblueworld.com/forum/showthread.php?9166-Service-has-been-bad-for-about-a-week-now&...
(Edited)
Photo of Old Labs

Old Labs

  • 3699 Posts
  • 3717 Reply Likes
Still experiencing issues and suspect the caper went down like this:

  • New modem software rolled out to selected beams around 12/16 consisting of the Accelenet client.
  • After being rolled out to those beams and around 12/25 changes to Accelenet server(s) servicing one or more of those beams were rolled out.
  • Problems started appearing shortly after that.
The problem with client/server technology in synchronizing change is often which comes first the client or the server, and what do you do when one or both are bad?

In this case seems pretty simple, roll back the server changes and we get a Christmas mulligan... but I could be wrong.

 
(Edited)
Photo of A. Everett Neuman

A. Everett Neuman

  • 430 Posts
  • 303 Reply Likes
Hi Lab,
My tracert returned this:


What is the ping to; edge2.losAngles9.Leve13 ??? Several timed outs also?
Photo of Old Labs

Old Labs

  • 3699 Posts
  • 3717 Reply Likes
Not sure but beyond your Accelenet server of Phoenix, which I;m betting hasn;t been updated yet explaining why you aren't having the problems. Don't worry about the timeouts.        
Photo of slowBill

slowBill

  • 54 Posts
  • 21 Reply Likes
thanks guys for commenting. I have sent my contact info as requested. unfortunately I am only at my property with the viasat on the weekends, so I will only be able to do very limited testing remotely when viasat's technical department contacts me.

I am an electrical engineer and embedded software programmer and have had very little problem with my Exede  service for the last ~2 years and have been largely pleased with it.

Lab Rescuer, I had set my router MTU from the default 1500 to 1492 which did seem to reduce the frequency  of the corruption.

I will post back with any progress or info I might obtain.
Photo of JEP

JEP

  • 987 Posts
  • 718 Reply Likes
slowBill - When I was having corruption issues, I had to set my MTU to 1490 based on the  Ping Yahoo.com -f -l 1472 test method.  That totally fixed my 100% repeatable corruption download errors.  Now, FWIW, I played with setting my MTU back to AUTO on my router and it doesn't make any difference.  Me thinks there are other factors going on in the Exede network.  Hopefully they will get the wrinkles ironed out.
Photo of A. Everett Neuman

A. Everett Neuman

  • 430 Posts
  • 303 Reply Likes
I just noticed that the hyper link we have been using to find the beam map's is dead.
https://www.wildbluetools.com/content/dealer/email/Beam_map-high-mid-low.html

Do they think we are too stupid to handle that much information? Is there a new beam map?
Photo of A. Everett Neuman

A. Everett Neuman

  • 430 Posts
  • 303 Reply Likes
Bill, that is some good info. I sure do not remember getting anything like that during my install. Do you have it in hand, or from an online source?

Kimberly, it would help all concerned if we could read/ see the information coming back from the Engineering team. Like Lab has mentioned many times, if the fix was found, please let us know so we do not waste time digging further into the problem.

And the email Lab got back, that is some very deep gibberish. When I paste that into Google, it returns: We are working on it.☺
Photo of Old Labs

Old Labs

  • 3699 Posts
  • 3717 Reply Likes
I think PHB used this: (or was it me?)

http://www.andrewdavidson.com/gibberish/

and I'm going to start using it for all replies now - well at least some if you know what I mean ;)
(Edited)
Photo of slowBill

slowBill

  • 54 Posts
  • 21 Reply Likes
Everett, I have a photo of the filled out paperwork we signed. that is a snip of the site information portion. the rest is mostly contact and billing info.

I just received word from ExedeKarmin that the issue has been escalated to engineering. 

who/what is PHB? someone in Colorado probably? 
Photo of A. Everett Neuman

A. Everett Neuman

  • 430 Posts
  • 303 Reply Likes
Bill, I did mine electronically, the detail are floating in a cloud somewhere I guess. maybe my little friend Kimberly can find it???♥♥♥

Lab, that would be a good one. You could even run for office:
If I am elected "insert paragraph of gibberish here" there will be a chicken in every pot pie!

(Edited)
Photo of Old Labs

Old Labs

  • 3699 Posts
  • 3717 Reply Likes
PHB: Pointy-Haired Boss - Everet doesn't take over Viasat until hitting the mother lode in early 2016 and then we call him HCB - Hairy Chinned Boss ;)

P.S. This is one place we NRTC susbscribers have and edge - all of that information is displayed on our account pages (including gateway, installer, etc.).
(Edited)
Photo of John_in_MO

John_in_MO

  • 3 Posts
  • 2 Reply Likes
Same problem here. Same firmware UT_2.2.3.0.13. Looks like we're on beam 316.

First attempt at submitting this failed due to the problem.

Symptoms for us include not being able to load web pages, sometime only parts of pages, corrupt images and PDFs, etc.
Photo of JEP

JEP

  • 987 Posts
  • 718 Reply Likes
Here's hoping that Microsoft and other developers are smart enough not to update their software on our computers with a corrupted download.
Photo of slowBill

slowBill

  • 54 Posts
  • 21 Reply Likes
and here's hoping Exede can still push new MODEM updates without bricking them because of corruption. ;-) yeah I know I'm just a worrywort.
Photo of Exede Kimberly

Exede Kimberly

  • 879 Posts
  • 202 Reply Likes
So small favor from everyone and anyone having these issues...not to sound moderator cliche...but..please send me your info to exedelistens@viasat.com. I am collecting customer info to send to our Engineers. Thanks you guys.
Photo of Old Labs

Old Labs

  • 3699 Posts
  • 3717 Reply Likes
John_in_Mo,

I'll echo JEP above in starting a new post particularly since this one as well as other related ones are now tagged as Solved.

Above in one of your posts you indicate your on Beam 316, and I don't recall that particular beam being mentioned either here or elsewhere on the other forums (and I was tracking this one pretty closely).

Regardless, you may want to do a tracert google.com (from the Dos command prompt assuming Windows).

The first recognizable city (generally the 5th or 6th hop) is an indicator of your Accelenet server - everyone reporting here and elsewhere was going through Salt Lake City.

P.S. While these errors were sometimes referred to aa connection errors in the several threads here and elsewhere, they technically weren't connection errors but rather data corruption errors exhibiting a number of specific error messages. Specifically, the connections were fine, the data traveling along those connections wasn't in too good shape.  
(Edited)
Photo of Steve Frederick

Steve Frederick, Champion

  • 2737 Posts
  • 1717 Reply Likes
John, the connection time is shown on the modem status page in the upper right. That will tell you how long the modem has been connected to the big shiny thing in the sky. If you have numbers in the minutes or hours, then the modem has had to reestablish communication. If it is showing days or even lots of hours, then it hasn't disconnected, and your problem is likely with your router or computer.

As the guys above have said, start a new topic, and send an email to the moderators at exedelistens@viasat.com explaining what you are going through. They will help you so much quicker than if you call phone support, or even use the Exede website to ask for help.

Good luck. 
Photo of John_in_MO

John_in_MO

  • 3 Posts
  • 2 Reply Likes
Thanks, guys. I didn't realize that you declared that this problem was solved for some of you. I'll open a new thread with the same info to re-open it for those of us still experiencing the corruption problems. Glad some of you are up and running. Hope the rest of us will be soon too. Cheers!
Photo of Steve Frederick

Steve Frederick, Champion

  • 2737 Posts
  • 1717 Reply Likes
We do not have the ability to declare problems are solved, that is done be the moderators who are Exede employees. We are just customers who try to help out with issues that people are having. If you are still experiencing problems, do start a new thread and the moderators will see it when they come in tomorrow.
Photo of Old Labs

Old Labs

  • 3699 Posts
  • 3717 Reply Likes
Good luck John-in_MO,

It took a sustained, coordinated effort (or a "ruckus" as someone else said) both here and in the Wildblue World forum to narrow the problem to the SLC server and UT_2.2.3.0.13) and get it fixed - it's logical to assume it may have been extended to other servers unless it was solely a hardware malfunction.

Note that when doing a tracert, Salt Lake City isn't being displayed anymore but rather it appears airport codes are being used (SLC in my case but that may be temporary).  As Steve states, we only declared it as a problem - not solved.
(Edited)

This conversation is no longer open for comments or replies.