The following email has been sent to all users:
Dear Linode customer,
Linode administrators have discovered and blocked suspicious activity on the Linode network. This activity appears to have been a coordinated attempt to access the account of one of our customers. This customer is aware of this activity and we have determined its extent and impact. We have found no evidence that any Linode data of any other customer was accessed. In addition, we have found no evidence that payment information of any customer was accessed.
We have been advised that law enforcement officials are aware of the intrusion into this customer’s systems. We have implemented all appropriate measures to provide the maximum amount of protection to our customers. Out of an abundance of caution, however, we have decided to implement a Linode Manager password reset. In so doing, we have immediately expired all current passwords. You will be prompted to create a new password the next time that you log into the Linode Manager. We also recommend changing your LISH passwords and, if applicable, regenerating your API key.
The following represent best practices in creating new passwords:
- Avoid using simple passwords based on dictionary words
- Never use the same password on multiple sites or services
- Never click on ‘reset password’ requests in unsolicited emails – instead go directly to the service
We apologize for the inconvenience. If you have any questions, please do not hesitate to contact our support team at support@linode.com.
Comments (34)
> Out of an abundance of caution, however, we have decided to implement a Linode Manager password reset.
Caution is always appreciated, but taking such a drastic measure only makes sense if Linode was compromised to a more severe extent than just one single customer getting his virtual machine hacked into.
If Linode truly values its customers confidence in its services, it should disclose to what extent the intruder managed to escalated their privileges, and to exactly which services these privileges gave him or her access to.
Couldn’t have picked a worse time to switch up your email distribution.
I spent a good 5 mins trying to figure out if the email was legit or not.
New distribution mechanism, html content, and a security alert — all at the same time.
I would have preferred you actually opened a support ticket, in that case I would have got the actual content about the issue and a reason to log in and change my password.
The execution of this did not instill confidence in your customers who have been with you for years.
Preparing myself for endless moans in the comments asking for more detail, more features etc.
Strong recommendation to _not_ use third-party remailers for this type of message; “e2ma” – who? Sounds like a spammer to me…
Firstly, I appreciate the openness of you about this, although I would be interested in the exact details in what happened, I know that you probably can’t tell us. It’s probably just a dictionary attack on the site or something. Resetting everyone’s password is probably overkill, but better safe than sorry, I guess.
Anyway, there is no reason to not trust the email, if it turned out to be a spam email, the worse that could happen is that you log into your Linode thing and it won’t prompt you for a new password. The email doesn’t contain any links to the Linode manager, so unless someone wants to send a spam email to confuse people and maybe make them change their passwords; not very profitable.
@TJ Fontain wrote, “””New distribution mechanism, html content, and a security alert — all at the same time.””” Yes, definitely, and you hit the nail on the head more than anyone else.
My own response, after initially simply believing it and thinking to report phishing, was:
“””This looks an awful lot like phishing. Can you confirm to me that the email, which suspiciously did not come from the Linode domain, represents a real request for a password reset rather than “We think your account has been cracked; for security reasons, please change your password to ‘ID10T'”?”””
Disappointing this isn’t accompanying support for two step authentication process; Google Authenticator; Yubikey, SMS; etc. This action doesn’t address the real problem.
One P.S.
I have a JavaScript in-browser password generator at:
http://JonathansCorner.com/passwords/
(Yes, I expect that the more security-conscious will download a copy, inspect their copy, and run it only if it appears to be doing neither more nor less than what it is said to do). It takes three dictionary words, joins them by a digit and a special character, and twiddles a little with capitalization. And it gives you a choice of 10 or so, with a button to regenerate new.
The resulting password, in exchange from being about 20 characters to type, is relatively quick to memorize, and may possibly be faster than typing with the cognitive load of remembering characters in a random line-noise password.
This presents at least concrete examples of good passwords–if you’re paranoid about security and not confident you could detect malicious code in a downloaded copy, you can do just as well by pulling a paper, dead trees dictionary and opening to a random page, placing your finger on the page, and record the word. Repeat three times, join them with a digit and a punctuation mark, and maybe tweak capitalization.
It’s easy enough to say what good passwords aren’t: they aren’t too short, based on a dictionary word, lacking either an uppercase, lowercase, digit, or punctuation character, repeating the same letter more than twice, and so on ad infinitum. Some psychologists say that only telling people what they cannot do is frustratingly difficult to obey rather than saying “Do something like this” and pulling an exemplar. And I believe my password generation page is good at concretely showing what is easy to remember, hard to guess and secure, and may be faster to type than a line noise password that you don’t use all the time (typing a word you know is faster than my typing speeds for remembering my line-noise passwords at least).
You can access my code under the terms of the MIT license, if you want to check my code for malware or simply observe the apparent algorithm and use a dictionary with paper and pencil to achieve the same effects in a way that I could not be pulling something other than what my password generator appears to make.
My reactions were like this:
See “From: .*@linode.com” and “Subject: .* Password Reset .*”.
Ok, it’s a scammer, I think to myself.
Then I saw the email address I use especially for my Linode account as the recipient.
This is getting nasty, I thought, let’s check the message headers.
There, I find “Received: from mc023.e2ma.net” and think to myself “Great, they’ve been compromised…”
More references to “app.e2ma.net” AND “e2.ma” in the message body don’t help either.
I only felt a little relief when I saw there were no “Click here to reset your password” links. After that I decided to check your official channels, which confirmed the legitimacy of the message.
However, I still feel uncomfortable that you shared my email address with a generic 3rd party remailer for such a trivial reason. Setting up Mailman under a linode.com domain name and importing your account database would have taken no more than 30mins – especially for highly technically capable people like you.
I fully expect to start receiving spam at this email address in the near future.
While I agree using a third-party mailer is not fantastic, do not mind the passphrase reset and further appreciate the expiration option. It would be pretty awesome if you guys offered two-factor authentication option for your customers accounts.
I would like to ask that someone looks into the password requirements. It should be valid these days to use passphrases without numbers or strange symbols. correctbatteryhorsestaple anyone?
Another frustrated customer:
> Setting up Mailman under a linode.com domain name and importing your account database would have taken no more than 30mins – especially for highly technically capable people like you.
This is so naïve as to beggar belief.
Google authenticator should be added, whitelist isnt enough
While I agree with everyone above for added security with authenticators, SMS, etc. and that Linode’s guys could really have their own mailer, let’s all keep in mind that Linode is on a rampage of new features, I doubt this situation will stay like this. Moreover, their support is of very high quality, fast and personalized.
They even had the decency to do a public announcement about the whole thing. If anything, I feel even more secure than before
@William Budd is spot on. Please tell us what is up Linode. Or at least tell us who is forcing you not to tell. Or at the very least that *someone* is forcing you not to tell. Or just tell us.
Really wish Linode would support 2-factor. It’s trivial to integrate with Twilio to send a 6-digit code to a mobile device. Even I’ve done it. C’mon guys, seriously. One week’s worth of your web developer’s time will pay serious dividends. I’d even be willing to pay for the feature.
Don’t use google authenticator as the company is not trustworthy, please.
Thanks for being cautious, guys.
James – Google Authenticator is merely an implementation of the TOTP standard. When Linode implements this, you will be able to use any TOTP client you’d like, Google Authenticator, Yubikey, etc.
I should also mention that the Google Authenticator app has **no** network communication capabilities. It runs completely standalone, and in fact, it will happily run on an iPod touch with no network connections. The algorithm only needs an accurate clock to function properly. So while your paranoia about google may be justified for other reasons, this should not be one of them.
Have to say, I thought this was a more sophisticated phishing scam than most and when the headers didn’t match a linode.com domain I deleted the email and moved on.
You might want to reconsider the use of off site mailing services for things like this.
@James Printer: The google authenticator is open source. I hope Linode doesn’t partake in such company bashing non sense.
(No, the actual app on the market right now isn’t, as far as I’m aware, but it has been open sourced partly or fully before, and if you didn’t trust it, it wouldn’t be hard to use another program).
Didn’t like receive a mail about “Compromissed things” from a third party mailer.
As other commented already, you email seemed like a phishing attempt. I almost didn’t read it.
Read the email, saw the bogus (non Linode) links in the mail, discarded it.
Turns out Linode are using spam-looking company that embeds links in their emails.
That’s *extremely* not good. The amount of user education IT professionals do, to try and get less clueful people to *not click links in emails* and Linode does this?
Wtf? Bad form Linode.
Turns out I had just been reading the news on Krebs’ site, as well as /. and a few others, about a wave of Wordpress brute force attacks from a growing botnet of Wordpress servers. Third party mailer or not, obviously I’m going to check things out when that mail comes in.
Turns out the suckers have set their efforts to my server as well. Fortunately, I’m not dumb enough to leave the default password or use a common password, but all this was enough to make me toughen it up a bit.
So folks, maybe Linode could have handled this differently, but at least they handled it. And though it wasn’t related to my issue, at least I got a heads up and now I know to keep an eye on my server and watch for any developments on the WP botnet.
Don’t just reset your Linode password, check your WP installs and toughen your passwords, folks.
Cheers!
+1 for needing 2-factor auth on the Linode manger. Right now it’s the weakest link in the chain, and the IP whitelisting is not suitable for use in many cases.
Ok cool
The source of the warning email didn’t bother me at all. If it was fake, it would have been obvious as soon as I tried to log in – had there been no password reset.
I’m glad you were cautious. Oh, and thanks for the forthcoming RAM upgrade in Fremont.
There’s an article on /. claiming that CC#s were compromised:
http://slashdot.org/firehose.pl?op=view&type=submission&id=2603667
Should we be contacting our banks?
A response from Linode is a must — immediately. It has already been way too long.
Were credit cards and password hashes accessed as alleged? Yes or no?
What encryption was used to secure both?
http://slashdot.org/firehose.pl?op=view&type=submission&id=2603667
Most people would have expected a follow-up on the latest rumours. However “We have been advised that law enforcement officials are aware of the intrusion into this customer’s systems.” stood out to me as unusual when I first read it a few days ago. It suggests that there may be reasons why Linode cannot make a statement at this time.
Certainly Linode’s public handling of the issue could improve, but use your paid support option if you actually have a need to be addressed. So open a support ticket, ask your questions, and if you can’t be satisfied by the support (which, reiterating, you are paying for) then make your decision to stay or move.
A few messages back I was praising your technical prowess and someone called me naive.
Now blog.linode.com has been replaced by slashdot.org apparently, guess I should have posted there instead.
I need an answer to just this question:
Were past CC details stored (and leaked) or just the ones currently set in our accounts?