Security Of the Spark Cloud

I was running a apache server and it got attacked by someone and they gained root access. Spark says on there website their cloud is secure but how security is it?

I think @zachary would love to explain this :smile:

1 Like

Security is hard. But I’m going to go out on a limb here, and say that generally speaking, a team that has the skills necessary for creating a cloud infrastructure like Spark’s in the first place probably knows a thing or two about security. :smile:

1 Like

Good question @rpiswag and thanks for the ping @Moors7.

The word “security” covers lots of areas of Spark’s stack and infrastructure, and I won’t go into them all here. However, the specific concern you’re talking about—an intruder gaining root access to a server—is first of all only possible through ports that are open to the outside. All Spark servers are completely closed to the outside except for the bare minimum of whitelisted ports.

SSH (the typical way such an intruder gets in, by guessing a password) is not open to the outside world at all. Just a few of the many industry standard practices we use to prevent unauthorized ssh access:

  • The only entry point to the cloud is through a Bastion Host
  • Password authorization is not allowed
  • Only very few users are even allowed to SSH into the cloud
  • All of those users get seriously trained on best practices in key management

Even better, we can spin up an entire cloud of many interacting servers with a single command. If there were ever even the slightest hint of unauthorized access, we would simply destroy the cloud and rebuild it in a few seconds. :wink: (Actually we do this all the time in development.)

At Spark we have been deeply concerned about security from the very beginning. You should rest assured we’ve got you covered.

8 Likes

What about the Spark Core, Photon, Electron themselves and not the Spark cloud? Even if the Spark cloud is secure is the devices themselves secure? Is the data sent to the Spark devices secure? Can I capture packets and read plain text information or is the data sent to the Spark devices or is the data encrypted and it it is what encryption is used?

I can’t speak about anything but the core yet:

For the code you write, you have to program it to add any encryption you want to use (which is likely to be none).

For the Spark cloud connection, data is protected by AES with a session key generated in the cloud and sent to the core using RSA public/private keys that are set in the factory. Every core has a unique private key in flash for which the public key is stored by the cloud server and every core has a copy of the cloud public key. Every time your core comes online to cloud, you get a new AES session key. This is very similar to SSL/TLS security. Like a lot of things, with physical access to the core and the FLASH chip or USB port, these keys can be read out, but then you could only really imitate one core.

Can I make a copy of the private key from the core?

If you have physical access to the core, you can copy that core’s private key. Or you can write a program to spit out the core’s private key anyway you want. Either way, you can only really take over that one core’s connection to the cloud. Nothing else. Every core’s private key is unique to within the statistics of generating large primes.

Could you clone a core? Sure I bet you could. Would the Spark cloud notice? Yes I think eventually it would and things would not work right. But you can send a core’s public key to cloud anyway so nothing going there, really.

Like I said, it is basically SSL/TLS level security without the certificates. The trust is established in the Spark factory instead of via certificate.

You can keep changing the private key and upload the public key to the :cloud: as and when you desire…

How is the code that you type in the Spark cloud sent to the core. I love computer security and I would like to know how the code it sent. Where it is stored and how the code is ran and when the code is being send over a network to the core could it be modified in some way for instance packet injection?

The entire communication library is here: https://github.com/spark/core-communication-lib

All the source code is open-source: https://github.com/spark

You might want to take time and study them :wink:

Here’s a demo of the encryption at work: https://community.spark.io/t/aes-encryption-demo-and-secure-tcpclient-usage/6092

I remember Zachary explaining about security and how things work but haven’t found the thread yet.

1 Like

The short answer is that the code you send over the air to your core is encrypted with the AES session key used for all core to/from cloud communication in a particular session. The code is decrypted on the core and stored in flash on your core.

Packet injection is not possible, similar to the way that SSL/TLS is not susceptible to packet injection.

I can recommend Bruce Schneier’s books on security as a good starting point. If you want to learn the crypto part, which is just one aspect of the security part, I would recommend Alfred Menezes books. If you want to learn the math behind the crypto, I recommend Neil Koblitz’s “A Course in Number Theory and Cryptography” but that is a challenging topic.

3 Likes

I understand that the Particle cloud is secure but what about social engineering? Many major breaches happen at least in part thanks to social engineering.

Hi @Greg123,

Good question! We intentionally compartmentalize internal access and system privileges to sensitive information, and we also limit what information we store. Since the beginning we also train all employees on the risks of social engineering, and we also play internal security games to keep everyone on their toes!

We’re also considering starting a formal bug bounty program (but responsible disclosures to us are always encouraged, and typically rewarded).

My plan long term is to continue to develop features that reduce risk for everyone, and to provide more and more security features. :slight_smile:

Thanks,
David

4 Likes

I thought I’d add few more questions here instead of creating totally new thread with the same topic…

Basically our threat model is maybe slightly different than most Particle devices currently. In our case we are installing our products around the world to customers who can be anything from homes, schools or companies. Each device is controlling living plants which can be worth anything from 100€-10 000€ depending on the amount of plants and the installation places can be extremely public and high profile (lobbies of malls/VIP spaces etc so failure(=killing plants) is not an option). Now this raises interesting questions if the system is operated via Particle cloud. So far we’ve operated these via our own system so we’re able to do IP blocking based on country and the system isn’t remotely FOTA updateable so hacking isn’t a big concern other than if our command and control server gets hacked and thus someone could send false commands to the devices.

With Particle we could of course keep using our current system via TCP client but your cloud system brings interesting things to the table so I really would like to use it. However for that I’d need good answers/ideas on how to work around these potential threats and how they are being taken into account currently. One of the main issues is that as Particle is growing your becoming bigger target for hackers so at the same time all your clients are in higher risk also.

  1. What kind of continuous security measures you have in place to actively detect hackers who have been able to breach your system? A) the cloud platform which handles the devices/data from them b) Compiler/build platform?
  2. Do you do any penetration testing?
  3. In case of DDOS, what happens? Will all of the Particle devices basically be inaccessible?
  4. Do you do or plan to do code reviews to find out potential problems beforehand?
  5. Is there any fool proof way to detect firmware updates?

Of course some of these problems could be at least partly mitigated via private cloud and private mobile APN so we can better control and monitor the data traffic between our devices and cloud. Do you provide private cloud?

Hi @tuxie,

Great questions! We take our security very seriously, and we treat all devices and products built on Particle as needing the same level of vigilance and importance. We’ve always known that success would paint a larger target on our backs, and even if we do everything right, anyone online eventually gets compromised, especially the larger players.

Our strategy has revolved around both preventing potential compromises, but also limiting any potential damage from a compromise. I won’t necessarily share all our security practices, but I’m happy to be transparent and answer questions as best I can:

1.) We continuously monitor and alert on access and access attempts on all pieces of our infrastructure. Only a small chunk of the team is allowed direct access, and we mandate strict whole-disk encryption, password managers, 2-factor auth, and total physical isolation and monitoring of that engineer’s workstation at all times.

2.) Yup! We’ve had pen tests done on our infrastructure, and we’re looking forward to coordinating another large test hopefully early next year.

3.) Our systems are designed to be deeply horizontally scalable. I won’t share our whole DDOS playbook, but it’s something we keep in mind. Any connected device that uses anything resembling a network will at some point be disconnected for various reasons. We strongly encourage all developers to plan for some or all of the network to be unavailable at some time. Most devices can also be directly engaged with over a local tcp / udp wifi network if needed.

4.) For most features we use internal pull-requests, and are constantly reviewing each-others code for issues. We consult with experts and follow the latest technology when available, and we constantly review our SSL certificates and other interconnects to make sure we’re staying current.

5.) Each piece of firmware is now packaged with a SHA hash of the firmware, and that hash is stored in the cloud when the device connects. This uniquely identifies a specific revision of firmware, and makes it easy to know exactly what is running on that device. We’re also working on user and organizational signing certificates, so you can be sure your device is only running firmware you’ve signed.

6.) We also actively support responsible disclosure of any known security issues or vulnerabilities. Please reach out to us if you think you found an issue and if we confirm it’s a major issue we tend to send free stuff and kudos. :slight_smile:

We do also offer private clouds! Feel free to email sales@particle.io if that’s something you want to pursue. I recommend our normal enterprise cloud offering though, since that will benefit from all our monitoring and upgrades. :slight_smile:

Thanks,
David

1 Like

Thanks Dave for the comprehensive answer.
Regarding answer 5, if someone was to have access to your systems could he reflash the particle and make it reply the same SHA even if the firmware is changed?

One more question popped in to my mind. If someone gets access to your systems, is there anything one could do to prevent him sending empty firmware to every Particle device? I guess in any other scenario problems can be fixed remotely but this scneario would be quite painful and our maintenance guys would have to go locally to each product to fix this problem and that would be tens of thousands of kilometers of driving so it’s not easily doable in few days :wink:

One interesting idea would be to have second server (Address and trigger customizable per particle and not hosted by you) which would host some kind of trigger which would answer to the particle if FW update currently is allowed. Or perhaps you have something already implemented?

Hi @tuxie,

Great questions! Getting a SHA hash collision is still fairly expensive both cost wise and computationally, and would generally be impractical. We would also increase hash sizes and rotate algorithms to stay current.

My plan to combat a bad flash scenario is by integrating code-signing certificates from both organizations and developers. In this way your device would reject unsigned firmware, and / or reject any unsigned commands. This would protect all devices very well in that scenario.

We have and continue to add lots of servers in various regions around the world, the devices currently automatically fail over and balance across our infrastructure. :slight_smile:

Thanks,
David

Hey,

Regarding the SHA I was just wondering if you can just spoof the SHA because if you can reprogram then you can decide what you send to server so the actual SHA can be anything… Though at least part of this should be mitigated by the certificates, that model sounds very interesting! You’ve clearly put a lot of thought on security. Now it’s just interesting what the first breach will be (: Unfortunately nowadays the only question is when and not if there will be a breach but at least you seem to be a lot to avoid it. Might be good to open these things up somewhere on the webpage a bit more. Could help with companies wondering if Particle platform is good fit for them.

Hi @tuxie,

That really depends on how the firmware is built and stored, it should be possible to ship a product where spoofing the hash is not be possible. It’s something we’re always thinking about. But that’s also why we have an open source, local cloud. You can always deploy your own local cloud for your devices and disconnect the internet connection from that network if you want a gapped deploy. :slight_smile:

Thanks,
David