Dimitri McKay’s Nerd News

Securing your Private Cloud Environment

Fri, 28 Jan 2011 09:47:06 -0500

On the back end of private cloud environments you’ll find multiple flavors of virtual software loaded directly onto hardware. This virtual software is essentially the host operating system. VM Host is the base hypervisor and hardware. Think of it as the house. The guest operating systems (Guest OSs) are the virtual machines living in the house.

As the basis for all public and private clouds, virtual infrastructure is how it’s done. So this conversation we’re about to have is related to the back end of the private cloud. If you’re building one, it is important for you and your organization to understand how to maximize the benefits and mitigate the risks of a private cloud infrastructure. There are several key things to keep in mind when trying to secure the virtual environment before even loading guest operating systems.

Most virtual solutions are transparent, by design, to the guest operating systems. The same way machines are secured in physical environments, they are secured in a virtual environment. This includes segregating networks, defining domain security policies and installing antivirus.
But unlike their physical hardware cousins, Virtual Machine infrastructure security seems to be lagging behind, and although this virtualization is consuming datacenters worldwide, many organizations fail to recognize that security basics are still security basics.

According to Gartner, 16% of server workloads were running on virtual machines by the end of 2009. Gartner expects this to grow 50% (to 58 million) by 2012. Unfortunately, Gartner also predicts that 60% of these virtual machines will be less secure than their physical hardware predecessors.

Why are Virtual Machines insecure?
The reason that VM servers are less secure than their traditional hardware counterparts are as follows:
Security isn’t considered at the beginning of the project, which is often the case. In many situations a public cloud project is begun, and from there each project becomes a knee jerk reaction.
If the VM host OS layer is compromised, all guest OSs can be compromised. This is called Hyperjacking. More on that later.

Although most public cloud vendors maintain adequate controls for admin access to the virtual machine monitor, many private clouds do not.

Segregate and separate.
VM hosts create flat networks. You’ll need to change that. In a non-virtual world, traditional data-centers had segregation and network traffic could be inspected, filtered and monitored by a number of security products. In a virtual world, these are a rare commodity. The local communication between virtual servers is largely untouched and unmonitored. If the traffic runs through a virtual switch it’s practically invisible because it never hits the wire. It’s just traffic between virtual hosts on virtual links. So virtual traffic between virtual machines needs to be monitored.

Separation of duties is something that we in security often push. Unfortunately, in a virtual server environment, the back-end of a private cloud environment, you’ll often find that the server team and the operations team are the same people who do both provisioning of machines and managing virtual switches. So that means that there’s rarely any integration between the tools and security controls to be implemented for the network and security groups. And what THAT means, is that without visibility into configuration and policy changes, topology specifications and audits, the network and security team has zero view into what’s taking place at the access layer.

In security circles, we also talk about the "principle of least privilege." This says that you should not give anyone more security than the minimum security they need to do their job. Defining roles that can be used to give different levels of security will make life much easier.

How do you secure a traditional server?
First, lock down the server OS (usually Windows or Linux). Now, as you go virtual, instead of just securing the server OS, you also have to lock down VMKernel and the VM layer (the host OS), as well as the console. The same thing you’d do with your weekly Microsoft patch plan is what you should do with your VM Infrastructure. Although extremely secure, stay up to date on patches. There are security updates in there, not just bug fixes.
What you really want to avoid is Hyperjacking, which involves compromising the hypervisor. This is the lowest level of the OS stack, and the hypervisor has more privileges than any other account. At this level it’s impossible for any OS running on the hypervisor to even detect that a hack has taken place. So the hacker can control any guest VM running on the host.

When you go virtual, you add that other layer to the mix, the hypervisor. So again, the hypervisor needs to be secured at all costs. It’s mission critical because an attack on the hypervisor can lead to the compromise of all hosted workloads, and successful attacks on virtual workloads can lead to a compromised hypervisor. Another concern would be a VM Escape, which is an exploit that is run on a compromised guest OS to attack and take over the underlying hypervisor, which can then result in a hyperjacking.

Moving up the stack are those OS patches. Although it’s super easy to spin up another guest operating system, admins sometimes forget that the virtualization software is there. So make sure that just as fast as virtual machines are spun up, patch distribution software should also be installed, and antivirus, service packs and security policy changes should be made to all of those virtual guest operating systems.
The biggest security concern, however, is the insecurity of the underlying virtual guest OS. The VM software you use will separate the guests from both each other as well as the VM Host, so if one of the guest VMs does get compromised it’s unlikely it could affect the host, with the exception of using more memory/processor/network resources.

Be aware that the easier move for a hacker to steal data would be VM Hopping. This is a situation where an attacker to compromise a virtual server and use that as a staging ground to attack other servers on the same physical hardware.

The last threat to VMs themselves would be VM Theft. This is the virtual equivalent of stealing a physical server. Take the whole box and run off with it. Then fire it up later and steal the data. Same concept, however, in this situation it is theft of the virtual machine file electronically, and then attack it later.
How can you make the private cloud more secure?

Start with the base layer. The lower stack. The hardware and traffic. Force all traffic between hosts to be inspected by an IDS (intrusion detection system). Each VM Host should also have a different ingress/egress VLAN pair. Then the IPS should be set with VLAN translation to configure each ingress VLAN and egress VLANs. The goal is to define all VM-to-VM traffic being sent across the wire where it can be inspected, monitored and potentially filtered. Of course, as the private cloud grows, it can become complicated and costly between data-centers and DR sites.

Another option is to define an IPS and firewall on each VM Host, and policies be configured to inspect the traffic. This makes sure all intra-virtual communication is inspected. Of course, there are some performance hits you’ll take running all those additional VM IPS and Firewalls, plus the monitoring for all traffic, however, in the end security should be paramount.

Next up is a ‘love connection’ between the above two. It’s a mix of both. Route traffic to an actual IPS where it’s filtered, can be monitored, etc. Then send the traffic off to a destination VM.

Another security option as you move up the stack after traffic would be securing your hypervisor. Keep your hypervisor console patched. Just like any OS, VM Servers will have security patches that need to be deployed. The majority of these patches are related to the Linux-based OS inside most service consoles.

Additionally, ensure that virtual machines are fully updated and patched and all provisioning is done with security tools etc before they are turned on in a production environment. Although VMs are much easier to move around and faster to deploy than physical machines, there is a greater risk that a VM that is not fully updated or patched might be deployed. To manage this risk effectively, use the same methods and procedures to update VMs as you use to update physical servers.

Another tip would be to use a dedicated Network Interface Card (NIC) for management of the virtualization server. By default, NIC0 is for the parent partition. Use this for management of the Host machine. Don’t expose it to untrusted network traffic and please don’t allow any VMs to use this NIC card. Use one or more different dedicated NICs for VM networking.

Lastly, before installing the private cloud services, (the reason you build this infrastructure) I’d recommend using disk encryption to protect the VMs. If one is stolen, it’s worthless to the thief, and the data at rest is also safe.

Summary
When building a private cloud, it’s important to remember that going VM isn’t as easy as moving servers to a host machine. Security is a bit different, but security practices are still the same. Recognize the additional components. Recognize that going VM often ends up in a flat network, and you’ll need to change that for security's sake. Lock down your console and hypervisor. And remember the same lock down procedure you used in traditional servers is also critical, if not paramount. Follow these tips and you’ll be ready to host those apps to your user community in no time. Securely.

Why Cloud Tenancy and Apartments have more in common than you think:

Mon, 24 Jan 2011 09:33:42 -0500

One of the most common questions about cloud security is around privacy and regulatory compliance. Questions around government mandates and industry requirements abound from IT managers considering a shift to the cloud—most of which relate to multi-tenancy.
Since there’s been so much discussion about multi-tenancy in the cloud lately, I thought I’d explain what it means for both cloud providers and cloud customers.
But first, a tangent:

I live in an apartment. I have a small apartment that has everything I need. I enjoy living in an apartment because I don’t have to maintain the plumbing or electrical. I don’t have to rake the yard or clean the gutters. I don’t have to clean the pool or fix broken equipment in the gym. I don’t have to worry about security; it’s provided as part of my rent. I only have to worry about my belongings in my apartment. As a result, I get to enjoy the cost savings of a shared environment, and the robust amenities of my building. My apartment building has lots of other residents. Some work different jobs, some work at different companies. But we all share the same building, with the same resources and amenities.

My apartment building is a multi-tenant cloud. More on that later.

Cloud providers (landlords) love multi-tenancy clouds (apartment buildings) because they rent the same resources to a large number of renters, and the renters get to enjoy all the financial savings of those shared resources. The cost savings is good news to all parties involved. The cloud, as you know, can provide amazing cost savings, fantastic up-times and someone else to blame or sue when there is a mistake, when there has been an accident or if problems go unfixed.

Today we’ll talk about security, reliability, audit-ability, quality of service and regulatory compliance in multi-tenant solutions vs. single-tenant solutions.

Reliability:
In my apartment building, there is a bank of washers and dryers. Everyone can use them—10 washers and 10 dryers for 300 families. But if the power goes out in the laundry room, we’re all wearing dirty underwear. This is a multi-tenant problem.

In the cloud, when a multi-tenant app goes down, it takes everyone with it. Take Wordpress. They went down a few months back. One app down, everyone went with it. Imagine other Web apps out there. What if Salesforce went down? What if Gmail went down? One App. Life stops. Now what if that’s an industry specific app everyone is using. You see where this is going. The upside? One application upgrade, one application maintenance: one application across the board saves time and money for the customer.

But what about single tenancy? Let’s talk again about washing machines. If we go single tenant on a washing machine, then each apartment that wants to pay for a washing machine can have one. It’s their washing machine, they don’t share it. It’s a single tenant solution. If their washing machine goes down, it doesn’t affect the other washing machines in the building. In this case, the cost savings obviously isn’t as pronounced as in a multi-tenant setting. Because in a building of 300 families, if even half of them want their own washer/dryer, we’re looking at 150 washers and 150 dryers, all of which need to be maintained, all of which can fail, all of which need to be supported individually and all of which carry their own price tag.

Security
How about security in a multi-tenant vs. single-tenant situation?
Securing a building is one thing, but what about securing each apartment from other tenants? That also needs to be considered. Firewalls at the front of the network keep external threats out, much like a doorman. But what’s to stop your neighbor from breaking into your apartment? There’s no doorman to stop someone on the inside.

So, because of shared resources, security needs to be handled at a much lower level: segmentation of resources. You have to segment your apartment from your neighbor’s apartment. On the network side that would be segmenting those shared resources using Mac Address Control address pools, VLAN tagging (Virtual Local Area Networks) with more advanced security controls such as tag zoned segments, private VLANS and ACLs (access control lists) to define a secure environment, enforce the policies of the secure environment and maintain that secure environment.

For storing your business data, your critical data and your customer data, you’ll want to make sure that the architecture users LUN (Logical Unit Number) masking, at rest encryption, zoning and VSANs (Virtual Storage Area Networks) to keep cloud insiders and cloud outsiders out. Ultimately, there needs to be as much security between you and your neighbor as there is from an outsider trying to break into the building.

Auditability
If you enter the lobby of my apartment building, the doorman will either allow you in, or he’ll turn you away. In other buildings, you need to authenticate yourself by using a key fob for entry. And make no mistake: there is always an audit trail:
“Dimitri McKay entered the building at 3:05am”
“Ms. Jameson came to visit Dimitri McKay at 4:15am”
If your business is governed by industry mandates or government regulatory compliance, you need to make sure you have data such as raw logs to keep your auditors (and upper management) happy. Local or in the cloud, it’s your responsibility to practice due diligence. There are providers that offer security and accountability. You can have your Kate and Edith too.

Quality of Service
My neighbor complains constantly about noise from my apartment. And he should. My subwoofer at volume level 10 shakes the apartment, his apartment, the people upstairs, the mail room, the garage and three blocks away at the local watering hole. I tend to use it at 3am when I have insomnia. You don’t want your cloud to have the “Dimitri McKay subwoofer” problem. In other words, I’m a tenant who is affecting the processes of another tenant—in this case, their sleep. By putting some quality of service in place, that segregation of work keeps my noise from impacting his sleep. It’s the same situation in the cloud: your workload shouldn’t be affected by your annoying neighbor.

The cloud is a shared environment, much like my apartment building. Where I’ve used a simple example of multi tenancy would be an apartment building. In an apartment building you have a “shared environment” where multiple “renters” share a common infrastructure (the plumbing, the electrical grid, hallways, etc.) but still have segregated areas where the users keep their stuff (host their applications and/or data).
Multi-tenancy is highly desirable to cloud providers because they can provide a platform or service (applications, infrastructure, etc.) and rent it to a large number of customers without having to make massive customizations, tons of labor-intensive upgrades, troubleshooting sessions and associated costs. Single tenancy has merit in situations where sharing the same app among a broad scope isn’t a viable option.

On a large scale such as the infrastructure side, the cloud provider will always opt for multi-tenant, but the customers themselves will likely seek single tenant in the following situations: custom apps, customers who are bound by specific regulatory compliance mandates, or those who care more about security than price.
One example of this is anyone who needs to have raw log data from all of their IT infrastructure, OS and apps in one place. They could have a single tenant log management tool in the cloud that only collects data from their specific cloud network devices, cloud applications and server operating systems. In this situation, segregation makes more dollars and sense.

Just as is the case with public, private and hybrid clouds, there’s no be-all, end-all situation when it comes to choosing between single- and multi-tenant deployments. It depends on what your goals are, as well as your budget.

What Type of Cloud is Right for Me?

Wed, 19 Jan 2011 09:39:05 -0500

The first known reference to the “Cloud” as it related to computing was in Douglas Parkhill’s 1966 book The Challenge of Computer Utility. Parkhill explained his conception of a "Private Computer Utility." He compared computing with the electrical industry and its extensive use of hybrid supply models. When the electricity grid was built, private on-site power generators were quickly cycled out. No longer did local businesses have to build, buy and maintain the hardware to create electricity, which was expensive both from a hardware as well as a human resource perspective. While it did carry some risk, electricity as a utility made sense in terms of finance and risk management.

In the world of Cloud Computing, there are three different types of “clouds” - public clouds, private clouds and hybrid clouds. Depending on what type of service or data you’re dealing with, you’ll want to compare the different options of what private, public and hybrid can offer. In most cases, the most important variable is the degree of security and management the hardware or application requires. While we as an industry like to think that Cloud Computing is new, it’s not. The concept was coined forty years earlier.

With that said, it’s time to figure out which cloud architecture is right for you.

Private Cloud
A private cloud is one in which the services and infrastructure are maintained on a private network—generally a local datacenter within an organization. These clouds offer the greatest level of security and control, but they still require the company to purchase and maintain all the software and infrastructure, which can significantly reduce cost savings. A private cloud is the obvious choice when:
·   Data is your business, so security and control are paramount on your list of requirements.
·   Your company is large enough to run a hyper-scalable cloud datacenter efficiently and effectively on its own. This generally implies large enterprises.
·   Your business is bound and gagged to conform to strict security and data privacy issues as well as compliance mandates like PCI-DSS and SOX.

Some vendors use the term “Private Cloud” to describe products and services as “cloud-like”, or that are described in their market-ecture as the ability to “emulate cloud computing on private networks.” These products are often virtualized solutions that have the ability to host applications and Virtual Machines in a company datacenter. Frankly, I see little value in “Private Clouds” as they’re more focused on virtualization than cloud computing.

Don’t get me wrong, I think virtualization has its place as well. It’s certainly used in cloud computing, but that doesn’t make cloud computing what it is. Virtual technologies are valuable to businesses but often tend to obscure the full capabilities of cloud computing. The term "private cloud" borders on deceptive advertising; it fails to deliver on the potential of cloud computing and those who attempt to use it are hanging onto the coattails of the cloud.

Depending on your industry, though, private clouds do offer some benefits including shared hardware costs, quick recovery from failure and upscaling/downscaling depending on demand. And that’s fantastic. But the organization still has to buy, build, support and manage the infrastructure. This solution doesn’t benefit from up-front capital costs and it lacks the economic model that makes cloud computing so compelling in the first place.

Public Cloud
A public cloud is one in which the services and infrastructure are provided off-site over the internet. At its essence, “Cloud Computing” refers to the public cloud. These clouds offer the greatest level of efficiency in shared resources as well as efficiency in cutting spending. However, they are also more vulnerable than private clouds. A public cloud is the obvious choice when:
·   You need incremental capacity, or, the ability to add computer capacity for peak times. When the proverbial crap hits the fan, you’ll have capacity available to handle that, but those resources can be used by other VMs for their own tasks when not in peak capacity mode.
·   Your standardized tools and applications are used by many employees. Examples include e-mail, contact management systems or a company intranet site.
·   You need a sandbox to develop applications across geographic locations. Development and testing are a great use case for Cloud, especially when collaboration is needed.
·   You have a SAAS (Software as a Service) application which is offered from a vendor who takes a hard line approach to security.

Public Cloud as a computing concept offers cheap, commoditized computing resources which outweigh the benefits of in-house resources that have limited added value (no capex, access to resources everywhere at any time, minimal support costs and employees for maintaining the resource, shared overall costs and no peak load concerns).

But one of the concerns associated with public clouds is security and reliability. Make sure you have your security and compliance/governance strategies well planned as the short term cost savings could become a long term nightmare.

Hybrid Cloud
A hybrid cloud offers a variety of public and private options with multiple providers. By using a hybrid approach, you’re able to spread things out over a number of providers to keep each aspect of your business in the most efficient possible environment. The major downside here is having to keep track of multiple security platforms and make sure all aspects of your business can communicate with each other. So, if the following situations describe your environment, then the hybrid cloud may be the best option for you:
·   Your company uses a SaaS application, but has security concerns. Private clouds are often used with VPNs (Virtual Private Networks) for additional security.
·   When your market is multiple verticals, you may be in a situation where you want to use private clouds for client interaction, but their sensitive data is kept in a Private cloud. This is an optimal use case for Hybrid Clouds.

When managing private, public and traditional datacenter models all at the same time, management can become complex. Maintaining a tool which will federate these separate pieces for the sake of SLAs and troubleshooting becomes the challenge.

Most of what people are calling "private clouds" share a number of qualities with public clouds and can thus be classed as a "hybrid cloud" architecture. Most large enterprises will be looking to run a hybrid architecture for several years to come (though many early adopters have already taken the plunge). The waters are tepid in different clouds for different reasons.

In summary, Public, Private and Hybrid cloud environments can all viable solutions based on your use case. Public clouds offer the greatest cost savings, but the least amount of security and control. Private clouds offer just the opposite, with costs being much higher due to hardware/software and maintenance costs; however, security and control are supreme. Hybrid is the best of both words, but can often be very complex to manage.
Take a step back, identify your use cases and requirements and then take the plunge. Cloud is not just the future. It’s today.

Disaster Recovery And The Cloud: A Recipe for Success

Thu, 13 Jan 2011 09:35:59 -0500

Cloud computing and disaster recovery are like peanut butter and chocolate - two great flavors that taste even better together. There are several companies that have recently entered the “Disaster Recovery in the Cloud” arena, offering services such as data backup, business continuity and disaster recovery services for MSPs packaged together into a single suite. Before jumping on that bandwagon, let’s deep dive into these three topics with a bit more detail.

When businesses hear the phrase “Cloud Computing,” their initial question is (understandably) how much control they will retain. There is the fear and uncertainty of added risk as well as the fear of losing control of their data. This is a common thought pattern, and is completely justified.

So why move to the cloud?
The promise of cost savings derived from cloud computing is very attractive, but concrete financial returns are not always quickly achieved. Except, perhaps, when it comes to disaster recovery.
Cloud Computing, by nature, is a distributed concept with some backup already available. However, the concern of the reduced reliance on local infrastructure on physical hardware, and the subsequent perceived risk of trusting another vendor with the business continuity of your business certainly gives some organizations pause. With due diligence and an understanding of the available feature-set, though, cloud disaster recovery is a very attractive solution. The additional cost savings doesn’t hurt, either.

At the end of the day, cloud-based disaster recovery allows you to add important capabilities to your IT infrastructure at a reduced cost—especially when you consider your alternative options.
Companies that have balked at the cost of building out their own disaster recovery infrastructure often find the cloud more cost effective. Offloading the expense hardware, software and network infrastructure to be a “what-if” solution can be very expensive. Think about it: your primary and secondary gear as well as the maintenance and support of lot can be tough to swallow, especially considering failover gear just sits in standby until something fails. Why pay for a room full of gear with the sole purpose of waiting for a failure?
Many companies do in fact use an outside vendor for disaster recovery, so a move to the cloud isn’t much of a change.

Here are some major points you should keep in mind when thinking about your approach to cloud disaster recovery:

1. Make sure your cloud provider offers business continuity as a necessary service, and that it’s part of your SLA.
2. The cloud provider should be in the know about its hardware/software and any sort of managed gear for failures. They should have multiple data-centers in multiple locations in order to quickly move data around or bring up backup and additional VMs if necessary.
3. Choose business continuity. Backup solutions are wonderful, but take it a step further with business continuity. Although they sound one-in-the-same, the key difference is offline backups vs. online, or online-accessible at a different location. Simply flip the switch, and you’re back in business.

While one of the key drivers for cloud computing is reduced cost and more feature-set, restoring data in the cloud is also much quicker than other disaster-recovery scenarios, and there’s no hardware to buy. A full disaster recovery solution at a reduced cost will sweeten the pot. Cloud computing and disaster recovery, much like peanut butter and chocolate, have a tasty future ahead of them, with the sweetest part coming when you see the savings on your bottom line. So, if you choose to dip your spoon into cloud security, these points can be your key ingredients for a delicious recipe that saves your organization money and offers a safe, more secure situation with greater accessibility.

“Big Data is information overload. Organized.”

Thu, 6 Jan 2011 12:39:02 -0500

Recently I was quoted in an article on CNet about “Big Data”. Dave Rosenberg made some excellent observations about how Big Data is being handled, and spotlighted some companies that are developing FOR Big Data.

But it got me thinking…Do most people really understand what Big Data is?

Big Data is a phrase becoming increasingly more popular. It’s a statement which implies that we’re moving from the Terabyte age to the Petabyte age. It has become the latest challenge for large enterprises and government. It’s not just a buzz word. It’s a real problem that IT departments everywhere are struggling with. And storage isn’t the hardest part of Big Data. In fact, storage is easy. We have the ability to store petabytes and exabytes of data today. But making SENSE of that data…that is the real challenge.

Big Data, as with most quantifications, is a relative term.

How do you know when you have Big Data? Here’s how. If you have to ask yourself “How are we going to store this, organize this and manage this? How are we going to get information out of this that’s useful?”...then you have Big Data.

Martin Wattenberg, a mathematician and computer scientist at IBM's Watson Research Center in Cambridge, Massachusetts says, “You can talk about terabytes and exabytes and zettabytes, and at a certain point it becomes dizzying. The real yardstick to me is how it compares with a natural human limit, like the sum total of all the words you'll hear in your lifetime. That's surely less than a terabyte of text. Any more than that and it becomes incomprehensible by a single person, so we have to turn to other means of analysis: people working together, or computers, or both.”

And he’s right. The more you have, the harder it is to work with. But, if analyzed, you can glean incredible information.

Data on a corporate network, whether it be database data, tons and tons of flat files, or even log data is often unstructured and hard to make sense of. For some, this is a nightmare. The capture and storage of mass amounts of data is a thorn in the side of the average CTO. But on the academic side, on the research side, on the private sector side – this data is a goldmine. Being able to trend events over time, to build predictive models, and to index the entire internet... that’s big. To use it as a performance tool and to identify throughput and use cases... that’s big. Big Data then becomes a decision making tool.

But what caused this?

Over time, disk prices dropped as data storage requirements went ever skyward. And with the advent of cheap storage, the need to delete that data went down. With more and more data being stored and going online every day, suddenly the focus shifted to data security. How do we protect our data? How do we know if our data has been stolen? If it’s been stolen, who stole it?

Before we knew it...storing data for the sake of forensics was on the rise, and after a rash of IP and user data thefts, compliance from the Payment Card Industry kicked in, as did the scourge of all public companies.... compliance to Sarbanes Oxley (SOX). Soon HIPAA grew some teeth in the healthcare industry, and ISO17799 came into effect. All of these mandates required audit trails for a period of time from three months to seven years. That’s when the log data piece of Big Data became a major part of the pie. Think about it. We’re talking about the storage of every log message from every device on a corporate network for up to seven years!

NOW we’re talking about BIG DATA.

Soon you may find yourself asking, “How are we going to store our data, organize our data and manage our data? How are we going to get information out that’s useful?”

It’s at that point you’ll realize that you too have Big Data.