Demande.tn

🔒
❌
Il y a de nouveaux articles disponibles, cliquez pour rafraîchir la page.
Hier — 22 septembre 2017OpenStack Superuser

Four ways to build open source community in your company

Par Anne Bertucio

LOS ANGELES — The Open Community Conference was one of five tracks at the Linux Foundation’s recent Open Source Summit North America. Chaired by Jono Bacon, the tracks covered community management, open source office establishment and open source culture in organizations.

If you missed it, here are four takeaways worth applying to your work.

Find the right community manager

Community managers seem to come from all walks of life: the engineer; the marketer; the team lead. But what makes the best community manager? Alan Clark, director of industry initiatives, emerging standards and open source at SUSE, has been involved in open source communities for more than 15 years. He’s seen trends in community management come and go, but the constant success is when a community manager reflects the values of the community.
“Community management is a little different everywhere. It depends on the personalities of the managers and the personalities of the community.” Clark has been involved with both openSUSE and OpenStack, and offered the two as examples. “In the openSUSE community there’s very little marketing. It’s very developer-focused and less about promotion, social outreach and marketing. Most [community managers in openSUSE] come from engineering backgrounds, and that’s very appropriate, but in OpenStack, that wouldn’t reflect the needs of the overall community.”

“OpenStack has set core values that it has embraced, particularly transparency. The successful community manager has to personify those and be the enforcer of those. Community managers are the front line and they set the tone and the attitude of the community.”

Delete the bias against marketing

Marketing is often perceived as a lower status role in the tech industry, as Deirdré Straughan called out in her presentation, which means that emerging open source projects or organizations working with open source often think they don’t need or want marketing. This is a naive mistake, as good marketing––not the kind that insults your intelligence or abuses your information––is what communicates your open source work to the world. Toolkits let developers access your project, blueprints and roadmaps help people evaluate, training gives the skills to use and contribute, and yes, this is all “marketing.” Straughan challenged the audience to reconsider their bias against marketing and to recognize that great marketing is part of a successful open source strategy.

 

Identify the invested parties in your organization

Nithya Ruff, senior director of the open source practice at Comcast, and Duane O’Brien, open source programs evangelist at PayPal, urged people trying to start open source program offices in their organization to spend time identifying who at their organization has a vested interest in formally engaging with open source communities.

Is the legal team getting overwhelmed with requests for contribution information? Is engineering eagerly consuming open source? Is marketing desperate for help to clearly communicate the open-source basis for their products? Finding who has this special interest can act as the impetus to kickstart more formal engagement with open source communities.

Start with “inner source”

Shilla Saebi, open source community leader at Comcast, has found success by establishing open source practices internally, “inner source” as she’s dubbed it, which helps the organization and individuals develop policies around open source and become comfortable with contributing so they can successfully engage with open source communities.

Inner source looks similar to externally facing open source: licensing discussions with the legal team and meetups. At Comcast there are internal slack channels dedicated to open source projects: the OpenStack channel has more 1,200 members; Cloud Foundry, 900. Inner source acts as a practice for engaging with open source communities, growing familiarity and confidence that translates to external engagement.

Share your thoughts on what makes open source communities tick with @WhyHiAnnabelle and @Superuser on Twitter.

Cover Photo // CC BY NC

The post Four ways to build open source community in your company appeared first on OpenStack Superuser.

À partir d’avant-hierOpenStack Superuser

Paths to autonomous workload management at the edge

Par Gregory Katsaros

It’s been estimated that in the next three to five years years the number of connected devices will reach a staggering 50 billion globally.

Even if that number sounds extreme, it’s undeniable that advancements in silicon technology (e.g. shrinking of computing and sensor components) and the evolution of 5G networks will definitely drive the rise of capable edge devices and create much more relevant use cases.

Given that scenario, when it comes to the technology to support it, several associated challenges need to be  identified and addressed first.

The recent OpenDev conference aimed to raise the awareness and foster collaboration in this domain. The topic of autonomous workload management at the edge was one of the working sessions at the conference. It focused on technical constraints, requirements and operational considerations when an application/workload has to be orchestrated at the edge. The main assumption is that several edge computing use cases (e.g. micro-edge/mobile edge such as set-top boxes, terminals etc.) will demand highly autonomous behavior due to connectivity constraints, latency, cost etc. The scale of edge infrastructure might also drive this autonomous behavior of the edge platform while the central management of all these edge devices will be an enormous task. To this end, several operational considerations were discussed as summarized below.

Workload orchestration

When it comes to autonomous workload management at the edge, effective orchestration is the most important issue. No matter if it is on bare metal, virtual machines or application containers, the need for automation and use of software-defined methodologies is apparent. In the NFVi world nowadays there’s a clear tendency towards model-driven declarative approaches such as TOSCA for addressing orchestration. The expectation is that the edge platform should include a functional component responsible not only for the orchestration and management of resources (e.g. VMs or container) but also the running applications or VNFs. Such an entity takes care of runtime healing and scaling as well as provisioning (edge-side or centrally triggered). Even if the goal is the autonomous operation of the workload orchestration, it’s expected that there will be some kind of central orchestration entity (or regional manager) that will still keep a reference of the edge state or drive provisioning of the edge. It feels like the absolute autonomous behavior of a mesh-like edge network is a bit futuristic and difficult to have in the short term, at least at large scale.

State and policy management

Autonomous workload orchestration also implies autonomous state management. In order to effectively orchestrate the state not only of the hosting platform (e.g. the virtual machine or container) has to be captured but also the services or application state should be monitored. Today, most of state management operations are handled by orchestration components (at the resources level) and the applications/VNF vendors themselves. However, there is no combined view of the state which results in pretty primitive fault handling: when the state of a workload is faulty, then the whole resource is restarted. In addition, the Service Function Chaining (SFC) or the applications’ micro-services paradigm introduce a composable state concept which potentially has to be considered. State abstraction is also important: a regional orchestrator might not need to keep the state of all components of an SFC but just an abstracted state of the whole SFC. On the other hand, the edge orchestrator must know the state of each service in the chain. The policy enforcement should also follow the same pattern with the state propagation. All-in-all, the above-mentioned points suggest the need for a more capable state management structure that can tackle these new requirements. Whether the existing orchestrators are responsible for these features or new modules and/or protocols have to be invented is up for discussion.

Managing packages of local repositories

Autonomous operation of the workload orchestration at the edge requires local repositories of images and software packages. Assuming that the connection of the edge device to the core network is either unreliable or very thin, only the control plane operations should be transferred over the air. It’s being suggested that the orchestration systems should explore cached or local repositories that would be synchronized on-demand by some central component. Multicast features for pushing updates to the edge repositories should be considered too, especially if the scale of edge devices increases exponentially.
Even if most of the issues and ideas discussed here are nor new neither unique, we can’t assume that the technologies and systems developed for data center operations can directly solve edge computing use cases. Perhaps it’s best to look at the problem with a fresh perspective and try to architect an edge computing platform that could serve all use cases (from micro to large edge), leveraging existing technology where and when possible but also investing in new.

About the author

Gregory Katsaros is a services, platforms and cloud computing expert who has been working for several years in research and development activities and projects related to the adoption of such technologies and transformation of the industry. Katsaros has special interest in services and resources orchestration as well as network function virtualization and software defined networking technologies.

He holds a Ph.D. from the National Technical University of Athens on “Resource monitoring and management in Service Oriented Infrastructures and Cloud Computing” and has been contributing to research communities with research and code.

In the last few years, he’s  been leading projects related to services and cloud computing, distributed systems, interoperability, orchestration, SDN, NFV transformation and more. He’s interested in transforming the telco and enterprise sector by embracing automation and orchestration technologies, as well as investigating edge computing.

Cover Photo // CC BY NC

The post Paths to autonomous workload management at the edge appeared first on OpenStack Superuser.

Takeaways from the first Diversity Empowerment Summit

Par Allison Price

LOS ANGELES — The expo hall booths were already packed up, but a few hundred Open Source Summit attendees returned to the JW Marriott to talk about some of the most persistent problems in tech at the first Diversity Empowerment Summit (DES).

The daylong get-together had a broad scope: diversity of gender, race, nationality and religion are all focus areas where initiatives aim to increase awareness and actions around increasing representation in open-source communities.

Only 3% of women are open source contributors in 2017 – @marinaz #OSSummit #DiversityEmpowermentSummit

— Alpha (@GaelleTjat) September 14, 2017

From game theory to chaos theory

Kate Ertmann kicked off DES by setting the stage for what’s happening when new generations enter the workplace.

“Millenials are making chaos in the workplace,” she said. “Generation Y and Generation Z will not tolerate when there is no diversity in the workplace and inauthentic attempts around being diverse. They will not tolerate when they are not reflected in the workplace.”

Ertmann said this illustrates a shift from game theory to chaos theory and the need to break down the 20th-century workplace to build anew.

Steps include reevaluating family leave, coverage for longer medical situations—particularly among all configurations of families—and flexible work hours. She applauded the onsite childcare offered at the Open Source Summit, which extended an opportunity to parents—and mothers in particular—who previously may have been unable to attend.

When asked by an attendee how to create a more welcoming workplace environment, Ertmann says “just ask. Creating feedback loops is the biggest opportunity to create a more inclusive workplace.”

Sometimes you need to break down old constructs to build up a new more inclusive world. @gok8 #LFDES pic.twitter.com/lpCXJUSMNR

— Nithya Ruff (@nithyaruff) September 14, 2017

Chasing Grace

A trailer for the upcoming documentary series “Chasing Grace” was played to a captivated audience. The series was created to share the stories of women who faced adversity and became change agents in the tech workplace. Inspired by pioneering computer scientist Grace Hopper, early sponsors include the Linux Foundation and the Cloud Foundry Foundation.

Producer Jennifer Cloer was joined in a panel by Comcast’s Nithya Ruff, the Cloud Foundry Foundation’s Abby Kearns and ScoutSavvy’s Kathryn Brown.

“My hope is that these stories will surface how women are navigating adversity and provide a blueprint for women who want to join the tech industry,” Cloer said.

When asked about the current culture of the tech industry, both Ruff and Brown encouraged attendees to include women in the conversation at the workplace around benefits, abandoning assumptions and processes that don’t engage women in the discussion.

Ruff did suggest that there has been a shift in conversation.

“The dialogue has changed from trying to change the woman to fit the workplace to changing the workplace to be welcoming for women,” she said.

Research from Chasing Grace indicates that women are twice as likely to leave the tech industry than men due to the lack of inclusivity in the industry, so Cloer asked panelists what has kept them in the industry.

Emphasizing the impact of having diverse voices represented, Ruff referenced a previous presentation by Amy Chen of Rancher Labs, who said that when she didn’t have a seat at the table, she brought her own table.

“I work with incredibly bright people and I get to change the world instead of just being a consumer of the world,” Ruff said. “Some of the biggest changes will come if our voices are just heard.”

Pay it back, pay it forward

Munira Tayabji introduced a concept early in the day that was repeated by several speakers and in the hallway track—remember to pay it back and pay it forward through mentoring and networking events.

Chen credited a female mentor who had helped her land an interview that changed her career.

“I would not be standing up here if I didn’t have others who publicly supported me,” said Chen, echoing Tayabji’s sentiment.

“sponsorship vs mentorship – both are needed” –@TheAmyCode – more details e.g. https://t.co/ivevDC8d2m #LFDES #OSSummit

— Erik Riedel (@er1p) September 14, 2017

During the “Chasing Grace” panel, Brown credited this notion of mentorship and the opportunity to mentor as the driving force that keeps her dedicated to the tech industry.

“Getting into tech made me realize that I can be that person that can do good things at scale,” she said. “I feel the responsibility to the women who have invested in me, to younger women to be a role model and a responsibility to do something big that positively impacts the world.”

Join the conversation – get involved in mentoring activities through the Linux Foundation or the OpenStack Foundation and learn more about how you can get involved in the conversations around tech diversity.

 

 

Cover Photo // CC BY NC

The post Takeaways from the first Diversity Empowerment Summit appeared first on OpenStack Superuser.

Building a collaborative community around open source

Par Allison Price

LOS ANGELES — In an ant colony,  members work together to create a thriving superorganism that’s exponentially more powerful than any single contributor. The same can be said for open source where vast networks of global members are transforming technology, science and culture.  This is one of the most striking takeaways from the first Open Source Summit held recently in Los Angeles and hosted by the Linux Foundation.

Hundreds of open-source enthusiasts, developers and end users gathered to discuss the principles of open source and how they can be applied to foster collaboration, innovation and solve real- world problems like the amount of water bottles that end up in the ocean. Under the umbrella Open Source Summit, the conference organized content into five sub-events: CloudOpen, ContainerCon, LinuxCon, the Open Community Conference and the Diversity Empowerment Summit.

Jim Zemlin, executive director of the Linux Foundation, kicked off the event and acted as emcee for the keynotes. Setting the stage for the Summit’s theme “inspiration everywhere,”  Zemlin hit on the explosive growth of open source.

“Open source isn’t growing – it’s actually accelerating exponentially in terms of its influence in technology and society,” Zemlin said, citing 23 million open source developers worldwide and 41 billion lines of code to back up his point.

You can’t have such an explosive growth of open source without a few challenges, however. Zemlin said the biggest bottleneck to open source growth in organizations is that they don’t know how to participate in open source. Employees need training on how to thrive in the “colony.”

“Projects with sustainable ecosystems are the ones that matter,” Zemlin said. “Successful projects depend on members, developers, standards and infrastructure to develop products that the market will adopt.”

How to build and measure success in open source 

After establishing the what—organizations must be involved and contributing to open source in order for the project to be successful—Zemlin answered the how by introducing the Open Source Guides for the Enterprise. Developed with the TODO Group, this resource includes best practices on starting with open source in your organization, including how to create an open source program and tools for managing open source.

To gauge the success of these open source initiatives, Zemlin also announced the Linux Foundation’s newest project, CHAOSS, Community Health Analytics Open Source Software.

New project CHAOSS working on community health metrics and code https://t.co/ZEsmyUtmi1 #opensource #ossummit pic.twitter.com/fz9aqfR9QS

— Mike Dolan (@mdolan) September 11, 2017

Achieved through collaboration of multiple organizations and communities like the OpenStack Foundation, Bitergia, Mozilla, the University of Missouri and more, CHAOSS will focus on creating analytics and metrics to help define community health.

Grow a community, not a crowd

On Tuesday, Joseph Gordon-Levitt, director, entrepreneur and most familiar to tech folks for his recent role in  “Snowden,” took the keynote stage to talk about the parallels between open source culture and his online collaborative production company HitRecord.

Although some of his insights were more focused on the entertainment industry, Gordon-Levitt made two key relatable points about growing a functioning community: Instead of thinking about the crowd, think about community, and instead of socializing, collaborate.

Joseph Gordon-Levitt sharing how working together fosters a new type of creativity at #ossummit @EventsLF pic.twitter.com/bI5zTNvVz5

— Bill Stark (@bill__stark) September 12, 2017

“Every member of a community is a unique individual,” he said. “The strength of a community is less about the quantity of people and more about the quality of people and their interactions.”

Around collaboration, he said the strength of people connecting online to work towards a common goal, whether it’s on HitRecord or GitHub—hitting on the importance of collaboration that Zemlin discussed the previous day.

“As we move forward in the future, what kind of impact will community have?” Gordon-Levitt said. “If people can do more than just connect but also collaborate, I start to feel optimistic about the future. It’s not about about the capabilities of the technology, it’s what the technology can help us do as human beings.”

Zemlin and Gordon-Levitt merely set the stage for a week centered around the power of open source, collaboration and the power that a community can have, but breakouts throughout the week continued to share real examples of how what open source collaboration can achieve.

Nithya Ruff from Comcast and Duane O’ Brien from PayPal presented the value of building open source departments within their own company, Joe Leaver from Mercedes-Benz Research and Development talked about the power of collaborating with a new open source partner, composure.ai to do research around autonomous driving and Chris Price from Ericsson discussed the value and impact of working with various open source communities like ONAP, OPNV and OpenStack to accelerate the shift from 4G to 5G networks.

For more about the inspiration of collaboration from the Open Source Summit, you can catch the lineup of keynote videos online and stay tuned on Superuser for more event coverage.

Cover Photo // CC BY NC

The post Building a collaborative community around open source appeared first on OpenStack Superuser.

Interested in reading up on OpenStack? Check out these books

Par Superuser

The OpenStack Marketplace — your one-stop shop for training, distros, private-cloud-as-a-service and more — is now offering a selection of technical publications, too. The listings are not affiliate links, but offered as a way to highlight the efforts of community members.

Under the new “books” heading, you’ll find titles by Stackers including “Mastering OpenStack,” “OpenStack for Architects,” “OpenStack Networking Essentials,” and “OpenStack: Building a Cloud Environment.”

Are you the author of a book on OpenStack? To have your book included in the Marketplace, email ecosystemATopenstack.org and remember to reach out to editorATopenstack.org for a profile of your work — we love to feature books!

The post Interested in reading up on OpenStack? Check out these books appeared first on OpenStack Superuser.

How to rock dirty clouds done cheap

Par Matthew Treinish

Matthew Treinish is part of IBM’s developer advocacy team and has been an active contributor to OpenStack since the Folsom cycle. He was previously a member of the OpenStack TC (technical committee) and a OpenStack QA program PTL (project technical lead). This post first appeared on his blog.

I gave a presentation at the OpenStack Summit in Boston with the same title. You can find a video of the talk here: https://www.openstack.org/videos/boston-2017/dirty-clouds-done-dirt-cheap.

This blog post will cover the same project, but it will go into a lot more detail, which I couldn’t cover during the presentation.

Just a heads up, this post is long!  I try to cover every step of the project with all the details I could remember. It probably would have made sense to split things up into multiple posts, but I wrote it in a single sitting and doing that felt weird. If you’re looking for a quicker overview, I recommend watching the video instead.

The project scope

When I was in college, I had a part time job as a sysadmin at a HPC research lab in the aerospace engineering department. In that role I was responsible for all aspects of the IT in the lab, from the workstations and servers to the HPC clusters. In that role I often had to deploy new software with no prior knowledge about it. I managed to muddle through most of the time by reading the official docs and frantically google searching when I encountered issues.

Since I started working on OpenStack I often think back to my work in college and wonder if I had been tasked with deploying an OpenStack cloud back then would I have been able to? As a naive college student who had no knowledge of OpenStack would I have been successful in trying to deploy OpenStack by myself? Since I had no knowledge  of configuration management (like puppet or chef) back then I would have gone about it by installing everything by hand. Basically the open question from that idea is how hard is it actually to install OpenStack by hand  using the documentation and google searches?

Aside from the interesting thought exercise I also have wanted a small cloud at home for a couple of reasons. I maintain a number of servers at home that run a bunch of critical infrastructure. For some time I’ve wanted to virtualize my home infrastructure mainly just for the increased flexibility and potential reliability improvements.  Running things off a residential ISP and power isn’t the best way to run a server with a decent uptime.  Besides virtualizing some of my servers it would be nice to have the extra resources for my upstream OpenStack development, I often do not have the resources available to me for running devstack or integration tests locally and have to rely on upstream testing.

So after the Ocata release I decided to combine these 2 ideas and build myself a small cloud at home. I would do it by hand (ie no automation or config management) to test out how hard it would be. I set myself a strict budget of $1500 USD (the rough cost of my first desktop computer in middle school, an IBM Netvista A30p that I bought with my Bar Mitzvah money) to acquire hardware. This was mostly just a fun project for me so I didn’t want to spend an obscene amount of money. $1500 USD is still a lot of money, but it seemed like a reasonable amount for the project.

However, I decided to take things a step further than I originally planned and build the cloud using the release tarballs from http://tarballs.openstack.org/. My reasoning behind this was to test out how hard it would be to take the raw code we release as a community and turn that into a working cloud. It basically invalidated the project as a test for my thought exercise of deploying the cloud if I was back in college (since I definitely would have just used my Linux distro’s packages back then) but it made the exercise more relevant for me personally as an upstream OpenStack developer. It would give me insight as to where what we’re there are gaps in our released code and how we could start to fix them.

Building the Cloud

Acquiring the Hardware

The first step for building the cloud was acquiring the hardware. I had a very tight budget and it basically precluded buying anything new. The cheapest servers you can buy from a major vendor would pretty much eat up my budget for a single machine. I also considered building a bunch of cheap desktops for the project and putting those together as a cloud. (I didn’t actually need server class hardware for this cloud) But for the cost the capacity was still limited. Since I was primarily building a compute cloud to provide me with a pool of servers to allocate My first priority was the number of CPU cores in the cloud. This would give me the flexibility to scale any applications I was running on it. With that in mind I decided on the priority list for the hardware of:

  1. Number of Cores
  2. Amount of RAM
  3. Speed of CPU

The problem with building with desktop CPUs is (at the time I was assembling pieces) the core count / USD was not really that high for any of the desktop processors. Another popular choice for home clouds is the Intel NUCs, but these suffer from the same problem. The NUCs use laptop processors and while reasonably priced you’re still only getting a dual or quad core CPU for a few hundred dollars.

It turns out the best option I found for my somewhat bizarre requirements was to buy used hardware. A search of eBay shows a ton of servers from 8 or 9 years ago that are dirt cheap. After searching through my various options I settled on old Dell PowerEdge R610, which was a dual socket machine. The one I ordered came with 2 Intel Xeon E5540 CPUs in it. This gave me a total of 8 physical cores (or 16 virtual cores if you count HyperThreading/SMT) The machines also came with 32 GB of RAM and 2x 149GB SAS hard drives. The best part though was that each machine was only $215.56 USD. This gave me plenty of room in the budget, so I bought 5 of them. After shipping this ended up costing only $1,230.75. That gave me enough wiggle room for the other parts I’d need to make everything working. The full hardware specs from the eBay listing was:

Although, the best part about these servers were that I actually had a rack full of basically the same exact servers at the lab in college. The ones I had back in 2010  were a little bit slower and had half the RAM, but were otherwise the same. I configured those servers as a small  HPC cluster my last year at college, so I was very familiar with them. Although back then those servers were over 10x the cost as what I was paying for them on eBay now.

The only problem with this choice was the hardware, the Xeon E5540 is incredibly slow by today’s standards. But, because of my limited budget speed was something I couldn’t really afford.

Assembling the Hardware

After waiting a few days the servers were delivered.  That was a fun day, the FedEx delivery person didn’t bother to ring the door bell. Instead I heard big thud outside and found that they had left all the  boxes in front of my apartment door. Fortunately I was home and heard them throw the boxes on the ground, because it was raining that day. Leaving my “new” servers out in the rain all day would have been less than an ideal way to start the project . It also made quite the commotion and several of my neighbors came out to see what was going on and watched me as I took the boxes inside.

After getting the boxes inside my apartment and unboxed, I stacked them on my living room table:

My next problem with this was where to put the servers and how to run them. I looked at buying a traditional rack, however they were a bit too pricey. (even on eBay) Just a 10U rack looked like it would cost over $100 USD and after shipping that wouldn’t leave me too much room if I needed something else. So I decided not to go that route. Then I remembered hearing about something called a LackRack a few years ago. It turns out the IKEA Lack table has a 19 inch width between the legs which is the same as a rack. They also only cost $9.99 which made it a much more economical choice compared to a more traditional rack. However, while I could just put the table on the floor and be done with it, I was planning to put the servers in my “data closet” (which is just a colorful term for my bedroom closet where I store servers and clothing) but I didn’t want to deal with having to pick up the “rack” every time I needed to move it. So I decided to get some casters and mount them to the table so I could just push the server around.

Once I got the table delivered, which took a surprisingly long time, I was able to mount he casters and rack the servers. As I put each server on the table I was able to test each of them out. (I only had a single power cable at the time, so I went one at a time) It turns out that each server was slightly different from the description and had several issues:

  • 4x8GB of RAM not 8x4GB
  • Memory installed in wrong slots
  • Dead RAID controller battery
  • Came with 15k RPM hard drives not 10k RPM

Also, the company that is “refurbishing” these old servers from whatever datacenter threw them away totally strips the servers down to the minimum possible unit. For example, the management controller was removed, as was the redundant power supply. Both of these were standard feature from Dell when these servers were new. Honestly, it makes sense, the margins on reselling old servers can’t be very high so the company is trying to make a little profit. I also really didn’t need anything they took out as long as the servers still booted.  (although that management controller would have been nice)

Once I put all 5 servers on the rack:
After getting everything mounted on the rack it turns out I also needed a bunch of cables and another power strip to power all 5 at once. So I placed an order with Monoprice for the necessary bits and once they arrived I wired everything up in the data closet:

After everything was said and done the final  bill of materials for all the hardware was:

Part Name Link Price Number Shipping & Tax Total
Dell PowerEdge R610 Virtualization Server 2.53GHz 8-Core E5540 32GB 2x146G PERC6 http://www.ebay.com/itm/191545700823 $215.56 5 $152.95 $1,230.75
LACK Side table, yellow http://www.ikea.com/us/en/catalog/products/40104270/#/10324278 $9.99 1 $13.65 $23.64
6 Outlet Power Strip https://www.monoprice.com/product?c_id=109&cp_id=10907&cs_id=1090703&p_id=13692&seq=1&format=2 $7.90 1 10.49 $18.39
3ft 16AWG Power Cord https://www.monoprice.com/Product?p_id=5285 $1.60 5 0 $8.00
FLEXboot Series Cat6 24AWG UTP Ethernet Network Patch Cable, 10ft Orange https://www.monoprice.com/product?p_id=9808 $1.76 10 0 $17.60
Casters https://www.amazon.com/gp/product/B01FJ97E64/ref=od_aui_detailpages00?ie=UTF8&psc=1 $29.99 1 0 $29.99
Grand Total: $1,328.37

Installing the Operating System

After getting the working set of hardware the next step was to install the operating system on the servers.  As I decided in the original project scope I was planning to follow the official install guide as much as possible.  My operating system choice would therefore be dictated by those covered in the guide, the 3 Linux distributions documented were OpenSUSE/SLES, RHEL/CentOS, and Ubuntu. Of those the 3 my personal choice was Ubuntu which I personally find the easiest to deal with out of the choices. Although, looking back on it now if I were to do an install during job college I definitely would of have used RHEL. Georgia Tech had a site license for RHEL and a lot of software we had commercial licenses for only had support on RHEL. But, my preference today between those 3 options is to use Ubuntu.

I created a boot usb stick for Ubuntu Server 16.04 and proceeded to do a basic install on each server. (one at a time) The install itself just used the defaults, the only option I made sure was present was the OpenSSH server. This way once I finished the initial install I didn’t have to sit in front of the server to do anything.  I would just install any other packages I needed after the install from the comfort of my home office.  For the hostname I picked altocumulus because I think clouds should be named after clouds. Although, after I finished the project I got a bunch of better suggestions for the name like closet-cloud or laundry-cloud.

It’s worth pointing out that if the servers had come with the management controller installed this step would have been a lot easier. I could have just used that to mount the installer image and ran everything from the virtual console. I wouldn’t have had to sit in front of each server to start the install. But despite this it only took an hour or so to perform the install on all the servers. With the installs complete it was time to start the process of putting OpenStack on each server and creating my cloud.

Installing OpenStack

With the operating system installed it’s time to start the process of building the servers out. Given my limited hardware capacity, just 40 physical cores and 160GB of RAM, I decided that I didn’t want to sacrifice 1/5 of that capacity for a dedicated controller node. So I was going to setup the controller as a compute node as well.  My goal for this project was to build a compute cloud, so all I was concerned about was installing the set of OpenStack projects required to achieve this. I didn’t have a lot of storage (the 2 149GB disks came configured out of the box with RAID 1 and I never bothered to change that) so providing anything more than ephemeral storage for the VMs wasn’t really an option.

OpenStack is a large project with a ton of different projects, (the complete list of official projects can be found here) But, I find some people have trouble figuring out exactly where to get started or for configuration X where to get started. The OpenStack Foundation actually has a page with a bunch of sample service selections by application. The OpenStack Technical Committee also maintains a list of projects needed for the compute starter kit which was exactly what I was looking for. The only potential problem is the discoverability of that information.  It kinda feels like a needle in the haystack if you don’t know where to look.

It also turns out the install guide is mostly concerned with building a basic compute cloud (it also includes using cinder for block storage, but I just skipped that step) so even if I didn’t know the components I needed I would have been fine just reading the docs The overview section of the docs covers this briefly, but doesn’t go into much detail.

The basic service configuration I was planning to go with was:

With a rough idea of how I was planning to setup the software I started following the install guide on setting up the server.  https://docs.openstack.org/ocata/install-guide-ubuntu/environment.html# walks you through setting up all the necessary Operating System level pieces like configuring the networking interfaces and NTP. It also goes over installing and configuring  the service prerequisites like MySQL, RabbitMQ, and memcached. For this part I actually found the docs really easy to follow and very useful. Things were explained clearly and mostly it was just copy and paste the commands to set things up. But, I never felt like I was blindly doing anything for the base setup.

Installing and Configuring the OpenStack Components

After getting the environment for running OpenStack configured it was time to start installing the OpenStack components. Keystone is a requirement for all the other OpenStack services so you install this first. This is where I hit my first issue because I decided to use the release tarballs for the install. The install guide assumes you’re using packages from your Linux distribution to install OpenStack. So when I got to the second step in the Installing Keystone section of the install guide it said run “apt install keystone” which I didn’t want to do. (although it definitely would have made my life easier if I did)

Installing From Tarballs

It turns out there isn’t actually any documentation anywhere that concisely explains the steps required to installing an OpenStack component on your system from source. I started doing searching on Google to try and find any guides. The first hit was a series of blog posts on the Rackspace  developer blog on installing OpenStack from source. However, a quick look at this showed this was quite out of date, especially for the latest version of OpenStack, Ocata, which I was deploying. Also, some of the steps documented there conflicted with the configuration recommended in the install guide. The other searches I found recommended that you look at devstack or use automation project X to accomplish this goal.  Both of these were outside the scope of what I wanted to do for this project. So for the tarball install step I decided to ignore the premise of just following the install guide and just used my experience working on OpenStack to do the following steps to install the projects:

  1. Download the service tarballs. I found the releases page has a good index by project to get the latest tarball for each project. Then extract that tarball to an easy remember location. (I created a tarballs directory in my home directory to store them all)
  2. Create the service user for each project. I ran:
    useradd -r -M $service
  3. Create the /etc and /var/lib directories for the service. For example on the controller node I used the following for loop in bash to do this for all the services:
    for proj in keystone glance nova neutron ; do
        sudo mkdir /etc/$proj
        sudo mkdir /var/lib/$proj
        sudo chown R $proj:$proj /etc/$proj /var/lib/$proj

    done

  4. Install the binary package requirements for the project. This is things like libvirt for nova, or libssl. Basically anything you need to have installed to either build the python packages or to run the service. The problem here is that for most projects this is not documented anywhere. Most of the projects include a bindep.txt which can be used with the bindep project (or just manually read like I did) to show the distro package requirements on several distributions, but few projects use it for this. Instead it’s often just used for just the requirements for setting up a unit (or functional) test environment. I also didn’t find it in any of the developer documentation for the projects. This means you’re probably stuck with trial and error here. When you get to step 6 below it will likely fail with an error that a library header is missing and you’ll need to find the package and install that. Or when you run the service something it’s calling out to is missing and you’ll have errors in the service log until you install that missing dependency and restart the service.
  5. Copy the data files from etc/  in the tarball into /etc/$service for the project. The python packaging ecosystem does not provide a way for packages to install anything outside of the python lib/ directories. This means that to install the required configuration files (like policy.json files or api paste.ini files) have to be copied manually from the tarball.
  6. After you do all of those steps you can use pip to install the tarball. One thing to note here is that you want to use constraints when you run pip. This is something I forgot installing the first few services and it caused me a ton of headaches later in the process. You can avoid all of those potential problems up front by just running:

    pip install -U -c “https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt?h=stable/ocata” $PATH_TO_EXTRACTED_TARBALL

If you’re using a different OpenStack release just replace “ocata” at the end of the url with that release name.

I wrote down these steps after I did the install mostly based on all of the issues I had during the install process. As you read through the rest of this post most of the issues I encountered could have been completely avoided if I did all of these up front.

It’s also worth noting that all of these steps are provided by the distro packages for OpenStack. This is exactly the role that packaging plays for users, and I was just going through the motions here because I decided to use tarballs. Python packages aren’t really designed for use in systems software and have a lot of limitations beyond the basic case of: put my python code in the place where python code lives. If you want more details on this Clark Boylan gave a good talk on this topic at the OpenStack summit in Boston.

I have also been trying to make a push to start documenting these things in the project developer docs so it’s not an exercise in misery for anyone else wanting to install from source. But, I’ve been getting push back on this because most people seem to feel like it’s a low priority and most people will just use packages. (and packagers seem to have already figured out the pattern for building things)

Creating systemd unit files

One thing that isn’t strictly a requirement when installing from source is creating systemd unit files. (or init scripts if you’re lucky enough to have a distro that still supports using SysV init) Creating a systemd unit file for each daemon process you’ll be running is helpful so you don’t have to manually run the command for each daemon. When I built the cloud I created a unit file for each daemon I ran on both the controller as well as all of the compute nodes. This enabled me to configure each service to start automatically on boot, but also encode the command for starting the daemons, so I could treat it like any other service running on the system. This is another thing that distro packages provide for you, but you’ll have to do yourself when building from source.

For an example this is the contents of my nova-api systemd unit file which I put in /etc/systemd/system/nova-api.service:

[Unit]
Description=OpenStack Nova API
After=network.target
[Service]
ExecStart=/usr/local/bin/novaapi configfile /etc/nova/nova.conf
User=nova
Group=nova
[Install]
WantedBy=multiuser.target
All the other service follow this same format, except for anything running under uwsgi (like keystone, more on that in the next section) , but you can refer to the uwsgi docs for more information on that.

Configuring Keystone

With the formula worked out for how to install from tarball I was ready to continue following the install guide.  The only other issue I had was setting up running the wsgi script under apache. By default keystone ships as a wsgi script that requires a web server to run it. The install guide doesn’t cover this because the distro packages will do the required setup for you. But, because I was installing from tarballs I had to figure out how to do this myself. Luckily the keystone docs provide a guide on how to do this, and include sample config files in the tarball. The rest of configuring keystone was really straightforward, the keystone.conf only required 2 configuration options. (one for the database connection info and the other for the token type) After setting those I had to run a handful of commands to update the database schema and then populate it with some initial data. It’s not worth repeating all the commands here, since you can just read the keystone section of the install guide. In my case I did encounter one issue when I first started the keystone service. I hit a requirements mismatch which prevent keystone from starting:

20170329 15:27:01.478 26833 ERROR keystone Traceback (most recent call last):
20170329 15:27:01.478 26833 ERROR keystone   File “/usr/local/bin/keystone-wsgi-admin”, line 51, in <module>
20170329 15:27:01.478 26833 ERROR keystone     application = initialize_admin_application()
20170329 15:27:01.478 26833 ERROR keystone   File “/usr/local/lib/python2.7/dist-packages/keystone/server/wsgi.py”, line 132, in initialize_admin_application
20170329 15:27:01.478 26833 ERROR keystone     config_files=_get_config_files())
20170329 15:27:01.478 26833 ERROR keystone   File “/usr/local/lib/python2.7/dist-packages/keystone/server/wsgi.py”, line 69, in initialize_application
20170329 15:27:01.478 26833 ERROR keystone     startup_application_fn=loadapp)
20170329 15:27:01.478 26833 ERROR keystone   File “/usr/local/lib/python2.7/dist-packages/keystone/server/common.py”, line 50, in setup_backends
20170329 15:27:01.478 26833 ERROR keystone     res = startup_application_fn()
20170329 15:27:01.478 26833 ERROR keystone   File “/usr/local/lib/python2.7/dist-packages/keystone/server/wsgi.py”, line 66, in loadapp
20170329 15:27:01.478 26833 ERROR keystone     ‘config:%s’ % find_paste_config(), name)
20170329 15:27:01.478 26833 ERROR keystone   File “/usr/local/lib/python2.7/dist-packages/keystone/version/service.py”, line 53, in loadapp
20170329 15:27:01.478 26833 ERROR keystone     controllers.latest_app = deploy.loadapp(conf, name=name)
20170329 15:27:01.478 26833 ERROR keystone   File “/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py”, line 247, in loadapp
20170329 15:27:01.478 26833 ERROR keystone     return loadobj(APP, uri, name=name, **kw)
20170329 15:27:01.478 26833 ERROR keystone   File “/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py”, line 272, in loadobj
20170329 15:27:01.478 26833 ERROR keystone     return context.create()
20170329 15:27:01.478 26833 ERROR keystone   File “/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py”, line 710, in create
20170329 15:27:01.478 26833 ERROR keystone     return self.object_type.invoke(self)
20170329 15:27:01.478 26833 ERROR keystone   File “/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py”, line 144, in invoke
20170329 15:27:01.478 26833 ERROR keystone     **context.local_conf)
20170329 15:27:01.478 26833 ERROR keystone   File “/usr/local/lib/python2.7/dist-packages/paste/deploy/util.py”, line 55, in fix_call
20170329 15:27:01.478 26833 ERROR keystone     val = callable(*args, **kw)
20170329 15:27:01.478 26833 ERROR keystone   File “/usr/local/lib/python2.7/dist-packages/paste/urlmap.py”, line 31, in urlmap_factory
20170329 15:27:01.478 26833 ERROR keystone     app = loader.get_app(app_name, global_conf=global_conf)
20170329 15:27:01.478 26833 ERROR keystone   File “/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py”, line 350, in get_app
20170329 15:27:01.478 26833 ERROR keystone     name=name, global_conf=global_conf).create()
20170329 15:27:01.478 26833 ERROR keystone   File “/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py”, line 362, in app_context
20170329 15:27:01.478 26833 ERROR keystone     APP, name=name, global_conf=global_conf)
20170329 15:27:01.478 26833 ERROR keystone   File “/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py”, line 450, in get_context
20170329 15:27:01.478 26833 ERROR keystone     global_additions=global_additions)
20170329 15:27:01.478 26833 ERROR keystone   File “/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py”, line 559, in _pipeline_app_context
20170329 15:27:01.478 26833 ERROR keystone     APP, pipeline[1], global_conf)
20170329 15:27:01.478 26833 ERROR keystone   File “/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py”, line 454, in get_context
20170329 15:27:01.478 26833 ERROR keystone     section)
20170329 15:27:01.478 26833 ERROR keystone   File “/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py”, line 476, in _context_from_use
20170329 15:27:01.478 26833 ERROR keystone     object_type, name=use, global_conf=global_conf)
20170329 15:27:01.478 26833 ERROR keystone   File “/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py”, line 406, in get_context
20170329 15:27:01.478 26833 ERROR keystone     global_conf=global_conf)
20170329 15:27:01.478 26833 ERROR keystone   File “/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py”, line 296, in loadcontext
20170329 15:27:01.478 26833 ERROR keystone     global_conf=global_conf)
20170329 15:27:01.478 26833 ERROR keystone   File “/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py”, line 328, in _loadegg
20170329 15:27:01.478 26833 ERROR keystone     return loader.get_context(object_type, name, global_conf)
20170329 15:27:01.478 26833 ERROR keystone   File “/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py”, line 620, in get_context
20170329 15:27:01.478 26833 ERROR keystone     object_type, name=name)
20170329 15:27:01.478 26833 ERROR keystone   File “/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py”, line 640, in find_egg_entry_point
20170329 15:27:01.478 26833 ERROR keystone     pkg_resources.require(self.spec)
20170329 15:27:01.478 26833 ERROR keystone   File “/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py”, line 943, in require
20170329 15:27:01.478 26833 ERROR keystone     needed = self.resolve(parse_requirements(requirements))
20170329 15:27:01.478 26833 ERROR keystone   File “/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py”, line 834, in resolve
20170329 15:27:01.478 26833 ERROR keystone     raise VersionConflict(dist, req).with_context(dependent_req)
20170329 15:27:01.478 26833 ERROR keystone ContextualVersionConflict: (requests 2.13.0 (/usr/local/lib/python2.7/distpackages), Requirement.parse(‘requests!=2.12.2,!=2.13.0,>=2.10.0’), set([‘oslo.policy’]))

This was caused solely because I forgot to use pip constraints at first when I started installing the controller node (I remembered later). Pip doesn’t have a dependency solver and just naively installs packages in the order its told. This causes all sorts of conflicts if 2 packages have the same requirement with different versions. (even if there is overlap and a correct version can be figured out) Using constraints like I recommended before would have avoided this. But after resolving the conflict keystone worked perfectly and I was ready to move on to the next service.

Installing Glance

The next service to install by following the install guide is Glance. The process for configuring glance was pretty straightforward. Just as with keystone it’s not worth repeating all the steps from the install guide section on Glance.  But, at a high level you just create the database in mysql, configure glance with the details for connecting to MySQL, connecting to Keystone, and how to store images. After that you run the DB schema migrations to set the schema for the MySQL database, and create the endpoint and service users in keystone. After going through all the steps I did encounter one problem in Glance when I first started it up. The glance log had this traceback:

2017-03-29 16:21:52.038 29647 ERROR glance.api.v2.image_data Traceback (most recent call last):
2017-03-29 16:21:52.038 29647 ERROR glance.api.v2.image_data File “/usr/local/lib/python2.7/dist-packages/glance/api/v2/image_data.py”, line 116, in upload
2017-03-29 16:21:52.038 29647 ERROR glance.api.v2.image_data image.set_data(data, size)
2017-03-29 16:21:52.038 29647 ERROR glance.api.v2.image_data File “/usr/local/lib/python2.7/dist-packages/glance/domain/proxy.py”, line 195, in set_data
2017-03-29 16:21:52.038 29647 ERROR glance.api.v2.image_data self.base.set_data(data, size)
2017-03-29 16:21:52.038 29647 ERROR glance.api.v2.image_data File “/usr/local/lib/python2.7/dist-packages/glance/notifier.py”, line 480, in set_data
2017-03-29 16:21:52.038 29647 ERROR glance.api.v2.image_data _send_notification(notify_error, ‘image.upload’, msg)
2017-03-29 16:21:52.038 29647 ERROR glance.api.v2.image_data File “/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py”, line 220, in __exit__
2017-03-29 16:21:52.038 29647 ERROR glance.api.v2.image_data self.force_reraise()
2017-03-29 16:21:52.038 29647 ERROR glance.api.v2.image_data File “/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py”, line 196, in force_reraise
2017-03-29 16:21:52.038 29647 ERROR glance.api.v2.image_data six.reraise(self.type_, self.value, self.tb)
2017-03-29 16:21:52.038 29647 ERROR glance.api.v2.image_data File “/usr/local/lib/python2.7/dist-packages/glance/notifier.py”, line 427, in set_data
2017-03-29 16:21:52.038 29647 ERROR glance.api.v2.image_data self.repo.set_data(data, size)
2017-03-29 16:21:52.038 29647 ERROR glance.api.v2.image_data File “/usr/local/lib/python2.7/dist-packages/glance/api/policy.py”, line 192, in set_data
2017-03-29 16:21:52.038 29647 ERROR glance.api.v2.image_data return self.image.set_data(*args, **kwargs)
2017-03-29 16:21:52.038 29647 ERROR glance.api.v2.image_data File “/usr/local/lib/python2.7/dist-packages/glance/quota/__init__.py”, line 304, in set_data
2017-03-29 16:21:52.038 29647 ERROR glance.api.v2.image_data self.image.set_data(data, size=size)
2017-03-29 16:21:52.038 29647 ERROR glance.api.v2.image_data File “/usr/local/lib/python2.7/dist-packages/glance/location.py”, line 439, in set_data
2017-03-29 16:21:52.038 29647 ERROR glance.api.v2.image_data verifier=verifier)
2017-03-29 16:21:52.038 29647 ERROR glance.api.v2.image_data File “/usr/local/lib/python2.7/dist-packages/glance_store/backend.py”, line 453, in add_to_backend
2017-03-29 16:21:52.038 29647 ERROR glance.api.v2.image_data verifier)
2017-03-29 16:21:52.038 29647 ERROR glance.api.v2.image_data File “/usr/local/lib/python2.7/dist-packages/glance_store/backend.py”, line 426, in store_add_to_backend
2017-03-29 16:21:52.038 29647 ERROR glance.api.v2.image_data verifier=verifier)
2017-03-29 16:21:52.038 29647 ERROR glance.api.v2.image_data File “/usr/local/lib/python2.7/dist-packages/glance_store/capabilities.py”, line 223, in op_checker
2017-03-29 16:21:52.038 29647 ERROR glance.api.v2.image_data raise op_exec_map[op](**kwargs)
2017-03-29 16:21:52.038 29647 ERROR glance.api.v2.image_data StoreAddDisabled: Configuration for store failed. Adding images to this store is disabled.

I forgot to create the /var/lib/glance dir, so there was no directory to store the images in. Again something else which would have been fix if I followed the steps I outlined in the installing from tarballs section. But after creating the directory everything worked.

One thing I do want to note here is that I have small issue with the verification steps for Glance outlined in the install guide. The steps there don’t really go far enough to verify the image uploaded was actually stored properly, just that glance created the image. This was a problem I had later in the installation and I could have caught it earlier if the verification steps instructed you to download the image from glance and compare it to the source image.

Installing Nova

The next service in the install guide is Nova. Nova was a bit more involved compared to Glance or Keystone, but it has more moving parts so that’s understandable. Just as with the other services refer to the install guide section for Nova for all the step by step details. There are more steps for nova in general so it’s not worth even outlining the high level flow here. One thing you’ll need to be aware of is that Nova includes 2 separate API services that you’ll be running, the Nova API and the Placement API. The Placement API is a recent addition since Newton which is used to provide data for scheduling logic and is a completely self contained service. Just like keystone, the placement API only ships as a wsgi script. But unlike keystone there was no documentation (this has changed, or in progress at least) about the install process and no example config files provided. It’s pretty straightforward to adapt what you used to keystone, but this was another thing I had to figure out on my own.

After getting everything configured according to the install guide I hit a few little things that I needed to fix. The first was that I forgot to create a state directory that I specified in the config file:

2017-03-29 17:46:28.176 32263 ERROR nova Traceback (most recent call last):
2017-03-29 17:46:28.176 32263 ERROR nova File “/usr/local/bin/nova-api”, line 10, in <module>
2017-03-29 17:46:28.176 32263 ERROR nova sys.exit(main())
2017-03-29 17:46:28.176 32263 ERROR nova File “/usr/local/lib/python2.7/dist-packages/nova/cmd/api.py”, line 59, in main
2017-03-29 17:46:28.176 32263 ERROR nova server = service.WSGIService(api, use_ssl=should_use_ssl)
2017-03-29 17:46:28.176 32263 ERROR nova File “/usr/local/lib/python2.7/dist-packages/nova/service.py”, line 311, in __init__
2017-03-29 17:46:28.176 32263 ERROR nova self.app = self.loader.load_app(name)
2017-03-29 17:46:28.176 32263 ERROR nova File “/usr/local/lib/python2.7/dist-packages/nova/wsgi.py”, line 497, in load_app
2017-03-29 17:46:28.176 32263 ERROR nova return deploy.loadapp(“config:%s” % self.config_path, name=name)
2017-03-29 17:46:28.176 32263 ERROR nova File “/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py”, line 247, in loadapp
2017-03-29 17:46:28.176 32263 ERROR nova return loadobj(APP, uri, name=name, **kw)
2017-03-29 17:46:28.176 32263 ERROR nova File “/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py”, line 272, in loadobj
2017-03-29 17:46:28.176 32263 ERROR nova return context.create()
2017-03-29 17:46:28.176 32263 ERROR nova File “/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py”, line 710, in create
2017-03-29 17:46:28.176 32263 ERROR nova return self.object_type.invoke(self)
2017-03-29 17:46:28.176 32263 ERROR nova File “/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py”, line 144, in invoke
2017-03-29 17:46:28.176 32263 ERROR nova **context.local_conf)
2017-03-29 17:46:28.176 32263 ERROR nova File “/usr/local/lib/python2.7/dist-packages/paste/deploy/util.py”, line 55, in fix_call
2017-03-29 17:46:28.176 32263 ERROR nova val = callable(*args, **kw)
2017-03-29 17:46:28.176 32263 ERROR nova File “/usr/local/lib/python2.7/dist-packages/nova/api/openstack/urlmap.py”, line 160, in urlmap_factory
2017-03-29 17:46:28.176 32263 ERROR nova app = loader.get_app(app_name, global_conf=global_conf)
2017-03-29 17:46:28.176 32263 ERROR nova File “/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py”, line 350, in get_app
2017-03-29 17:46:28.176 32263 ERROR nova name=name, global_conf=global_conf).create()
2017-03-29 17:46:28.176 32263 ERROR nova File “/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py”, line 710, in create
2017-03-29 17:46:28.176 32263 ERROR nova return self.object_type.invoke(self)
2017-03-29 17:46:28.176 32263 ERROR nova File “/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py”, line 144, in invoke
2017-03-29 17:46:28.176 32263 ERROR nova **context.local_conf)
2017-03-29 17:46:28.176 32263 ERROR nova File “/usr/local/lib/python2.7/dist-packages/paste/deploy/util.py”, line 55, in fix_call
2017-03-29 17:46:28.176 32263 ERROR nova val = callable(*args, **kw)
2017-03-29 17:46:28.176 32263 ERROR nova File “/usr/local/lib/python2.7/dist-packages/nova/api/auth.py”, line 57, in pipeline_factory_v21
2017-03-29 17:46:28.176 32263 ERROR nova return _load_pipeline(loader, local_conf[CONF.api.auth_strategy].split())
2017-03-29 17:46:28.176 32263 ERROR nova File “/usr/local/lib/python2.7/dist-packages/nova/api/auth.py”, line 38, in _load_pipeline
2017-03-29 17:46:28.176 32263 ERROR nova app = loader.get_app(pipeline[-1])
2017-03-29 17:46:28.176 32263 ERROR nova File “/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py”, line 350, in get_app
2017-03-29 17:46:28.176 32263 ERROR nova name=name, global_conf=global_conf).create()
2017-03-29 17:46:28.176 32263 ERROR nova File “/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py”, line 710, in create
2017-03-29 17:46:28.176 32263 ERROR nova return self.object_type.invoke(self)
2017-03-29 17:46:28.176 32263 ERROR nova File “/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py”, line 146, in invoke
2017-03-29 17:46:28.176 32263 ERROR nova return fix_call(context.object, context.global_conf, **context.local_conf)
2017-03-29 17:46:28.176 32263 ERROR nova File “/usr/local/lib/python2.7/dist-packages/paste/deploy/util.py”, line 55, in fix_call
2017-03-29 17:46:28.176 32263 ERROR nova val = callable(*args, **kw)
2017-03-29 17:46:28.176 32263 ERROR nova File “/usr/local/lib/python2.7/dist-packages/nova/api/openstack/__init__.py”, line 218, in factory
2017-03-29 17:46:28.176 32263 ERROR nova return cls()
2017-03-29 17:46:28.176 32263 ERROR nova File “/usr/local/lib/python2.7/dist-packages/nova/api/openstack/compute/__init__.py”, line 31, in __init__
2017-03-29 17:46:28.176 32263 ERROR nova super(APIRouterV21, self).__init__()
2017-03-29 17:46:28.176 32263 ERROR nova File “/usr/local/lib/python2.7/dist-packages/nova/api/openstack/__init__.py”, line 243, in __init__
2017-03-29 17:46:28.176 32263 ERROR nova self._register_resources_check_inherits(mapper)
2017-03-29 17:46:28.176 32263 ERROR nova File “/usr/local/lib/python2.7/dist-packages/nova/api/openstack/__init__.py”, line 259, in _register_resources_check_inherits
2017-03-29 17:46:28.176 32263 ERROR nova for resource in ext.obj.get_resources():
2017-03-29 17:46:28.176 32263 ERROR nova File “/usr/local/lib/python2.7/dist-packages/nova/api/openstack/compute/cloudpipe.py”, line 187, in get_resources
2017-03-29 17:46:28.176 32263 ERROR nova CloudpipeController())]
2017-03-29 17:46:28.176 32263 ERROR nova File “/usr/local/lib/python2.7/dist-packages/nova/api/openstack/compute/cloudpipe.py”, line 48, in __init__
2017-03-29 17:46:28.176 32263 ERROR nova self.setup()
2017-03-29 17:46:28.176 32263 ERROR nova File “/usr/local/lib/python2.7/dist-packages/nova/api/openstack/compute/cloudpipe.py”, line 55, in setup
2017-03-29 17:46:28.176 32263 ERROR nova fileutils.ensure_tree(CONF.crypto.keys_path)
2017-03-29 17:46:28.176 32263 ERROR nova File “/usr/local/lib/python2.7/dist-packages/oslo_utils/fileutils.py”, line 40, in ensure_tree
2017-03-29 17:46:28.176 32263 ERROR nova os.makedirs(path, mode)
2017-03-29 17:46:28.176 32263 ERROR nova File “/usr/lib/python2.7/os.py”, line 157, in makedirs
2017-03-29 17:46:28.176 32263 ERROR nova mkdir(name, mode)
2017-03-29 17:46:28.176 32263 ERROR nova OSError: [Errno 13] Permission denied: ‘/usr/local/lib/python2.7/dist-packages/keys’

This was simple to fix and all I had to do was create the directory and set the owner to the service user. The second issue was my old friend the requirements mismatch:

2017-03-29 18:33:11.433 1155 ERROR nova Traceback (most recent call last):
2017-03-29 18:33:11.433 1155 ERROR nova File “/usr/local/bin/nova-api”, line 10, in <module>
2017-03-29 18:33:11.433 1155 ERROR nova sys.exit(main())
2017-03-29 18:33:11.433 1155 ERROR nova File “/usr/local/lib/python2.7/dist-packages/nova/cmd/api.py”, line 59, in main
2017-03-29 18:33:11.433 1155 ERROR nova server = service.WSGIService(api, use_ssl=should_use_ssl)
2017-03-29 18:33:11.433 1155 ERROR nova File “/usr/local/lib/python2.7/dist-packages/nova/service.py”, line 311, in __init__
2017-03-29 18:33:11.433 1155 ERROR nova self.app = self.loader.load_app(name)
2017-03-29 18:33:11.433 1155 ERROR nova File “/usr/local/lib/python2.7/dist-packages/nova/wsgi.py”, line 497, in load_app
2017-03-29 18:33:11.433 1155 ERROR nova return deploy.loadapp(“config:%s” % self.config_path, name=name)
2017-03-29 18:33:11.433 1155 ERROR nova File “/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py”, line 247, in loadapp
2017-03-29 18:33:11.433 1155 ERROR nova return loadobj(APP, uri, name=name, **kw)
2017-03-29 18:33:11.433 1155 ERROR nova File “/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py”, line 272, in loadobj
2017-03-29 18:33:11.433 1155 ERROR nova return context.create()
2017-03-29 18:33:11.433 1155 ERROR nova File “/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py”, line 710, in create
2017-03-29 18:33:11.433 1155 ERROR nova return self.object_type.invoke(self)
2017-03-29 18:33:11.433 1155 ERROR nova File “/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py”, line 144, in invoke
2017-03-29 18:33:11.433 1155 ERROR nova **context.local_conf)
2017-03-29 18:33:11.433 1155 ERROR nova File “/usr/local/lib/python2.7/dist-packages/paste/deploy/util.py”, line 55, in fix_call
2017-03-29 18:33:11.433 1155 ERROR nova val = callable(*args, **kw)
2017-03-29 18:33:11.433 1155 ERROR nova File “/usr/local/lib/python2.7/dist-packages/paste/urlmap.py”, line 31, in urlmap_factory
2017-03-29 18:33:11.433 1155 ERROR nova app = loader.get_app(app_name, global_conf=global_conf)
2017-03-29 18:33:11.433 1155 ERROR nova File “/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py”, line 350, in get_app
2017-03-29 18:33:11.433 1155 ERROR nova name=name, global_conf=global_conf).create()
2017-03-29 18:33:11.433 1155 ERROR nova File “/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py”, line 710, in create
2017-03-29 18:33:11.433 1155 ERROR nova return self.object_type.invoke(self)
2017-03-29 18:33:11.433 1155 ERROR nova File “/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py”, line 203, in invoke
2017-03-29 18:33:11.433 1155 ERROR nova app = context.app_context.create()
2017-03-29 18:33:11.433 1155 ERROR nova File “/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py”, line 710, in create
2017-03-29 18:33:11.433 1155 ERROR nova return self.object_type.invoke(self)
2017-03-29 18:33:11.433 1155 ERROR nova File “/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py”, line 146, in invoke
2017-03-29 18:33:11.433 1155 ERROR nova return fix_call(context.object, context.global_conf, **context.local_conf)
2017-03-29 18:33:11.433 1155 ERROR nova File “/usr/local/lib/python2.7/dist-packages/paste/deploy/util.py”, line 55, in fix_call
2017-03-29 18:33:11.433 1155 ERROR nova val = callable(*args, **kw)
2017-03-29 18:33:11.433 1155 ERROR nova File “/usr/local/lib/python2.7/dist-packages/nova/wsgi.py”, line 270, in factory
2017-03-29 18:33:11.433 1155 ERROR nova return cls(**local_config)
2017-03-29 18:33:11.433 1155 ERROR nova File “/usr/local/lib/python2.7/dist-packages/nova/api/metadata/handler.py”, line 49, in __init__
2017-03-29 18:33:11.433 1155 ERROR nova expiration_time=CONF.api.metadata_cache_expiration)
2017-03-29 18:33:11.433 1155 ERROR nova File “/usr/local/lib/python2.7/dist-packages/nova/cache_utils.py”, line 58, in get_client
2017-03-29 18:33:11.433 1155 ERROR nova backend=’oslo_cache.dict’))
2017-03-29 18:33:11.433 1155 ERROR nova File “/usr/local/lib/python2.7/dist-packages/nova/cache_utils.py”, line 96, in _get_custom_cache_region
2017-03-29 18:33:11.433 1155 ERROR nova region.configure(backend, **region_params)
2017-03-29 18:33:11.433 1155 ERROR nova File “/usr/local/lib/python2.7/dist-packages/dogpile/cache/region.py”, line 413, in configure
2017-03-29 18:33:11.433 1155 ERROR nova backend_cls = _backend_loader.load(backend)
2017-03-29 18:33:11.433 1155 ERROR nova File “/usr/local/lib/python2.7/dist-packages/dogpile/util/langhelpers.py”, line 40, in load
2017-03-29 18:33:11.433 1155 ERROR nova return impl.load()
2017-03-29 18:33:11.433 1155 ERROR nova File “/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py”, line 2301, in load
2017-03-29 18:33:11.433 1155 ERROR nova self.require(*args, **kwargs)
2017-03-29 18:33:11.433 1155 ERROR nova File “/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py”, line 2324, in require
2017-03-29 18:33:11.433 1155 ERROR nova items = working_set.resolve(reqs, env, installer, extras=self.extras)
2017-03-29 18:33:11.433 1155 ERROR nova File “/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py”, line 859, in resolve
2017-03-29 18:33:11.433 1155 ERROR nova raise VersionConflict(dist, req).with_context(dependent_req)
2017-03-29 18:33:11.433 1155 ERROR nova ContextualVersionConflict: (pbr 1.10.0 (/usr/local/lib/python2.7/dist-packages), Requirement.parse(‘pbr>=2.0.0’), set([‘oslo.i18n’, ‘oslo.log’, ‘oslo.context’, ‘oslo.utils’]))

In this instance it was a pretty base requirement, pbr,  that was at the wrong version. When I saw this I realized that I forgot to use constraints (because pbr is used by everything in OpenStack) and I quickly reran pip install for nova with the constraints argument to correct this issue.

The final thing I hit was a missing sudoers file:

2017-03-29 18:29:47.844 905 ERROR nova Traceback (most recent call last):
2017-03-29 18:29:47.844 905 ERROR nova File “/usr/local/bin/nova-api”, line 10, in <module>
2017-03-29 18:29:47.844 905 ERROR nova sys.exit(main())
2017-03-29 18:29:47.844 905 ERROR nova File “/usr/local/lib/python2.7/dist-packages/nova/cmd/api.py”, line 59, in main
2017-03-29 18:29:47.844 905 ERROR nova server = service.WSGIService(api, use_ssl=should_use_ssl)
2017-03-29 18:29:47.844 905 ERROR nova File “/usr/local/lib/python2.7/dist-packages/nova/service.py”, line 309, in __init__
2017-03-29 18:29:47.844 905 ERROR nova self.manager = self._get_manager()
2017-03-29 18:29:47.844 905 ERROR nova File “/usr/local/lib/python2.7/dist-packages/nova/service.py”, line 364, in _get_manager
2017-03-29 18:29:47.844 905 ERROR nova return manager_class()
2017-03-29 18:29:47.844 905 ERROR nova File “/usr/local/lib/python2.7/dist-packages/nova/api/manager.py”, line 30, in __init__
2017-03-29 18:29:47.844 905 ERROR nova self.network_driver.metadata_accept()
2017-03-29 18:29:47.844 905 ERROR nova File “/usr/local/lib/python2.7/dist-packages/nova/network/linux_net.py”, line 606, in metadata_accept
2017-03-29 18:29:47.844 905 ERROR nova iptables_manager.apply()
2017-03-29 18:29:47.844 905 ERROR nova File “/usr/local/lib/python2.7/dist-packages/nova/network/linux_net.py”, line 346, in apply
2017-03-29 18:29:47.844 905 ERROR nova self._apply()
2017-03-29 18:29:47.844 905 ERROR nova File “/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py”, line 271, in inner
2017-03-29 18:29:47.844 905 ERROR nova return f(*args, **kwargs)
2017-03-29 18:29:47.844 905 ERROR nova File “/usr/local/lib/python2.7/dist-packages/nova/network/linux_net.py”, line 366, in _apply
2017-03-29 18:29:47.844 905 ERROR nova attempts=5)
2017-03-29 18:29:47.844 905 ERROR nova File “/usr/local/lib/python2.7/dist-packages/nova/network/linux_net.py”, line 1167, in _execute
2017-03-29 18:29:47.844 905 ERROR nova return utils.execute(*cmd, **kwargs)
2017-03-29 18:29:47.844 905 ERROR nova File “/usr/local/lib/python2.7/dist-packages/nova/utils.py”, line 297, in execute
2017-03-29 18:29:47.844 905 ERROR nova return RootwrapProcessHelper().execute(*cmd, **kwargs)
2017-03-29 18:29:47.844 905 ERROR nova File “/usr/local/lib/python2.7/dist-packages/nova/utils.py”, line 180, in execute
2017-03-29 18:29:47.844 905 ERROR nova return processutils.execute(*cmd, **kwargs)
2017-03-29 18:29:47.844 905 ERROR nova File “/usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py”, line 400, in execute
2017-03-29 18:29:47.844 905 ERROR nova cmd=sanitized_cmd)
2017-03-29 18:29:47.844 905 ERROR nova ProcessExecutionError: Unexpected error while running command.
2017-03-29 18:29:47.844 905 ERROR nova Command: sudo nova-rootwrap /etc/nova/rootwrap.conf iptables-save -c
2017-03-29 18:29:47.844 905 ERROR nova Exit code: 1
2017-03-29 18:29:47.844 905 ERROR nova Stdout: u”
2017-03-29 18:29:47.844 905 ERROR nova Stderr: u’sudo: no tty present and no askpass program specified\n

Nova needs root priveleges to perform some operations. To do this it leverages a program called rootwrap to do the privelege escalation. But it needs sudo to be able to leverage rootwrap. I was able to to fix this by creating a sudoers file for nova like:

nova ALL=(root) NOPASSWD: /usr/local/bin/novarootwrap /etc/nova/rootwrap.conf

After correcting those 3 issues I got Nova running without any errors (at least with the verification steps outlined in the install guide)

Installing Neutron

The last service I’m installing from the install guide (I skipped cinder because I’m not using block storage) is Neutron. By far this was the most complicated and most difficult service to install and configure. I had the most problems with neutron and networking in general both during the install phase and also later when I was debugging the operation of the cloud.  In the case of Neutron I started by reading the install guide section for neutron like the other services, but I also often needed to read the OpenStack Networking Guide to get a better grasp on the underlying concepts the install guide was trying to explain. Especially after getting to the section in the install guide where it asks you to pick between “Provider Networks” or “Self Service Networking”.

After reading all the documentation I decided that I wanted use provider networks because all I wanted was all my guests on a flat Layer 2 and for the guests  to come on my home network with an IP address I could reach from any of my other computer I have at home.  When I saw this diagram in the Networking Guide:

it made my decision simple. This networking topology was exactly what I wanted. I didn’t want to have to deal with creating a network, subnet, and router in neutron for each tenant to be able to access my guests. With this decision made I went about following the configuration guide lke for the previous services.

Unfortunately I hit an issue pretty early on. These were related to Neutron’s default configuration being spread across multiple files. It makes it very confusing to follow the install guide. For example, it says you want to write one set of config options into /etc/neutron/neutron.confthen a second set of config options into /etc/neutron/plugins/ml2/ml2_conf.ini and a third set of config options into /etc/neutron/plugins/ml2/linuxbridge_agent.ini, etc. This process continues for another 2 or 3 config files without any context on how these separate files are used. Then what makes it worse is when you actually go to launch the neutron daemons . Neutron itself consists of 4-5 different daemons running on the controller and compute nodes. But, there is no documentation anywhere on how all of these different config files are leveraged by the different daemons. For example, when launching linuxbridge-agent daemon which config files are you supposed to pass in? I ended up having to cheat  for this and look at the devstack soure code to see how it launched neutron there. After that I realized neutron is just leveraging oslo.config‘s ability to specify multiple config files and have them be concatenated together at runtime. This means that because there are no overlapping options that none of this complexity is required and a single neutron.conf could be used for everything. This is something I think we must change in Neutron, because as things are now are just too confusing.

After finally getting everything configured I encountered a number of other issues. The first was around rootwrap, just like nova, neutron need root privileges to perform some operations, and it leverages rootwrap to perform the privilege escalation.  However, neutron uses rootwrap as a separate daemon, and calls it over a socket interface. (this is done to reduce the overhead for creating a separate python process on each external call, which can slow things down significantly) When I first started neutron I hit a similar error to nova about sudo permissions. So I needed to create a sudoers file for neutron, in my case it looked like this:

neutron ALL=(root) NOPASSWD: /usr/local/bin/neutron-rootwrap /etc/neutron/rootwrap.conf *
neutron ALL=(root) NOPASSWD: /usr/local/bin/neutron-rootwrap-daemon /etc/neutron/rootwrap.conf

But it also turns out I needed to tell neutron how to call rootwrap. I found this bug on launchpad when I did a google search on my error and it told me about the config options I needed to set in addition to creating the sudoers file.  These weren’t in the install documentation as I expect by default the neutron distro packages set these config options.  After creating the sudoers file and setting the config flags I was able to get past this issue.

The next problem was also fairly cryptic. When I first started neutron after fixing the rootwrap issue I was greeted by this error in the logs:

2017-03-30 11:57:05.182 4158 ERROR neutron.plugins.ml2.drivers.agent._common_agent Traceback (most recent call last):
2017-03-30 11:57:05.182 4158 ERROR neutron.plugins.ml2.drivers.agent._common_agent File “/usr/local/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/agent/_common_agent.py”, line 453, in daemon_loop
2017-03-30 11:57:05.182 4158 ERROR neutron.plugins.ml2.drivers.agent._common_agent sync = self.process_network_devices(device_info)
2017-03-30 11:57:05.182 4158 ERROR neutron.plugins.ml2.drivers.agent._common_agent File “/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py”, line 153, in wrapper
2017-03-30 11:57:05.182 4158 ERROR neutron.plugins.ml2.drivers.agent._common_agent return f(*args, **kwargs)
2017-03-30 11:57:05.182 4158 ERROR neutron.plugins.ml2.drivers.agent._common_agent File “/usr/local/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/agent/_common_agent.py”, line 203, in process_network_devices
2017-03-30 11:57:05.182 4158 ERROR neutron.plugins.ml2.drivers.agent._common_agent device_info.get(‘updated’))
2017-03-30 11:57:05.182 4158 ERROR neutron.plugins.ml2.drivers.agent._common_agent File “/usr/local/lib/python2.7/dist-packages/neutron/agent/securitygroups_rpc.py”, line 277, in setup_port_filters
2017-03-30 11:57:05.182 4158 ERROR neutron.plugins.ml2.drivers.agent._common_agent self.prepare_devices_filter(new_devices)
2017-03-30 11:57:05.182 4158 ERROR neutron.plugins.ml2.drivers.agent._common_agent File “/usr/local/lib/python2.7/dist-packages/neutron/agent/securitygroups_rpc.py”, line 131, in decorated_function
2017-03-30 11:57:05.182 4158 ERROR neutron.plugins.ml2.drivers.agent._common_agent *args, **kwargs)
2017-03-30 11:57:05.182 4158 ERROR neutron.plugins.ml2.drivers.agent._common_agent File “/usr/local/lib/python2.7/dist-packages/neutron/agent/securitygroups_rpc.py”, line 139, in prepare_devices_filter
2017-03-30 11:57:05.182 4158 ERROR neutron.plugins.ml2.drivers.agent._common_agent self._apply_port_filter(device_ids)
2017-03-30 11:57:05.182 4158 ERROR neutron.plugins.ml2.drivers.agent._common_agent File “/usr/local/lib/python2.7/dist-packages/neutron/agent/securitygroups_rpc.py”, line 157, in _apply_port_filter
2017-03-30 11:57:05.182 4158 ERROR neutron.plugins.ml2.drivers.agent._common_agent security_groups, security_group_member_ips)
2017-03-30 11:57:05.182 4158 ERROR neutron.plugins.ml2.drivers.agent._common_agent File “/usr/local/lib/python2.7/dist-packages/neutron/agent/securitygroups_rpc.py”, line 173, in _update_security_group_info
2017-03-30 11:57:05.182 4158 ERROR neutron.plugins.ml2.drivers.agent._common_agent remote_sg_id, member_ips)
2017-03-30 11:57:05.182 4158 ERROR neutron.plugins.ml2.drivers.agent._common_agent File “/usr/local/lib/python2.7/dist-packages/neutron/agent/linux/iptables_firewall.py”, line 163, in update_security_group_members
2017-03-30 11:57:05.182 4158 ERROR neutron.plugins.ml2.drivers.agent._common_agent self._update_ipset_members(sg_id, sg_members)
2017-03-30 11:57:05.182 4158 ERROR neutron.plugins.ml2.drivers.agent._common_agent File “/usr/local/lib/python2.7/dist-packages/neutron/agent/linux/iptables_firewall.py”, line 169, in _update_ipset_members
2017-03-30 11:57:05.182 4158 ERROR neutron.plugins.ml2.drivers.agent._common_agent sg_id, ip_version, current_ips)
2017-03-30 11:57:05.182 4158 ERROR neutron.plugins.ml2.drivers.agent._common_agent File “/usr/local/lib/python2.7/dist-packages/neutron/agent/linux/ipset_manager.py”, line 83, in set_members
2017-03-30 11:57:05.182 4158 ERROR neutron.plugins.ml2.drivers.agent._common_agent self.set_members_mutate(set_name, ethertype, member_ips)
2017-03-30 11:57:05.182 4158 ERROR neutron.plugins.ml2.drivers.agent._common_agent File “/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py”, line 271, in inner
2017-03-30 11:57:05.182 4158 ERROR neutron.plugins.ml2.drivers.agent._common_agent return f(*args, **kwargs)
2017-03-30 11:57:05.182 4158 ERROR neutron.plugins.ml2.drivers.agent._common_agent File “/usr/local/lib/python2.7/dist-packages/neutron/agent/linux/ipset_manager.py”, line 93, in set_members_mutate
2017-03-30 11:57:05.182 4158 ERROR neutron.plugins.ml2.drivers.agent._common_agent self._create_set(set_name, ethertype)
2017-03-30 11:57:05.182 4158 ERROR neutron.plugins.ml2.drivers.agent._common_agent File “/usr/local/lib/python2.7/dist-packages/neutron/agent/linux/ipset_manager.py”, line 139, in _create_set
2017-03-30 11:57:05.182 4158 ERROR neutron.plugins.ml2.drivers.agent._common_agent self._apply(cmd)
2017-03-30 11:57:05.182 4158 ERROR neutron.plugins.ml2.drivers.agent._common_agent File “/usr/local/lib/python2.7/dist-packages/neutron/agent/linux/ipset_manager.py”, line 149, in _apply
2017-03-30 11:57:05.182 4158 ERROR neutron.plugins.ml2.drivers.agent._common_agent check_exit_code=fail_on_errors)
2017-03-30 11:57:05.182 4158 ERROR neutron.plugins.ml2.drivers.agent._common_agent File “/usr/local/lib/python2.7/dist-packages/neutron/agent/linux/utils.py”, line 128, in execute
2017-03-30 11:57:05.182 4158 ERROR neutron.plugins.ml2.drivers.agent._common_agent execute_rootwrap_daemon(cmd, process_input, addl_env))
2017-03-30 11:57:05.182 4158 ERROR neutron.plugins.ml2.drivers.agent._common_agent File “/usr/local/lib/python2.7/dist-packages/neutron/agent/linux/utils.py”, line 115, in execute_rootwrap_daemon
2017-03-30 11:57:05.182 4158 ERROR neutron.plugins.ml2.drivers.agent._common_agent return client.execute(cmd, process_input)
2017-03-30 11:57:05.182 4158 ERROR neutron.plugins.ml2.drivers.agent._common_agent File “/usr/local/lib/python2.7/dist-packages/oslo_rootwrap/client.py”, line 129, in execute
2017-03-30 11:57:05.182 4158 ERROR neutron.plugins.ml2.drivers.agent._common_agent res = proxy.run_one_command(cmd, stdin)
2017-03-30 11:57:05.182 4158 ERROR neutron.plugins.ml2.drivers.agent._common_agent File “<string>”, line 2, in run_one_command
2017-03-30 11:57:05.182 4158 ERROR neutron.plugins.ml2.drivers.agent._common_agent File “/usr/lib/python2.7/multiprocessing/managers.py”, line 774, in _callmethod
2017-03-30 11:57:05.182 4158 ERROR neutron.plugins.ml2.drivers.agent._common_agent raise convert_to_error(kind, result)
2017-03-30 11:57:05.182 4158 ERROR neutron.plugins.ml2.drivers.agent._common_agent RemoteError:
2017-03-30 11:57:05.182 4158 ERROR neutron.plugins.ml2.drivers.agent._common_agent —————————————————————————
2017-03-30 11:57:05.182 4158 ERROR neutron.plugins.ml2.drivers.agent._common_agent Unserializable message: (‘#ERROR’, ValueError(‘I/O operation on closed file’,))
2017-03-30 11:57:05.182 4158 ERROR neutron.plugins.ml2.drivers.agent._common_agent —————————————————————————

Which isn’t helpful at all. It turns out that this error means that neutron can’t find the ipset command, but it’s not at all clear from the traceback. I was only able to figure this out after tracing through the neutron source code (by following the calls in the traceback) I realized that this error is being emitted after neutron calls the rootwrap daemon. I had to turn debug log level on in the separate rootwrap.conf (which is something packaged in the tarball) to get the rootwrap daemon to log the error message it’s encountering, which in this case was that ipset could not be found. After installing ipset this was corrected.

After all of these headaches I finally got neutron running. But, I quickly found that my choice for provider networks was causing issues with DHCP on my home network. I only have a single 24 port unmanaged switch at home and the bridge interfaces for the guests were on the same Layer 2 network as the rest of my home infrastructure, including my DHCP server. This meant that when I created a server in the cloud the DHCP request from the guest would go out and be recieved by both the neutron DHCP agent as well as my home DHCP server because being on the same Layer 2 meant they shared a broadcast domain. Luckily neutron’s default security group rules blocked the DHCP response from my home server, but there was still a lease record being created on my home server. Also if I ever loosened the security group rules and DHCP traffic was allowed then there would be a race condition between my server and the neutron agent. It turns out there was a small note (see step 3) on this potential problem in the networking guide. So my solution for this was to disable DHCP in neutron and also stop running the DHCP agent on my cloud. This had a ripple effect in that I couldn’t use the metadata service either because it depends on DHCP to set the route for the hardcoded ip address for the metadata server. (this will come up later) Luckily I was able to leverage the force_config_drive option in Nova to make sure the metadata service wasn’t necessary.

I modified the network diagram above for what I ended up with in my cloud:

(note I’m terrible at art, so you can clearly tell where I made changes)

If all of the above didn’t make it clear I still find Neutron the roughest part of the user experience for OpenStack. Besides complexity in configuration it also has a presumption of a decent understanding of networking concepts. I fully admit networking is hard, especially for clouds because you’re dealing with a lot of different pieces, but this is somwhere I feel we need to make improvements. Especially in my use case where my requirements were pretty straightforward. I just wanted to have a server come up on my home network when it was booted so I could log into it right after it booted.  In my opinion this is what the majority of cloud consumers (people using the API) care about. Just getting an IP address (v4 or v6 it doesn’t really matter) and being able to connect to that from their personal machines. After going through this process I’m pretty sure that my college student self who had a much more limited understanding of networking than I do now would have had a very difficult time figuring this out.

Booting the first server

After getting everything running on a single node it was time to boot my first server. I eagerly typed in the

openstack server create

command with all the parameters for my credentials the flavor and the image I had uploaded and waited for the server to go ACTIVE state by running:

openstack server list

a few times. Once the server went into the ACTIVE state I tried to login into the guest with ssh, and got nothing. The ssh connection just timed out and there wasn’t any indication why. Having debugged a ton of issues like this over the years my first guess was ok I screwed up the networking, let me look at the console log by running:

openstack console log show testserver

and it returned nothing.  I was a bit lost as to why, the console log should show the process of booting the operating system. I figured that I made a configuration mistake in the nova, so to double check I logged into the compute node and checked the libvirt state directory and confirmed that the console log file was empty. But this left me at an impasse, why would the guest not be logging anything to the console on boot? So I just started sanity checking everything I could find. When I looked at Nova’s local image cache and saw the cirros image was 0 bytes in size. A cirros image should be about 13MB in size, so 0 bytes was clearly wrong. From there I started tracing through the glance logs to figure out where the data was getting lost (was it nova downloading the image from glance, or did glance have an empty image) when I found:

DEBUG glance_store._drivers.filesystem [req3163a1a74ca947e89444cd8b865055fb 20f283024ffd4bf4841a8d33bdb4f385 6c3fc6392e0c487e85d57afe5a5ab2b7 default default] Wrote 0 bytes to /var/lib/glance/images/e673563643d94fb0a302f3710386b689 with checksum d41d8cd98f00b204e9800998ecf8427e add /usr/local/lib/python2.7/distpackages/glance_store/_drivers/filesystem.py:706

Which was the only hint I could find in the glance logs. It wasn’t even that useful all it said was that glance wrote 0 bytes to disk for the uploaded image. Which at least confirmed that glance wasn’t storing any data from the image upload. But, I couldn’t find any other information about this. So I decided to re-upload the image to glance and use tcpdump on both my desktop and the server to make sure the data was getting sent over the wire to glance. The output of the tcpdump showed all the data being sent and received. This at least meant that the data is getting to the glance api server, but it didn’t really help me figure out where the data was going.

With no other ideas I decided to “instrument” the glance code by manually adding a bunch of log statements to the installed python code in

/usr/local/lib/python2.7/sitepackages/glance

by hand to the trace the data flow through the glance code to find where the image goes from 13MB to 0 bytes. When I did this I was able to figure out that the image data was being lost outside of the glance code in one of it’s requirement libraries either webob, paste, or something like that. When I saw that I realized that I forgot to use constraints when installing glance. I quickly rushed to reinstall glance from the tarball using the constraints parameter and restarted the service. After doing this and re-uploading the image everything worked!

My only mistake in that process was in my over-eagerness to fix the problem I forgot to take notes of exactly what I reinstalled to see where the actual problem was. So all I can say for sure is that make sure you use constraints whenever you install from source, because clearly there was an issue with just using pip install by itself.

After getting glance working I was able to re-run the openstack command to create a server and this time I was able to get a console log, but ssh still didn’t work.

Networking woes

At this point I had the servers booting, but I wasn’t able to login to them. I’ve personally had to debug this kind of issues many times, so when I saw this my first step was to ping the IP address for the guest, just to rule out that it was an issue with the ssh daemon on the server. Since the ping didn’t work I wanted to see if there were was an entry in my arp table for the ip address. Again, there was nothing on that IP after running the arp command. So this either meant there was an issue with Layer 2 connectivity to the guest from my desktop, or the guest didn’t know it’s IP address. (I’ve personally seen both failure conditions) My next step was to check the console log to see if it was setting an IP address correctly. When I got to the cloud-init section of the console log it showed that the IP address was never getting assigned. Instead the server was timing out waiting for a DHCP lease. If you remember the neutron section above I had to disable DHCP on the guests because it was conflicting with my home’s DHCP server so this clearly wasn’t right.

It turns out that cloud-init doesn’t know how to deal with static networking configuration from a config drive. (it might work with a metadata server, but I was not able to check this) So when the guest boots it just ignores the static networking information in the config drive and then tries to get a DHCP lease. This meant that cirros, the recommended image for testing and what the install guide tells you to use, wasn’t going to work. Also the majority of cloud images you can download weren’t going to work either. The only cloud image I was able to get working was the official ubuntu cloud image. This was because Nova was doing file injection to write a the networking information directly into the guest file system. I found a useful blog post on this in my searching: http://blog.oddbit.com/2015/06/26/openstack-networking-without-dhcp/ (although the translation didn’t work on RHEL like that post indicates) But, even if I got ubuntu to work, having a cloud that was only able to boot a single type of image isn’t really that useful.

Luckily the OpenStack Infrastructure team has a similar problem on some public OpenStack clouds they run things on, and they created the Glean project to be an alternative for cloud-init that can properly use the static networking information from a config drive. All I had to do was leverage the Disk Image Builder project to create the images I uploaded into my cloud with glean instead of cloud-init. While not ideal solution, because you can’t take anyone’s random pre-existing cloud image, this worked well enough for me because I can remember to do this as the primary user of my cloud.

It’s also  worth pointing out that all of these networking issues would have been completely avoided if I chose self service networking back in the setting up neutron section. (because it creates a separate Layer 2 network for each tenant) But, given my goals with the cloud and the way the documentation lays out the options I had no way to know this. This connects back to my earlier complaints with neutron being too complex and presuming too much prior knowledge.

But, at this point I had a working single node cloud and could successfully boot guests. All that was left before I finished the cloud deployment was to replicate the installation on the remaining 4 servers.

Setting up the compute nodes

Once I confirmed to have a working configuration and got all the services figured out on the controller node (which included nova-compute and the necessary neutron services for a compute node because it was an all in one) and got everything running there, it was time to setup the compute nodes. This was pretty straightforward and just involved configuring nova-compute and neutron services. It was pretty formulaic and basically just copy and paste. The exact procedure that I wrote down in my notes for this process was:

  1. add provider network interface config
  2. disable apparmor
  3. reboot
  4. download tarballs
  5. create system users
  6. add nova user to libvirt group
  7. install all binaries (libvirt, qemu, ipset, mkisofs, libssl-dev, pip)
  8. make service dirs /etc/ /var/lib for both neutron and nova
  9. copy etc dirs from tarballs to /etc
  10. pip install code with upper-constraints
  11. write config files (basically just copy from another compute node)
  12. set permissions on /etc and /var/lib
  13. create sudoers files for nova and neutron
  14. create systemd unit files
  15. start services
  16. run nova discover_hosts

This is basically just copying and pasting things across the remaining 4 servers. But there were a couple of lessons I learned from the initial install were reflected in these. The only one I haven’t talked about before was disabling apparmor. (or SELinux on other linux distros) I learned the hard way that the default apparmor rules on Ubuntu prevent nova and libvirt from doing the necessary operations to boot a guest. The proper way to fix this issue (especially for better security) would be to create your own apparmor rules to allow the operations being blocked. But, I have always been confused by this, especially on SELinux and didn’t even bother trying. I just disabled apparmor and moved on.

After repeating these steps across the 4 compute nodes I had a fully operational cloud. Nova was showing me the full capacity of 80 vCPUs and I could interact with the cloud and launch guests across all of them. My project was complete! (at least for the first phase)

Conclusion

So after writing all of this down, I came to the realization that I likely give the impression that installing OpenStack by hand is an impossibly complex task. But, honestly it wasn’t that bad of an experience. Sure, OpenStack is complex software with a lot of moving pieces, but in total I got everything working in two-three days.  (And I wasn’t dedicating all my time during those days either). The majority of the issues were caused solely by my insistence on installing everything from tarballs. If I actually followed my original thought experiment and just followed the install guide, the only issue I probably would have hit was with networking. Once you understand what OpenStack is doing under the covers, the install is pretty straightforward. After doing my first OpenStack install a few years ago I found I had a better understanding of how OpenStack works which really helped me in my work on the project. It’s something I recommend that everyone does at least once if they’re planning on working on OpenStack in any capacity. Even just in a VM for playing around. (devstack doesn’t count)

For comparison, that rack of similar Dell servers I deployed back in college took me so much longer to get running. In that case I used xCAT for deployment automation. But, it still took me over a month to get the inifiband cards working with RDMA using OFED, setting up SLURM for MPI job scheduling, connecting everything to our central LDAP server, and having users able to launch jobs across all the nodes. While, it’s not entirely a fair comparison since I have almost a decade more of experience now, but I think it helps put into perspective that this is far from the most grueling experience I’ve had installing software.

After going through the whole exercise I don’t actually run this cloud 24/7, mostly because it heats up my apartment too much and I can’t sleep at night when it’s running.  The power consumption for the servers is also pretty high and I don’t really want to pay the power bill. This basically means I failed the second half of the experiment, to virtualize my home infrastructure. Since I can’t rely on the cloud for critical infrastructure if it’s not always running. But I have found some uses for the cloud both for development tasks as well as running some highly parallel CPU tasks across the entire cloud at once.

Moving forward I intend to continue working on the cloud and upgrading it to future releases as they occur. Also, one of my goals for this entire exercise was going back to the OpenStack community with feedback on how to improve things and submitting patches and/or bugs on fixing some of the issues. This will be an ongoing process as I find time to work on them and also encounter more issues.

Cover Photo // CC BY NC

The post How to rock dirty clouds done cheap appeared first on OpenStack Superuser.

It’s that time again! Cast your vote for the Sydney Superuser Awards

Par Ashlee Ferguson

Voting is now closed. Stay tuned to find out who wins!

The OpenStack Summit kicks off in less than eight weeks and seven deserving organizations have been nominated to be recognized during the opening keynotes. These organizations are competing to win the Superuser Award that will be presented by the most recent winner from the OpenStack Summit Boston.

For this cycle, the community (that means you!) will review the candidates before the Superuser editorial advisors select the finalists and ultimate winner. Finalists will be recognized onstage during the OpenStack Summit Sydney keynotes.

Check out the nominations in alphabetical order below and click through to see each organization’s full application. Then, rate the nominees to select who you think should be recognized at the OpenStack Summit Sydney.

You have until Wednesday, September 20 at 11:59 p.m. PT to rate them. Cast your ratings here.

China Railway Corporation

China Railway’s deployment of OpenStack cloud has helped them save millions of dollars and their application launch cycle has been shortened from several months to one-two days, with much quicker response to business requirements. Cloud computing has improved their resource utilization rates and measurements have shown that energy consumption has been saved by approximately 50 percent. They are also adjusting architecture for large-scale deployment, verifying that 800 servers hosting 100,000 VMs in the same region can be operating stably, supporting high availability of control nodes.

China UnionPay

China UnionPay is a pivotal element in China’s bankcard industry and has extended its card acceptance to 162 countries and regions. They have a scale of 1,200 compute nodes across two data centers, which make up 10,000 cores and about 2.0 petabytes of storage, all running on OpenStack. Some 80 percent of UnionPay’s production applications, including many critical ones, have been migrated onto UnionPay Cloud, accounting for over 150 applications. Their cloud is now supporting 500 million users, averaging 50 million transactions per day, 3,000 transaction per second at peak, with 100 billion RMB per day.

City Network

City Network runs their public OpenStack based cloud in eight regions across three continents. All of their data centers are interconnected via private networks. Apart from their public cloud, they run a pan-European cloud for  finance verticals solving all regulatory challenges. Over 2,000 users of their Infrastructure as a Service (IaaS) solutions run over 10,000 cores in production.

Insurance Australia Group Data Team

IAG is the highly trusted name behind many of the APAC region’s leading insurance groups. Their adoption of OpenStack has enabled the move to a real-time data architecture that allows for continuous integration and building of data products in days instead of months, changing their culture and way of delivering both internal and external customer value. The performance for data workloads is four times the performance and one-fifth the cost of their VMware environment, currently supporting workloads from micro to 8xlarge with volumes from 40G to 18TB per node.

Memset Hosting

OpenStack Swift has been part of Memset’s infrastructure since the Essex release. It forms the core of their multi-award winning distributed cloud storage platform, Memstore, which serves about 6.7 million requests a day. Memset operates multiple geographically diverse Swift platforms in production, each with varying capacity totaling around .75PB. Alongside this, their OpenStack IaaS deployment has approximately 2TB of production compute available over two geographically diverse regions advertising 300TB of raw Ceph storage through Cinder.

Tencent TStack Team

Tencent is a leading provider of Internet value added services in China. Their OpenStack-based private cloud platform cuts server costs by 30 percent and operator and maintainer costs by 55 percent, and saves them RMB100 million+ each year. It shortens resource delivery from two weeks to 0.5 hours and supports the development teams (such as QQ, WeChat, and Game) for services that generate tens of billions of revenues.

VEXXHOST

VEXXHOST is a leading Canadian public, private and hybrid cloud provider with an infrastructure powered by 100 percent vanilla OpenStack. Since the migration to an OpenStack infrastructure in 2011, VEXXHOST has been offering infrastructure-as-a-service without any vendor lock-in or proprietary technology. OpenStack compute, network and storage are the backbone powering all of their managed solutions.

The post It’s that time again! Cast your vote for the Sydney Superuser Awards appeared first on OpenStack Superuser.

Sydney Superuser Award Nominee: China Railway Corporation

Par Ashlee Ferguson

Voting is now closed. Stay tuned to find out who wins!

It’s time for the community to determine the winner of the Superuser Award to be presented at the OpenStack Sydney Summit. Based on the community voting, the Superuser Editorial Advisory Board will review the nominees and determine the finalists and overall winner.

Now, it’s your turn.

The China Railway Corporation is among the seven nominees for the Superuser Awards. Review the nomination criteria below, check out the other nominees and rate the nominees before the deadline Wednesday, September 20 at 11:59 p.m. Pacific Time Zone.

Please specify the team and organization for nomination.

The team for nomination: China Railway Information Technology Center(CRITC)
The joint team: CRITC, Beijing SinoRail Information Technology Co. Ltd.(BJSRIT), Beijing T2Cloud Technology Co. Ltd., with a total of about 200 engineers.

CRITC:
Mingxing Gao/Project Director
Yang Liu/System Architect
Liang Liu/Senior Engineer

BJSRIT:
Wei Rao/R&D Manager
Guangqian Li/Technical Support Manager
Yahong Du/OpenStack Leader
Minhong Wang/IAD of OMS Group Leader
Gang Xu/MON & AMS of OMS Group Leader

T2Cloud:
Jinyang Zhao/R&D Manager
Yahui Hou/CMP Group Leader
Chao Xie/OpenStack Leader
Hanchen Lin/Testing Leader
Tony Xu/Software Architect

How has OpenStack transformed the organization’s business?

OpenStack is the key element to promote transforming the construction model of China railway information system from the traditional project-driven to platform-driven. It also supports the high-speed rail as the foundation of China’s growth strategy. The adoption of OpenStack marked the first time CRITC has fully and embraced open source technology with an open mind. The deployment of OpenStack cloud saved us millions of dollars and the application launch cycle has been shortened from several months to 1-2 days, with much quicker response to business requirements.Cloud computing has improved resource utilization rates, and measurements have shown that energy consumption has been saved by approximately 50 percent.

How has the organization participated in or contributed to the OpenStack community?

In 2014, CRITC started developing an open source cloud solution based on OpenStack and started to actively participate in OpenStack community activities from 2016 on. This included sharing a topic at the 2016 OpenStack Days China, two topics at 2017 Boston Summit, two topics at 2017 Sydney Summit to be held and giving a keynote presentation at 2017 OpenStack Days China. In 2016 and 2017 they also hosted OpenStack meetups in Beijing and Nanjing respectively. We have contributed 734 patch sets, 5,979 lines of code, and submitted and resolved 47 bugs for the OpenStack community. We have shared some practice experience of stress testing and performance optimization analysis at 2017 Boston summit and 2017 OpenStack Days China and written whitepapers for the communities.

What open source technologies does the organization use in its IT environment?

In addition to OpenStack, the China Railway Cloud depends on KVM, OpenVSwitch/LinuxBridge, Hadoop, Kafka, Flume, Spark, CentOS, LXC, Docker, Kubernetes, OpenShift, CEPH, GlusterFS, Redis, MongoDB, MySQL/MariaDB, Ansible, Open-Falcon, ELK, ZeroMQ/RabbitMQ, etc..

What is the scale of the OpenStack deployment?

Deployed about 5,000 physical server nodes, including about 800 KVM nodes and about 730 VMware nodes; 20PB SAN storage, 3PB distributed storage (Ceph). An additional 2,000 physical server nodes are to be deployed in the end of 2017..

Our OpenStack cloud platform, with a scale of 800 physical nodes, hosts thousands of VMs and a dozen of mission critical applications, which covers 18 railway bureaus and over 2,000 railway stations and powers the production. OpenStack cloud platform also well undertook the huge pressure brought by Spring Festival peak to the system, especially the over 31 billion daily average page view, and it has also supported stable, safe and 24/7 uninterrupted operation of real-time dispatching management for all the trains, locomotives and vehicles.

What kind of operational challenges have you overcome during your experience with OpenStack?

Suspension of OpenStack service: By monitoring the service process and the status of log generation, we can check whether the OpenStack service is hung up or not.

Version upgrade problems: Previously, we had already upgraded the cloud software from Essex to Juno version and from Juno to Liberty. With lots of changes made on the community edition, we are now upgrading the software from the Liberty version to Ocata and we will upgrade the production system online after the above work are done.

High availability of cloud: We realized there was a function of automatic failover to achieve high availability of VMs. Compute nodes would be isolated and their VMs would be evacuated to the other nodes in the same zone if a certain network is down or is unresponsive for two minutes.

How is this team innovating with OpenStack?

Comprehensive cloud solution plan: Based on open source components, we developed the Operation Management System (OMS) complementary to our cloud software. Currently OMS consists of monitor, automation and analysis.

Multidimensional and customer-oriented improvements: Modified front-end functions to optimize customer experience, added operation type logs for easier archiving by administrators, added permissions control, added failover function, etc..

Architecture adjustment for large-scale deployment: We verified that 800 servers hosting 100,000 VMs in the same region can be operating stably, supporting high availability of control nodes.

How many Certified OpenStack Administrators (COAs) are on your team?

There is currently one COA on our team. Our team plans to be more involved in the OpenStack community, and cultivate more talent, aiming to obtain 3-5 COAs in 2018 and 5-10 COAs in 2019.

The post Sydney Superuser Award Nominee: China Railway Corporation appeared first on OpenStack Superuser.

Sydney Superuser Awards Nominee: China UnionPay

Par Ashlee Ferguson

Voting is now closed. Stay tuned to find out who wins!

It’s time for the community to determine the winner of the Superuser Award to be presented at the OpenStack Sydney Summit. Based on the community voting, the Superuser Editorial Advisory Board will review the nominees and determine the finalists and overall winner.

Now, it’s your turn.

China UnionPay is among the seven nominees for the Superuser Awards. Review the nomination criteria below, check out the other nominees and rate the nominees before the deadline Wednesday, September 20 at 11:59 p.m. Pacific Time Zone.

Please specify the team and organization for nomination. 

UnionPay plays a pivotal role in China’s bankcard industry and has extended its card acceptance to 162 countries and regions. Deployed in 2012, UnionPay Cloud was the first financial cloud powered by OpenStack in China’s financial industry. UnionPay has a dedicated OpenStack team with more than 50 members and many of them with COA training and upstream skills.

How has OpenStack transformed the organization’s business? 

With the introduction of OpenStack, UnionPay formed a cloud computing team, focusing on computing, storage, networking etc. OpenStack significantly shortened the time for UnionPay’s data centers to provide resources for the applications, going from a few days to a few minutes. OpenStack’s resilient ability has helped UnionPay to support users and greater business agility. UnionPay Cloud has supported the applications in cooperation with China Eastern Airlines, meanwhile UnionPay also introduced OpenStack to Bank of Shanghai and helped its deployment.

How has the organization participated in or contributed to the OpenStack community? 

We are frequent participants of OpenStack Days China and we also participated in most of the meetups in Shanghai and Beijing. We have delivered three keynote speeches in OpenStack Days China to share our experience as a typical OpenStack financial user in China. The request to build the Financial Working Group from UnionPay has been approved by the User Committee and we also participated Large Contributing OpenStack Operators (LCOO) Work Group to make more contribution to community. UnionPay works closely with EasyStack and Intel in the community work and we will give a speech with EasyStack, titled “UnionPay’s five years in the Financial Cloud Powered by OpenStack” at the OpenStack Sydney Summit.

What open source technologies does the organization use in its IT environment?

In terms of cloud computing, we use OpenStack, CentOS, KVM, Xen, Libvirt, Qemu, Open vSwitch, Ceph, Pacemaker, Corosync, HAProxy, Cobbler, Puppet, Ansible, Rabbitmq, Memcached, Mongodb, Mysql, Apache2, Zabbix, etc.
As for OpenStack, we use many components such as Nova, Cinder, Neutron, Keystone, Glance, Ceilometer, Ironic.
For Big Data, we use Hadoop, Spark, Impala, Kudu, etc.

We also use many other open source projects in applications such as Kafka, JBoss Application Server, Netty, Spring, Hibernate, Struts, etc.

What is the scale of the OpenStack deployment? 

We have the scale out of 1,200 compute nodes across two data centers, which make up 10,000 cores and about 2.0 petabytes of storage, all running on OpenStack. 80 percent of UnionPay’s production applications, including many critical ones, have been migrated onto UnionPay Cloud, accounting for over 150 applications. Our cloud is now supporting 500 million users, averaging 50 million transactions per day, 3000 transaction per second at peak, with 100 billion RMB per day. In order to support more new functions and more powerful management of UnionPay Cloud, we have completed the deployment of UnionPay Cloud 2.0 powered by OpenStack Liberty, with many applications already running on it.

What kind of operational challenges have you overcome during your experience with OpenStack? 

We designed several different network data planes and set corresponding QoS restrictions in both switches and servers to make sure that the network traffic will not affect each other.
We integrated Neutron with Huawei SDN solutions through driver to improve the entire network performance. It is the first mature case and put onto the critical production in China’s financial industry. We modified Keystone’s code to integrate it with UnionPay’s existing SSO system to unify the authentication mechanism. It is risky and complicated to upgrade the existing old OpenStack version directly, so we deployed the new UnionPay Cloud 2.0 paralleled with the old one and make a detailed plan to migrate the applications gradually.

How is this team innovating with OpenStack? 

We are able to manage F5 LTM device through LBaas V2 API and implemented the advanced function of LTM beyond LBaas on portal directly through F5 SDK. We are able to manage Cisco ASA devices which locate between different OpenStack regions through FWaas API and custom developed ASA SDK to automate the management of firewall. We developed a unified portal to manage all OpenStack regions for more convenient operation and maintenance. We extended Nova to implement the function of live resize for the virtual machine and it is very important for the critical applications in the financial industry. We are able to make sure the host high availability by deploying a custom daemon process on every controller node to detect the status of compute nodes.

How many Certified OpenStack Administrators (COAs) are on your team?

None, but dozens of engineers in our team have taken COA training and are preparing for COAs tests in 2018.

 

Cover photo courtesy UnionPay.

The post Sydney Superuser Awards Nominee: China UnionPay appeared first on OpenStack Superuser.

Sydney Superuser Award Nominee: City Network

Par Ashlee Ferguson

Voting is now closed. Stay tuned to find out who wins!

It’s time for the community to determine the winner of the Superuser Award to be presented at the OpenStack Sydney Summit. Based on the community voting, the Superuser Editorial Advisory Board will review the nominees and determine the finalists and overall winner.

Now, it’s your turn.

City Network is among the seven nominees for the Superuser Awards. Review the nomination criteria below, check out the other nominees and rate the nominees before the deadline Wednesday, September 20 at 11:59 p.m. Pacific Time Zone.

Please specify the team and organization for nomination.

City Network’s Operations and DevOps team: Marcus Murwall, Tobias Rydberg, Magnus Bergman, Tobias Johansson, Alexander Roos, Johan Hedberg, Joakim Olsson, Emil Sundstedt, Kriss Andsten.

How has Openstack transformed the organization’s business?

Shifting to 100% focus on OpenStack has been key to the global expansion of our organization in general and our cloud offerings in particular. With OpenStack and all its features, ease of use and popularity as the catalyst we have added value through multiple data centers in Europe, US, Asia and UAE as well as a clear strategy and implementation for Data protection and regulatory aspects.

With OpenStack we also get the benefit of providing open APIs which prohibits our customers to be locked in with only us as an IaaS provider. It´s easy for the customer to start and easy to leave, as we are convinced that this is a must for all providers in order to stay relevant going forward.

How has the organization participated in or contributed to the OpenStack community?

Our CEO Johan Christenson was recently elected as a member of the OpenStack Foundation Board. His goal is to help the community and the ecosystem to leverage this open platform and drive the transformation for a more open, and future proof IT infrastructure.

In 2016, we initiated and organized the first ever OpenStack Days Nordic event in Stockholm and this year we are once again leading the way to take the event to Copenhagen.

We also participate in the summit user groups public cloud and the security project.

What open source technologies does the organization use in its IT environment?

We are very pro open source and use it every time where open source is a viable option.

A selection of the Open Source technologies we are currently using: CentOS, OpenBSD, Ubuntu, Nginx, Apache, php, Python, Ansible, MySQL, Mariadb, Mongodb and Ceph.

What is the scale of the OpenStack deployment?

We run our public OpenStack based cloud in eight regions across three continents. All of our data centers are interconnected via private networks. Apart from our public cloud, we run a Pan-European cloud for the finance vertical solving all regulatory challenges. Over 2,000 users of our infrastructure-as-a-service (IaaS) solutions run over 10,000 cores in production.

What kind of operational challenges have you overcome during your experience with OpenStack?

Since we are running OpenStack as Public IaaS there have been a lot of hurdles to overcome as OpenStack is not yet fully adapted for public clouds. We had to build our own APIs in order to get network connectivity over several sites to work and also we had to add features such as volume copy and the ability to move volumes between sites. We have also had our fair share of issues with upgrading to new OpenStack versions, however we do feel as this process have been getting better with each upgrade.

How is this team innovating with OpenStack?

We innovate with OpenStack on two main focus areas:

One of them is figuring out how we can interconnect all our OpenStack data centers over a global, private network and all the benefits that comes from doing so—one of which is being able to provide our customers with direct, private access to our cloud services.

The other focus area is helping regulated companies, mainly in the financial and healthcare industries, with their digital transformation and cloud adoption. By building completely separated cloud services compliant with regulations such as ISO 9001, 27001, 27015, 27018, Basel, Solvency and HIPA, we allow for these industries to go cloud with a pay-as-you-go model and be truly agile.

How many Certified OpenStack Administrators (COAs) are on your team?

Three.

The post Sydney Superuser Award Nominee: City Network appeared first on OpenStack Superuser.

Sydney Superuser Award Nominee: Insurance Australia Group (IAG) Data Team

Par Ashlee Ferguson

Voting is now closed. Stay tuned to find out who wins!

It’s time for the community to determine the winner of the Superuser Award to be presented at the OpenStack Sydney Summit. Based on the community voting, the Superuser Editorial Advisory Board will review the nominees and determine the finalists and overall winner.

Now, it’s your turn.

The Insurance Australia Group (IAG) Data Team is among the seven nominees for the Superuser Awards. Review the nomination criteria below, check out the other nominees and rate the nominees before the deadline Wednesday, September 20 at 11:59 p.m. Pacific Time Zone.

How has OpenStack transformed the organization’s business? 

OpenStack has enabled the move to a real time data architecture that allows for continuous integration and building of data products in days instead of months. This work is all tightly integrated with GitHub for pair programming and peer review and then automatically deploying new data and analytics products to serve our internal users better and faster. OpenStack has also allowed us to enable new ways to interact with our external customers and make their world a safer place. The OpenStack solution has also improved our performance stats and enabled new ways to deliver open source software solutions point in time for our internal teams. Overall it has changed our culture and our way of delivering both internal and external customer value.

How has the organization participated in or contributed to the OpenStack community? 

IAG is active in many open source communities and has been on the mailing lists and attending community events as well as presenting at several conferences on our transformative solution. In addition to partnering with external companies to commit code to enable other to leverage our solutions IAG will be open sourcing our first piece of software in early September directly that has all been built using our OpenStack solution.

What open source technologies does the organization use in its IT environment?

IAG leverages a number of data as well as web open source solutions today and contributes back to a number of them as well thru our digital and data teams.

What is the scale of the OpenStack deployment? 

OpenStack is deployed in three separate tenants within IAG: Pre-production, production and analytics. The analytics tenant consists of 12 server with high performance and archival storage leveraging commercial ScaleIO SDS solution. The pre-production is 10 nodes and uses a nl-sas solution using open source ScaleIO SDS solution. The production tenant has 18 node of a mixed SSD, SAS and nl-SAS the later two leverage commercial ScaleIO SDS solution. The performance for data workloads is four times the performance and one-fifth the cost of our VMWare environment. Currently we support workloads from micro to 8xlarge and have volumes from 40G to 18TB per node.

What kind of operational challenges have you overcome during your experience with OpenStack? 

The ability to stand up a new complete environment end to end do a quick validation of proof of concept and fail fast has been highly valuable and the upgrade and migration of the platform has been seamless.

How is this team innovating with OpenStack? 

The data workloads and adoption of quick fail fast prototypes to deliver data products has aided in our pricing, customer service, marketing and analytics space. The solution has also allowed IAG to focus on core customer facing capability using developers and code, while not worrying about the infrastructure underneath.

How many Certified OpenStack Administrators (COAs) are on your team?

Five.

The post Sydney Superuser Award Nominee: Insurance Australia Group (IAG) Data Team appeared first on OpenStack Superuser.

Sydney Superuser Award Nominee: Memset Hosting – OpenStack Public Cloud Team

Par Ashlee Ferguson

Voting is now closed. Stay tuned to find out who wins!

It’s time for the community to determine the winner of the Superuser Award to be presented at the OpenStack Sydney Summit. Based on the community voting, the Superuser Editorial Advisory Board will review the nominees and determine the finalists and overall winner.

Now, it’s your turn.

Memset Hosting is among the seven nominees for the Superuser Awards. Review the nomination criteria below, check out the other nominees and rate the nominees before the deadline Wednesday, September 20 at 11:59 p.m. Pacific Time Zone.

Please specify the team and organization for nomination.

Ross Martyn
George Stamoulis
Simon Weald
Stewart Perrygrove
Nick Craig-Wood

Memset Hosting has been focused on open source since our inception in 2002. Given the open nature of OpenStack, Memset naturally became heavily interested early in OpenStack’s history.

We had a clear focus from day one; deploy a highly secure, highly available public cloud platform, that meets the strict requirements of the UK Government, then offer that same rock solid level of security to the remainder of our customer base.

How has OpenStack transformed the organization’s business?

OpenStack Swift has been part of Memset’s infrastructure since the Essex release. It forms the core of our multi-award winning distributed cloud storage platform, Memstore, serving ~6.7 million requests a day.

Memset has since developed a fully self-service OpenStack IaaS platform, offering our customers highly performant and tightly-secured public cloud, perfect for UK public and private sector needs.

OpenStack’s rapid development and fantastic community support enables us to implement new features, performance improvements and security updates quicker than ever before, and, excitingly, is increasingly working its way into other parts of our business, most recently being used to breathe new life into our older proprietary Xen- based Cloud VPS platform.

How has the organization participated in or contributed to the OpenStack community?

Alongside our presence in the IRC community, Memset regularly attends a variety of OpenStack events around the globe, including Summits, working groups and operators meetups. We also visit other open source events including Ceph, Kubernetes meet-ups and Gophercon. We are a regular sponsor of events including the London OpenStack Meetup and the upcoming OpenStack Days London conference.

As well as the social aspect of the community, Memset’s team strives to feedback bugs, patches and reviews to the community wherever possible, with a string of contributions to OpenStack since early 2012 to multiple projects.

Memset recently became a corporate sponsor of the OpenStack Foundation, offering our highly secure public IaaS and award winning Cloud Storage (Memstore) from the OpenStack Marketplace.

What open source technologies does the organization use in its IT environment?

Open source is built into the very core of Memset. We extensively utilize an open-source approach wherever possible. This is evident in every layer of our business, from the Ubuntu powered Operations and Development team, through to the heavy Python development and Django management backend that powers Memset’s business critical systems. We strive to build and maintain great public services, using brilliant people, community driven software and commodity hardware.

Memset’s co-owner and technical director Nick Craig-Wood has been able to personally rock the open source object storage community with his pet project rclone – (an rsync tool for cloud storage), and the widely used Go Swift Library, providing an easy way for new Go projects to interact with Swift environments.

Whilst not an exhaustive list, most commonly used technologies at Memset include: OpenStack, Puppet, Ansible, Terraform, Jenkins, Gerrit, Git, SVN, Python, Django, Packer, Docker, HaProxy, Nginx and much more…

What is the scale of the OpenStack deployment?

Memset operates multiple geographically diverse Swift platforms in production, each with varying capacity totaling around .75PB. Alongside this, our OpenStack IaaS deployment has approximately 2TB of production compute available over two geographically diverse regions advertising 300TB of raw Ceph storage through Cinder. We integrate NVME, SSD and HDD storage, all built on latest generation Dell equipment.

Memset operates at two sites in the UK, including its privately owned data center located in Dunsfold. Memset strives to operate in an environmentally friendly manner, regularly winning awards for doing so, aided by our utilization of a large local solar farm to offset a portion of our high power requirements.

What kind of operational challenges have you overcome during your experience with OpenStack?

OpenStack has been fairly easy to integrate into our mature business, mostly due to the great documentation, administration and security guides. Whilst times can become stressful during upgrades and patching cycles, and especially when dealing with customers in production, OpenStack gives us easy access to intelligent management capabilities that allow us to maintain customer uptime and in turn, their satisfaction.

OpenStack’s APIs can give us the ability to dynamically schedule migrations of workloads in order to attempt zero downtime patching, software and hardware upgrades. We strive to minimize this even further for the future, especially as recent OpenStack releases have focused so heavily on improving these processes.

How is this team innovating with OpenStack?

Memset’s OS team focus is currently moving their production environments towards a container based control plane. This will allow us to be able to keep pace with the fast OpenStack release cycle easier than ever before.

Utilizing what we have learned, we aim to be able to stay closer to trunk, and plan to offer Magnum and Octavia as self-service offerings to our loyal customers very soon.

How many Certified OpenStack Administrators (COAs) are on your team?

At Memset we have more than half a dozen Mirantis certified OpenStack Administrators and currently have two engineers studying for the COA exam.

The post Sydney Superuser Award Nominee: Memset Hosting – OpenStack Public Cloud Team appeared first on OpenStack Superuser.

Sydney Superuser Awards Nominee: Tencent TStack Team

Par Ashlee Ferguson

It’s time for the community to determine the winner of the Superuser Award to be presented at the OpenStack Sydney Summit. Based on the community voting, the Superuser Editorial Advisory Board will review the nominees and determine the finalists and overall winner.

Now, it’s your turn.

The Tencent TStack Team is among the seven nominees for the Superuser Awards. Review the nomination criteria below, check out the other nominees and rate the nominees before the deadline Wednesday, September 20 at 11:59 p.m. Pacific Time Zone.

Please specify the team and organization for nomination.

The Tencent TStack team is comprised of 76 members who developed the OpenStack-based TStack private cloud platform. It consists of four sub-teams:

  • Product design: Responsible for requirement analysis and interaction design
  • Technical architecture: Responsible for solution design and technology research
  • Product development: Responsible for feature design and implementation
  • Operations support: Responsible for deployment, monitoring and troubleshooting.

We built private cloud to provide services for internal IT environments and testing environments (e.g. QQ, WeChat).
We also provide complete hybrid cloud services for government departments and enterprises in China.

How has Openstack transformed the organization’s business?

The OpenStack-based private cloud platform cuts server costs by 30% and O&M costs by 55%, and saves RMB100 million+ each year for Tencent. It shortens resource delivery from two weeks to 0.5 hours and supports the development teams (such as QQ, WeChat, and Game) of the services that generate tens of billions of revenues for Tencent. It optimizes global resource scheduling. For example, the deployment duration of the global mail system has been cut from 10 days to 1 day. It is an important cloud computing platform for Tencent to achieve its “Internet plus” strategy. It is deployed in multiple provinces in China, including Sichuan government cloud, Guangdong government cloud, Xiamen government cloud, and Yunnan government cloud. It serves more than 100 million users.

How has the organization participated in or contributed to the OpenStack community?

Tencent actively participates in community activities, such as OSCAR, OpenStack Days China. As a participant and sponsor, Tencent shared experience in using OpenStack. Currently, the TStack team is preparing to participate in OpenStack Sydney Submit. We compiled the OpenStack Usage White Paper which we shared with our customers. We created a WeChat official account to share our experience with OpenStack users, and submitted bugs and blueprints to the community. We cooperated with Intel to apply the latest features (such as RDT, DPDK, SPDK, FPGA, and Clear Container) to accelerate the production. Tencent plans to join the OpenStack Foundation. Tencent is one of the members of the CNCF, Linux Foundation, and MariaDB Foundation.

What open source technologies does the organization use in its IT environment?

Technology stacks used in the TStack team are built with open source tools, including OpenStack, KVM, Centos, Ironic, HAProxy, keepalived, Docker, Clear Container, Kubernetes, Rabbitmq, Mairadb, Nginx, Ansible, Jenkins, Git, ELK, Zabbix, Grafana, Influxdb, Tempest, and Rally. These tools are used in development, testing, and CI/CD.

What is the scale of the OpenStack deployment?

TStack has 6,000+ nodes, including 2,000 ironic nodes. It manages more than 12,000 VMs, provides 80,000+ cores, 360+ TB mem, and 20+ PB disks. It covers 14 clusters in 7 data centers in 4 regions. 1,000+ nodes managed in a single region. It carries 300+ online services, including OA, WeChat gateway, email, and ERP. For example, the email system on the TStack provides 24/7 services for 40,000 employees. It provides CI/CD environment for 20,000 developers and testing environment for large applications (such as QQ and WeChat), and supports concurrent access of more than 100 million users. Tencent promoted TStack to Chinese government and enterprise. It signed cooperation agreements with 15 provinces and 50 cities, for example, TStack-based Xiamen government cloud in the BRICS Summit.

What kind of operational challenges have you overcome during your experience with OpenStack?

Many different types of VMs, such as XEN and KVM, are deployed in Tencent. It is a huge challenge to manage VMs seamlessly on the OpenStack platform. The TStack team developed a set of tools to manage VMs on existing heterogeneous virtualization platforms without interrupting services. Earlier versions of OpenStack had bottlenecks, including message queues and keystones in large-scale deployment. After large-scale optimization, a single region can now support more than 1,000 computing nodes. In large-scale deployment, the VxLAN performance is a bottleneck. Tencent developed an OpenStack-based technology that is compatible with SDN hardware and software of multiple vendors to accelerate the VxLAN. The adaptive compression technology is used to cut the VM migration duration by 50 percent.

How is this team innovating with OpenStack?

Strict real-time rate limit to control CPU usage time and ensure VM QoS. VM scheduling policies are customized based on service tag, with VMs dispatched to different hosts to ensure high availability. The online resize function resizes VMs without interruption. The adaptive compression technology is used to cut VM migration duration by 50%. Cooperated with Intel to provide Clear Container on TStack for better security. The Neutron-based SDN controller supports heterogeneous networks and performs unified management on legacy networks, SDNs and NFVs. Integration of Tencent cloud security technologies with OpenStack to provide complete cloud security services. Implemented unified management of OpenStack-based private cloud, public cloud, and VMware.

How many Certified OpenStack Administrators (COAs) are on your team?

One team member is COA-certified, 28 have received COA training and nine will take the COA examination in October.

 

Cover image: Shenzen’s Tencent building, courtesy Tencent.

The post Sydney Superuser Awards Nominee: Tencent TStack Team appeared first on OpenStack Superuser.

Sydney Superuser Award Nominee: VEXXHOST

Par Ashlee Ferguson

It’s time for the community to determine the winner of the Superuser Award to be presented at the OpenStack Sydney Summit. Based on the community voting, the Superuser Editorial Advisory Board will review the nominees and determine the finalists and overall winner.

Now, it’s your turn.

VEXXHOST is among the seven nominees for the Superuser Awards. Review the nomination criteria below, check out the other nominees and rate the nominees before the deadline Wednesday, September 20 at 11:59 p.m. Pacific Time Zone.

Please specify the team and organization for nomination.

VEXXHOST is a leading Canadian public, private and hybrid cloud provider with an infrastructure powered by 100 percent vanilla OpenStack. Since the migration to an OpenStack infrastructure in 2011, VEXXHOST has been offering infrastructure-as-a-service without any vendor lock-in or proprietary technology. The VEXXHOST team, lead by CEO Mohammed Naser, delivers a high level of expertise to help users optimize cloud infrastructure so they can focus on their core competencies.

How has Openstack transformed the organization’s business?

OpenStack has helped VEXXHOST speed up the delivery of new services to customers by accelerating ability to innovate. We can focus on delivering greater customer solutions instead of dealing with infrastructure issues which are solved by OpenStack. Every release of OpenStack has offered new features that were utilized to deliver higher performance.

How has the organization participated in or contributed to the OpenStack community?

VEXXHOST have been contributing to the OpenStack community since its second release in 2011. We have had a presence in the community by regularly attending OpenStack summits and being part of the Interop challenge during the Boston summit in 2017. We have hosted OpenStack Canada day and help organize the Montreal OpenStack meetup. Our co-founder Mohammed Naser, who is the PTL for Puppet OpenStack, has given talks at Montreal, Ottawa and Toronto OpenStack meetups.

We also play a part in the community by actively contributing upstream code and sharing feedback with PTLs and developers. When we encounter bugs, we report them, diagnose them and work with the community to get a full fix in. We are active on the mailing list and provide feedback and fixes. We also contribute to the community.

What open source technologies does the organization use in its IT environment?

We run exclusively OpenStack services across our entire infrastructure. Our offering is fully open source without any proprietary licensed technology. Among many others, we use in our IT environment Nova with KVM with Libvirt, Ceph centralized storage, Pacemaker for high availability, MySQL Galera for database and Puppet for config management.

What is the scale of the OpenStack deployment?

Being a public cloud provider, we cannot disclose metrics regarding the scale of our users’ consumption. Our public cloud is able to handle several production-grade enterprise-scale workloads with private and hybrid cloud solutions delivering the same level of stability and robustness as our public cloud. Both our infrastructure and users production workloads are powered by OpenStack. OpenStack compute, network and storage are the backbone that is powering all our managed solutions.

What kind of operational challenges have you overcome during your experience with OpenStack?

Initially we have faced some challenges in terms of rolling upgrades as they were difficult though they have become much easier with new releases. After upgrading our infrastructure to Pike, we found a bug in the code which we reported. The developers at OpenStack were very responsive and happy to cooperate — as they always are — to help fix the bug. The bug was fixed in less than 24 hours in trunk and less than 48 hours in stable branches. This increases our trust the OpenStack CI and grows our confidence in the software.

How is this team innovating with OpenStack?

As a public and private cloud provider, we are heavily invested in improving and extending our list of managed services. Using OpenStack has helped us innovate in our managed services. In August 2017, we launched Kubernetes services using Magnum on the Pike release. We worked with the Magnum project team to ensure delivery of the best possible Kubernetes and OpenStack experience. VEXXHOST is currently one of the very few cloud providers to offer Magnum. OpenStack has also facilitated the delivery of big data solutions with the help of Sahara integration. We were also able to speed up the deployment of clusters with the help of transient clusters which provide huge cost savings.

How many Certified OpenStack Administrators (COAs) are on your team?

The VEXXHOST team does not have any Certified OpenStack Administrators on its team.

The post Sydney Superuser Award Nominee: VEXXHOST appeared first on OpenStack Superuser.

Collaboration and code beat hype and ego every time

Par Mark Collier

Tech may come and go in hype cycles but some models endure.

Take the LAMP stack, which gets its name from its original four open-source components: the Linux operating system, the Apache HTTP Server, the MySQL relational database management system (RDBMS) and the PHP programming language. It drove the creation and meteoric rise of an entire industry on open source.

LAMP provided a model that involved multiple open-source projects, each with their own communities. This was not one community creating a monolith and that’s why it worked. Each community was able to focus on what they were good at, which wasn’t feasible in just one community.

If we can’t put aside the egos, AWS will absolutely eat everyone’s lunch. In every industry. Period. Skeptical? We can meet at Whole Foods to discuss it.

Although it was somewhat opinionated as a concept, it did get the operational burden under control and yet it was still modular enough so that each layer could be swapped out. These communities seemed to have struck the right balance. We have every opportunity to replicate this success in the open infrastructure/open cloud area, but only if we work across communities and listen to users.

It’s clear from every user I talk to that they struggle with pulling all the pieces together, particularly in terms of operating the beast once it’s assembled. That’s what they want, but we just aren’t there yet. Within OpenStack itself, we have started to cull options and projects to essentially make it more opinionated, which is helping.

Our opportunity, which is far from trivial, is to rise to the occasion across communities to actually serve the needs of the users, rather than succumbing to ego-driven development. If we can’t put aside the egos, AWS will absolutely eat everyone’s lunch. In every industry. Period. Skeptical? We can meet at Whole Foods to discuss it.

To offer a concrete example, eBay is perhaps the largest OpenStack user in the world. It also happens to have what may be one of the largest deployments of Kubernetes. They deploy them together and it’s very powerful!

What works well about that opinionated combination and what doesn’t? How are the participants in each community doing when it comes to listening to what eBay actually needs and wants out of the combination? It will take more than Kubernetes+OpenStack to form the LAMP stack of the cloud, but they’re a great place to start.

That’s why we recently organized the OpenDev event in San Francisco, with members of many open-source communities looking to build the open edge computing stack. We heard from over 30 organizations, including users like eBay, Verizon, NTT, AT&T, Walmart, and Inmarsat, and industry leaders from Carnegie Mellon, Intel, VMware, Ericsson, Red Hat and Huawei.

They’re taking open infrastructure to some of the harshest environments imaginable: from ships at sea to appliances next to the ovens heating your bagels at coffee shops to massive retail stores handling hundreds of billions of dollars in commerce. For each of these use cases, the old cloud model simply doesn’t work and so it falls upon us to assemble an open infrastructure stack that will.

This event was one concrete step we’ve taken to build something that is both diverse and open, while also being opinionated enough for each use case to actually operate at scale.

Kelsey Hightower, a prominent leader in the Kubernetes community, is taking the next step by organizing a joint Kubernetes-OpenStack event during KubeCon in Austin in December. This is the precisely the kind of leadership we need.

There are many steps left, so join us!

Mark Collier, COO, OpenStack Foundation can be found at @sparkycollier or your local Whole Foods.

 

The post Collaboration and code beat hype and ego every time appeared first on OpenStack Superuser.

Developing edge computing use cases, reference architectures

Par Nicole Martinelli

SAN FRANCISCO — The edge will soon be everywhere — telecoms, retail, internet of things, supply chains – and an all-star group of industry experts has pledged to build use cases and reference architectures for it.

That’s one of the major outcomes of OpenDev, a recent two-day event sponsored by the Ericsson, Intel and the OpenStack Foundation. OpenDev was devised as more of a workshop than a traditional conference with the first day featuring sessions more like working groups based on key topics including “Thin Control Plane,” “Deployment Considerations” and “Zero-Touch Provisioning.”

Reference architecture was one of the “meatiest of all the sessions” said Jonathan Bryce, executive director of the OpenStack Foundation who facilitated the 90-minute closing session of the September 7-8 event. The takeaways from all sessions are summarized together on a single Etherpad and you can also check the event schedule for Etherpads from the individual sessions.

Participants from the reference architecture session said the next action is “to discuss and determine what functions are managed by OpenStack and what is managed by the layer above when managing edge nodes. The outcome is potentially a set of whitepapers (one per use case) with straw man deployment designs.”

Volunteers to push forward efforts here include veteran OpenStack members, employees of multinational tech conglomerates and telcos. Industries identified as solid terrain for edge include: Retail, supply chain, utilities, industrial telematics, autonomous vehicles, industrial IoT, agriculture and medical tech.

There were five major edge use cases identified to work on:

  • Micro Edge Device (Remote Radio Head / Remote Radio Unit, CPE / set top box, runs a single instance, instance changes infrequently)
  • Small Edge Device (coffee shop / POS for a store / cell tower site / FTTN cab, multiple instances, instances change occassionally)
  • Medium Edge Backhaul Critical Deployment (C-RAN / big cell site / NBN POI, multiple instances, instances change daily)
  • Medium Edge Backhaul Non-Critical Deployment (big box retail / cloudlet, multiple instances, instances change daily)
  • Large Edge Deployment (region DC, thousands of instances, instances changing constantly)

Verizon’s Beth Cohen noted in the closing session that “there isn’t really an ecosystem of vendors for edge, what we have now is coming out of internal requirements, there’s nobody to go to.” When another participant asked her if it should stay that way, she said “No! It’s very early and this is an opportunity.”

Stay tuned for more on how you can get involved and on these emerging edge computing reference architectures.

Cover Photo // CC BY NC

The post Developing edge computing use cases, reference architectures appeared first on OpenStack Superuser.

How edge computing can take the pain out of daily life

Par Nicole Martinelli

SAN FRANCISCO —- Some day soon, when you assemble a piece of Ikea furniture without tears you’ll have edge computing to thank.

That’s the vision of Mahadev Satyanarayanan, known as Satya, a computer science faculty member at Carnegie Mellon University. While the word “disruptive” gets passed around more than bitcoin in the Bay Area, he means it, expecting that edge computing will touch everything from troubleshooting equipment repair to helping people with Alzheimer’s live at home for longer.

Some great (and entertaining) use cases for #edge computing at the #OpenDev keynote by M. Satyanarayanan pic.twitter.com/zs7c6qFgaR

— Lisa-Marie Namphy🤓 (@SWDevAngel) September 7, 2017

With a halo of greying curls and wry sense of humor, Satya is considered the godfather of edge computing for his 2009 paper “The Case for VM-based Cloudlets in Mobile Computing.” He showed a demo to the 200-or so technical folks assembled for OpenDev where a researcher put together an Ikea lamp with the computer-assisted technology in the form of Google Glass. The headset showed the user a video of the assembly as he was putting it together, promptly rewarded him with “good job” when he got it right and caught him when he forgot a piece.

Satya outlines the perfect storm for edge computing at OpenDev.

“We have some distance to go before these are things we can count on, but edge is coming and it will disrupt,” he concluded.

It was an inspiring kick-off to the event held at DogPatch Studios. Sponsored by the Ericsson, Intel and the OpenStack Foundation, OpenDev was devised as more of a workshop than a traditional conference. The roughly two-hour keynotes featured talks from AT&T, eBay, Verizon and Intel. (Superuser will have more on those with the videos, stat.)

The first day featured sessions more like working groups based on key topics including “Thin Control Plane,” “Deployment considerations” and “Zero-Touch Provisioning.” On day two, the afternoon slots are set aside for unworking sessions, where participants will gather insights from previous work and discuss how to move forward. (Stay tuned to Superuser for those takeaways.)

OpenStack Foundation executive director Jonathan Bryce likes to back up his predictions with numbers. And while the numbers surrounding the edge computing can be a little fuzzy (will there be 10 or 50 billion devices connected to the internet in 2020?) the crystal ball gazers have one thing right: the “estimates may be off, but the point is that all those devices will be generating data and generating work that needs to be analyzed and put into a system.”

Experiencing some seriously powerful brainstorming at the #OpenDev edge computing conference working sessions. Competitors collaborating.

— Dan Sneddon (@dxs) September 7, 2017

Bryce outlined some important goals for the event:

  • Build relationships between practitioners working on edge and developers building infrastructure
  • Create and publish documentation
  • Develop use cases, architecture and deployment patterns
  • Build best practices around management, operations, security, etc
  • Devise triage needs for open projects

The outcome will be to “arrive at a vision for what this can be and where this is going to go.” More to come!

Cover Photo // CC BY NC

The post How edge computing can take the pain out of daily life appeared first on OpenStack Superuser.

Not just a road: The computing history behind “Pike”

Par Anne Bertucio

The OpenStack community recently released Pike, the 16th release of OpenStack. As always, the OpenStack Foundation team has been busy fielding questions about the release from press, analysts and OpenStack supporters. Most are about the latest features, but on occasion a brave soul has asked, “So…uh…what’s a ‘pike’?”

“The Massachusetts Turnpike. It’s a road near Boston.”

“So this release is named after a road?”

Stop the presses! This isn’t just any road. It’s the road, and it has a greater significance to Boston’s technology history than you might think.

To get the story from a local’s point of view, I called up my father, who we’ll call “Billy from Boston.” Billy was born and raised in the Boston suburbs. He was in his teens when they began building the Massachusetts Turnpike in 1955. At this time Wang Laboratories, headquartered in nearby Cambridge, was taking off. “I had a Wang Calculator while Bill Gates was still wearing diapers!” claims my father. While this claim can be refuted by some timeline checking and basic math, the point is well taken: Boston had a high-tech industry well before much of the US.

Industries take people and there were two major roads that made “the people” factor possible in Boston’s tech story: The Massachusetts Turnpike and Route 128. Route 128 runs north-south in a semicircle, connecting the western Boston suburbs. Route 128 was called “America’s Technology Highway” because of all the labs and computing companies along the route. The Turnpike, which, if you want to sound like a local should only be called the “Mass Pike,” stretched the length of the state and brought all east-west bound traffic into Route 128 and Boston.

For the next two decades, science, research and computing in Boston grew––wave to college freshman Richard Stallman at Harvard in 1970 as we zoom by on our history adventure––laying many of the foundations for modern computing. “The Mass Pike made access to all the high tech jobs on Route 128 way easier. You drove in during the day, out at night,” says Billy, “and it worked. It resulted in the great economic success and high tech industry of Massachusetts.”

The economic success Billy from Boston is referring to is called the “Massachusetts Miracle,” when in 1975, the unemployment rate in the state fell from 12 percent to 3 percent and kept a thriving economy through the 80s, due to Boston’s tech industry.

So Pike isn’t exactly “just a road.” It’s a transportation milestone for a state, a puzzle piece in computing history and a key to unlocking the economic prowess of technology.

Each OpenStack development cycle has a code name proposed and chosen by the community, this one is an homage to the long and winding road of tech.

 

Images // CC BY NC

The post Not just a road: The computing history behind “Pike” appeared first on OpenStack Superuser.

OpenStack operator spotlight: Adobe Advertising Cloud

Par Nicole Martinelli

We’re spotlighting users and operators who are on the front lines deploying OpenStack in their organizations to drive business success. These users are taking risks, contributing back to the community and working to secure the success of their organization in today’s software-defined economy. We want to hear from you, too: get in touch with editorATopenstack.org to share your story.

Here we catch up with Adobe’s cloud platform manager Joseph Sandoval. If you’re interested in hearing more, Adobe will be keynoting at the upcoming Sydney Summit.

Describe how are you using OpenStack. What kinds of applications or workloads are you currently running on OpenStack?

Adobe Advertising Cloud is running OpenStack in production across six data centers in the US, Europe and Asia. We’re running a high volume real-time bidding application that requires low latency and high throughput to meet our growing customers demands.

What business results, at a high level, have you seen from using OpenStack? What have been the biggest benefits to your organization as a result of using OpenStack? How are you measuring the impact?

We’re seeing higher performance with a purpose-built architecture that meets our application needs. As our business continues to grow, there are demands for increased compute. OpenStack allows us to scale in repeatable modules which scale up without scaling the team out. We tracked our application performance metrics running on OpenStack Icehouse and public cloud and both had performed equally. With our recent upgrade to Mitaka, we have measured a massive performance gain (three times faster) compared to our public-cloud instances.

What’s a challenge that you’ve faced within your organization regarding OpenStack and how did you overcome it?

Our engineering community understands public cloud and the benefits it brings but had concerns about going to private cloud. But by evangelizing the platform and providing tooling that helped them get productive quickly on a purpose-built private cloud, engineers experience the benefits and agility and build features that are optimized for a consistent compute platform.

 

Superuser wants to hear more from operators like you, get in touch at editorATopenstack.org

Cover Photo // CC BY NC

The post OpenStack operator spotlight: Adobe Advertising Cloud appeared first on OpenStack Superuser.

Getting started with OpenStack? Check out this cheat sheet

Par Superuser

Following the long tradition of one-page cheat sheets for common UNIX applications (remember EMACS, Bash, regular expressions?) Aviv Lichtigstein, head of product evangelism at Loom Systems recently made a nice-looking one for OpenStack where he covers some common commands for services Keystone, Glance, Nova, Neutron and Cinder.

From my own experience, having a proper cheat sheet will make your life so much easier,” he says over on the Loom Systems blog.  “When I just started out with software engineering (back in 2004), I had my own cheat sheet, too.” 

You can download the PDF here.

 

Hat tip/Reddit

Superuser is always interested in community content, get in touch: editorATopenstack.org

The post Getting started with OpenStack? Check out this cheat sheet appeared first on OpenStack Superuser.

❌