DOES17 LONDON FROM CODE COMMIT TO PRODUCTION WITHIN A DAY TRANSCRIPT

Size: px

Start display at page:

Download "DOES17 LONDON FROM CODE COMMIT TO PRODUCTION WITHIN A DAY TRANSCRIPT"

Lindsay Doyle
5 years ago
Views:

1 DOES17 LONDON FROM CODE COMMIT TO PRODUCTION WITHIN A DAY TRANSCRIPT

2 Gebrian: My name is Gebrian uit de Bulten, I m from Accenture Gebrian: Who has ever heard about Ingenco? Gebrian: Well, not a lot of people so I think Vincent has a good job for you to explain what Ingenico is. Vincent: Yeah, let me uh. Can you guys all hear me? Yeah? Cool. I found the clicker, this is the clicker right? I just found it over there. Vincent: So, Ingenico. How many hands did go up? Like 5 or 6? I m not sure. So during the break actually, I told Gebrian, I went out to quickly get a small Pedington bear for my kid at home. So I have something to bring tonight. And I went to pay, and what did I see? The terminal was an Ingenico one. It s one of those companies you have never heard of. But once you see it, you will never unsee it anymore. So tonight, if you go to the hotel or you go for dinner or whatever, and you have to pull your card and do a payment, take a look on the terminal and I bet a least one time tonight it will say Ingenico. So that s who we are, before I get to that. Ingenco is a French company, it was founded somewhere in the eighties and the core business is payment terminals and all the back-end systems. I think we have about 32 million terminals running right now globally. So that s a lot of payments every day. But, today I am not for that. So, I am for Ingenico e-payments and that s all the electronic payments. So, no cards, not terminals, everything online. Which is a very different business I would say. We call ourselves a Payment Service Provider. I don t know if anybody joined the session earlier with the world pay? No? So that was our competitor, so nobody was there. Gebrian: It s good that nobody was there! So, what we do is, for our merchants, for our customers, we make online payments very easy. We provide you one single API to be able to do online payments in 150 countries, in 150 currencies. With paypal or with credit card or a local bank transfer and we take all these interfaces and group them together into one API. Which you can imagine, is very challenging. And we also need to be available 24/7. We do billions of payments every year, right? So, we cannot, we cannot have one single second of downtime actually. Gebrian: No, I think, we had a discussion rather. I think if this system goes down, I won t say the internet goes down, but you will be aware that some of the biggest sites in the world will not work. It is not a homegrown small application, let s call it that way, it s the real deal. The biggest sites in the world are using this system. I think that s why it makes it quite exciting. Vincent: It s quite exciting to be honest and if you saw the title of the session now, it says: from code commit to production within a day, right? Well, to give you a spoiler: we are not there yet. But that s where we are heading towards. And you can imagine that on a system, where we have payments, real-time going, on the fly, it s a very delicate way to do. We come from a time where we will do 5 or 6 releases a year. We will show you where we are right now and where we are heading towards. So I won t bore you anymore with all the numbers, you can look it up on the website. So, I ll skip this marketing slide. But, what we want to tell today, is actually our journey. And our journey started as a start-up. In the mid-nineties, e-payments was, well, not a big business yet. It was a niche. The company started to develop some solutions and as the market for online payments was growing, we

3 were growing as well. You could say maybe we were a unicorn (*3.40) back then, but everything was going very smooth. Over time, we had more competitors entering the market, we started to, I would say, make our processes more complicated and it wasn t a start-up anymore. We had all kinds of different roles, all kinds of controls. And what we noticed is that, I would say around 2013, we saw Hey, our business, it s stabilizing; maybe even declining a bit. And that while the market is growing. That s not good, we need to change something, we need to do something. So that s when, we actually started to scratch our heads and see: What do we need to do next? So, we used this as a hook throughout the day. We covered the past a bit, we skipped the start-up part and what we decided back then is to, on an organizational level, architectural level and on delivery pipe-lines, we need to do things differently. We ll get into that later. Last year, so we didn t have a lot of people here that saw the presentation. So maybe if you like the talk today, look it up. Because we show you some of the techniques we used on making some technical improvements and changing organizations. We can already see that it s paying back. So, what we are going to do is: show you, where are we now and what will be. What s the outlook, where we going for, right? So, I think this picture shows a bit the journey we are going through. We are now at the point that things are getting much better. We have a really good momentum now to, I d say, to start to fly. I m really proud of all that. That s why we are here today, to share that, right? So, inspired by the best. Like last year, we have three major inspirations. Which on the one hand is Spotify. I don t know if people are familiar with the Spotify model? Let s see, some nodding. Yeah, so, Spotify is really organized around features and products and not around technologies. I ll show you how we imbed this in our organization. Architecture, microservices; last year we went quite in-depth on how we have been approaching this. To go from the (*5.50), to small micro services and this is also something which we made some great progress, but we are not there yet. Lots of things need to be done, but it s good enough already to actually enable feature teams and product teams. We ll show you in a bit. CI/CD, we come from a time where we did a couple of releases a year. Well, that s not where we are anymore and we have been investing a lot in automating the pipelines using all kinds of tools. From (6.20*) Dockereven, Jenkins, I don t know what. We have been trying everything and we made some good steps there. These are our great examples, microservices from Netflix, CI/CD from Facebook, the organization of Spotify. And what I can say is: Last time when we were presenting, is that, my team, we had some squads, they were actually ahead of these things. So, we were already having some feature teams in there. We automated the pipelines and that really showed the organization how it should be done, right? From just the teams that I was managing, that we were working with, but now we are actually implementing these things fully in Ingenico e-payments. Because we set an example, we showed the results and I m really proud of that, that we were able to do that together. I m gonna leave this to Gebrian. Ingenico is a French company, you see Napoleon sitting there, so I d rather not do this part. Gebrian: No else you would probably get fired so. I m from Accenture so, who gives a sh*t? So, this is where we came from and I am also not the creator of this slide, but I think overall the goal was that it was really hierarchy driven, right? It was really a hierarchy, top. So, this was a little bit where we come from, from the other items. So the organization, there were more people just looking at one developer that does some stuff but the rest had really no idea just talking and just drinking. Probably, we were also a little bit around this area.

4 Vincent: For sure. Gebrian: Architecture, it was really one big overall monolith. It was really a big monolith and it had a huge engine, but it couldn t drive. That was a little bit what was going on. And the delivery overall, as you can see, was like a train. But a train left, let s say a few times a year. And if you missed that train, well, a lot of stuff will happen. So, eventually, what is happening? Everyone wants to get on that release train right? Everyone wants to get to production. Because it s a big monolith, releasing to less than 5 years today. It took really months for a code commit to go to production and I think sometimes even maybe years. We had a high defect ratio of P1 incidents. And already we started to talk with hey, this system cannot be allowed to go down. We still had a lot of P1 incidents. So overall, if you then look at it, if we went to go with this one big release train to production. Even the CTO of the company was involved. So there s really the whole company would look into how we would bring something to production. So there was a lot of pressure etc. So, this is a little bit, the world we come from. Vincent: I m sure this is quite familiar to many of you, right? Gebrian: You probably, hopefully hopefully not. Vincent: But since you re here today, I guess it is. Gebrian: Yeah so for now, I see you missed one. Vincent: We are one version behind, so let s skip this one. Gebrian: Yeah, you can do the Spotify. Vincent: So, last time what we talked about is that we had some great successes with some feature teams. So, a feature team is really organized around a specific product and not specific technology. Take, for example, Credit Card payments. So we have, what we did, an example off of Spotify. Wehave been identifying tribes. And a tribe is a group of teams. And a tribe as a goal, as a purpose. So for example, for a (*10.04) credit card, is it to give our merchants the best credit card payment experience that they would ever have right? What we do is, we give the teams a very specific product related purpose so they can actually get better and better on the topic. Also what we do is, we get rid of the old way of organizing projects. You remember like, with the waterfall, we had the design stages, we needed to get budgets, we got rid of that. So, what we do is, we actually pre-finance for the entire year, the tribes and squads. We have a budget for development and we just say: oke, for acquiring we are going to spend x amount this year, to have x teams, that are going to do this. And then it s up to the business together with R&D to start prioritizing. But in that capacity. So, we have capacity as a given, we are not going to fight about budgets, this is the capacity we have. Next to we need to do is: set priorities. But in that specific product or feature that we want to work on. So that s what we have in the tribes. Gebrian: I think overall what makes you strong (*11.03), I try always to repeat this constantly, it s really about the products right. It is not anymore about, we have a back-end system, we have a middle-ware etc. We have a font-end system. It s really about the product itself. Vincent: So, for example, I used to be responsible for the java development in e-payments. Right now, I am not. I m responsible for two tribes as a servant leader. I am not telling them what to do, I am just looking after them they have all the means to work, that they can actually deliver and it s a very different concept of how we work. And we have different technologies inside the tribes to make them as autonomous as possible. Because we don t want them to go to other teams to develop this

5 or to go to another team to develop that. Sure, we still have dependencies, but we try to cut them down as much as we can. And this is actually something recent, because we redid that the entire e- payments R&D organization. We made a new blueprint of what tribes we should have and what squads we should have and we actually let the people choose themselves what teams they wanted to join. So, we just put the organization on the wall and people said: I want to be in this team, I want to be in that team. And it was really powerful and as management a bit of risky thing, I felt like that at least. But it worked out really well and this is something we did just in the last two months and what we want is autonomy in the teams. I am not telling them what to do, together with product, sitting together with the developers and the ops engineers. They have their own product, they own it, they need to make sure they automate things and we get to production as fast as possible. They set the road maps and I am just there to make sure they get the stuff they need to get going. Something that Gene actually asked us in the review of the slides is to go a little more in-depth on the collaboration between us and Ingenico. Because when I joined the company, there was lots of fighting going on. We were really collaborating in a traditional way I would say. Where we would have most of the things outsourced, we had contracts, everything was pinned down very specifically. This needs to be done. We had fights about the things that were not done. This didn t work out; we were more wasting our time on actually fighting each other rather than adding value to the company. So, on both sides we had some efforts to get rid of that. What we have right now is, within these squads, there are Accenture people in there, it s tribes and squads. We can t say, this is Accenture, this is Ingenico, no. The people, we hold them responsible as a team. So, we shared a responsibility for the delivery and we focused on the behavior of the teams, which is very important. We have hybrid teams, so mixed vendors, inhouse people, all together responsible to do whatever the purpose of their team is. What the added value, in this case, of Accenture is, is that they actually inject knowledge in our teams. Some knowledge, maybe some dev-ops transformations we don t have. Well, the guys join the teams, are part of the team and they make sure the level of maturity goes up. And we can adapt and learn from them and apply it also in the other teams. It s all about trust, I would say, and autonomy. In the end, it results in more value for our merchants. I think the collaboration works really well. Gebrian: I think a few things amazing here is that, especially shared responsibility is nice. In the past, I was responsible for the part. And you just have, like you have in your own organization as well, there is stuff that you have trouble with etc. But Ingenico didn t always see that. Now we went to a shared responsibility both Accenture and Ingenico and so they are really fighting for the team itself. So, it s not about the vendor or whatever, it is really about the teams themselves. I think that as well, we have hybrid teams. We have a team, one team consists of one in the Netherlands, one in Pune, one member in Mumbai. So, one team consists all over the world and we have got it to work. So overall, but that s a little bit from an organizational part. If you look at the architecture, I am not going into this too deep. If you really want to see a little bit more or have a lot of discussion, because this is quite a big topic overall, just talk to me after this talk. Or look at the one from last year. What we do is, what we call the strangler pattern. We had this old big Lexie (*15.32) system and we slowly created a new, a little bit green field architecture next to it, and move items from the Lexie * system to this new state of the art system. From that we created, instead of just choosing only the strangler pattern, to kill the existing tree (like the strangler tree), we also create new services next to it. To be honest, without that, we were not able to switch our teams in the way we did, because this way we have a service particular for, for example, paypal payments. So we can have a PayPal team as well, and that s where a lot of benefits come from.

6 Vincent: Yes, to give them autonomy, you have to give them their own part of the application, right? So, the only way to do it is to cut it down in pieces. Gebrian: So overall, I just wanted to add also a little bit about our pipeline. Probably you can t see it but this is the whole pipeline. This goes from checkout to setting up a complete development environment, to integration environment, to creating a staging environment. But what we also do in a pipeline, is given automation on approvals. If you want to approve something to production, this is currently to be honest still in the works. I got an two weeks ago from one of our Devops engineers here and he said: this is the wet dream of every Devops engineer. Because it goes all the way to production. It s fully to production and be aware, we are a payment system. Officially we are also a bank. So that means there is a huge amount of regulation on this, so we tried to automate almost everything to also handle all the regulations; and you can see the results. We have more than full test cases. We have some environments in the cloud, we are releasing to production at least every week and I think we had 2 months ago; we did 18 times in one month. Vincent: Yeah there is a next slide, I ll show you. Gebrian: Oke, next slide. Decoupled releases and deployment. So instead, what we mean with that, is that instead of just putting something to production and releasing it to the whole wide world, right. You deploy it and you release it to the whole world, we are not doing that. We are putting something on production, but it has almost the same functionality as is. So we are putting something on production and then later on, like with feature toggles, we set new functionality to production. That makes it a lot easier to go to production. So, 99% uptime, but I think this is really where it s all about. Vincent: I m really proud about this. So, this is for one of the teams, not all the teams. But this is the team where we started with the discussions last year. You can see we had some struggles in 2015, we didn t get a lot of changes to production and you can see every quarter it goes up and up. The speed is unbelievable right now. It actually results in bottlenecks elsewhere in the organization because we need to be able to think quickly, prioritize: what do we actually want to do?. How do we keep our merchants up to date about new features? How are we going to do those things? So, I am really proud of it, I looked at the numbers of Q2, I didn t put them in yet, but we going to beat Q1 again. It s going to go even more up than this and this is why we do it right? This allows us to give features to our merchants and beat the competition basically. Gebrian: And just to give you a little bit of the numbers. This is not, for me it s just 10% or 20% increase of what we bring to production. This is 40 times more elements that we bring to production. Is it defect fixes, is it new functionality, it could be even bigger projects that we bring eventually to production. I think this is the core, right? We saw, I am not sure if you were in the talk before as well, where you see that everywhere right? Big numbers? This is a company, this is a system that is so important, has so much impact and we are able to release an increase of 40 times. That means that, and to be honest, if we can do it you can do it as well, because we said there was fighting, I don t want to say physical fighting, but a huge amount of challenges within the organization. This, eventually, we overcame. A lot of sweat, a lot of tears as well probably. Vincent: I guess you need a good fight sometimes. Gebrian: Yeah, I need sometimes a good fight. But I think, and that s why we are here, and I think you heard my story also talking a lot about we. Well, I m still a vendor but I feel this is really something that we did together. And I think that s really what we should.. (*20.23).

7 Vincent: I think we have only got 3 minutes left, so let s do this quick. So what s going to be next? What are we going to do? So, 3 topics back. Organization, so we need to stabilize what we created right now. Because people still need to get mature, we need to improve there. So, we do a lot of coaching in the teams. Embed the culture, it s not just changing the organization structure but it s also a mindset. We really invest in that. From architecture point of view, we have microservices, we split things up, but we are not there yet. There is much more to divide, to really give autonomy to the teams, to develop their own bits and pieces without being impacted by the other teams. Delivery, 1 day from code commit to production. Well, we re now about a week, it used to be months. Maybe next year we are able to reach this, I am pretty confident about that. So that s really where we are going. Gebrian: And I think overall if you look from an architect perspective, I like the Lego blocks as well, because there is standardization, right? There is exactly the size of one of those elements and that s where we want to go within architecture as well. Plug and play. I think from an organization level as well, there is a lot that we change over the last year, people need to get used to it. Especially the culture is not something that you will change overnight. And I think the delivery, we have weeks that we have pushed 3 or 4 times to production. But we want to get into a more regular way. We have the big release that goes at least every week out and we call that a big release. In the past that was, an extremely small release. And we do smaller ones per day. We want to get into an even faster mode as well. I think overall that is it. Vincent: Yeah, I guess that was the last slide. So, I don t think we have any time for questions actually. So, if you want to reach out, or want to know something, just ping us in the app or whatever.

agilecxo.org Agile Leadership Podcast #4

agilecxo.org Agile Leadership Podcast #4 Agile Leadership Podcast #4 This is Joe Kirk. I m the CIO for the Tennessee Department of Transportation. Welcome to the Agile CXO, Agile Leadership Podcast. I m your host, Jeff Dalton. This month, we