Scaling a growing organization by reorganizing the teams

by | Syndicated

During this year’s QCon conference held in New York, I attended a full-day workshop on the scalability challenges a growing organization faces, hosted by Randy Shoup. In my previous two posts I discussed a model to understand the needs of an organization in its different life phases, as well as a migration strategy for getting from a monolith to a set of well-defined microservices.

The Universal Scalability Law…again

However, Randy also talked about people, or more specifically, how to reorganize the teams for scalability without ignoring the Universal Scalability Law. What this means is that you should be looking for a way to have lots of developers in your organization working on things in isolation (thereby reducing contention) without the need for a lot of communication (a.k.a. coherence). So any form of team structuring that involves a lot of coordination between teams is obviously out of the question, particularly skill-based teams, project-based teams or large teams.

For the same reason, Randy advises against geographically split teams or outsourcing to so-called job shops. Those do not only involve a lot of coordination, but local conversations will become disruptive in melding a team. Just like Randy, I find face-to-face discussions crucial for effective teams. But if your team is not co-located, those local conversations will never reach the rest of the team. Yes, you may persist on posting a summary of that discussion on some kind of team wiki, Flowdock/Slack or other team collaboration tool, but they will still miss the discussions that let to that summary. Even using a permanent video conferencing set-up doesn’t always solve that, particularly if the people in the team don’t share the same native language (which is already a problem for co-located teams).

The ideal team

He also said something about the effect of getting more people in the organization. In his view, 5 people is the ideal. That number of people can sit around a table, will benefit from high bandwidth communication and roles can be fluid. When you reach about 20 people, you require structure, which on turn, can be a potential trough of productivity and motivation. When you reach 100 people, you must shift your attention from coordinating individuals to coordinating teams. A clear team structure and well-defined responsibilities becomes critical. Knowing this, it’s kind of expected that Randy likes to size his teams using the “2 pizza rule”, the number of people you can feed from 2 pizza’s. So a team consisting of 4-6 people in a mix of junior and senior and (obviously) co-located has his preference.

Ideally he wants to have that team take ownership of a component or service, including maintenance and support as well as the roadmap for that component or service. This implies that all teams are full-stack from a technological perspective and are capable of supporting their component or service all the way into production. But Randy emphasizes that managers shouldn’t see teams like this as software factories. Teams should have an identity and be able to build up proud-of- ownership. This also implies taking responsibility over the quality of those services. He purposely mentioned the problem of not having the time to do their work right and taking shortcuts because of (perceived) pressure from management or other stakeholders. In his opinion, this is the wrong thing to do, since it means you’ll need to do the work twice. The more constrained the team is in time, the more important it is to do it the right way first.

The effect of team structure on architecture

Another argument for his ideas is provided by Conway’s Law. Melvin Conway observed that in a lot of organizations the structure of the software system closely followed the structure of the organization. This isn’t a big surprise, since quite often, cross-team collaboration requires some kind of agreed way of working, both on the communication level as well as on the technical level. Quite often, architectural seams like API contracts or modules emerge from this. So based on that observation, he advices organizations to structure the teams along the boundaries you want to accomplish in your software architecture. And this is how Conway’s Law is usually used. But in this workshop, Randy has already steered us towards the notion of using microservices for scaling the organization. So does Conway’s Law apply here? Each team owns one or more microservices, or, the API contracts I just discussed. They work in isolation, but negotiate about the features provided by their services. I would say that is a resounding yes!

All things considered, it should not come as a surprise that he believes microservices are the perfect architecture for scaling an organization on both the people and the technical level. So what do you think? Are microservices the right technology to scale teams? Or is this the world up-side down? Let me know by commenting below. Oh, and follow me at @ddoomen to get regular updates on my everlasting quest for better solutions

זמן ומוטיבציה Time and Motivation

by | Syndicated

דמיינו את הסיטואציה הבאה: אתם נמצאים ללא אינטרנט, אבל מול עמדת מחשב.למשך לילה שלם. בערך פעם בשעה מצלצל הטלפון הקווי ואתם צריכים לעדכן טבלת אקסל ומשהו בתוכנה ייעודית כלשהי. בערך חמש דקות עבודה. מה תעשו עם שאר הזמן? אפשר להביא ספר, אבל למי יש כוח לקרוא באמצע הלילה?
אי אז מזמן, בטרם היות הטלפונים החכמים, הייתה רק תשובה אחת – משחקים סוליטייר, או שולה מוקשים.
לא מזמן, הודות לצה”ל, מצאתי את עצמי בסיטואציה כזו – מול עמדת מחשב מנותקת מכל דבר שמזכיר אינטרנט (בסך הכל – מערכות צבאיות הן לא משהו שאנחנו רוצים שידלוף החוצה), בלי פלאפון ומחכה שמשהו מעניין יקרה. כדי להוסיף חטא על פשע, המחשב הוגדר כך שהמשתמש שניתן לי היה בעל הרשאות מוגבלות למדי – כל מה שאפשר לפתוח הוא התוכנה הצבאית ומגוון תוכנות אופיס. זהו.
כדי להילחם בשיעמום נקטתי בגישה בה נתקלתי אי אז אצל ג’יימס באך – “אני יודע מה זה, המחשב הזה הוא תחנת אימונים לבודקי תוכנה!”. פתאום יש לי מה לעשות. האתגר שבחרתי לעצמי – האם ניתן לעקוף את המגבלות שהוטלו על המשתמש שלי ולעשות דברים מעניינים יותר?
התשובה, בקיצור, היא “כן” (אחרת לא היה לי מה לכתוב). בערך אחרי שעה מצאתי את פריצת הדרך הראשונה שהובילה לכך שהיה לי ממשק של powershell פתוח על המחשב, עם גישה לכל מיני קבצים שאין אליהם גישה דרך הממשק הגרפי. לכאורה – ניצחתי, לא? הוכחתי שניתן לעקוף את מערכת ההרשאות עד מידה מסויימת. אבל כמו שהזכרתי קודם – היה לי משעמם, ורציתי לעשות משהו. ביליתי קצת זמן בלנסות להבין את מימדי הנזק שאני יכול לגרום (כמובן, בלי לגרום נזק של ממש – זו לא הייתה המטרה שלי), ואחרי ששכנעתי את עצמי שאני יכול לנסות להפיל את העמד&#
1492; שלי ע”י מילוי הכונן הקשיח, או לנסות להציף את הרשת בבקשות PING ברשת הפנימית הצבתי לעצמי אתגר מעניין יותר – האם אני יכול להפעיל סוליטייר על המחשב? אני באמת לא בטוח מתי בדיוק הידע הזה נכנס לי לראש, אבל זכרתי שההתקנה הסטנדרטית של סוליטייר נמצאת תחת system32 (בחלונות XP) ולכן גיליתי די מהר שהמשחק לא מותקן.
חבל מאוד. כאן הגעתי לגבול הידע שלי, והמשכתי להעביר את הזמן במשמרת
אחרי המשמרת שלי, ניצלתי את האינטרנט בפלאפון כדי לשאול “איך להתקין סוליטייר משורת הפקודה?” מסתבר שזה ממש פשוט. במשמרת הבאה כבר שיחקתי בסוליטייר.
בסיכומו של עניין, בתוך מספר מועט של שעות (פחות משלוש שעות עבודה נטו, כולל חיפושים באינטרנט בפלאפון שנעשו הרחק מהמחשב) הצלחתי להגיע לקבצים שלא הייתי מורשה לראות, ידעתי לומר שההגנה על הregistry נאכפת היטב, מצאתי דרך להריץ קוד על המחשב ודרך נוחה למדי לכתוב אותו, והצלחתי “להתקין” משחקים (שכבר היו מכווצים על המחשב – לא חיברתי שום דבר שהגיע מחוץ לרשת המאובטחת). עשיתי את כל זה למרות שבתחילת הדרך ביליתי כשעה בניסיונות סרק שלא הובילו אותי לשום מקום. למה המשכתי? כי היה לי זמן לשרוף, וכי שיעמום הוא מניע חזק – אני רציתי סוליטייר, ועד שלא הגעתי למטרה שלי, לא עצרתי. מסתבר שעקשנות, התמדה או איך שלא תרצו לקרוא לזה, היא כלי חזק להחריד.
עכשיו נשארו רק שני דברים לעשות – למצוא מישהו שאני יכול לדווח לו על חולשות האבטחה במערכת, ולהבין איך אני מתרגם את דפוס ההתנהגות הזה לעבודה היומיומית.

——————————-

Imagine the following situation: You are sitting in front of a computer, with no access to internet at all, for a whole night. Once an hour, or so, the phone rings, and you have to update something on an excel sheet and another on a custom management system. All in all, a phone calls takes 5 minutes to handle. 10 if you try to stretch it. What would you do with the rest of the time you have? Perhaps you would bring a book, but frankly – reading at the middle of night can be taxing. There was a time, before smartphones, where the most common answer to such a situation was “Solitaire” (or minesweeper).
Not long ago, at the courtesy of the IDF, I found myself in such a situation during reserve duty. Naturally, I won’t go into details, but I think that without breaking any rules or exposing classified information, I can safely say that I found myself in a similar situ
ation. A lot of dead time in front of a computer with an occasional interruption every now and then. No internet, phones are not allowed (since they can be used for remote eavesdropping) and nothing really interesting to do. Worse: I was using the computer with an underprivileged account that enabled me access only to what was necessary to do the work I was called to do – A dedicated piece of software and a couple of office programs. Bummer.
So, to fight boredom off, I took an approach I saw at one of James Bach’s videos – “I know what this is. This is a tester workout station”.Suddenly, I had something to do. The challenge I chose for myself (or my charter, if you would prefer it), was to try and bypass the extreme limitation imposed on my user. Could I do something more interesting? The answer, by the way, is yes. I could (otherwise – nothing to write about, write?) It took me about an hour to get my first breakthrough that gave me a powershell interface.  Regular DOS was blocked for my user, so opening powershell was definitly something I was not supposed to do. In addition, Through powershell, I could access files that I was not able to access through the GUI.
As it seemed – victory was mine, right? opening a shell is usually refereed to as the hacker’s end-game. But, as I have mentioned, I was bored, and thus I started looking around to see how extensive can I damage the system (with the obvious limitation of not actually wanting to do any damage, or any noise that might be noticed from the outside). I have convinced myself that I could, in theory, flood the internal network in ping requests, learned a bit about the underlying technologies used in the software installed on this computer and then went looking for solitaire under system32. Sadly, it wasn’t there, and I concluded this shift.
During my off time, away from any sensitive data, I used my phone to google out some instructions and found out that I could actually install (or rather, extract) solitaire manually without doing something I know was monitored such as plugging in a USB drive (besides, if you ever work on a sensitive system, don’t insert any media that was in contact with the outside world, you might end up with stuxnet). Thus, 10 minutes after my next shift, I got to play solitaire & minesweeper.

What really got my attention, except for the fact that in less than 3 hours of work with very limited resources, I knew quite a bit more about a system than I was supposed to (for instance, I know that the registry was write protected properly, but I could read the entire tree, which could have been helpful to an attacker familiar with the registry), was that I managed to do so despite my first 30 minutes were terribly frustrating – everything I tried was either blocked or futile. In a normal situation, I would have probably gone to check other things and say “this is safe enough”. This time, as I had both plenty of time and motivation (boredom is almost a strong motivator as laziness), I just went on and on until something gave in. Persistence seems to be quite a strong tool.
Now, i have to do two things – find someone to report this bug to, and figure out how to translate this work pattern to my daily job. 

The Promise and Failure of Record and Playback

by | Syndicated

I came across the below video of Bret Pettichord’s keynote presentation to the Selenium Conference in 2011 called “Science and Stories and Test Automation”. Much of the talk covers his experience with Test Automation, specifically the promise and failure of record and playback over the last 20 years (I think). Just this historical perspective makes […]