LOEBlogger Carly James was “thrown into the deep end” to lead an impact evaluation of World Vision Somalia’s long term shelter work in the state of Puntland. Her work is on display at the Design for Recovery: Fieldwork in Nepal & Somalia presentation, reception, and exhibit on April 4 at 6:30-8:00 pm in CGIS South S050.
We went around to look at how they were moving on. And we found this lady. Her house had become a shop. I asked her, “why would you give up your house?” ”I no longer have any relatives. I came from the rural area. Here, unless I beg, I cannot buy milk or food. When I rent, I am able to get $50. With that money, I can buy my clothes; I can buy my milk; I can buy my food. I am not a beggar.” “‘But is it warm enough for you?" we asked. And she took me inside her house. She had extra blankets, reinforced her walls. She told me, “I am secure and I am warm.”
What does it mean to have a permanent shelter? What does it meant to feel protected in your community? What does it mean to be truly “at home” in a location you did not choose? These questions sit at the core of the Puntland Shelter Project conducted by World Vision Somalia from 2010 through 2013. Chronic drought and ongoing conflict have displaced millions for decades in Somalia; at the time the World Vision International project began, Puntland state was defined by temporary habitation. WVI was one of the only organizations looking at this crisis as an opportunity to innovate solutions for relief.
As of July 2010, the number of internally displaced people–or IDPs–in Somalia was estimated to be 1.4 million, with 92 percent of the displacement linked to conflict. The central state of Puntland hosted approximately 125 thousand of these IDPs. Chronic drought compounded with a resulting famine in August 2011 led to a population swell, as northern pastoralists who lost their livelihoods and animals migrated south, and those who fled violence in South-Central Somalia migrated north.
After a global emergency was declared in the region, World Vision Somalia conducted a number of assessments revealing the complexity of links between conflict-related displacement, drought-related displacement, and the long term displaced population. WVS undertook a phased approach which combined immediate provision of non-food items for those in formal and informal camps, and a longer term transitional shelter program that dealt with the protracted needs of the most vulnerable people. Quickly, however, there was recognition that the conditions within the IDP camps, as well as their precarious location (e.g. along riverbanks), exacerbated a wide range of urgent sanitation, protection, and livelihood issues. The WVS team switched to a multi-sector approach that eventually led to a broader “community build” strategy.
Though temporary shelter (known as “CGI” due to the iconic corrugated iron roofing) was the recommended design by the United Nations High Commission on Refugees, promises by Puntland local authorities for permanent land rights persuaded the WV Shelter Team, led by Loeb Fellow Brett Moore, that a permanent shelter solution could be explored.
Three years after the handover ceremony, I was invited to help WVS conduct an impact evaluation of its work in Puntland. The original plan was for WVS to hire an evaluation consultant, and for me to assist that person with research design and data analysis. A week after I arrived, I was told that WVS was unable to hire a consultant for this work and that it wanted to move forward with me leading the entire evaluation. It was a large undertaking, but the learning-by-doing process that I have been involved in since this began in January has been incredibly meaningful.
After the research tools–household questionnaire, focus group discussion questions, key information interview guides, instructions for activity with youth–were developed, we completed the research design with scheduling and enumerator (data collector) training. Owing to the travel restrictions on Somalia, data collection was not something that I could participate in directly, but I had regular Skype check-ins with the local field team.
Impact evaluations are summative, retrospective reviews. In a typical international development context, this would mean measuring an organization’s theory of change–which activities they conduct, which outcomes and outputs they expect from those activities, and ultimately what impact they are trying to make through that work–against the actual outcomes and impacts of the project. Because evaluation reports are often used to secure further funding, they are taken seriously and held to a high standard of rigor by the organizations that commission them. This generally translates to heavy econometrics and strategically selected indicators. For a project like the Puntland Shelter Project, there was an iterative and creative process as the team’s goals rapidly adapted to local needs. Not every project works this way, which is what made this one particularly difficult to evaluate. Moreover, quantitative data generally anchor the evaluation and structure the narrative, while qualitative data fill in gaps of understanding and round out the narrative. In this evaluation most of the richness of our data came from the qualitative data–the anecdotal evidence, the direct quotes, the unprompted stories, the human element.
This project involved securing permanent landholdings for the former IDPs, which was rather unusual for Somalia. The notion of land ownership and rights in Puntland have long been defined by extralegal power structures and clan-based networks, intertwined with fluid connections of space, home, and place that are historically rooted in nomadic pastoralist livelihood practices. Within the Puntland context, the evaluation had to highlight the fact that there was no precedent for a permanent, sedentary ”community build” led by a humanitarian organization. This evaluation has shed light on the ways in which the holding of title deeds at the local scale and the legitimization of private land ownership at the regional government scale have changed the way people think about shelter.
WVI was most interested in understanding the relationship between permanent shelter and:
- Livelihood opportunities
- Child and maternal health
- Safety and protection
- Food security and nutrition
- Household resilience
- Social cohesion and social capital
- Identity and membership
Among many other findings, the evaluation revealed that the sense of ownership over shelter affects how people think about the future. Beneficiaries confirmed that when living in the IDP camps, they could only plan on relatively short timescales. A woman in Jilab Village said, “Yes, of course every human being has plans...” and explained that those plans were limited to feeding and clothing their family when she was in the camp. However, women in both permanent shelter sites reported that their aspirations had gotten bigger and they were able to plan for longer timescales: “Most of us we didn’t have good plans for the future, but now because we have a permanent shelter we think beyond just feeding our children but also how we could give them better health and education,” said a woman in Burtinle.
Regarding gender equity, female-headed households reported that having title deeds had helped remove the fear of eviction. Personal security, which had been almost nonexistent in the IDP camps, was another major benefit of living in sites where each shelter has lockable windows and doors.
The project took a participatory design approach to the shelter itself, holding co-design workshops and inviting beneficiaries to give feedback on demonstrations and prototypes. The beneficiaries were also invited to participate in the construction of their own homes and hire friends and family in order to activate the local labor market. Our evaluation showed that this approach did not have as much impact on the local labor market as we would have thought, though the spirit of the approach was appreciated by beneficiaries. WVI worked primarily through the government for this work, which has very little influence on daily life in Somalia. It’s possible that the idea of civic duty or voluntary participation may not have been communicated appropriately in this design process. The recommendation here is to analyze the approach’s successes and failures more closely to see which design decisions were truly aligned with local social and economic dynamics.
WVI was also able to introduce a healthcare center, a commnunity center (which doubled as a school for a period of time), communal latrines (which provide privacy and security, particularly to women), transportation infrastructure (linking the new shelter site with its host community and thereby increasing commuter traffic), and more.
Most people think of Somalia within a narrative of conflict, chronic malnutrition, and cyclical poverty. The evaluation revealed the degree to which WVS faced resistance when it came to pursuing a permanent shelter project in such a complex context, where most intervention comes in the form of strictly defined humanitarian and relief services. After conducting key informant interviews with former project managers and the WV field team in Somalia, one thing became clear: that there is a deep desire to begin promoting an entirely new narrative about Somalia. “It’s not what people think...women are the backbone of life here...and we are not fighting everyday,” said a Somali member of the project team. It’s time for a new story of Somalia.
We don’t make those fine divisions between emergency and development. When we’re doing emergency, we still have those aspirations of doing development projects. So even at 180 days, we’re already thinking about 1 year, 2 years down the road, etc.
Operations Manager, Puntland Project
So what does all this mean for the future? I am of the mind that an evaluation means little if the lessons are not operationalized. WVS is interested in a feedback loop for lessons learned in the future, as there doesn’t exist a formal way of applying them at the moment. Moreover, it is worth thinking through how the innovation that came about through this project can be applied in future WVS work. In emergency relief situations, it’s uncommon for humanitarian organizations to not only provide permanent shelter but also other soft and hard community infrastructure. One major criticism related to WVS’s early and relatively ambitious long term development goals, whereas most aid organizations would have confined themselves to cost-effective, temporary solutions. Conversely, WVS saw this as one of the strengths that allowed them to push through the goals they set for the project.
I’d like to probe this tension more in future research, asking questions about what it takes to have a “culture of innovation” by design, and what that means for building long term resilience. Brett Moore has written on the topic of building resilience and social capital in a stateless context based on his experience with WVI in Somalia, and he and I will continue working together on this topic. Frances Westley, writing for the Stanford Social Innovation Review, illuminates the basic theory of change:
Part of building resilience in complex systems is strengthening cultures of innovation. These are cultures that value diversity, because as any bricoleur knows, the more (and more different) the parts, the greater the possibility of new and radical combinations. But these cultures also need to encourage the kind of communication and engagement that allows disparate elements to meet and mingle, and that allows for experimentation and support rather than blame. Such cultures support social innovation, and social innovation in turn builds resilience.
After all, if we cannot promote an appetite for change, and build in a capacity to make change, then how can we expect to reach a more resilient future?