Technical Reports

Background and Justification:[Back to table of contents]

The Internet recently passed the four billion user mark [1]. Despite this massive accomplishment, growth is rapidly slowing as dense, urban markets become saturated with cellular broadband signals. As the GSM Association noted in 2016: “In most countries, even in Africa, mobile operators have already rolled out 2G and 3G network coverage as far as possible within the envelope of a commercially sustainable business model." [2] Similarly, LTE rollouts will slow down as operators shift their focus to metropolitan 5G. This slowing leaves literally over three billion people, primarily in rural and developing regions, without broadband Internet access. The reasons for this are myriad, touching on low population density and socioeconomic status, high install cost, and lack of existing infrastructure. Affordably providing broadband Internet to this long tail of rural communities worldwide is an open research challenge. One particularly promising solution is Community Networking. Community Networks, largely defined as networks built and operated by local actors in a community-centric and often cooperative fashion, mitigate many of the economic concerns of operating in rural areas. The high cost of rural installations is reduced with local “know-how”, skills, and infrastructure. The low density of subscribers is mitigated by strong community participation, often engaging with core “anchor tenants," such as local governments and schools, to ensure long-term sustainability. In this work, we focus specifically on Community Cellular Networks (CCNs). These networks are particularly well suited to rural and developing areas due to their wide-area coverage, centralized repair and failure structure, and support for low-end handsets. There exist numerous examples of successful CCNs in the world, most notably Rhizomatica [3] (2G GSM) in Mexico, CoCoMoNets [4] (2G GSM) in the Philippines, and Tucan3G (3G UMTS) in Peru [5]. Despite their advantages, the limitations of existing cellular technologies in these contexts are becoming apparent. First and foremost, both 2G and 3G rely on a variety of cellular primitives, including phone numbers and interconnection agreements that require interoperating with incumbent carriers [6]. Additionally, CCNs operate in licensed bands that are often inaccessible to small organizations. Lastly, they provide only limited connectivity over voice, SMS, and low-bandwidth circuit-switched IP. In this work we propose, implement, and deploy Community LTE: an LTE-based community networking solution. CoLTE is motivated by our belief that LTE is uniquely well suited to community networking for many reasons: it is wide-area, inexpensive, high-bandwidth, can use IP primitives that remove the need for telecom interconnect, and has recently developed a robust uptake of client devices even in remote areas [7]. LTE is also available in over forty different bands, a number of which are unlicensed or available to small operators. Despite these advantages, LTE is still fundamentally a telecom technology, designed for highly centralized operation wherein the cellular radios (eNodeBs) are managed by a set of specialized network functions kept in a single location under the operator's control. In cell networks these functions are commonly referred to as the “core," and in LTE as the Enhanced Packet Core (EPC). To resolve this and other constraints, CoLTE reimagines and optimizes the LTE core network towards rural operation in a number of ways. These optimizations include 1) an on-site EPC co-located with the radio access network (RAN) to reduce backhaul costs, 2) support for only over-the-top (OTT) telephony to remove the need for phone numbers and telecom interconnect, 3) all IP-based billing and local services (including support for zero-rating), and 4) leveraging LTE SIM-based auth primitives for network and service authentication, removing the need for passwords in local services. To evaluate CoLTE, we deployed our system in the rural community of Bokondini in remote Papua, Indonesia over a period of six months. This network is operated by a local NGO, sustainably provides broadband Internet access to a community with only existing 2G voice and SMS coverage (no GPRS), and currently connects over forty users to the Internet. We examine the system and show that our design decisions 1) reduced the network backhaul requirements, 2) scale gracefully as more users enter the system, 3) allow for communication through common services like WhatsApp, and 4) do so in an economically viable manner that recoups all operational and capital costs. We have released the entire CoLTE system as a fully open source project [8], and maintain .deb packages for Debian 9 and Ubuntu 18.04 LTS.

---

Project Implementation:[Back to table of contents]

RURAL NETWORK CONSTRAINTS Most infrastructure development is focused on dense, urban areas where power, backhaul, and support are readily available. In contrast, we focus our work on relatively small communities (1000 or fewer residents) in remote, hard to reach locations; hundreds of millions of people live in such communities around the world. In this section, we describe the constraints present in these contexts that inform the design of CoLTE. Limited Infrastructure: Many presently disconnected areas have constrained infrastructure, such as intermittent and/or dirty power [9] or scarce building supplies. These challenges impede conventional telecom rollouts, which deploy one-size-fits-all solutions designed for rapid scalability, because these solutions cannot leverage local context and flexibility. In contrast, community network operators are also community members and intuitively understand how to adapt within the local context to navigate issues such as transportation, power, and easements. Remote and rural areas also often lack highly-available, high-speed Internet backhaul infrastructure, and typically rely on satellite or long-distance microwave links. The characteristics of these links, particularly highly variant latency and downtime with weather, are problematic as more and more user-facing Internet and Web services are built for high-bandwidth, low-latency, and always-on contexts. As these services become more centralized and cloud-native, they often implicitly enforce these requirements, even when they are not strictly necessary, via protocol-level design decisions such as overly chatty protocols, short timeout values, and system-level failure in the face of client disconnection. Low Density & Budget: By definition, rural areas have lower population density than urban areas. At the same time, infrastructural costs are often higher, as equipment and labor must be brought from nearby urban centers. Since economically sustainable rural networks must amortize higher costs across fewer and often less wealthy users, it follows that inexpensive deployment and operation are critical requirements. Increasing the coverage area of the equipment is a natural goal, as is designing systems that are durable, repairable, and low cost. These financial realities limit the feasibility of custom solutions and one-off protocols, since rural-only solutions lose the economies of scale and technical ecosystem that exist with widely-deployed urban standards. Scale Mismatches: When local organizations take up the mantle of connectivity and implement their own solutions, these solutions will inherently be small-scale and local. This creates problems when using technologies and protocols designed for global-scale communications. In legacy 2G and 3G community cellular networks, a remarkable amount of system complexity was introduced specifically by the need to support interconnect with existing telephony networks. The earliest instances of these networks [42, 44] did not interconnect with existing phone networks at all, and provided only in-network communication. Later instances of these networks provided interconnect via a wide range of designs, including (1) purchasing Swedish numbers via Twilio [43], (2) building a custom system that assigns every subscriber an extension to a single public phone number [3],[1] or (3) building a large, cloud-based system in partnership with a national scale telecom [6]. Unfortunately, each of these solutions comes paired with significant drawbacks: purchasing Twilio numbers immediately became the dominant operating expense of the network, number sharing with extensions does not allow SMS messages into the community, and partnering with an existing telecom company required a tremendous and complex engineering and organizational effort. Local Customization: In a less technical sense, community networks are owned and operated locally, and have the need to be customized to meet local development, sustainability, or social goals [45, 46, 19, 20]. Traditional centralized telecom architectures prohibit this customization as most services and configuration are placed at the core. Innovations such as fog computing [47] bring compute closer to the user, but not necessarily within their administrative control, and still disallow development and deployment of local network services. DESIGN Our design seeks to enable community cellular networks while addressing the constraints of remote contexts through a novel integration of cellular (3GPP) and IP networking technologies, a non-traditional allocation of physical network resources, and the removal of unnecessary complexity to ease network deployment and customization. We accomplish this by embracing three core ideas: “dumb pipe" [48] forwarding, edge compute, and LTE authentication. Embracing the “Dumb Pipe” In contrast to that of telecom, the relationship and interconnect between two IP networks is orders of magnitude simpler [49, 50]. Basic techniques such as NAT and hole-punching enable address-space scalability, and packet-switched forwarding [51] supports high network utilization and resource scalability. Together, these characteristics enable easy internetworking, which in turn has driven global Internet adoption and scalability. This motivates our system design choice to provide only IP based connectivity and remove any support for telecom interconnect or mobility. OTT-Only Networking By supporting only IP traffic, CoLTE does not interconnect with legacy telecom services like voice and SMS, and bills all traffic equally: as IP traffic that is drawn down from a user's data balance (in the case of usage-based plans) or simply logged (in the case of rate-based plans). In billing all traffic as bytes, CoLTE explicitly makes no distinction between in-network VoIP (i.e. VoLTE), over-the-top VoIP (e.g. WhatsApp), or general Internet traffic. Our decision to not support voice or SMS stems primarily from feedback that these services are a low priority for our users. In Indonesia, as in many other countries (most famously India), the organizational barriers and confusing billing practices associated with voice interconnect between competing telecom companies have driven explosive adoption of WhatsApp. In contrast to the service offered by the national telecom companies, WhatsApp offers users a single and consistent identity, universal reachability over all IP-based networks, and clearly understood billing practices.[2] When faced with the daunting proposition of negotiating expensive interconnection agreements with Indonesian telecom companies, combined with the reality of near-universal WhatsApp adoption within our target communities, we simply chose to follow our users' lead. This also allows us to operate with low budget by avoiding costly numbering and interconnect fees. Relying on over-the-top services has the added practical benefit of insulating our local operating partner from the responsibility of securing, safeguarding, and sometimes reporting [52] user traffic. These services provide strong end-to-end encryption since they assume an untrusted public Internet substrate, a feature that is notably absent in traditional voice and SMS. Application Layer Mobility While LTE networks traditionally are engineered to support seamless link layer handover of flows within the RAN, CoLTE explicitly does not support mobility between eNodeBs (LTE radio base stations). While handover efficiency is important when networks are very dense or user mobility occurs at high speed, it also introduces a large amount of complexity into the network. More fundamentally, link layer mobility simply does not match our scale. Additionally, in our context, the practical difference between a link layer reconnection and a handover is minimal. While reconnection breaks existing link layer circuits and drops network-native phone calls, it has little impact on OTT-based telephony applications, which are already engineered for robustness against intermittent packet loss. Additionally, in practice these reconnections are rare, since there exist only a small number of eNodeBs in each community and user mobility typically occurs at walking speed. Embracing the Edge Given that backhaul is so limited in our target context, CoLTE moves everything to the edge, including the LTE “core network" itself. CoLTE co-locates the EPC within the community, very close to the eNodeBs and specifically downstream of any constrained backhaul links. This approach is important in the context of rural networks, providing us with the benefits of reliable LTE resource signaling, lower backhaul utilization, and local breakout and services. Local Core Network An important consequence of co-locating the EPC with the RAN is that the backhaul link is used only for Internet-bound traffic, since local network traffic (i.e. traffic between users or between a user and the EPC) and LTE signaling overhead is kept within the community. Given that our target deployment contexts are backhaul-constrained, it is paramount to use this backhaul link as optimally as possible to mitigate the limited infrastructure. In DEPLOYMENT we evaluate the network bandwidth savings of co-locating the EPC with the RAN, and find that these savings are particularly impactful in low-bandwidth scenarios, reaching observed savings of up to 45%. Similarly, since LTE tunnels all user equipment (UE) generated network traffic through the core, in a traditional core network all local traffic ceases during any periods of backhaul dis-connectivity. By co-locating the EPC with the eNodeB, we minimize the consequences of backhaul interruption and preserve local connectivity regardless of backhaul availability. Local Breakout and Services A co-located EPC also gives us a powerful mechanism for supporting local communities. By terminating the UE-EPC tunnel at the access point (i.e. “Local Breakout" [53]), we can route and “zero-rate" (i.e., free [54]) specific traffic flows that are of utility to the community. An obvious extension is to do this for local services that are provided by NGO partners or generated by local community actors, thereby providing local customization. Embracing LTE Security Finally, an interesting property of the LTE architecture is that it enforces cryptographic verification of end user devices in the network through its Subscriber Identity Module (SIM). Once authenticated, CoLTE can map each UE to a known IP, and ensure that only this UE generates traffic from its assigned IP. This enforcement prevents IP spoofing in the CoLTE network, and makes the IP address a reliable identity proxy for the user involved with each flow. Using IP addresses as user identities simplifies the implementation of CoLTE and allows reuse of the rich ecosystem of performance optimized IP networking tools, detailed in IMPLEMENTATION section. Basing network management on validated IP addresses also allows for easy customization and extension of the services offered in the network (again supporting local customization) with basic web programming skills. User and administrator interaction with the network occurs via a locally hosted web interface. Web technologies allow a flexible and visually pleasing UI when accessed both via smartphone or attached computer, and CoLTE's integration of LTE authentication allows the interface to strongly authenticate users by their access device without typing a username and password into the portal, a common complaint with the school's existing WiFi hotspot. Community LTE IMPLEMENTATION Community LTE is comprised of a set of modifications to, and companion software packages for, OpenAirInterface (OAI) [55]. OAI is an open source, all-in-one software EPC compatible with a wide range of commercial-off-the-shelf (COTS) handsets and eNodeBs. Our primary contributions include hardening OAI to be production ready and capable of handling active user traffic, extending OAI to meet our above design goals, and developing Haulage, our prepaid network administration system that enables usage-based or rate-based billing of IP traffic and network management. Extending The OpenAirInterface EPC The OpenAirInterface codebase was initially designed as a reference implementation for researchers and telecom engineers, and was not intended for production systems. As such, our work with OpenAirInterface involved stabilizing the codebase, stress-testing compatibility with different handsets and eNodeBs, building tools to automate and simplify system configuration and orchestration, and writing systems code to ensure that OAI recovers into a running state after a wide range of crashes and/or failures. Enforcing IP Address Assignment In LTE, all UE-generated network traffic (both control and data-plane) is transmitted to the EPC over the GPRS Tunneling Protocol (GTP). Community LTE uses kernel-level GTP encapsulation and de-encapsulation via a Linux virtual network interface. De-encapsulation occurs before any routing or filtering decisions are made, and CoLTE explicitly enforces IP address assignment at this point, to ensure that a malicious UE cannot spoof its address. Default Bearer Traffic As a part of GTP encapsulation, LTE uses the concept of bearers to provide QoS guarantees to specific traffic flows (e.g. VoLTE calls). CoLTE explicitly does not utilize this feature of LTE, and instead routes all traffic to/from the UE over the default bearer, which provides only best-effort IP delivery without any QoS. This decision to support only IP traffic via the default bearer channel ties-in with our decisions to not provide network-native support for VoLTE or handover, and to leverage end-to-end principles for simplicity. Policy-Blind Forwarding The EPC is traditionally responsible for enforcing a network's administrative and business policies, known as the Policy and Charging Control (PCC) System. However, in CoLTE we explicitly separate this logic and functionality into a separate codebase, Haulage. It follows that the EPC is responsible only for establishing and maintaining a radio link and routing/forwarding packets accordingly. Our decision to divide these functions into separate codebases was driven primarily by engineering considerations. First and foremost, separating Haulage from the EPC allowed rapid development and deployment based on IP primitives and standard Linux kernel interfaces (specifically iptables), whereas integration with the EPC would have required a much more complex engineering effort focused on the bearer establishment process. Second, this split enables us to distribute each package separately without introducing any operational co-dependencies: a CoLTE network without Haulage is simply an unmetered network, and Haulage's reliance on IP primitives means that it can meter non-LTE networks. Finally, this split reflects the standard engineering practice of separating mechanism from policy, and enables each system to evolve independently of the other. Haulage PCC Haulage handles the tracking and enforcement of the operator's business policies in the network. It is divided into three logical functions: a Traffic Detection Function (TDF), a Policy and Charging Rules Function (PCRF), and a Policy and Charging Enforcement Function (PCEF) Currently, all three components are deployed as a single userspace program written in Golang; this program runs on the same physical machine as the EPC and lies on the dataplane for all network traffic. Haulage enacts its policies via kernel networking hooks to minimize its impact on latency. This approach minimizes cost while giving the local network operator an environment they are familiar with administering, monitoring, and debugging. The TDF observes the data-plane of the network, aggregates the bytes used by each user, and reports it to the PCRF after a configurable amount of time or traffic. The TDF captures traffic with libpcap and the gopacket packet processing framework; this results in sub-optimal packet header duplication but is not a bottleneck in the current implementation. All resources are garbage collected as users leave the network, which allows the TDF to run continuously. Expiration of user update timers or a sufficient number of observed events triggers a callback into the Haulage PCRF. The PCRF ensures that every user is still in compliance with operator policies and updates the user state in a failure tolerant SQL database for recovery after power outages. Policies are encoded as functions which take current user state as input and output a new desired state, and can contain arbitrary logic like zero-rated services or special promotions. In our current deployment, user state consists of the current data balance, lifetime data used, and whether Internet access is allowed. The policy function generates warnings as the user's balance approaches zero, and updates the user policy with the PCEF to shut off access at zero. The PCRF also accepts updates from the administration and management interfaces to trigger re-enabling access when additional balance has been purchased. The PCEF enacts policy by installing rules in the appropriate network forwarding appliances (in this deployment, Linux net filter rules via iptables). Since policy enforcement occurs asynchronously to detection and determination, there exists a brief window when a user may have an out of date policy. With the settings in our current deployment, this window is on the order of 10s, and is acceptable to the network operator. Timeout windows can be decreased at the expense of additional calls to the PCRF and higher overhead. Our main motivation for splitting Haulage into different logical components is to support larger-scale, higher-speed, and mixed-access networks by integrating with existing ISP approaches. For example, many commercial routers and switches support traffic reporting over RADIUS, and could replace the TDF provided by Haulage. Similarly, the PCEF could be extended to enact its policies at SDN-enabled switches via OpenFlow. IP Address Assignment Interface As described above, the EPC binds IP addresses to UEs and enforces this binding. Subsequently, Haulage uses this IP address to associate packets with a user for purposes of metering and billing. It follows that IP address assignment represents a remarkably nuanced interface in our system, because it is the only point at which the EPC directly interfaces with Haulage. The need for an interface for IP address assignment between the EPC and Haulage stems from our decision to fully separate the network control and data planes (powered by the EPC) from network management and administration tools (Haulage). This design is unique in the telecom space, which typically enacts all of these functions at the EPC. This design also mimics standard ISP architecture, where network management is split from the protocols responsible for IP assignment (e.g. DHCP or PPPoE). Initially, this “interface" was simply a shared MySQL database that mapped SIM cards to IP addresses. Understanding the brittleness of this approach, we moved towards a design wherein the EPC interfaced with Haulage using the RADIUS protocol, which is a standard interface for interconnection between ISP management tools and supports push- and pull-based IP assignment. As more and more ISP management software migrates to a REST API approach, we are planning to provide REST support and integration with a wider range of management software. IP-Based Service Authentication Because IP addresses are assigned to the UE by the EPC and verified during GTP de-encapsulation, most LTE networks explicitly do not support host-driven protocols for address assignment such as ARP or DHCP. This prevents UEs from learning the IP addresses of other nodes on the network or spoofing their IP address. Additionally, the wireless channel between the UE and the eNodeB is individually encrypted for each user; this prevents channel eavesdropping attacks (e.g. [56]) that are prevalent in WiFi. CoLTE explicitly assigns UEs the same static IP address when they leave or rejoin the network. When combined with the above two characteristics of LTE, this implies that CoLTE can reliably and securely bind an IP address to a specific SIM. This binding enables us to use IP addresses as user identity primitives, and therefore gives us fine-grained per-user traffic control via standard IP-based tools such as iptables or OpenFlow. Taking this a step further, we also leverage the security of the user-IP binding to provide a novel form of IP-based user identification for our locally hosted services. Rather than relying on usernames and passwords, or building a complex voice- or SMS-based system that relies on telephony primitives, users accessing local web services are automatically identified via their authenticated IP address. Users then use a simple website hosted on the EPC to perform basic account management operations, such as purchasing data packages or transferring money from one subscriber to another. DEPLOYMENT In mid-2018, work began in the remote Indonesian village of Bokondini to deploy the first CoLTE-based rural access network. Deploying our code in a live production setting provided us with invaluable experience, a deep understanding of both the logistical and technical constraints of the area, and helped us situate our work in the community cultural context. Local Context Bokondini (pop. 1,500) is located in the Highlands region of Indonesian Papua, at an altitude of approximately 2,000 meters. Bokondini is connected to Wamena (pop. 30,000), the major city in the region, via a three hour truck ride over unpaved roads and one river crossing. Aside from locally-farmed produce (pineapples are a local cash crop), all supplies and infrastructure are transported to Bokondini from Wamena. Bokondini largely lacks robust infrastructure. Power in Bokondini is provided though diesel generators scattered throughout the community and by a micro-grid consisting of solar panels and a micro hydro-electric generator connected to a battery bank for the large local private school. Power from the hydro is unavailable from 9pm to 6am, varying slightly based on weather, time of year, and event schedules. The connectivity situation in Bokondini is very constrained. In the cellular space, Telkomsel (Indonesia's largest national-scale provider) provides somewhat-irregular[3] 2G service in the community via a single tower. Internet is not available for general consumption. Slow Internet (1 Mbps with a 10:1 contention ratio) is provided by a VSAT satellite connection but available only to teachers for instructional purposes. Topographically, Bokondini exists on a mountainside shelf above a river bend. To the southwest, a mountainside quickly rises for approximately 1,000 meters; the other directions are bounded by an equally sharp drop-off of approximately 200 meters down to a river. These features essentially define the boundaries of the community. Deployment Platform To deploy CoLTE in Bokondini, we installed CoLTE and associated services onto an inexpensive Zotac ZBox B Series MiniPC. The ZBox is a mini-PC that comes with a 1.6GHz, 4-Core Intel Celeron processor, 8GB RAM, and a 250GB hard drive. This platform was chosen for cost and ease of replacement, and was literally the cheapest headless PC we could find as of April 2018. The CoLTE EPC is connected to two 1-watt BaiCells 850MHz Nova-233 eNodeBs with a basic unmanaged gigabit ethernet switch. Each eNodeB radiates to a 120 degree cross polarized (2xMIMO) antenna with 15dBi of gain, with the eNodeBs and antennas configured into the two sectors. We use 850 MHz due to a combination of factors, including previously measured handset support [8], long range and coverage, measured spectrum availability, spectrum legality via an experimental license, and the existence of COTS eNodeBs in this band. We deployed CoLTE with zero-rating and LTE auth for each user's personalized landing page. This page allowed users to transfer credit and buy data packages. Our partner asked us to extend this with a localized zero-rated educational media sharing service, which we added as a link from this home page. Geographical Coverage We installed our eNodeBs on a twenty-foot pole mounted on top of the local primary schoolhouse. This location was chosen primarily for logistical considerations, given that the same building also houses the school's electrical infrastructure that we used. This stems from the school's centrality in the community, both geographically and socially, and helped to build community interest in our work. The exact location of our deployment was sub-optimal from a radio perspective, given the relatively low height of the building and a large amount of surrounding foliage. However, in our field tests we found that the 850 MHz band gave us great penetration and coverage. Aside from a group of approximately five houses behind a hill in the northwest corner of the maps, our network covers the entire village with relatively strong (-90 dBm) signal, at points reaching distances of over a kilometer from the tower. Even more remarkably, the borders of coverage were dictated primarily by geography (i.e. the sloping uphill or downhill) rather than signal attenuation. While this observation is clearly anecdotal, it gives us good reason to believe these results will generalize, and implies that our model (which assumes a small set of co-located base stations) is sufficient to cover most small-scale rural Power Sharing Example: The standard telecom approach to rural installations with unreliable power is to provide reliable power for the tower via off-grid means, typically a diesel generator. This incurs additional installation cost (the generator itself) as well as remarkably high operational costs (purchasing diesel, transporting it to the tower, and guarding it against theft). This likely incurs additional operational and logistical challenges for the telecom company itself, which is far more specialized in supporting telecommunications infrastructure than power infrastructure. In contrast, we simply engineered our system for resilience in the face of unplanned outages, and then plugged into the community micro-grid. Similar to the above narrative, the availability of this solution to us depended heavily on our relationship with the community, and was only possible as the result of community discussion around power usage. Note that this same community discussion also makes our user-base more sympathetic to power outages: when our network fails due to power interruption, the community is not surprised, because the entire power infrastructure has failed.

---

---

[1] This solution can be considered a novel form of "phone number NAT" [2] Though WhatsApp is free, we refer here to the clear practices around IP network billing, which is either sold at a specific speed (rate-based) or amount of data (usage-based). [3] During our time in Bokondini, we heard several reports of (1) voice calls not succeeding and (2) SMS delivered hours after being sent, or dropped entirely.

Project Evaluation:[Back to table of contents]

PERFORMANCE EVALUATION The deployment of CoLTE in Bokondini provided us a unique opportunity to technically evaluate our system in an in situ deployment. In this section, we explore the efficacy of our designs through a variety of metrics including traffic measurements, protocol overhead calculations, and attachment message counts. Our data collection and analysis was conducted in compliance with University of Washington IRB review process. EPC Platform Network and system observations were taken on our production system in Bokondini at 15-minute intervals over the course of two days, with an average of 26.2 attached users over the duration of the evaluation. System Performance: The CPU and Memory metrics show that the EPC is barely being used. CPU load averaged over 15-minute intervals using top reveals that the system hovers around 2%, with peaks of up to 6% utilization during “high-use" hours (i.e. 12:00-14:00 and 17:00-19:00 local time). Similarly, memory utilization for the entire system, not just our specific software, stays constant at slightly under 0.7 Gigabytes. Throughput Measurements: The throughput tests above illustrate three separate points. First, loopback throughput reveals the maximum packet-processing rate the system can handle in software; this rate (14 Gbps) is higher than expected and exists well above other system bottlenecks (i.e. our 3Mbps backhaul link). Second, throughput across the Ethernet and USB-Ethernet interfaces reveals that total system throughput is defined by the physical limits of the forwarding interfaces (note that the native Ethernet port can forward at approximately 1 Gbps, whereas our USB-Ethernet adapter only supports 100 Mbps). Implications: This comparison highlights the most significant bottleneck of our deployment platform: it comes with only one native ethernet port. 100 Mbps is likely to remain well above our available backhaul for the foreseeable future, and this problem is easily resolvable by migrating to a miniPC platform with two gigabit Ethernet ports. However, this limit to our system invites an architectural discussion towards the future. In our current network design, the EPC is a gateway that routes/forwards all network traffic, but further separation of the control- and data-planes could enable less constrained designs. For example, both the EPC and Haulage's data-plane operations could be replaced by an SDN application that interfaces with a high-performance router, thereby completely removing CoLTE from the data plane. We note that fifteen-minute intervals do not capture the bursty nature of network processing and signaling, and throughput tests are not always representative of actual forwarding capacity. However, these metrics still substantiate our claim that EPC functionality can feasibly and affordably be enacted at the edge of the network, particularly when the edge is behind a constrained network link. This argument becomes even stronger when it is taken into account that we specifically optimized our platform for cost rather than computational power. Daily Network Usage Figure 6 provides a graph of total backhaul usage per day, and Figure 7 provides the same graph normalized by the number of users in the system. The means are calculated separately depending on the number of SIMs distributed, noting that they were released in bundles of ten into the community. At the highest level, these figures demonstrate robust usage of the network by the community. They also illustrate that the average usage per-user stays relatively constant, ranging between 104 and 124 megabytes per user per day, while the total bandwidth used per day increases correspondingly. This is surprising to us, as we expected the backhaul to be immediately saturated by even a small number of users. The prospect of over forty active users sharing a lowspeed high-contention Internet backhaul opens many usage questions, and requires a much deeper investigation well outside the scope of this paper. Daily RAN Attachments When a user hits a zero balance and is cut off from network access, the user can still attach to our RAN; this is because Haulage enforces access policies at the IP forwarding layer of the EPC. Even though our network is clearly backhaul constrained, this decision invites an evaluation of RAN utilization. Figure 8 provides a graph, collected over two days, of the hourly mean Connected (actively transmitting) and Attached (including idle) devices. This graph shows standard usage patterns, with zero devices when the network is off. However, this graph also shows us that (1) there exists a large difference between the mean number of attached (19.2) and connected (6.25) devices, and (2) both of these values are well below the total number of SIMs in the system (42). Each eNodeB supports 255 connected users and 150 Mbps of throughput, leading us to believe that our architecture can support many more users (and a backhaul improvement of up to 100x the current rate of 3Mbps) before we encounter resource contention in the fronthaul. LTE-Induced Overhead Table 3 examines the traffic overhead induced by LTE in our network, collected over two days. The top two lines provide raw collected metrics, and the bottom three lines provide a breakdown of the signaling between the EPC and eNB. Overhead is calculated against the UE-Internet traffic, which would simply be forwarded as-is in a non LTE network. These results are explained primarily by (1) our constrained network backhaul and (2) the fact that the LTE control plane scales with users, not tra_c. If, for example, our network served twice the upstream capacity (i.e. 12.2GB per day, resulting in 670MB of GTP overhead) while holding the number of network users (and control plane operations) constant, the total network overhead would drop to 3.09GB / 12.2GB = 25%. While these results are not necessarily applicable to generic LTE networks, they are pertinent to networks with limited infrastructure and support EPC colocation in backhaul-constrained environments. Individual Network Usage Figure 9 illustrates how much total bandwidth each user has consumed, broken into three “Phases" that correspond to the number of users in the network. The single outlier in the first phase (and two outliers in the second phase) are known credit re-sellers who also sell hotspot access off of their phone. We presume that the “shelf" in the second phase represents multiple users sharing a SIM for access. The gradual linear shaping of the curve over time leads us to conclude that as more SIMs enter the network, users (1) normalize their consumption and (2) stop purchasing hotspot access or otherwise sharing SIMs. Telephony Services We analyzed a week's worth of network traffic with the goal of quantifying the impact of OTT telecom services. We focused exclusively on WhatsApp due to local knowledge that WhatsApp is by far the most widely used OTT telephony service in the community. The results in Table 4 show that WhatsApp traffic consumes a relatively low amount of total network bandwidth, yet the large number of flows indicates that our OTT-only approach is actively used by the community. The fact that the normalized bytes are larger than the normalized number of connections also implies that our subscribers were making use of higher-bandwidth services such as voice and video calls rather than just text.

---

Gender Equality and Inclusion:[Back to table of contents]

The Community LTE project benefited from past knowledge and experience with local culture and gender issues.  Upon the advice of all interested groups, we adopted a strategy of implementing small groups for discussions, information gathering, training and feedback.  This strategy proved to be most effective and removed the troublesome issues of powerful local men dominating the public meetings. The small group approach was readily accepted, as traditional family talks are conducted in a similar manner. Communications provided to the whole community is inclusive by nature and anyone with access to a handset can easily use the system.  We found that the younger generation was quick to learn the features and applications necessary to use the LTE system.  These younger members of the community gained new respect and attention as they passed along their knowledge and helped train their elders.

Project Communication Strategy:[Back to table of contents]

Yanobama received permission from  Direktorat Jenderal Informasi dan Komunikasi Publik (KOMINFO, Indonesian Ministry of Communications) for research and experimental licensing. Yanobama greatfully thanks KOMINFO for allowing this work. Our research results will be submitted to KOMINFO. The University of Washington is in the process of publishing a public paper on Community LTE in Papua.  UW has released the entire CoLTE system as a fully open source project, and maintain .deb packages for Debian 9 and Ubuntu 18.04 LTS.  We are hopeful that the results from this project will open doors for other communities around the world.

Recommendations and Use of Findings:[Back to table of contents]

Field Observations Handset Heterogeneity Is A Long Tail: During the network rollout, our team encountered a wide range of diverse bugs. This can be attributed to the large number of optional header-fields and potentially different codepaths that exist in LTE, as well as widespread heterogeneity in handset models and manufacturers who sometimes implement optional features incorrectly. This heterogeneity continues to be the bottleneck and primary motivation for our team to restrict SIM card distribution as a hedge and triage technique against future bugs. Interestingly, this heterogeneity was mitigated by the observation that users want Internet and will actively work to meet you halfway. At one point, the team was informed that a specific make of phone was not connecting to our network. Within a week (and well before we were able to debug the issue) the complaint disappeared; the two affected users had simply sold their phones on the used market and purchased a make that was already known to be compatible. Hotspots Both Help And Hinder: Almost immediately after our team distributed the initial set of 10 SIM cards, a robust "secondary" hotspot market emerged, with network users selling access to other users. This finding surprised us at first, particularly seeing individual users consume such large amounts of data. However, it simply continues the above-mentioned theme of community members meeting us where we were. Unfortunately, hotspotted access had the secondary effect of “fuzzing" our understanding of per-user usage patterns, as well as our total network usage as we added users. However, overall, the hotspot market significantly helped alleviate some of the community pressure for SIM distribution, especially since our initial impetus to restrict SIM cards stemmed primarily from handset compatibility issues. Community Knowledge Drives Analysis: A consistent conclusion of community network analyses is that you must understand the community dynamics at play before you can make sense of the usage data. Our deployment was no exception to this rule. Confusing data-points, such as the massive usage “spike" on 23 Jan in Figure 6 (the mayor celebrating a community event by purchasing and sharing literally 3 GB of data in a single day) or the two heavy users in Phase 2 of Figure 9 (the reseller employing a relative to help him sell data), were explained only by asking the members of the community. VoLTE and OTT Services In comparing network-native voice to OTT services, we note that starting with VoLTE (Voice over LTE), all network-native voice is powered by VoIP and SIP. The true differences between telecom-powered VoLTE and OTT VoIP are twofold. First, VoLTE relies on the network layer to provide security and guarantee specific QoS metrics, whereas OTT services enact both of these tasks at the network endpoints. While certain network optimizations can in fact improve performance, the proliferation of cross-network telephony traffic backhauled by the Internet makes a compelling case for the end-to-end argument [57], which is still understood to be best practice in packet-switched network protocol design. Second, LTE tags and bills telecom-powered VoIP as minutes, and OTT-powered VoIP as bytes. This difference in billing is astounding: In the United States in October 2018, the median price per gigabyte was 7.02 USD [58] and the cheapest phone plan was 0.10 USD per minute [59]. At this rate, a HD Voice VoLTE call costs approximately 16,000 USD per gigabyte – literally over 2,000 times the median data price, in addition to any roaming or international fees. While VoLTE does provide certain QoS guarantees, this comparison quantifies the user-borne costs of such service An IP Approach to 3GPP Access Modern (LTE and beyond) telecom systems expose the S2, GX, and SW interfaces for “Trusted Non-3GPP Access." This interface is designed to allow telecom companies to extend and seamlessly integrate their networks with any generic IP-based access technology, such as WiFi or Ethernet, without sacrificing any of the fine-grained billing or monitoring primitives provided by their PCC. The “Trusted Non-3GPP Access" interfaces represent an effort to unify telecom and IP networks under a single management and administrative domain. While this unification is appealing, the “Trusted Non-3GPP Access" paradigm turns IP access networks into nothing more than a subset of telecom networks, and necessarily imposes telecom architectural choices on the network, such as QoS-enforced bearer channels, event-based billing, and network-layer security. This work is part of a continued effort to push back on the wisdom of imposing these legacy telecom architectural primitives on users, especially given that (1) mobile network traffic is increasingly dominated by data, and (2) mobile networks are now built on top of IP networks. In contrast, we firmly believe that the inverse approach is preferable: the architecture and design of future telecom networks should reflect the dominance of these IP primitives and therefore resemble ISP management systems with voice and SMS added as a subset of the network service provided. The architecture of our billing system (and billing decisions) reflects this vision. Comparing to Cloud-based EPCs A variety of researchers and practitioners have been developing cloud based EPCs that could be applicable to rural areas. Examples include BaiCells' CloudCore [32], Echo [24], and Nokia's Kuha [33]. These centralized solutions promise easier setup and configuration, potentially a boon for less tech-literate installers. In practice, these solutions have issues. BaiCells, for instance, extended their platform to include a local EPC (called HaloB) to handle disconnections to the cloud. Our work supports this, showing that on low-bandwidth backhauls, LTE signaling can consume problematic amounts of capacity. Similarly, the commercial solutions are expensive, often charging a substantial per-user monthly fee. Though rural backhaul may eventually improve to support a purely cloud-based solution, it seems likely that local cores will remain the most viable solution for rural areas in the near future. Additionally, CoLTE offers communities flexibility, agency, and system ownership; centralized cloud solutions fundamentally do not provide any of these. Comparing to WiFi Most existing community networks today, including the largest community network on earth [16], are built using 802.11 WiFi technology running static mesh protocols. While these solutions are clearly successful in their environment, we find that they are problematic in the more rural environments covered in this work. Their distributed nature invites hardware failure (as seen in the Village Telco project [14] and AirJaldi [9]) and their use of commercial WiFi makes wide-area coverage more difficult. As such, we believe that cellular protocols, designed to cover wide areas, are best suited to rural community networks.

Bibliography:[Back to table of contents]

REFERENCES [1] World internet users statistics and 2018 world population stats. https://www.internetworldstats.com/stats.htm . Accessed March 2019. [2] GSMA Intelligence. Connected society unlocking rural coverage: Enablers for commercially sustainable mobile network expansion. 2016. [3] rhizomatica. https://www.rhizomatica.org. [4] Mary Claire Barela, Josephine Dionisio, Kurtis Heimerl, Manuel Victor Sapitula, and Cedric Angelo Festin. Connecting Communities through Mobile Networks: The VBTS-CoCoMoNets Project. In Alan Finlay, editor, Global Information Society Watch 2018: Community Networks. Association for Progressive Communications, 2018. [5] Andres Martinez Fernandez, Jose Vidal Manzano, Javier Simo Reigadas, Ignacio Prieto Egido, Adrian Agustin de Dios, Juan Paco, and Alvaro Rendon. The tucan3g project: wireless technologies for isolated rural communities in developing countries based on 3g small-cell deployments. IEEE communications magazine, 54(7):36{43, 2016. [6] Shaddi Hasan, Mary Claire Barela, Matthew Johnson, Eric Brewer, and Kurtis Heimerl. Scaling community cellular networks with CommunityCellularManager. In Proceedings of the Sixteenth Symposium on Networked Systems Design and Implementation (NSDI). USENIX, 2019. [7] Kushal Shah, Philip Martinez, Emre Tepedelenlioglu, Shaddi Hasan, Cedric Festin, Joshua Blumenstock, Josephine Dionisio, and Kurtis Heimerl. An investigation of phone upgrades in remote community cellular networks. In Proceedings of the Ninth International Conference on Information and Communication Technologies and Development, page 6. ACM, 2017. [8] Anonymous Author. Anonymized for review. Anonymous Results, 1(1):1{2, 2018. [9] Sonesh Surana, Rabin K Patra, Sergiu Nedevschi, Manuel Ramos, Lakshminarayanan Subramanian, Yahel Ben-David, and Eric A Brewer. Beyond pilots: Keeping rural wireless networks alive. In NSDI, volume 8, pages 119{132, 2008. [10] Rabin K Patra, Sergiu Nedevschi, Sonesh Surana, Anmol Sheth, Lakshminarayanan Subramanian, and Eric A Brewer. Wildnet: Design and implementation of high performance wifi based long distance networks. In NSDI, volume 1, page 1, 2007. [11] Somprakash Bandyopadhyay, Kazuo Hasuike, S Horisawa, and S Tawara. An Adaptive MAC Protocol for Wireless Ad Hoc Community Network (WACNet) using Electronically Steerable Passive Array Radiator Antenna. In Global Telecommunications Conference, 2001. GLOBECOM'01. IEEE, volume 5, pages 2896{2900. IEEE, 2001. [12] Paul Gardner-Stephen and Swapna Palaniswamy. Serval mesh software-WiFi multi model management. In  Proceedings of the 1st International Conference on Wireless Technologies for Humanitarian Relief, ACWR '11, pages 71{77,New York, NY, USA, 2011. ACM. [13] Shree Om, Carlos Rey-Moreno, and William David Tucker. Towards a scalability model for wireless mesh networks. 2015. [14] Michael Adeyeye and Paul Gardner-Stephen. The village telco project: a reliable and practical wireless mesh telephony infrastructure. EURASIP Journal on Wireless Communications and Networking, 2011(1):78, 2011. [15] Johnathan Ishmael, Sara Bury, Dimitrios Pezaros, and Nicholas Race. Deploying rural community wireless mesh networks. IEEE Internet Computing, 12(4):22{29, 2008. [16] Roger Baig, Ramon Roca, Felix Freitag, and Leandro Navarro. Gui_. net, a crowdsourced network infrastructure held in common. Computer Networks, 90:150{165, 2015. [17] David L Johnson, Veljko Pejovic, Elizabeth M Belding, and Gertjan van Stam. Villageshare: Facilitating content generation and sharing in rural networks. In Proceedings of the 2nd ACM Symposium on Computing for Development, page 7. ACM, 2012. [18] Ali Raza, Yasir Zaki, Thomas Potsch, Jay Chen, and Lakshmi Subramanian. xcache: Rethinking edge caching for developing regions. In Proceedings of the Ninth International Conference on Information and Communication Technologies and Development, page 5. ACM, 2017. [19] Talal Ahmad, Edwin Reed-Sanchez, Fatima Zarinni, Alfred Afutu, Kessir Adjaho, Yaw Nyarko, and Lakshminarayanan Subramanian. Greenapps: A platform for cellular edge applications. In Proceedings of the 1st ACM SIGCAS Conference on Computing and Sustainable Societies, page 45. ACM, 2018. [20] Adisorn Lertsinsrubtavee, Mennan Selimi, Arjuna Sathiaseelan, Lloren_c Cerd_a-Alabern, Leandro Navarro, and Jon Crowcroft. Information-centric multi-access edge computing platform for community mesh networks. In Proceedings of the 1st ACM SIGCAS Conference on Computing and Sustainable Societies, page 19. ACM, 2018. [21] Dario Sabella, Peter Rost, Yingli Sheng, Emmanouil Pateromichelakis, Umer Salim, Patricia Guitton-Ouhamou, Marco Di Girolamo, and Giovanni Giuliani. Ran as a service: Challenges of designing a flexible ran architecture in a cloud-based heterogeneous mobile network. In 2013 Future Network & Mobile Summit, pages 1{8. IEEE, 2013. [22] Felicia Lobillo, Zdenek Becvar, Miguel Angel Puente, Pavel Mach, Francesco Lo Presti, Fabrizio Gambetti, Mariana Goldhamer, Josep Vidal, Anggoro K Widiawan, and Emilio Calvanesse. An architecture for mobile computation offloading on cloud-enabled lte small cells. In 2014 IEEE Wireless Communications and Networking Conference Workshops (WCNCW), pages 1{6. IEEE, 2014. [23] Zafar Ayyub Qazi, Melvin Walls, Aurojit Panda, Vyas Sekar, Sylvia Ratnasamy, and Scott Shenker. A high performance packet core for next generation cellular networks. In Proceedings of the Conference of the ACM Special Interest Group on Data Communication (SIGCOMM), pages 348{361. ACM, 2017. [24] Binh Nguyen, Tian Zhang, Bozidar Radunovic, Ryan Stutsman, Thomas Karagiannis, Jakub Kocur, and Jacobus Van der Merwe. Echo: A reliable distributed cellular core network for hyper-scale public clouds. In Proceedings of the 24th Annual International Conference on Mobile Computing and Networking, pages 163{178. ACM, 2018. [25] Mohamed M Kassem, Mahesh K Marina, and Bozidar Radunovic. Diy model for mobile network deployment: A step towards 5g for all. In Proceedings of the 1st ACM SIGCAS Conference on Computing and Sustainable Societies, page 47. ACM, 2018. [26] Adisorn Lertsinsrubtavee, Nunthaphat Weshsuwannarugs, Nisarat Tansakul, Attaphongse Taparugssanagorn, and Kanchana Kanchanasut. Wireless edge network for sustainable rural community networks. In Proceedings of the Applied Networking Research Workshop, pages 40{42. ACM, 2018. [27] Cloud packet core - nokia mobile networks. https://networks.nokia.com/solutions/packet-core . Accessed 24 February 2019. [28] Cloud packet core - ericsson. https://www.ericsson.com/en/portfolio/cloud-software--services/cloud-core/packet-core/cloud-packet-core. Accessed 24 February 2019. [29] Athonet networks. https://www.athonet.com/ Accessed 24 February 2019. [30] Tecore networks. https://www.tecore.com/ . Accessed 24 February 2019. [31] Magma: A platform for building access networks and modular network services. https://github.com/facebookincubator/magma [32] Baicells - halob modularization solution. https://na.baicells.com/modularization Accessed 24 February 2019. [33] Kuha mobile network. https://www.kuha.io/industry . Accessed 24 February 2019. [34] Halob user guide. https://baicells.zendesk.com/hc/en-us/article_attachments/360019418254/HaloB_User_Guide__SRv1.7_6-Dec-2018_.pdf  .Accessed 6 March 2019. [35] Mehrdad Moradi, Karthikeyan Sundaresan, Eugene Chai, Sampath Rangarajan, and Z Morley Mao. Skycore: Moving core to the edge for untethered and reliable uav-based lte networks. In Proceedings of the 24th Annual International Conference on Mobile Computing and Networking, pages 35{49. ACM, 2018. [36] Marisa Elena Duarte. Network Sovereignty: Building the Internet across Indian Country. University of Washington Press, 2017. [37] Sarbani Banerjee Belur, Meghna Khaturia, and Nanditha P Rao. Community-led Networks for Sustainable Rural Broadband in India: the Case of Gram Marg. In Community Networks: the Internet by the People, for the People, page 193. Association for Progressive Communications, 2017. [38] Davide Vega, Llorenc Cerda-Alabern, Leandro Navarro, and Roc Meseguer. Topology patterns of a community network: Gui_. net. In 2012 IEEE 8th International Conference on Wireless and Mobile Computing, Networking and Communications (WiMob), pages 612{619. IEEE, 2012. [39] Adisorn Lertsinsrubtavee, Liang Wang, Arjuna Sathiaseelan, Jon Crowcroft, Nunthaphat Weshsuwannarugs, Apinun Tunpan, and Kanchana Kanchanasut. Understanding internet usage and network locality in a rural community wireless mesh network. In Proceedings of the Asian Internet Engineering Conference, pages 17{24. ACM, 2015. [40] Altermundi. https://www.altermundi.net/ Accessed 24 February 2019. [41] Kurtis Heimerl, Shaddi Hasan, Kashif Ali, Eric Brewer, and Tapan Parikh. Local, sustainable, small-scale cellular networks. In Proceedings of the Sixth International Conference on Information and Communication Technologies and Development (ICTD), pages 2{12. ACM, 2013. [42] Kurtis Heimerl and Eric Brewer. The village base station. In Proceedings of the 4th ACM Workshop on Networked Systems for Developing Regions (NSDR), page 14. ACM, 2010. [43] Kurtis Heimerl, Shaddi Hasan, Kashif Ali, Tapan Parikh, and Eric Brewer. A longitudinal study of local, sustainable, small-scale cellular networks. Information Technologies & International Development, 11(1):pp{1, 2015. [44] Abhinav Anand, Veljko Pejovic, Elizabeth M Belding, and David L Johnson. Villagecell: Cost effective cellular connectivity in rural areas. In Proceedings of the Fifth International Conference on Information and Communication Technologies and Development (ICTD), pages 180{189. ACM, 2012. [45] Elinor Ostrom. Governing the commons. Cambridge university press, 2015. [46] Lilly Irani, Janet Vertesi, Paul Dourish, Kavita Philip, and Rebecca E Grinter. Postcolonial computing: a lens on design and development. In Proceedings of the SIGCHI conference on human factors in computing systems, pages 1311{1320. ACM, 2010. [47] Flavio Bonomi, Rodolfo Milito, Jiang Zhu, and Sateesh Addepalli. Fog computing and its role in the internet of things. In Proceedings of the first edition of the MCC workshop on Mobile cloud computing, pages 13{16. ACM, 2012. [48] Hendrik Luth. Toll road or dumb pipe? Economic perspectives on net neutrality. Review of Economics, 66(3):303{329, 2015. [49] Vinton Cerf and Robert Kahn. A protocol for packet network intercommunication. IEEE Transactions on communications, 22(5):637{648, 1974. [50] David Clark. The design philosophy of the darpa internet protocols. ACM SIGCOMM Computer Communication Review, 18(4):106{114, 1988. [51] Robert M Metcalfe and David R Boggs. Ethernet: Distributed packet switching for local computer networks. Communications of the ACM, 19(7):395{404, 1976. [52] Stephen Gleave. The mechanics of lawful interception. Network Security, 2007(5):8{11, 2007. [53] Seung-Que Lee and Jin-up Kim. Local breakout of mobile access network traffic by mobile edge computing. In 2016 International Conference on Information and Communication Technology Convergence (ICTC), pages 741{743. IEEE, 2016. [54] Genevieve Gebhart. Zero-rating in emerging mobile markets: Free basics and wikipedia zero in Ghana. In Proceedings of the Eighth International Conference on Information and Communication Technologies and Development, page 16. ACM, 2016. [55] Navid Nikaein, Mahesh K Marina, Saravana Manickam, Alex Dawson, Raymond Knopp, and Christian Bonnet. Openairinterface: A flexible platform for 5g research. ACM SIGCOMM Computer Communication Review, 44(5):33{38, 2014. [56] Eric Butler. Firesheep packet sniffer. https://codebutler.com/2010/10/24/firesheep/ Accessed 5 March 2019. [57] Jerome H Saltzer, David P Reed, and David D Clark. End-to-end arguments in system design. Technology, 100:0661, 1984. [58] Rewheel Research. The state of 4g pricing - 2h2018. http://research.rewheel.fi/downloads/The_state_of_4G_pricing_DFMonitor_10th_release_2H2018_PUBLIC.pdf . Accessed 4 March 2019. [59] Kelsey Sheehy. Best cheap cell phone plans of 2018. https://www.nerdwallet.com/blog/utilities/cheap-cell-phone-plans/ . Accessed 4 March 2019.