Larry Hosken. Technical writer. Puzzlehunt enthusiast.
87 stories

Have Your Cake and Eat it Too

1 Share

How we open-sourced (some of) our monorepo

When we decided to open-source some of our code, our goal was to bring many of the benefits of developing in a monorepo to our open-source projects, while keeping each project relatively small and purpose-driven. We wanted to find a way to make a similar guarantee of cross-project compatibility, without forcing contributors into a large repo with a custom tool chain. We wanted contributors to be able to build, modify, and test individual projects easily, and to spread out into neighboring projects just as easily. In short, we wanted to have our cake and eat it too. This post describes how we bake that cake.

There Can Be Only One

The source of truth for code at Turbine Labs is our internal monorepo. This is not really a blog post about monorepos, and I will avoid defending them beyond stating that, for us, a monorepo provides atomicity of commit across multiple projects, which lets us make a strong guarantee about the integrity of our source code at any point in time. We like this guarantee. A lot.

From the README of our monorepo

We partition and push commits from our monorepo as patches to smaller open-source projects, at a regular cadence. This is a common pattern — so common that we were able to use Facebook’s FBShipIt more or less off the shelf to do the actual partitioning and pushing. With a few tweaks to the workflow and the addition of an important invariant, we’re able to regularly ship a collection of open-sourced projects that can be used individually or in concert, with strong guarantees of cross-project compatibility, and a full commit history.

Commits from our monorepo are partitioned, filtered, and published into public repos, preserving commit history.

The Guarantee

We release and version all Turbine Labs open-source projects and public artifacts in lock-step, guaranteeing the following invariant:

All Turbine Labs projects and artifacts tagged with the same version can be safely used together.

The version tags follow Semantic Versioning rules, but applied to the aggregation of all projects. We push all projects in every open-source release (though some will be unchanged), after which we tag them all with the new version. We push new docker images of tbncollect, tbnproxy, and our all-in-one demo as well. Everything works well together, because it already works together in our monorepo.

The Result

The main benefit of this approach is that contributors can work comfortably in individual projects, while enjoying the compatibility guarantees of our monorepo as they extend their reach. The downside is that the version increments will seem pretty chatty, especially for lower-velocity projects. We may introduce project-specific semantic version numbers in the future, but for now we are avoiding the operational overhead.

Because our open source export is one-way, pull requests from contributors are not merged directly into the open-source repositories; Instead, approved pull requests are applied as a patch to our monorepo, and then pushed back out. A side benefit of this approach is that it’s possible to submit a suite of pull requests across several projects to make a broader change. We’ll merge them as a single commit, knowing we can build and test our entire tree with the new code.

Tell Me More!

For a detailed run-down of the projects we’ve open-sourced, and more about how we did it and why, please head over to our developer repository. You can also read our documentation at, which is itself open-sourced. If you’re interesting in enabling your team to ship more often with greater confidence, visit our website and sign up for a free 30 day trial.

Have Your Cake and Eat it Too was originally published in Turbine Labs on Medium, where people are continuing the conversation by highlighting and responding to this story.

Read the whole story
17 days ago
San Francisco, USA
Share this story

The only option left to the Senate is to make health care reform someone else’s problem

1 Share

I wish I had more time to blog. I really do. But I’m consumed with grant submissions, mentoring researchers, and writing columns about health policy because it’s clear that someone’s got to do it.

That said, I’m taking ten minutes out of my day here to rant. Read it or don’t.

If it hasn’t become abundantly clear, the only thing left for Republican Senators to try is to kick the can down the road. Again. They’re going to try and pass a bill which gives less money overall to states, a lot less money to some states, and then tells them to “figure it out”. Later, they can claim that they gave the states all the tools they needed to fix the health care system, so now it’s THEIR fault things don’t work.

This is ridiculous.

There is no magic. There is no innovation. If there was a way to make the health care system broader, cheaper, and better, we would do it right now. We would have done it years ago. No matter what you may think of Democrats in 2009, they didn’t choose the ACA because they wanted to keep states from fixing the health care system. The ACA was the best they could get.

There are no governors, of red or blue states, who have a magic plan for health care innovation. There are no state legislators (who likely work part-time) who have a secret plan to unleash the power of federalism. The Republicans in Congress have had seven years, all the money in the world, the phone numbers of every conservative wonk in the country, the CBO, experts eager to offer their help… If they couldn’t figure this out, do they think that Montana will? Oklahoma? Indiana? In less than two years?


Pretty much every health care organization is against this bill. The organization of Medicaid directors is against this bill. Most governors are against this bill. Nearly every wonk I can think of is against this bill.

There are legitimate ways to reform the health care system according to conservative principles. They all involve tradeoffs. The people defending this bill refuse to acknowledge that. Many are – at this point – seemingly just making stuff up and promising the moon. And – I’ll bet all the money in my pocket on this – when they can’t deliver, it’s going to be someone else’s fault.

Welcome to public service in 2017.



Read the whole story
24 days ago
San Francisco, USA
Share this story

VPN Crackdown Hits Science, Business Communities

1 Share

Internet censorship has long been a thorn in the side of activists, journalists, and others who seek out news and information that may be banned in China. The ongoing crackdown on online speech under Xi Jinping is also increasingly impacting the day-to-day work of business people, academics, scientists, and others who rely on the internet to conduct research or business with associates abroad. At this year’s Two Sessions parliamentary meetings in Beijing in February, Chinese People’s Political Consultative Conference vice-chairman Luo Fuhe issued a proposal to improve access to foreign websites, citing the scientific and economic cost of current internet controls. Yet since then, an intensified crackdown on VPNs has hit the scientific and business communities especially hard. New regulations in January initiated a campaign against Virtual Private Networks, which allow internet users in China to bypass blocks in the Great Firewall that bar access to certain websites hosted outside China. VPNs have become a main target in a campaign which appears aimed at limiting access to outside information in the run-up to the 19th Party Congress meetings in October. In August, five domestic internet companies were told to cease selling VPNs and a year earlier, Deng Jiewei was arrested in Guangdong and later sentenced to nine months in prison for selling illegal VPNs.

AP’s Joe McDonald reports on how such internet restrictions impact business owners in China:

Chen’s 25-employee company sells clothes and appliances to Americans and Europeans through platforms including Facebook, one of thousands of websites blocked by China’s web filters. Chen reaches it using a virtual private network, but that window might be closing after Beijing launched a campaign in January to stamp out use of VPNs to evade its “Great Firewall.”

“Our entire business might be paralyzed,” said Chen by phone from the western city of Chengdu. Still, he added later in a text message, “national policy deserves a positive response and we fully support it.”

The crackdown threatens to disrupt work and study for millions of Chinese entrepreneurs, scientists and students who rely on websites they can see only with a VPN. The technology, developed to create secure, encrypted links between computers, allows Chinese web users to see a blocked site by hiding the address from government filters.

Astronomers and physicists use services such as Google Scholar and Dropbox, accessible only via VPN, to share research and stay in touch with foreign colleagues. Merchants use Facebook and other blocked social media to find customers. Students look for material in subjects from history to film editing on YouTube and other blocked sites. [Source]

ZDNet’s Asha McLean reports on a recent survey by consumer research company GlobalWebIndex about the habits of China’s VPN users:

The firm’s survey of Chinese internet users found that 14 percent use a virtual private network (VPN) daily. For China’s online population of 731 million, this means 100 million regular users.

Chinese astronomers and physicists surveyed said they use services such as Google Scholar and Dropbox, accessible only via VPN, to share research and stay in touch with foreign colleagues.

Similarly, merchants said they use Facebook and other blocked social media to find customers, while students use YouTube and other blocked sites for subjects such as history and film editing.

[…] The VPN crackdown is part of a campaign to tighten political control that activists say is the most severe since the 1989 suppression of the Tiananmen Square pro-democracy movement. [Source]

Last month, a report in Science Magazine looked at how the VPN crackdown is impeding scientific research in the country:

Many scientists in China routinely bypass the Great Firewall using VPN software that routes traffic through foreign servers. The central government had long tolerated VPNs, but these are now in the crosshairs.

[…] Internet access “has definitely gotten worse,” says a geneticist who splits his time between institutions in China and overseas. The new restrictions make working in China “a total disaster,” he says. And they are likely to be a shock for both foreign and Chinese scientists working overseas who might apply for positions in China. The astronomer who spoke to Science recalls instances in which applicants suddenly couldn’t access materials needed for presentations. “This affects their performance and discourages future applicants through word of mouth,” he says.

Even before the crackdown, scientists had to cope with slow internet speeds. With an average connection speed of 7.6 megabits per second (Mbps), China ranks 74th globally, according to a recent study by Akamai Technologies of Cambridge, Massachusetts. That is less than a third as fast as South Korea, the world leader at 28.6 Mbps. [Source]

While VPN use has been officially restricted in China for years, authorities have largely tolerated their use. Government officials, including Fang Binxing, the creator of the Great Firewall censorship system, have been known to use VPNs to navigate the World Wide Web. Read about the history of the Great Firewall and technologies used to circumvent it via Global Voices, and listen to a discussion with Adam Segal on the Sinica podcast on China’s tightening control over cyberspace.

© Sophie Beach for China Digital Times (CDT), get_post_time('Y'). | Permalink | No comment | Add to
Post tags: , , ,
Download Tools to Circumvent the Great Firewall

Read the whole story
30 days ago
San Francisco, USA
Share this story

Durable Design

1 Share


It seems like small thing, but it’s an engineering detail I’ve always had a lot of respect for.

That picture is of a Flip video camera with the lid off, a product from about nine years ago. It was a decent little video camera at a time that phones weren’t up to it, storing a bit over an hour of 720p video with decent sound. The company that made them, Pure Digital Technologies, was bought by Cisco in 2009 for about $590M and shut down less than two years later. Their last product – that ultimately never shipped – could stream video live to the Web, something we wouldn’t really see from a pocket-sized device until Periscope and (now-dead) Meerkat took a run at it five years later.

The thing I wanted to call attention to, though, is the shape of that case. The Flip shipped with a custom rectangular battery that had the usual extra charging smarts in it and you could charge off USB, like all civilized hardware that size. But it also gave you the option of putting in three absolutely standard, available-everywhere AAA batteries instead, after that exotic square thing finally died.

You only get to run the camera about two-thirds as long, sure. But long after they’ve stopped making those custom batteries or supporting the device itself, the fact of the matter is: you can still run it at all. It may not be the best thing around, but it’s also not in a landfill. It still does everything it said it would; my kids can make movies with it and they’re good fun. It didn’t suddenly become junk just because the people who made it aren’t around anymore.

I’ve often wondered what those product meetings looked like at Pure Digital. Who pushed for that one extra feature that might give their product a few extra years of life, when so many market forces were and are pushing against it. What did they see, that convinced them to hold the line on a feature that few people would ever use, or even notice? You see it less and less every day, in software and hardware alike – the idea that longevity matters, that maybe repair is better than replace.

If you’re still out there, whoever made this what it was: I noticed. I think it matters, and I’m grateful. I hope that’s worth something.

Read the whole story
31 days ago
San Francisco, USA
Share this story

The Trump Administration Was Ordered to Disclose the Legal Basis for its Syria Strike. It Handed Over Squat.

1 Share

After President Donald Trump launched a cruise missile strike against Syria in April, his administration struggled to justify the legal basis for the attack. For months, a watchdog group has hounded the Trump administration for its legal reasoning. Under court order, the government has finally produced documents that reveal little, if anything.

One document the administration saw fit to release is simply an aggregation of praise for Trump’s strike from pundits, lawmakers, and world leaders. It was prepared by Trump’s National Security Council.

On April 6, the United States fired 59 tomahawk missiles at Syria’s Shayrat air base in response to the regime’s use of chemical weapons in Khan Shaykhun, Syria, two days earlier.

The day after the missile strike, an advocacy group called the Protect Democracy Project filed a request with multiple agencies for documents that outline the administration’s legal basis for the attack. After the Pentagon and State Department denied the group’s request for expedited processing, a judge ruled in July that there was a “compelling need” for the information to get out and ordered the administration to provide answers “as soon as practicable.”

In response, the administration on Friday released nearly 60 pages of responsive documents, none of which contain any legal reasoning beyond what appears in the White House’s public statements. The Protect Democracy Project has published the documents on its website.

The documents consist mostly of transcripts of the administration’s public statements, briefings, and press conferences about the strike, as well as emails from a Justice Department spokesperson that are almost entirely redacted.

In the Trump administration’s tradition of being nontransparent, the Department of Justice even redacted memos with talking points that are traditionally circulated throughout the government and sometimes to members of the media.

Allison Murphy, a lawyer with Protect Democracy and former White House attorney for the Obama administration, told The Intercept that the release demonstrates how little the American people know about the strikes, and she urged Congress to step in and provide clarity.

“The founders gave Congress the power to declare war precisely because they wanted to ensure such a momentous decision was subject to public debate,” said Murphy. “This is why Congress must reassert its authority now and not wait until a dangerous decision is made with Congress and the American people left in the dark.”

The State Department, Pentagon, and Justice Department all withheld documents, some due to classification, and others on the grounds that they revealed internal government deliberation, which is privileged under the Freedom of Information Act. Protect Democracy is considering whether to seek further legal action to obtain them.

The White House’s initial statements about the strike did not include any legal justification, and it is unclear whether any legal analysis was done beforehand. White House Press Secretary Sean Spicer later claimed that Article 2 of the Constitution granted the president the “full authority to act” whenever military force is “in the national interest.”

For months, Trump’s strike has confounded legal experts, who questioned what possible legal basis there was for the attack. The day after the strike, national security blog JustSecurity surveyed more than 10 leading experts in national security law, all of whom expressed doubts about its legality, citing a lack of congressional authorization or authority under the U.N. Charter.

The lone dissenter in JustSecurity’s survey was Harold Koh, a former Obama administration lawyer known for his expansive views of war powers. In 2011, Koh helped write a memo  arguing that the Obama administration could continue bombing Libya past the limits set by the War Powers Resolution, simply because the bombings did not amount to “hostilities.”

Even though the Constitution grants Congress the power to “declare” war, the U.S. Supreme Court has ruled that the “commander in chief” role inherently allows the president to take military action in defense of the United States. But Congress has never authorized direct war against the Syrian government, and the administration’s stated rationale was not to act in self-defense, but to punish the Assad regime for using chemical weapons. (“No child of God should ever suffer such horror,” Trump said at the time.)

While Congress has not passed a resolution authorizing strikes against the Syrian government, it passed an expansive authorization to fight terror groups long ago. Former presidents George W. Bush and Barack Obama repeatedly cited the 2001 Authorization for Use of Military Force, a resolution passed in the days after the 9/11 attacks, to justify an ever-expanding war on terror.

Top photo: In this image provided by the U.S. Navy, the guided-missile destroyer USS Porter (DDG 78) launches a tomahawk land attack missile in the Mediterranean Sea in April.

The post The Trump Administration Was Ordered to Disclose the Legal Basis for its Syria Strike. It Handed Over Squat. appeared first on The Intercept.

Read the whole story
31 days ago
San Francisco, USA
Share this story

NSA’s Quiet Presence at a Base in England’s Countryside Revealed in Snowden Documents

1 Share

Civil war was raging in Libya, and Col. Moammar Gadhafi was in hiding. Two thousand miles away, on the outskirts of a small village in the English countryside, British and American spies were monitoring the chaos, listening in on the Gadhafi regime’s phone calls.

The spies were part of a group known as Joint Service Signal Unit Digby, operating from within a nearly 100-year-old military base near the village of Ashby de la Launde in Lincolnshire, a county in England’s east midlands. About a three-hour drive north of London, it is a scantly populated area encompassing flat fields that stretch across the landscape. The British government says publicly that the Digby facility conducts “research into new communications systems.” A more truthful account is that it is an important part of the sprawling covert surveillance network maintained by British and American spy agencies, GCHQ and the National Security Agency.


Aerials at Royal Air Force Digby on July 9, 2007.

Photo: Ian Paterson/Wikipedia

Digby has attracted little media scrutiny in the United Kingdom, perhaps because it is smaller in size and not as visually striking as other surveillance bases in the country. It consists of a nondescript, single-story building and a series of attached offices, all in the southwest corner of a larger military compound. There is an adjacent field equipped with about two dozen antennas, according to maps of the area, but these cannot be seen from a distance. Unlike better-known NSA and GCHQ bases in England, such as the facilities in Bude, Cornwall, and Menwith Hill, Yorkshire, there are no massive, golf ball-like domes visible from miles away; instead, Digby blends in with the landscape.

The NSA’s presence at Digby has never been reported before. And secret documents published Wednesday by The Intercept show that the agency has prized its low profile at the obscure site. The base “does not attract the kind of press attention lavished on RAF Menwith Hill to the north — and long may that continue,” one NSA employee wrote in a March 2005 article for SIDtoday, an internal NSA newsletter. “The American presence here is a quiet one,” the employee added, “but [it] has a profound impact as soldiers, sailors and airmen work together with their British counterparts to produce critical intelligence on an amazing variety of targets.”

The documents, obtained from NSA whistleblower Edward Snowden, indicate that Digby focuses on monitoring communications across the Middle East and North Africa. In recent years, it has targeted Lebanese, Sudanese, and Palestinian communications. And through the Arab Spring uprisings that spread across the region between 2010 and 2011, the base was at the forefront of British and American government efforts to get a handle on events.

National Transitional Council (NTC) fighters confiscate posters of former Libyan leader Moammar Gadhafi in the apartment of a man suspected of being a loyalist in the restive Abu Salim district of Tripoli October 15, 2011. AFP PHOTO/MARCO LONGARI (Photo credit should read MARCO LONGARI/AFP/Getty Images)

National Transitional Council fighters confiscate posters of former Libyan leader Moammar Gadhafi in the apartment of a man suspected of being a loyalist in the restive Abu Salim district of Tripoli, Oct. 15, 2011.

Photo: Marco Longari/AFP/Getty Images

In Libya, where pro-democracy protests spiraled into a full-blown civil war, Digby’s electronic eavesdroppers were at one point working 24 hours a day, seven days a week to analyze what was happening on the ground. In mid-May 2011, with the conflict at its height and a NATO-led coalition enforcing a no-fly zone across the North African country, GCHQ employees at the Lincolnshire spy base identified their primary target as “Libyan regime command and control use of mobile satellite phones.” By late June, the Digby team was still tapping into the regime’s calls, but it was also focused on gathering intelligence about the command and control structure of opposition forces located in the mountainous Jabal Nafusa area, about 120 miles southwest of Tripoli.

Similarly, through the conflict in Syria, personnel at Digby closely watched the situation. A top-secret April 2013 document, authored by senior NSA officials, noted that GCHQ staff at the base were paying particular attention to Syrian President Bashar al-Assad’s security services in the country’s coastal region. A small team of five Digby-based analysts were tasked with the Syrian surveillance mission, according to the document, after the intelligence community assessed that the “regime will relocate and create a strong-hold there [on the coast] should Damascus fall.”

Digby’s function is not limited to that of a passive observer, however. Its central mission, one GCHQ document explains, is to “produce and deliver near-real time intelligence … in order to support military and contingency operations.” It has been integral to a program code-named “AIRHANDLER,” for example, which uses surveillance equipment on Predator and Reaper drones to gather data that is then passed to military commanders. During one six-month period in 2009, there were 148 AIRHANDLER missions flown out of Digby – averaging about five each week. The base was also equipped with a capability enabling it to perform “(near) real-time co-location” of GSM cellphones.


An aerial photograph of the Digby surveillance center, sourced from the Snowden archive.

Photo: NSA

The documents indicate that Digby’s assistance to military forces on the ground has centered around missions in Iraq and Afghanistan, where British and American troops have been deployed. But the facility also carries out a broader global role when it comes to providing tactical military support. One confidential document describes Digby as a “unique” site because it has a joint British and American navy surveillance department within it. This department – known as a Maritime Cryptologic Integration Center – backs up mobile sea, air, and land units operating in parts of the North Atlantic Ocean, the Barents, Baltic, and Black seas, and across North and sub-Saharan Africa.

According to the 2005 NSA report, in addition to providing direct support to operations in Iraq and Afghanistan, Digby performs an “incredibly diverse” mission that is tailored to the intelligence needs of U.S. Strategic Command, Central Command, Northern Command, and European Command. In recent years, it is likely that its capabilities have been expanded. A GCHQ planning document dated from 2011 revealed that the British agency was working to broaden Digby’s function, “growing a cyber mission” out of the base. The agency also wanted to develop a “center of excellence” at the facility, which it said would be a key hub for delivering intelligence to “military partners and customers.”

The NSA and GCHQ declined to comment for this story. GCHQ said in a statement that its work “is carried out in accordance with a strict legal and policy framework, which ensures that our activities are authorised, necessary, and proportionate.”

Other NSA documents — some 294 — and related stories released by The Intercept today are available here.


Documents published with this article:

Top photo: Image of U.K. Digby base from Google Maps.

The post NSA’s Quiet Presence at a Base in England’s Countryside Revealed in Snowden Documents appeared first on The Intercept.

Read the whole story
31 days ago
San Francisco, USA
Share this story
Next Page of Stories