jeffschnitzer - Tumblr blog

jeffschnitzer · 9 years

Text

Guice and Jersey 2, the easy way

It's 2015, almost two years since the release of Jersey 2.0, and it still doesn't play nicely with Guice. The Jersey/Glassfish team decided to write their own dependency injection system from scratch and made a mess of it ("Static globals? Check. Configuration files in META-INF directories? Check. J2EE locked and loaded, SIR!"). There have been a few attempts at Guice integration but nothing that makes it easy.

Well here it is, easy:

<dependency> <groupId>org.gwizard</groupId> <artifactId>gwizard-jersey</artifactId> <version>${gwizard.version}</version> </dependency>

import com.google.inject.AbstractModule; import com.google.inject.Guice; import org.gwizard.jersey.JerseyModule; import org.gwizard.web.WebServer; import javax.ws.rs.GET; import javax.ws.rs.Path; public class Main { /** A standard JAX-RS resource class */ @Path("/hello") public static class HelloResource { @GET public String hello() { return "hello, world"; } } public static class MyModule extends AbstractModule { @Override protected void configure() { bind(HelloResource.class); } } public static void main(String[] args) throws Exception { Guice.createInjector(new MyModule(), new JerseyModule()).getInstance(Run.class).start(); } }

Run this and you have a REST service running on port 8080.

Now the bad news: gwizard-jersey uses the jersey2-guice adapter, which works by using reflection to punch values into private final fields in Jersey's static globals. I didn't even know you could do that! While passing the Kobayashi Maru test, this intrepid effort may fail for some future point release of Jersey. So we don't recommend you use it unless you absolutely 100% need Jersey.

Now the silver lining: If you change gwizard-jersey to gwizard-rest in the pom.xml and replace new JerseyModule() with new RestModule(), the same code works with the reliable RESTEasy implementation of JAX-RS. It comes with Guice integration builtin, and it implements the very same JAX-RS specification.

GWizard throws in many more features as optional guice modules, including configuration files, logging, and Hibernate/JPA. They're all very thin and noninvasive; even if you decide not to use GWizard, you may want to use the source code as a pattern.

https://github.com/gwizard/gwizard

https://github.com/gwizard/gwizard-example

#gwizard #java #jaxrs #jersey #resteasy

0 notes

jeffschnitzer · 9 years

Text

"TDD is dead", he said

One of the issues that drove me out of Dropwizard was the inability to properly test code. I like deep tests that cut across a wide swath of code, with a minimum of mocking and stubbing. My favorite level to test code is the JAX-RS api itself!

Unfortunately, because Dropwizard uses Jersey to implement aspects like transaction management, you can't just casually fire up a test using an in-memory H2 database without instantiating the whole web container - and even then you can only access it using the clunky http client api. This is a mess:

public class LoginAcceptanceTest { @ClassRule public static final DropwizardAppRule RULE = new DropwizardAppRule(MyApp.class, resourceFilePath("my-app-config.yaml")); @Test public void loginHandlerRedirectsAfterPost() { Client client = new Client(); ClientResponse response = client.resource( String.format("http://localhost:%d/login", RULE.getLocalPort())) .post(ClientResponse.class, loginForm()); assertThat(response.getStatus()).isEqualTo(302); } }

I don't want to write tests like this. And I don't want to write tests that mock and stub DAOs - half the business logic of a typical application is wound around using Hibernate (or whatnot) properly. You might as well just stub the whole test green and go home! I think this is the problem that DHH describes as TDD is dead; heavily mocked micro-tests are nowhere near as useful as whole-system tests. The problem is that we have crappy tools for making system tests (eg, all that Client gunk above).

This is the test I want:

public class LoginAcceptanceTest { private Injector injector; @BeforeMethod public void setUp() { injector = // create the injector with test modules } @Test public void loginHandlerRedirectsAfterPost() { try { LoginResource loginResource = injector.getInstance(LoginResource.class); loginResource.login(loginForm()); assert false; } catch (WebApplicationException e) { assertThat(e.getResponse().getStatus(), equalTo(302)); } } }

But even this is wrong; an modern AJAX app doesn't issue redirects on login. Javascript submits a login form via AJAX, gets a result, and displays the appropriate response to the user. The JAX-RS test is even prettier, especially after we abstract the injector setup into a base class and include creation of the user, which is an essential part of the test!

public class LoginAcceptanceTest extends TestBase { @BeforeMethod public void setUp() { instance(UserCreator.class).createUser("user", "password"); } @Test public void goodCredentialsProduceSuccessfulLogin() { LoginResource loginResource = instance(LoginResource.class); LoginResponse response = loginResource.login(new LoginRequest("user", "password")); assertThat(response.getToken(), isValidToken()); } }

The great thing about writing tests like this is that they are fully typesafe. If you refactor any of the types or methods in your IDE, your tests are included. And it's easy to write hundreds of business-level tests, just like fine-grained unit tests - but a thousand times more useful.

You can see an example of this in the GWizard example application here:

https://github.com/stickfigure/gwizard-example/blob/master/src/test/java/com/example/app/resource/ThingsResourceTests.java

#java #gwizard #jaxrs #jersey #resteasy #testing #tdd

0 notes

jeffschnitzer · 9 years

Text

GWizard, like Dropwizard but Guicier

If you've been using Guice for a while, you probably can't imagine how you could ever again live without dependency injection. At this point I pretty much "think in DI". So it was with some trepidation that I started a client project using Dropwizard.

Dropwizard is a fantastic idea - make some opinionated decisions (ones which I for the most part agree with) and cut out most of the boilerplate of web applications. Unfortunately there's one critical opinion missing - DI - and consequently there's still more boilerplate than I can stand. Furthermore, the heavy reliance on Jersey for AOP really complicates testing (more in the next post) - you can't, for example, access a database without starting up the whole web stack!

After progressively replacing piece after piece of Dropwizard with Guice-friendly alternatives, I finally took the final plunge and built a Dropwizard-like framework, based on Guice. GWizard is the result.

The first thing to note is that this is a minimal framework - there are more lines of documentation than lines of code. However, it squarely hits the pain points of building a JAX-RS application with Guice. Here is a complete REST service using GWizard:

import com.google.inject.AbstractModule; import com.google.inject.Guice; import com.voodoodyne.gwizard.rest.RestModule; import com.voodoodyne.gwizard.web.WebServer; import javax.ws.rs.GET; import javax.ws.rs.Path; public class Main { /** A standard JAX-RS resource class */ @Path("/hello") public static class HelloResource { @GET public String hello() { return "hello, world"; } } public static class MyModule extends AbstractModule { @Override protected void configure() { bind(HelloResource.class); } } public static void main(String[] args) throws Exception { Guice.createInjector(new MyModule(), new RestModule()) .getInstance(Run.class) .start(); } }

This is about the amount of boilerplate that I can stand. It could be better; using Reflections to auto-discover @Path-annotated resources would help. But I'm pretty ok with this.

GWizard does more; check out the example application. You'll find easy Dropwizard-style configuration files, JPA integration, logging, and an example of how to write typesafe tests against your JAX-RS api without mocking and stubbing!

Check it out: https://github.com/stickfigure/gwizard

#java #gwizard #jaxrs #jersey #resteasy

0 notes

jeffschnitzer · 11 years

Text

JSOG: Encode complex object graphs in JSON

I keep having this problem over and over. I have a complex, interlinked object graph on my server and I need to transfer a representation to my (usually Javascript) client. This is a long-solved problem in RPC protocols from the 90s, but like the cure for survy, we seem to have forgotten something in progress towards JSON-based protocols.

Consider this example graph of people and their secret santas:

[ { "name": "Sally", "secretSanta": <link to Bob> },{ "name": "Bob", "secretSanta": <link to Fred> },{ "name": "Fred", "secretSanta": <link to Sally> } ]

Notice that it has cycles. Even if somehow it didn't, there's still likely going to be a lot of duplicated data both on the wire and in memory. Typically this is where programmers tediously add id fields and write code to cross-link the entries manually - but wouldn't we rather just transfer the object graph as-is?

Yes we would! So Jon and I created JSOG, a laughably simple convention that lets us represent an interlinked, cyclic object graph in JSON:

JSOG is 100% JSON and uses standard platform JSON parsing tools.

JSOG is human readable; graphs without cycles look like regular JSON.

JSOG does not require or interact with pre-existing id fields.

JSOG is fully self-describing; no external metadata or IDL is required.

JSOG is easy to implement in any language or platform.

JSOG-decoding a simple JSON structure leaves the JSON unchanged.

This is the JSOG representation of the aforementioned graph:

[ { "@id": "1", "name": "Sally", "secretSanta": { "@id": "2", "name": "Bob", "secretSanta": { "@id": "3", "name": "Fred", "secretSanta": { "@ref": "1" } } } }, { "@ref": "2" }, { "@ref": "3" } ]

To encode an object graph: Each time a *new* object is encountered, give it a unique string @id. Each time a *repeated* object is encountered, serialize as a @ref to the existing @id.

To decode an object graph: Track the @id of every object deserialized. When a @ref is encountered, replace it with the object referenced.

The "catch" is that you cannot have real keys '@id' or '@ref' in your structure. When the graph is serialized, @id and @ref are added; when the graph is deserialized, @id and @ref are removed.

JSOG is designed to be trivial to implement on any platform with a JSON parser. @ids are arbitrary strings unique only within the context of a single graph serialization. To facilitate implementation on platforms with ordered dictionary structures, @id definitions must come before @ref references.

We have provided four production-ready implementations:

Javascript, available in NPM

Java, available in Maven

Python, available in PyPI

Ruby, available in Rubygems

The logic of JSOG encoding and decoding is simple; these implementations are a couple dozen lines of code each. The license is MIT. We are striving for ubiquity.

Care to contribute an implementation for your platform-of-choice? Email me.

Discuss this at Hacker News

0 notes

jeffschnitzer · 12 years

Text

Beware cutesy two-letter TLDs for your domain name

Update: After over 8 hours of downtime, DNS authority is once again delegating properly. No response from support emails. Also: According to one HN commenter, other .st domains were affected.

Every .com domain with remote phonetic value is held by a squatter looking for a $1m payout. So, like many other startups, we chose a clever domain for Voost: https://www.voo.st/. It was perfect - short, relevant, and (best of all) available.

One year later, we are discovering the downside of this decision. If you just clicked on that URL, you probably got an error message. It's been that way for the last six hours... and there's absolutely nothing I can do about it except whine on Hacker News.

I presume that the .com domain nameservers are run by smart people with a well-oiled support organizations. Unfortunately, the .st domain is run by a Swedish company named Bahnhof.

Normally (and for a small percentage of requests still), DNS resolution requests for www.voo.st look like this:

legba:~ jeff$ dig www.voo.st ; <<>> DiG 9.7.3-P3 <<>> www.voo.st ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 32274 ;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 2, ADDITIONAL: 4 ;; QUESTION SECTION: ;www.voo.st. IN A ;; ANSWER SECTION: www.voo.st. 300 IN CNAME cf-ssl7797-protected-www.voo.st. cf-ssl7797-protected-www.voo.st. 300 IN A 141.101.126.152 cf-ssl7797-protected-www.voo.st. 300 IN A 199.27.134.152 ;; AUTHORITY SECTION: voo.st. 14744 IN NS eric.ns.cloudflare.com. voo.st. 14744 IN NS vera.ns.cloudflare.com. ;; ADDITIONAL SECTION: eric.ns.cloudflare.com. 28038 IN A 173.245.59.112 eric.ns.cloudflare.com. 89754 IN AAAA 2400:cb00:2049:1::adf5:3b70 vera.ns.cloudflare.com. 84894 IN A 173.245.58.147 vera.ns.cloudflare.com. 118688 IN AAAA 2400:cb00:2049:1::adf5:3a93 ;; Query time: 100 msec ;; SERVER: 166.102.165.11#53(166.102.165.11) ;; WHEN: Mon Aug 20 17:08:19 2012 ;; MSG SIZE rcvd: 242

The request for www.voo.st is delegated to the authority for voo.st - CloudFlare's nameservers. The .st servers are configured to use eric.ns.cloudflare.com and vera.ns.cloudflare.com in the control panel at www.nic.st.

Right now, most requests for www.voo.st resolve like this:

legba:~ jeff$ dig www.voo.st ; <<>> DiG 9.7.3-P3 <<>> www.voo.st ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 5497 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 4, ADDITIONAL: 4 ;; QUESTION SECTION: ;www.voo.st. IN A ;; ANSWER SECTION: www.voo.st. 600 IN A 94.254.0.40 ;; AUTHORITY SECTION: st. 66968 IN NS ns1.bahnhof.net. st. 66968 IN NS dns-au.st. st. 66968 IN NS auth61.ns.uu.net. st. 66968 IN NS ns2.bahnhof.net. ;; ADDITIONAL SECTION: ns1.bahnhof.net. 162265 IN A 195.178.160.2 ns2.bahnhof.net. 57951 IN A 212.85.64.4 auth61.ns.uu.net. 3132 IN A 198.6.1.182 dns-au.st. 68302 IN A 203.19.59.140 ;; Query time: 120 msec ;; SERVER: 166.102.165.11#53(166.102.165.11) ;; WHEN: Mon Aug 20 17:08:18 2012 ;; MSG SIZE rcvd: 203

For some reason, Bahnhof's DNS servers are not delegating authority for 'voo.st' to the 'voo' nameservers at CloudFlare. 94.254.0.40 is something inside Bahnhof's network that refuses HTTP connections. I've also noticed some requests intermittently going to a server that responds with "This domain may be available to regster". It makes our business look terrible.

I can speculate as to what is broken inside Bahnhof. We just came up on the year anniversary of our domain registration; even though we renewed the registration a couple weeks before expiration, perhaps some part of their automated system is trying to reclaim the domain. Just in case, we tried renewing for an additional year - no effect. We tried changing the nameservers - no effect.

Naturally, you're thinking: Contact support!

...and this is where I get angry. Multiple emails to the advertised email address ([email protected]) go unanswered several hours later. There is a web form for submitting support requests, all of which have gone unanswered. There is a fax number, which has been disconnected. There is a link to www.bahnhof.se - the "English Version" button takes you to this website:

What can I do? My only recourse is to - what, change the domain and rebrand my entire business? Wait 12 hours for some Swedish dude to wake up and check the support email? What if he doesn't?

This is crazy.

Discuss this on Hacker News

2 notes · View notes

jeffschnitzer · 12 years

Text

Joy of Python

# first: grep '^......$' /usr/share/dict/words > words f = open('words') lines = f.readlines() lines = [line.strip().lower() for line in lines] letters = 'ecmlobkidepx' def hasall(line): llist = list(letters) for letter in list(line): try: llist.remove(letter) except: return False return True for line in lines: if hasall(line): print(line)

Can you guess what game I'm playing?

0 notes

jeffschnitzer · 12 years

Text

SSL for your domain on Google App Engine

As Jon and I prepared to launch Voost, we knew we needed an always-on SSL solution. In the age of Firesheep, it is simply no longer reasonable to provide any kind of authenticated service, no matter how trivial, over unencrypted channels. Even though our website hands off authentication (Facebook and BrowserID) and purchasing (WePay) to third parties, there are two good reasons why our online athletic registration system needs always-on SSL:

Even if the authentication and purchase processes are securely encrypted, it's not always clear to the user. This is especially true for purchase flows that run embedded in a "normal" page like WePay and Stripe - the URL bar shows no padlock.

Every time you think to yourself "who would bother exploiting this loophole to annoy other users?", the answer is probably sitting in the cafe next to you. One thing I learned at EA is that in any community of appreciable size, there are always a handful of sociopaths that delight in causing trouble for their fellow humans. As unlikely as it is, the chance that someone could steal an auth token and use it to change somebody's race entry is simply not acceptable - it could ruin the day that an athlete has spent months training for.

Unfortunately, lack of support for SSL has been a consistent complaint of serious Google App Engine developers since the beginning. You can publish your app to the world as https://yourobscureappid.appspot.com/, but not https://www.yourdomain.com/ - and appspot.com in the URL bar looks like amateur hour to all of your users who actually care about security and encryption.

Google has had SSL on the roadmap since I first started using App Engine three years ago. The feature is actually in the "trusted tester" phase - but unfortunately this particular test group is nearly impossible to get into. My repeated enquiries through various channels both official and personal have failed - and due to open source work I'm about as well-connected as anyone gets outside the G-plex. Good luck.

Fortunately, you don't actually need Google's help to have SSL on your custom domain.

We discovered CloudFlare because their Platform Lead is one of Jon's cycling teammates (John Roberts has since become a valuable advisor to us, although we're still paying list price!). Their business is normally about providing worldwide CDN services and DOS attack protection, but they will proxy SSL for paying customers.

The basic information you should know:

SSL services are offered as part of the "Pro" plan, not the free plan. It costs $20/mo for the first domain, $5/mo for each additional.

CloudFlare generates the SSL certificate for you. They actually generate a cert that includes 20+ domains together (check the cert details at https://www.voo.st/ for an example), which makes a fair bit of sense. Otherwise they would need a separate IP address per customer. End-users don't see this unless they closely examine the certificate.

With GAE, you use the "Flexible SSL" option instead of the "Full SSL" option. This provides encryption between the browser and CloudFlare, but plain HTTP between CloudFlare and Google. Voost sends passwords and credit card numbers directly from the browser to the respective service providers (encrypted), so this half-SSL approach effectively solves our realistic threat models. If Echelon really wants to know what marathons you're signed up for, they can just crawl the public registration records.

In theory, you can use CloudFlare to serve naked domains on GAE. We're cautious so we didn't choose to try this, despite the fact that adding "www." practically doubles the length of our domain name (sigh).

You use CloudFlare's DNS servers for your domain rather than pointing individual DNS records at CloudFlare. This is actually a bonus - CloudFlare's management interface is rather nice, in sharp contrast to other hosted DNS systems I have used (I'm looking at you, Rackspace). It also makes setup quite a lot easier than it is with other CDN systems.

Here's the HOWTO:

Sign up for the "Pro" plan at CloudFlare ($20/mo). Add your website domain. You should have a panel that looks like this: Note the DNS settings, CloudFlare settings, and Page rules menu items on the dropdown - you will need those in a moment.

Go to your domain registrar and point your domain's nameserver entries at CloudFlare's DNS servers. You will manage DNS for your domain entirely at CF.

In the CloudFlare DNS settings for your domain, add the usual entries for Google App Engine. For example, a CNAME for www -> ghs.google.com.

In the CloudFlare settings for your domain, set SSL to "Flexible SSL".

After a few minutes you should successfully be able to visit https://www.yourdomain.com/. However, you aren't done.

You still need to redirect http:// requests to https:// requests. This will require a Page Rule. There's a special Page Rule for doing exactly this:

Last but not least, you must do something about the naked domain. Google Apps has a built in redirector for naked domains to www (or whatever), but it does not work with SSL - it just hangs. Instead we will add a CloudFlare PageRule which takes care of naked domain redirection for us - both SSL and non-SSL. As a bonus, it will preserve the remainder of the URL. Map yourdomain.com/* to https://www.yourdomain.com/$1 like this:

At this point you should be up and running. You can turn on and off CloudFlare per-domain by clicking on the little cloud in DNS settings (see the right side of the picture in #3). Note that it may take a few minutes for DNS entries to propagate when you make changes.

Since CloudFlare IP addresses will show up as the request origin in your GAE logs, CloudFlare adds special headers to each request. Here is an example:

CF-Connecting-IP: 173.190.104.220 CF-IPCountry: US CF-Visitor: {"scheme":"https"} X-Forwarded-For: 173.190.104.220

Will we continue to use CloudFlare when Google rolls out SSL for App Engine? Without any visibility into what Google plans to offer, that's a difficult question to answer. However, it's hard to imagine switching:

The SSL prices Google floated as a trial-balloon in a user survey were several times higher than what CloudFlare charges.

CloudFlare offers compelling features in its own right beyond SSL:

CF has a global edge cache with documented behavior. GAE has some sort of built-in edge cache, however, its behavior is undocumented and has been experimentally discovered to be rather quirky.

CF offers a lot of nifty little performance-enhancing tricks like rewriting our pages to eliminate extra whitespace, minifying css, etc. We compile and minify our own JS but it's nice not to have to worry about the rest.

CF has analytic tools that are significantly more sophisticated than the GAE dashboard. It's like Google Analytics-lite but based on actual traffic served rather than cookies and javascript.

CF's threat-protection could become incredibly valuable if we ever needed it. GAE's builtin make-your-own-IP-blacklist is not especially reassuring.

CloudFlare answers their support email in minutes (as in, less than 5). It's shocking.

Our experience is still young, but so far all lights are green.

Discuss this at Hacker News

1 note · View note

jeffschnitzer · 12 years

Text

The Facebook Platform Is A Trainwreck, Example #871

There are few APIs more painful to work with than the Facebook API. As the author of a couple deeply integrated Facebook applications and an opensource Java integration library, I suffer with it almost every day. The problem is not that the API is buggy and inconsistent. The problem is that Facebook doesn't care if the API is buggy and inconsistent.

I could pick from hundreds of examples from the last few years, but here's why Facebook wasted a couple hours of my valuable time today:

The Javascript SDK lets you subscribe to events 'auth.login' and 'auth.logout'. They are documented like so:

auth.login - fired when the user logs in

auth.logout - fired when the user logs out

Pretty simple, right? Of course it isn't. In fact, auth.logout fires when you log out and when you log in. Depending on how complicated your authentication system is, it may take you a couple hours of debugging to figure this out.

You might consider logging this as a bug in Facebook's issue tracker. Lo and behold, someone already tried this:

https://developers.facebook.com/bugs/291382317558560

https://developers.facebook.com/bugs/148271701940380

According to Facebook, this broken behavior is By Design. And watch Facebook's typical issue response flow:

Close as Won't Fix without comment.

Ignore developer comments asking for clarification.

Developer posts a duplicate bug asking for clarification.

Close as By Design with a cryptic explanation.

Ignore developer comments asking for clarification.

It's worth looking at the "explanation": If you look at the actual source instead of the minified source, the param "c" is really called "both", so that's totally correct. This is by design. This makes absolutely no sense to anyone outside of the Facebook bubble: Facebook doesn't publish non-minified source for their javascript SDK. They used to, but this project appears to be all but abandoned. The production source has diverged significantly from the github source, which has been annotated "We have no plans to update this repository until December 2011." Well it's xmas, where's my f*cking gift?

I wish this was an isolated incident, but nearly every experience I've had with the Facebook platform is equally unsatisfying. Bugs are closed or simply ignored. Of the issues which get a response, nearly always the first round is "need a repro case" even though the bug is *clearly* spelled out in the description. Do I need to drive down to Palo Alto and type it in on your keyboard?

Another trick that Facebook's platform support engineers have learned is that bugs in "need repro" state automatically close after a week. So even when you provide shell scripts that a 12 year-old could copy-paste to demonstrate the issue, all they need to do is wait you out and your little problem will simply go away. Of course, when I tried to search for one of my issues to illustrate this point, here's what I get:

Sigh. Every little step is a struggle.

Edit: Second bug was a link to the wrong issue. Fixed.

#facebook

6 notes · View notes

jeffschnitzer · 13 years

Text

The Unofficial Google App Engine Price Change FAQ

I don't work for Google, but I read the mailing lists and pay attention. Also, I show up at places where Google buys beer. Here's what I've learned:

What is changing?

Google is changing the way it charges for App Engine. Previously, you were charged for three things:

Bandwidth in/out

Data stored in the datastore

"CPU time" of all your programs

The new billing model charges you for:

Bandwidth in/out

Data stored in the datastore

Wall-clock time spent running application server instances

Number (and type) of requests to the datastore

In addition, there is a 15-minute charge ($0.02) every time* an instance starts and $9 per application per month (as a minimum spend) if you enable billing.

* It isn't quite "every time". See this thread for more.

How are these pricing models different?

For most GAE applications, the biggest charge has always been "CPU time". This number was composed of two parts:

CPU time directly consumed by your frontend web application instance

A crude approximation of CPU time consumed by the datastore and other APIs ("api_cpu_ms")

In fact, api_cpu_ms has never been a real measure of CPU activity - the numbers are based on simple heuristics like "each index write costs 17ms". So the change from api_cpu_ms billing to per-request billing for the datastore really isn't much of a change.

The most significant change to the pricing model is that instead of billing you for CPU time consumed by your frontend web application, you will now be charged for every minute of wall-clock time that each instance runs, irrespective of how much CPU it consumes.

What was wrong with the old pricing model?

The old pricing model charged for the wrong thing. The overwhelming vast majority of web applications use very little CPU; processes spend most of their time blocked waiting for I/O (datastore fetches, url fetches, etc). The App Engine cluster is not limited by CPU power, it is limited by the number of application instances that can be fit into RAM. This is particularly problematic with single-threaded instances like the current Python servers and Java servers without <threadsafe>true</threadsafe>, which require multiple instances to serve concurrent requests.

Charging for CPU time created architectural distortions. Imagine your single-threaded Python application makes a URL fetch that takes 2s to complete (say, Facebook is having a bad day). You would need 200 instances (consuming large quantities of precious RAM) to serve 100 requests per second, but you would pay almost nothing because instances blocked waiting on I/O consume nearly no CPU time. Google's solution was simply to refuse to autoscale your application if average request latency was greater than one second, essentially taking down your application if third-party APIs slow down.

Because the new instance-hour pricing model more accurately reflects real costs, App Engine can now give you those 200 instances -- as long as you're willing to pay for them.

Is this a move away from usage-based billing?

No. You're still being charged for resources you consume, but now you're being charged for the scarce resources that matter (occupied RAM) rather than the overabundant resources that are irrelevant (CPU time).

But my instance only uses 20MB! Why not charge per megabyte-hour?

It's tempting to imagine that Google can just add instances to a machine until it runs out of RAM, then start adding instances to the next machine. In practice, you can't architect a system like this. Your frontend instance may use 20MB now but nothing stops it from growing to 100MB without warning. If a couple application instances did this suddenly, it could push the box into swap and effectively halt all instances running on it. An application instance must reserve not just the RAM it actually uses, but the RAM it *could* use. Oversubscribing creates the risk of incurring unpredictable performance problems, and at the huge scale of App Engine, even low-sigma events become inevitable. I suspect that Google is very conservative about oversubscribing RAM reservations, if they do it at all.

Will my bill go up?

Almost certainly, especially if you are using single-threaded instances (Python, or Java without <threadsafe>true</threadsafe>). Google really was charging an absurdly low price for App Engine before, letting us occupy many hundreds of megabytes of RAM for pennies a day. It was nice, but it wasn't sustainable.

Does this mean App Engine is more expensive than other hosts?

It depends. If you're looking at just the cost of computing, then yes GAE will be more expensive than services like AWS. On the other hand, you can run large-scale applications without the need to hire a system administrator or DBA - so that frees up a couple hundred thousand dollars per year from the budget.

It's also really hard to make an apples-to-apples comparison. With the high-replication datastore, GAE provides a redundant, multihomed, fault tolerant system that can transparently survive whole datacenter crashes. It's already happened. Setting up an equivalent system requires significant engineering effort, and you have to pay someone to wear a pager.

As App Engine has matured, it has gone from being a low-end hosting solution to a high-end hosting solution. Compared to a VPS at dreamhost, App Engine is very expensive. Compared to building your own HRD, App Engine is still comically cheap.

What can I do to lower my bill?

Google has created an article for this, but here's some blunt advice.

There are two aspects to this, driven by the two separate aspects of billing:

Lower the number of instances your app needs

Reduce your datastore footprint

Most developers freaking out about their bill on the appengine mailing list are Python users shocked by the number of instances running to serve their application. You may be able to optimize this somewhat by tuning the scheduler but this will at best provide a small improvement. The stark reality is that single-threaded web servers are a huge expensive waste of resources. Processes that occupy big chunks of RAM while they sit around blocked on I/O don't scale cost-effectively.

The only practical way to significantly lower your instance count is to use multithreading. This allows one instance to serve many concurrent requests, theoretically right up until you max out a CPU. Dozens of threads can block on I/O within a single process without consuming significant additional chunks of precious RAM.

If you are using Java, put <threadsafe>true</threadsafe> in your appengine-web.xml NOW

If you are using Python, beg/bribe/extort your way into the Python 2.7 Trusted Tester group

If you have turned on multithreading, it's unlikely that the scariest line item in your bill will be instance-hours. Instead you will be wondering why you are being charged so much for the datastore. The good news is that there's nothing new about optimizing datastore usage, this is what you should have been doing all along:

Cache entities in memcache when you can.

Remove unnecessary indexes.

Denormalize where you can. It's much cheaper to load/save one fat entity than 20 small ones.

Why are developers complaining about the scheduler?

The scheduler is a red herring. It may very well have issues, but no amount of tweaking the scheduler will change the fact that in order to serve multiple concurrent requests with a single-threaded process, you need to run multiple instances. At best, the scheduler can trade off a crappy user experience for a lower bill.

Forget about the scheduler. Turn on multithreading ASAP.

Is there any good news about the price change?

Google says that higher prices will allow them to increase their commitment to App Engine and devote more resources to its development. This is probably true. If you're building a business (as opposed to a hobby), the new pricing is probably not going to make or break you; salaries and overhead are probably still your biggest concern. However, the addition of significant new features (say, cross-entity-group transactions or spatial indexing) could allow you to improve your product in ways that were too expensive or difficult before. To the extent that more money means more features sooner, paying more might be worth it. Time will tell.

#appengine #google

13 notes · View notes