Don G. Park

I came to thailand to be warm instead of cold. That was pretty much the only goal - not be cold. I like my routine of working at collective agency, and living off of Hawthorne. I like the project I work on, icecondor.com, and living off of savings (for now).

The city I'm in is the second largest city in thailand and my usual pattern is not simply applied here, translated by 16000 kilometers (the metric system is stupid). There is something else, and for someone who deals with a lot of anxiety, its kind of awesome. Its so incredibly social here. My first goal is to find a cafe that has good work qualities - somewhere i can stay on my laptop for hours and get stuff done. In checking out a cafe lead I met an Aussie who's been here for six months and told me about things in the city he finds interesting. We chatted for a couple hours.

The 'Keeping It Realtime' conference just finished. It reignited some ideas about how to move data around in a stream-oriented way.

The reworking of icecondor.com will have API endpoints that are indented to hold a TCP(or ZMQ?) connection open and stream results as they happen. Laying out the end points feels like object oriented programming for the first time - a new paradigm that takes a bit of thought how to lay things out.

Identify the basic nouns in the system as Location, Device, User, Group and Fence. There is a channel for each noun. Serialized objects come down the channel as they are received by the system from other actors. Assuming a default of all available location objects coming down the location pipe, that results in a firehose. A filter can be sent up the channel, such as "Device X", and "Inside bounding box X,Y". Filters are stacked/applied with an "and" operator. To listen to the results of two different filter stacks, make two connections to the API. As I write this I'm realizing the default should be no objects and filters expand what comes down the pipe, rather than a wide open default with filters limiting what comes down.

The server side of a web site should be scalable in processing power and storage. The app logic is fine with running multiple copies of itself. The datastore is the hang-up. CouchDB is the solution I'm focued on. The setup is simple, a 'server' exists as a pairing of an application server and a CouchDB instance. Servers can come online at will and the couchdb instances are set to replicate with each other. It should be trivial to change the number of running servers, including across data centers. One performance metric here is time to consistency.

When two servers with conflicting data are syncing documents, a merge conflict can occur. CouchDB has a default merge conflict resolution - longest history wins and for the same history count, highest ascii md5hash/rev value wins.

My next server-side project, a rewrite of the icecondor.com back-end, will use this approach.

One of your greatest assets is your internal motivator.

The best way to check on the health of your internal motivator is to create an environment of as much freedom as possible, then look at what you're motivated to do. Below are some possible responses when you achieve that environment.

"I dont know."

You feel glad for the freedom, but unsure about where to go from here. Your internal motivator is weak from lack of use. Chances are you've been working in order to get away from something rather than working to move toward something. Avoiding fear, such as the fear of being unable to pay bills, is a common motivation and is damaging to your internal motivator. Build your internal motivator strength by moving towards the positive instead.

When you are in a work environment that is damaging to your internal motivator, its important to recognize it and make changes.

turntable.fm is one of those sites that the moment I read about it, a huge light turned on. of course! its so obvious! the pieces were right in front of us, they just had to be put together in a new way.

what i find especially interesting is live365. live365 has been around since '99. you could create your own streaming radio station and upload your own music to play. once there, anyone could listen. live365 did the hard part of getting arrangements with the record labels to allow this sort of playing. no more than two of the same artists within a half hour or something along those lines.

turntable.fm is a refresh of that idea, this time being much more collaborative and social. in a world of infinite content, curation creates value. this is crowdsourced curation thats easy to consume. i listen to music all day but to one or two favorites streaming stations. I miss out on a lot of innovation. crowdsourced DJing builds real-world social ties because people I know are (sometimes) DJing and I get to hear music I would not normally choose.

Looking at different web stacks (ruby mostly), and seeing if/how they work in parallel.

class Plel
  def call(env)
   sleep 1 # snooze
   [200, {'Content-Type'=>'text/plain'}, StringIO.new("Hello World!\n")]
  end
end

run Plel.new

Concurrency Level:      2
Time taken for tests:   1.006 seconds
Complete requests:      2
Failed requests:        0
Write errors:           0
Total transferred:      364 bytes
HTML transferred:       26 bytes
Requests per second:    1.99 [#/sec] (mean)

Two requests take one second. Apparrently Rack will process requests in parallel. How does rack do this?

## config.ru
require 'paral'
run Paral

## paral.rb
require 'sinatra/base'

class Paral < Sinatra::Base
  get '/' do
    sleep 1
    "Hi"
  end
end

Concurrency Level:      2
Time taken for tests:   1.013 seconds
Complete requests:      2
Failed requests:        0
Write errors:           0
Total transferred:      366 bytes
HTML transferred:       4 bytes
Requests per second:    1.97 [#/sec] (mean)

Same result for a sinatra app in rack. Two requests, one second.

went back to the simplest of all webservers, straight webrick

require 'webrick'

class Paral < WEBrick::HTTPServlet::AbstractServlet
 
  def do_GET(request, response)
    status, content_type, body = do_stuff_with(request)
   
    response.status = status
    response['Content-Type'] = content_type
    response.body = body
  end
 
  def do_stuff_with(request)
    sleep 1
    return 200, "text/plain", "Hello World"
  end
 
end

server = WEBrick::HTTPServer.new({:Port=>8081})
server.mount "/", Paral
server.start

Concurrency Level:      2
Time taken for tests:   1.012 seconds
Complete requests:      2
Failed requests:        0
Write errors:           0
Total transferred:      360 bytes
HTML transferred:       22 bytes
Requests per second:    1.98 [#/sec] (mean)

Still just one second. Whats going on with this?

Now that multi-core CPUs are the norm, the challenge of taking advantage of those cores is pervasive in web development. Asynchronous software frameworks and libraries like node.js and eventmachine, and concurrency approaches of multiple processes, threads (lightweight processes), and coroutines, is something I'm trying to understand better.

Ruby Parallelism With Threads

First thing I wanted to look at was running methods in parallel in ruby using threads. The following bit of ruby launches two threads that do division twenty million times per thread.

# parallel

work=["A","B"]

# Launch threads
work.map! do |job|
 Thread.new do
   puts "I got #{job}"
   20000000.times { 4.0/2.0 }
 end
end

# Wait for each thread to finish
work.each { |thread| thread.join }

The results of running this code on different ruby interpreters are below.

$ time /usr/local/ruby-enterprise-1.8.7-2010.01/bin/ruby parallel.rb
I got A
I got B

real 0m9.226s
top says RSS 2340k, 1 core

Ruby 1.8 has "green threads" which is a kind of cooperative multitasking.

$ time /usr/local/ruby/1.9.2-p136/bin/ruby parallel.rb
I got BI got A


real 0m7.580s
top says RSS 2648k, 1 core

Ruby 1.9 has operating-system threads, yet only one core is in use (according to top, dual core CPU was 50% busy). Ruby's Global Interpreter Lock keeps things running sequentially.

$ time /usr/local/ruby/rbx-1.2.3/bin/ruby parallel.rb
I got A
I got B

real 0m5.463s
top says 26,000k 1 core

Rubinius, has I believe the same functionality and limitations as ruby 1.9 as far as threading. Look at that memory usage, wow.

$ time /usr/local/java/jruby-1.6.0/bin/jruby parallel.rb
I got A
I got B

real 0m7.130s
top says 29,000k *2 cores*

Java is the winner here for using both cores. The same crazy memory usage. It didnt take any less time to do the same work. Another confusing result.

The UNIX multiuser system was the original social network.

$ who
bryceo   ttyp1    Mar 14 03:32   (vps1.cobryco.com)
hugiload ttyp2    Mar 17 13:22   (us02-cip.synopsys.com)
pazzed   ttyp4    Mar 14 03:39   (proxy1.dcu.ie)
jalh     ttyqK    Mar 18 12:01   (sb.med.harvard.edu)

User profiles too.

$ finger bryceo
Login: bryceo                           Name: Jean-Luc Picard
Directory: /home/bryceo                 Shell: /usr/local/bin/bash
Office: U.S.S. Enterprice NCC-1701-D 
On since Mon Mar 14 03:32 (EDT) on ttyp1, idle 2 days 22:17,
    from vps1.cobryce.com
No Mail.
Project:
Nothing sepcific at the moment, at least not on devio.us.

Plan:
Finger?? Really?? Who the HECK uses FINGER anymore?
Come to think of it, why do *I* even remember it?
And while I'm at it, what's with all the questions?

Something's value can be measured by how much it is shared. Last month I was experimenting with using an IRC Bouncer and setup ZNC. Then I thought hey what if others want a bouncer but dont have a server to run it on? Or just dont want to set one up? So I made a signup page for ZNC on my box that instantly created a ZNC account for whomever wanted it. My approach to sharing limited resources is to start wide open and crank it down if it becomes a problem.

Then another friend wanted to use IRC from work but couldn't because of a networking issue. I gave him a shell account on my server to run screen/irssi. I thought about an account creation screen for anyone who wanted a shell account. There seemed to be an opportunity to ask for something in return for an account.

Zynga and Cityville has an approach to goals in the game, you can pay with social connections or you can pay with credits. Usually people are gifted in one or the other so it satisfies most people. In exchange for a shell account, the system could ask for a Bitcoin payment, or require a minimum connectedness ranking for the tech community that I'm a part of - the Portland tech community.

Building a connectedness rating (if a single vector is even applicable, it might take multiple) is tricky and arbitrary but the Portland tech community has a great catalog of information to help build a rating - epdx.org.

Twitter's new terms of service restricts the use of tweets to the determent of the twitter ecosystem.

Specifically I feel overly constrained by:

4. You will not attempt or encourage others to: "use or access the Twitter API to aggregate, cache (except as part of a Tweet), or store place and other geographic location information"

I was just about to archive the contents of my tweetstream for analysis of various sorts, including (eventually) location.

5. A. Your Client must use the Twitter API as the sole source for features that are substantially similar to functionality offered by Twitter. Some examples include trending topics, who to follow, and suggested user lists.

I find that to be crossing the line. If someone making a better source of trending topics, and I have a twitter client, I would want to pull information from that service to enhance the experience of my twitter client.

5. E. You may not use Twitter Content or other data collected from end users of your Client to create or maintain a separate status update or social network database or service.

This sounds like it kills any cross-posting app. If my app collects a status update and posts it to two networks, twitter is now trying to kill that. Its these cross-posting apps that will help the migration from twitter to new status/microblog networks.

Its time to get more serious about moving to a federated system.

Using steganography in an existing image is so much more visually appealing than a QR Code. I'm puzzled as to why it hasnt taken off yet. Perhaps the whole idea is patent encumbered. QR Codes are bulky and ugly to look at. If my business card had my photo on it and my linkedin url were encoded in the photo, that sounds better in just about every way. Whats even better is a small, appealing logo could signify the presence of embedded information in the image.

This morning I thought it could be a killer app to have a wordlens effect in the image-code-reader app so that when one looks at the poster on the wall, the embedded URL would be "written" on the poster. Like Bilbo baggins looking "moon letters" on the map of Lonely Mountain. A royalty-free standard for embedding data into an image would be fantastic and create a spot for one or two "killer" mobile apps.