Goals

My servers tend to run Debian GNU/Linux, because I am totally in love with dpkg and APT. I generally like being able to rely on automatic updates to my software, and Gitweb should be no exception, so this configuration uses the copy of Gitweb that is maintained by dpkg.

git.ithinksw.org also runs git-daemon to support the git:// protocol. One of the features of git-daemon is support for per-user public_git directories. Of course, Apache has the mod_userdir module to provide a similar feature over HTTP. With a little mod_rewrite hackery, the git:// and http:// clone URLs for every repository are identical, and every repository has a Gitweb interface.

The configuration

This configuration relies on two things:

1. A valid Gitweb configuration, including a directory that contains symbolic links to repositories and public_git directories for use as the Gitweb root. Unfortunately, this does require some manual maintenance, to ensure that the desired repositories and users show up in Gitweb. That said, I find this to be useful: I can “hide” repositories or users from Gitweb whilst maintaining their ability to be cloned.
2. The VirtualHost configuration in Apache.

Here is an example of the layout of the server’s filesystem (paths followed by lines that begin with -> are symbolic links to the following path):
/home/user/public_git/repository.git
/srv/git/git.ithinksw.org/ITFoundation.git
/srv/www/git.ithinksw.org/gitweb/gitweb.conf
/srv/www/git.ithinksw.org/gitweb/root/ITFoundation.git
-> /srv/git/git.ithinksw.org/ITFoundation.git
/srv/www/git.ithinksw.org/gitweb/root/\~user
-> /home/user/public_git

Here is the relevant gitweb.conf:
$projectroot = "/srv/www/git.ithinksw.org/gitweb/root";
$git_temp = "/tmp";
$home_text = "indextext.html";
$projects_list = $projectroot;
$stylesheet = "/gitweb.css";
$logo = "/git-logo.png";
$favicon = "/git-favicon.png";
@git_base_url_list = qw(git://git.ithinksw.org http://git.ithinksw.org);
$site_name = "git.ithinksw.org";
$home_link_str = "git.ithinksw.org";
$projects_list_description_width = "200";
$feature{'pathinfo'}{'default'} = [1];

Here is the VirtualHost configuration for Apache:
DocumentRoot /srv/git/git.ithinksw.org
UserDir public_git
UserDir enabled
# gitweb
SetEnv GITWEB_CONFIG /srv/www/git.ithinksw.org/gitweb/gitweb.conf
Alias /gitweb.css /usr/share/gitweb/gitweb.css
Alias /git-logo.png /usr/share/gitweb/git-logo.png
Alias /git-favicon.png /usr/share/gitweb/git-favicon.png
ScriptAlias /gitweb /usr/lib/cgi-bin/gitweb.cgi
RewriteEngine on
# redirect userdir listing to gitweb search
RewriteRule ^/?(~[^/]+)/?$ /?s=$1 [R]
# rewrite all other paths that aren't git repo internals to gitweb
RewriteRule ^/$ /gitweb [PT]
RewriteRule ^/(.*\.git/(?!/?(HEAD|info|objects|refs)).*)?$ /gitweb%{REQUEST_URI} [L,PT]

Closing

That’s all there is to it! I do hope that you’ll read the documentation for the tools I’ve employed here, so that you may fully understand what is going on. This will give you a very basic setup similar to the one at git.ithinksw.org, but you’ll need to know what’s going on to further customize or improve on what I’ve provided. Good luck, and do let me know if you improve on these beginnings or have any suggestions!

Currently, the following projects are available:

• ITFoundation: GNU Lesser General Public License
• ITKit: GNU Lesser General Public License
• ITMac: GNU Lesser General Public License
• MenuTunes: GNU General Public License
• ITTSW.growlView: GNU General Public License

MenuPrefs will be added as soon as I’ve converted the old CVS and Subversion backups of its history into a single Git repository (a process that I will be documenting at some point in the future for others looking to do the same). Otherwise, I intend to use this new platform to further maintain and develop these projects, as well as any new software that I may be developing in the future (including some of the work I’m currently doing for Oberlin College).

My desk at work

On February 16th, I started my new job as Web Developer in the Oberlin College Office of Communications. The only position of its kind in the office, I am charged with maintaining and extending the CMS used by their new website, and providing the benefits of my experience in the form of suggestions and proposals in related technical matters.

This being my first proper “office job”, wherein I physically work in a shared location with my co-workers, has provided me with an interesting new experience in which I’m learning and socializing daily. I believe I’ve adapted smoothly, and I now find the change of pace considerably enjoyable. Although the bureaucratic protocol of the office environment leaves much to be desired, my co-workers are good people, and the knowledge I bring to my position seems well-suited to affecting positive change.

My desktop at work

My one major workflow problem regards my now having two primary desktop computers: one at home, and one at work. Due to the lack of a satisfactory solution for synchronizing my data between the two, I’m currently remoting my home desktop on my work computer for things that are primarily stored on my home computer. It’s not the most elegant solution, so I hope to come up with something better soon. Maybe Haven?

havend

havend instances manage a local database of objects, and connect to (and are connected to by) havend instances on other computers. When a libhaven client, or a remote havend instance, requests an object from the local havend instance, that instance first checks its local database, and returns any match. If it does not find any matches, or if the request was not by UUID but by a key/value query, it forwards the query on to the other havend instances known to it. It determines what havend instances are known to it by introspecting its own local database for objects that specify an appropriate reference to another havend (i.e., IP address and port). havend instances can be externally (that is, external to its introspection) configured to automatically download references from remote havend instances that connect to it, only allow connections from remote havend instances for which it has a reference, and only consider references that are signed by a particular private key. While this does allow the creation of private Haven databases with certain configurations, unconfigured havend instances will contribute to the public database.

When a request is made for an object that does not exist in the local havend instance’s database, and a remote havend instance provides it, the local havend instance will copy it into its local database. Depending on external (again, external to introspection) configuration, the havend instance can have such “cached” objects expire to conserve local storage for locally-generated objects, or only expire after an alloted cache size has been exceeded. This way, oft-accessed objects are replicated on many nodes, and provides redundancy against individual havend instance failures and increased access speed for the whole database. When multiple remote havend instances are capable of providing an object, the local havend instance can request pieces from each one for distributed copying. Once copied, the local havend instance becomes a provider for similar requests by remote havend instances. In this, Haven effectively duplicates the function of BitTorrent.

libhaven

Internally, havend instances provide the notion of “published” objects to local clients. Published objects are objects that follow the rules, and are immutable upon UUID assignment. Published objects are visible to remote havend instances, and contribute to the database. Unpublished objects do have UUIDs, but they are mutable prior to being published. The unpublished object space is provided as a “testing ground” for data structuring and relationship establishment prior to insertion into the database, and is only accessible by local libhaven clients. Unpublished objects can be long lived, in which case their unpublished status reflects their use as a private data store. Unpublished objects can still reference published objects, and other unpublished objects, and generally appear identical to published objects.

libhaven provides clients with APIs for finding and accessing objects, monitoring havend-to-havend copy progress, and editing unpublished objects. libhaven provides higher-level acknowledgement of pedigree objects, and can be provided with cryptographic information such as public and private keys to automate the creation of signature objects

It is encouraged that libhaven be wrapped in the creation of other libraries that provide higher-level support for specific data storage needs. As an example, I plan to wrap libhaven to provide an interface to standard hierarchical filesystem calls, so that applications designed around the hierarchical filesystem metaphor can work with objects in the Haven database.

Object references

Objects can be referenced using a URL of the form:

haven:UUID[?key=value][#fragment-key]

If a query string is provided, with key=value pairs, it will be used to ensure the validity of the referenced object before being returned to the referencer. This will be most useful with cryptographic hashes, which while not necessarily stored as a part of the object will be provided by havend instances upon object requisition as part of the content.

If the fragment-key is provided, the reference will be to the value of the specified key within the object, rather than to the entire object. This is to provide Xanadu-style transclusion.

Object contents and semantics

I will be referring to the immediate objects managed by the storage system as “core” objects, and standard relationships consisting of multiple core objects which provide higher-level features as “abstract” objects. The simplicity of the core object intends to allow for future migration to new, unforeseen usage semantics. Although I will be describing what I believe to be necessary abstract features to higher level clients, by keeping the definition of the core object simple features can change in the future in a reasonably automated fashion. All of the following abstract object features will use objects that at the very least use the “haven.class” key.

Signed objects

One step up from a core object is the abstract “signed object”. This consists of a primary core object that contains the data to be signed, and a secondary core object (with the haven.class key set to haven.signature) that contains a reference to the primary object, as well as a cryptographic signature of the primary object.

A benefit of this system versus a system wherein the signatures are included as a part of the objects themselves is that the authority of a given object’s content can be validated by multiple people, not just the creator, allowing for a distributed web-of-trust.

Revisioned objects

One step up from a core object, again, is the abstract “revisioned object”. This consists of two primary core objects, one representing the previous revision, and another representing the next revision, and one secondary object (with the haven.class key set to haven.pedigree) that contains appropriate references to both primaries.

The benefit of the pedigree being stored separately from the content of the revisions themselves is that the creator of an object has no absolute control over the history represented by the database. This avoids problems where the creator of an object makes a mistake about what prior object is appropriately the progenitor.

Signed and revisioned objects

Both signed and revisioned object semantics can be combined to provide signed and revisioned objects. In addition to both revisions being signed, the pedigree object is signed as well.

Most general purpose computers use the standard hierarchical filesystem model. This is the system you’re all undoubtedly familiar with, where you have directories (folders), which contain files and other directories. Files are individual containers of data, wherein the data is generally stored in a use-specific format. Having used computers for more than half my life, I have accumulated hundreds of thousands of files, in tens of thousands of directories, in many different formats, and I have found this model to be unsustainable.

I hate files

Filesystems, as a rule, do not impose any sort of structure on the data stored within files. They are “content agnostic”. Files are mere collections of bytes. Only applications/programs designed to interpret certain file formats can accurately represent the information stored within them. Due to this, the information can become inaccessible if you do not have such an application. Furthermore, depending on the application and how it utilizes files, a given file may contain multiple discrete pieces of information.

As an example, the music jukebox application iTunes works with audio files stored in your filesystem, each one representing a single piece of music. In order to allow the user to see “albums” or “artists” or “playlists” as conceptual entities, and search within them, iTunes also manages an “iTunes Library” file. The “iTunes Library” file contains a database which contains metadata about and references to the audio files. Although iTunes does have settings which the user can enable to have iTunes automatically store their audio files in a standard hierarchy (in which audio files are stored in directories named after their album, and the album directories are stored within directories named after their artist), if you cease using iTunes you lose access to a lot of the data that iTunes otherwise stores in its database. Your customized playlists, ratings, comments, and play counts are all stored within the “iTunes Library” file, and you must use iTunes to store and access that data.

Personally, I’ll often have a friend send me a piece of music. I keep all of my friends’ biographical information managed with Mac OS X’s Address Book application. That application stores all of that information in a database file, similar to the “iTunes Library” file used by iTunes except designed for managing information about people rather than pieces of music. So, let’s say that when a friend sends me a piece of music, I’d like to have my computer remember that. How can this be done? Although iTunes knows about “tracks” and “albums” and “artists” and “playlists”, it knows nothing about ordinary “people”. Apple would have to give iTunes the ability to read my Address Book database so as to allow me to select someone from my Address Book as the “Source” of a given track. And, even if that ability was added to iTunes, if I were in the Address Book looking at my friend, it would be non-trivial for me to “ask” my computer for all pieces of music sent to me by that friend unless it, too, was granted the ability to read the iTunes Library file.

Of course, these are problems that appear with only two applications. What if I’m using Finder (Mac OS X’s file management application) to look at a particular audio file directly, rather than using iTunes, and have the same questions? More often than not, when I receive a piece of music from a friend, it is via e-mail or instant message, mediums which inherently contain identifying information about the person I’m corresponding with. Shouldn’t my computer be able to automatically create these relationships for me, without me having to even go into iTunes and specifying a “Source” of a particular track, even if iTunes otherwise supported that feature? Well, in order to do that, my e-mail or instant messaging application would need to know about the iTunes Library and Address Book database file formats, as well as the audio file itself. Furthermore, applications such as Address Book and iTunes generally assume that they are the “owners” of their respective databases, so if another application tried to directly access their database files to modify them, things could go wrong and you may end up losing your entire database due to an error.

Even worse, given that iTunes and Address Book (and even my instant messaging application, iChat) are proprietary, closed-source applications, there is no technical documentation available to programmers that defines the structure of their files. As a programmer, if I wanted to create my own application that works with any of the data managed by those applications, it would be very difficult. So, I am not only limited by what a given application allows me to do with my data, but I am also limited, in many cases, to using those applications forever. If I wanted to switch to another application, there is no guarantee that my data will migrate over, and I may effectively lose it. To me, this is wrong. It is my data and should not be under the effective control of the producers of the applications that I use to manage it.

I hate hierarchies

Even in cases when files do represent individual pieces of information, users are often left with the directory hierarchy system as their only tool with which to conceptually relate them. Directory hierarchies provide two concepts: the concept of directories and their ability to contain both files and other directories, and the concept of directory and file names. If you’re navigating through your filesystem trying to find something, you often have only its location in the hierarchy and its name to help you. This demands that you manage your filesystem carefully, to ensure that files are named and placed well enough so that you will be able to identify them later. And, depending on your usage, you may find that no one conceptual hierarchy fits all of your needs.

So, as an example, I enjoy writing essays. I’ll store those essays in a directory named after the context of my essay (e.g., Metaphysics, Politics, Sociology, Psychology, etc.), which are stored in a directory named “Essays”, which is stored in a directory named “Documents”. Now, I also enjoy working with others on projects, and I’ll create directories inside a directory named “Projects”, also stored in the directory named “Documents”, to represent each project. One project that I have ongoing with a friend of mine is related to some of the political essays I’ve written. Now, let’s say that I’m looking at the project’s directory, and I want to see that. Well, by default, I cannot. I need to remember that the essays are stored in a different hierarchy, with their own classification system. Most filesystems do support a “symbolic link”, or “alias”, or “shortcut” metaphor which allows you to put a reference to a file in multiple places, but the problem there is that the file itself still needs to be stored in an original location. If I write a political essay while working on the project, should I save it in the project’s directory, and then create a link to it in my “Essays” directory? Or should I do the reverse? Also, should I name it the same thing in both directories?

What about keeping track of who is contributing to a given project? If I’m working with my friend, and he sends me some files related to the project we’re working on, I’ll store them in that project’s directory. But how do I remember which files he worked on, and which ones I worked on? I wouldn’t want to accidentally create a link in my “Essays” directory to an essay my friend wrote, as the “Essays” directory is supposed to be for essays that I wrote. But then, what if I do want to start storing essays written by others on my computer? Should I create a new hierarchy? Should I modify my old one? What about the earlier example with Address Book? What if I want to select a file in the project’s directory and get to my friend’s telephone number so that I can call him and talk about it?

What about sharing?

So far, I’ve only described some problems that an individual might encounter when managing their own files on their own computer and keeping them, at best, personally meaningful and accessible. What about sharing? If I’m using a directory hierarchy to store my data in a way that I can understand, does that guarantee that it’s the most useful hierarchy for someone else? What if I want to share my essays with someone, but they’d prefer to see them in chronological order? What if I alone want to see them in chronological order, if only temporarily? What if I used an application that allowed me to embed graphics into my essays, but the person that I’m sending them to doesn’t have an application that can read the file format my essays are in? What if the person is blind, and none of the applications available that can decode the file format support screen reading? What if I send my friend’s essay to a third friend, and they want to call them and talk to them about it? Did whatever system I set up to remember that it was my friend’s essay translate to their computer, so that they can easily figure out who to call?

Solving the problems

These are the sorts of problems and questions that led to the design that I will be describing in future posts, the design I refer to as Haven. I believe that I’ve come up with something that takes the best from every system of data storage and access that I’ve encountered, and which not only solves all of the problems I have with the status quo of filesystems, but is also capable of improving many other concepts, such as e-mail, instant messages, and the world wide web. Stay tuned!

Innovation is achieved by creative individuals, who almost always work as part of for-profit institutions to implement their innovations. These institutions have their own preservation and profits, and not the well-being of human society as a whole, as their motivation. However, I believe that most individuals are socially conscious, and that they work for these institutions not for the institution’s gain, but to fulfill their very basic needs to provide for themselves financially. This reveals a lack of options for these individuals to work in an environment which can provide for them, and allow them to provide their innovations directly for the public good.

Government has traditionally been seen as the platform upon which public works can be implemented. This is the basis for the political system known as socialism. Unfortunately, most socialistic governments have been incompatible with the basis for capitalism, individual liberty, as they are implemented with state-owned and controlled enterprises. Such governments only exist by essentially stealing from those they govern to obtain the financial and material support needed to implement their works. The result is that the creative individuals that are the true source of innovation in society are still stifled by the government’s need to sustain itself.

Proposed is an alternative, a practical system for achieving ends supportive of the public good, which is compatible with the capitalistic economy that predominates.

Why not form a non-profit institution dedicated to the implementation of innovations beneficial to the public, and the support of those creative individuals doing the innovating?

Such an institution could be financed completely voluntarily by society as a whole, via donations from conscientious individuals, and an endowment would be created for long-term stability. The intellectual products of this institution would be offered into the public domain, or under licensing terms similar to those used by open-source/free software projects. Physical products would be manufactured by for-profit entities in the relevant industries, but offered to the public at the cost of manufacturing and distribution, with no profit motive for the institution itself. Alternatively, the price of the products could be subtly increased, with any income being diverted to the institution’s endowment. Further financial surplus would allow for public information and educational campaigns, aimed at increasing the collective intelligence and wisdom of society as a whole.

Ideally, anyone with a good idea and the means to contribute to the implementation of that idea as an individual would be welcomed. Additionally, skilled professionals looking for work would be hired into the pool of individuals available to work on the institution’s public work projects. All workers in this institution would make a stable living, and work in an ad-hoc fashion on the various projects that they could contribute to. This financial stability would allow them the peace of mind necessary to intuit and implement the great innovations needed for the future of society, and as I believe this is the true calling of most anyone, there is no doubt that they would do the best that they are capable of.

The organization and long-term plans of this institution would be transparent and open to the public. The leadership of the institution as a whole would likely be accomplished by a subset of the worker pool being elected democratically by the entire pool, preferably using a Schulze method voting system, to a board of governors who would then decide on future projects for the length of their term. This is the “representative democracy” model. Alternatively, the projects themselves could be decided by the worker pool in a similar process, a “direct democracy” model. Either way, a non-binding public polling system would allow the public to contribute to the decision making process in an advisory role.

Projects would be managed by a task force, the subset of the worker pool that primarily contribute to that particular project. If help is needed by a task force from an outside worker in the pool, the task force can nominate that worker for inclusion in the project, or request that they work on a particular individually-manageable subtask. Projects would be formally described in a mission statement that clearly outlines the goals and purpose of the project, the incidental auxiliary products of the project, and the human, material, and time resources believed to be required for implementation (i.e., a budget). Conflicts between projects and task forces due to a contradiction or overlap of goals or material requirements would be handled by the aforementioned institution-wide leadership system. Within a task force, irreconcilable differences between workers would be handled similarly, though their management would be otherwise fairly autonomous with regards to the institution as a whole, and decided upon by the task force itself. This way, larger projects can implement any larger formal systems as necessary, but smaller projects will be free to be developed by smaller task forces more intimately.

So, what is needed for this institution to become a productive reality? Well, workers, the money to support them, and an institutional identity.

Enter Curnon.

In attempting to realize the idea of this institution, we have chosen to start a public discussion around our website, Curnon. The name Curnon is derived from the Latin for “Why not?”, and we believe this accurately reflects our attitude. We are open to any and all suggestions, including rebuttals to our proposals. And, of course, we are open to donations, and notices of intent from people who would be willing to join us. Everything is on the table. Our all-encompassing goal is to make the world a better place for everyone living in it, and to do so by the most efficient and appropriate means. What those means are is what this platform will help us realize.