I’m looking to buy a few more computer related books, any suggestions? Kinda hard to answer, considering you don’t know what I already own…but still…any computer books that are highly recommended? It can be anything really, algorithms, graphics, data structures, perl, lisp, smalltalk, the list goes on.

I’ve been working on setting up my virtual server which runs this site and Kiwi, and I’ve been tinkering with the mail system (of course). I have Postfix running as the MTA, which sends incoming mail to Procmail, which deposits my mail into a series of Maildir directories. From there, I can use the wonderful Dovecot (maybe the best IMAP server out there right now) to view my mail. The only thing that is missing is SPAM protection. When I was on Dreamhost I had SpamAssassin doing the SPAM filtering for me, this past week without any sort of filter and I’ve been hit with a barrage of junk e-mail.

I’ve done the basics, like configured Postfix to reject senders without a valid domain. I installed and ran SpamAssassin for about a day, but SpamAssassin is huge! I mean, really, it gobbles up RAM. Normally this wouldn’t be a problem, but I’m running on a VPS with 96MB of RAM, and I’ve got a bunch of services, I can’t afford to have SpamAssassin taking up 40 MB! By default it’s even worse, since SpamAssassin starts with 5 processes on Debian, each of which take 20MB of RAM. Yikes! I did the natural thing and scaled it back as much as possible, 2 processes, but that is still 40MB of RAM.

So what’s the solution? I’m happy to say that I’ve found something that uses fewer resources and has filtered out every piece of junk mail: SpamBayes What is SpamBayes? It’s server side Bayesian filtering software that you train based off of the mail you receive. I’m going to present here a few simple directions for setting up SpamBayes, however I’m making a few assumptions:

• That you have an MTA (mail transfer agent) such as Postfix or Exim setup
• You have Procmail configured to be your MDA (mail delivery agent)
• You’re using Maildir (these steps can be easily modified for mbox or mh)
• You’re using IMAP

So on with the show:

1. Get SpamBayes from the website (it’s a series of Python scripts), or if you are on Debian apt-get install spambayes

2. Create a SpamBayes database by running sb_filter.py -n in your home directory

3. Create a simple configuration file ~/.spambayesrc which tells SpamBayes where to find your database. Here is the config file I used:

   [Storage]
   persistent_use_database = True
   persistent_storage_file = ~/.hammiedb
   

4. Edit your .procmailrc file so that it invokes SpamBayes when e-mail is delivered. I also added some filters to filter out e-mail that SpamBayes tags as Spam, and e-mail that it is unsure about. Edit the path to sb_filter.py as appropriate, and note for my setup this puts Spam in a Spam folder and unsure mail into, you guessed it, Unsure:

   :0 fw:hamlock
   | /usr/bin/sb_filter.py
   
   
   :0
   * ^X-SpamBayes-Classification: spam
   .Spam/
   
   
   :0
   * ^X-SpamBayes-Classification: unsure
   .Unsure/
   

5. Add a line to your crontab file so that every night SpamBayes learns by looking at e-mail you’ve put in the Spam folder, and your Inbox. Run crontab -e and add this line:

   10  0   *   *   * /usr/bin/sb_mboxtrain.py -g /home/mronge/Maildir/cur -s /home/mronge/Maildir/.Spam/cur
   

What does that line do? It says that every night, 10 minutes after midnight, run sb_mboxtrain.py (despite the name, it works on mbox, Maildir, and mh), where good mail is stored in the Inbox located at /home/mronge/Maildir/cur and Spam is stored in /home/mronge/Maildir/.Spam/cur. Of course you’ll have to adjust your paths above as necessary for your own system. This way SpamBayes gets smarter every day but scanning your e-mail. You can even add other folders so that SpamBayes can train off of your archives or mailing list folders, for me I filter out mailing list e-mail in my .procmailrc, so I don’t train off of it.

After that, you’re done. E-mail which is left in your Inbox will be used to train what a “good” message looks like, and mail that you move to your Spam folder will be used to train the system on what spam looks like. Also, make sure you check your unsure IMAP folder, and move any mail that SpamBayes is unsure of to it’s proper location. Hopefully, like me, you’ll have success in filtering out Spam without the performance penalty of running SpamAssassin. As a final note: I realize these directions assume quite a bit of unix and sysadmin knowledge, if you have any trouble with the above feel free to leave a comment and I’ll do my best to help out. Another final note: Let me know if you spot any grammatical mistakes (or if I’ve got some of the technical details screwed up).

I’ve been working on migrating my sites over to my shiny, new virtual private server. Like always, it’s taking longer than anticipated, mostly because of a lack of sysadmin experience on Debian. I’m learning quickly, and I have a few things setup, like the main Kiwi website, and a wiki run off of MoinMoin.

I’m in the process of setting up SMTP, IMAP and migrating this TheRonge over as well. There are a number of things I need to do, like setup Mailman, configure Exim and Dovecot, and move this Wordpress install. I’m going to be working on all of these things this weekend, and I want to have it done soon so things can return to equilibrium around here.

I’ve got a Kiwi development plan etched out, but I’m not quite ready to post a link to the development plan quite yet. I need to get some things ironed out before the server is ready to take any sort of serious traffic. :)

Check out this talk by Martin Fowler on domain specific languages. It’s directed at Java developers, but the same ideas can very easily be applied to Objective-C. Link via Ralph Johnson

Progress has been slow on Kiwi lately, and I apologize for that. I’m a student as many of you know, and I seem to have a ton of projects and homework assignments floating around. Recently, there’s been a bit of progress, like builds no longer fail (yaay!). The client is still not functional in it’s current state. I’ve started the process of removing glue code and moving to a nearly 100% bindings built app. I’ve learned two things from this experience:

1. The repository should always be left in a build-able state
2. There shouldn’t be a loss of functionality with a subversion commit

If either situation occurs, development should be done on a branch or work should be done in a working copy until both of the conditions are satisfied. This might be obvious to most developers, but it wasn’t until recently that I learned this particular lesson.

With that out of the way…

A new server

Just today I signed up for a virtual private server through Quantact. I’ve also registered a domain name, kiwiclient.org, which I will be moving the Kiwi resources over to. I’m going to be setting up a bug tracking system, a wiki, http, mailman and etc, all with the sole purpose of supporting this project. So please be patient, some resources might go down for a day or so while I move to the new server, but it will be for the better.

What about my repository access?

I’ve granted a number of people access, but that will be changing with the move to the new server. I hate to revoke repository access, but I think it will work better if people submit patches to the list and I can approve them as deemed necessary. If people become very active in the project, I can then grant repository access as is needed.

What can I work on?

A number of people have asked me what they can work on, I’ll be putting together a to do list shortly, and I’ll be adding documentation on the architecture of Kiwi. Once that is up it will be easier for others to contribute to the project. I want to put together a series of milestones, so that we can incrementally add more functionality to Kiwi, and hopefully get off the ground sooner.

So hang in there, I’m working on turning this into a real open source project. This weekend I want to have the new server up and running and a project schedule with milestones.

I’ve been busy with school, a summer internship, and loads of personal projects and I’ve had very little time to write any blog posts, here’s goes nothing though:

Apple

I spent this past summer working as an intern for Apple in sunny Cupertino, California. I’ve lived in the Midwest all of my life and my stay was my first real visit to California, so it was a new experience for me. During my time at Apple I was the intern on the Video Foundation Team, which is part of the Interactive Media Group within Apple where I worked on QuickTime Video. I made contributions to an in-house tool that monitors video playback events in realtime. The event tracing system provides the video team with tons of information on what QuickTime is doing, but it doesn’t introduce any performance penalties, in fact tracing is enabled on all builds of QuickTime out in the wild. My tool helps visualize the huge amount of data available in these traces and presents it in a format that can best be described as an EKG of video playback. It was challenging, and best of all, it gave me a change to interact with the kernel team, the QuickTime team, Quartz Composer team, and others at Apple.

There are other interesting things that happened at Apple, like getting to know the Mail.app team, which is kinda funny considering I work on Kiwi. According to them I’m crazy. A few of them said that Kiwi is too much work for one person, but I disagree, I want Kiwi more than anyone else, and I’m determined to make it a success.

I also won a new MacBook during an intern idea contest. I can’t speak on what the idea was, but a small team of us spent a few late nights hashing it out. We presented it to some executives durign a competition, and somehow we won. In fact, news traveled rather quickly and we ended up meeting with Bertrand Serlet and others about the possibility of making the idea a reality. I wish I could tell you what it was, but if it makes it into a future Apple product it will just ruin the surprise, and according to an agreement with Apple I can’t tell.

Kiwi

Progress on Kiwi has, unfortunately, slowed to a standstill. The code base is getting larger and that honeymoon period during which you can produce alot very quickly has come to a close. Everyday that I am forced to use Mail.app or Thunderbird, I want Kiwi more. So stayed tuned, Kiwi will be completed but not as quickly as I had hope for.

The last bit of work I completed was a build system that is a combination of Xcode plus a few Python scripts which build some open source libraries in the repository. I’m moving the entire user interface over to bindings, and that’s still a work in progress. Hopefully, I’ll soon have more time to continue working on Kiwi, but those damn classes here at school keep getting in the way.

Cocoa Tutorial

I’m the president of a Mac users group at the University of Illinois Urbana-Champaign campus known as MacWarriors. Some of us are experienced Cocoa programmers, but most know very little about Objective-C and Cocoa. I’m hoping to change that with a talk and tutorial I’ve been crafting. Writing isn’t easy, and creating a good talk is no exception. It’s taking me a longer to write the tutorial then I expected, but I suppose that’s the nature of the beast. Like software, your always almost done and your still tweaking things when it’s ripped out of your hands. Once the actual talk is completed, I’m going to post it online for others to use, and if we continue with having talks we might eventually start recording and posting them on the MacWarriors website.

LaserLine 2.0

A few years ago, MacWarriors built LaserLine, that was a combination of software and hardware necessary to draw images with a pen style laser. The gist of the project was that we had two computer speakers with small mirrors attached on to them, such that when the speaker cone moves the mirror moves along with it. We hooked this speaker setup to a Mac portable and using our software we were able to draw simple drawings in our Cocoa drawing program which were converted into a series of equations using cubic splines. Once we had the parametric equations, we iterated over them constantly sending the result out the audio jack to the speaker setup. While the speaker vibrated we aligned a small laser with the mirrors and the laser drew the vector drawing from the computer.

This year, we are taking it a step further with LaserLine 2.0. Joey Hagedorn is building some wickedly cool hardware that is a set of galvanometers and other parts so that we’ll be able to control the mirrors far more precisely. The hardware will incorporate a standard interface which will be plugged into the computer via USB. On top of our new hardware we are developing a new version of our software that will output data to the USB device from a file known as an ILDA file, the industry standard for laser displays. We hope this all turns out to be a ridiculously cool project.

Mailman and CS427

I’m in CS427 Software Engineering this semester. It’s somewhat interesting, most of the class is about different development methodologies like Extreme Programming, and the Rational Unified Process. However, much of the class is bullshit, something not typical of CS classes here on campus. While, I don’t have enough industry experience to truly evaluate these methods, it’s very interesting that high profile companies like Apple and Google seem to shun these processes. While I was at Apple people avoided meetings and the man with the clipboard, the project manager.

For class we have to apply RUP or XP to a project of our choosing. My group chose to work on the next version of Mailman in hopes of providing a new user interface, a database backend and a more sophisticated mailing list archival software. Our proposal was excepted, but there was a lack of student interest in the project. Sadly, I’ve had to move onto a far less interesting project which hopes to develop some cross platform persistent chat software. So much for getting more experience with Python and Mailman.

C4

I’m going to be attending the C4 conference this weekend, which should be a lot of fun. Luckily I’m just a few short hours south of Chicago, so it’s going to be no big deal for me to drop into the city for the weekend, I wonder how many people from other parts of the country will be there?

I’m also lucky to have received a student scholarship to attend the conference. Some of you might have seen this posted on Rentzsch’s blog:

The page includes the responses to the question I posed to applicants: “why you want to attend C4?” It’s some interesting reading. Sneaky tidbit: one of the entries are fake. Well, not fake, but written by me and not the applicant. I pinged him about his missing text, and got this back:

Oh damn. I didn’t know about that. Hmm…. i just got back from a bar so I’m in no shape to write right now.

So I filled it in for him. Can you guess which one I made up?

You don’t have to guess any longer, that person was me. It’s amazing that I still managed to get a scholarship, there must have been virtual hanging chads.

Programming languages to learn:

• Lisp
• SmallTalk
• Haskell
• Ruby

• Python
• Lua

This year at WWDC Student Sunday, Aaron Hillegass gave a talk on Cocoa design patterns and data structures. It was an interesting talk, as usual, but the most interesting part is the contest he is sponsoring:

The Bonzo Collection of Reusable Classes for Cocoa I think we all agree that it would be good if there were a large collection of tested, documented classes that could be used in a variety of Cocoa applications. To kick-start this effort, Big Nerd Ranch is holding a contest. Give us a few classes (or categories or functions) for the project, and you may win a free seat in a Big Nerd Ranch class. The contest is open to anyone. You need ideas? Data structures: balanced trees, ordered sets and dictionaries, etc. Wrap C libraries: gzip, OpenSSL, BerkeleyDB, Thunderbird client libraries, etc. Artificial Intelligence: classes for neural nets, simulated annealing, Hidden Markov Models, etc. (Read “AI Application Programming” by Tim Jones.) Big Nerd Ranch

Totally awesome, I’m hoping that this contest will spawn a whole collection of code, like the former MiscKit. I’d love to participate, however, I’ve got enough on my plate with Kiwi. Are you participiting?

A new version of MailCore is available. This release fixes an issue with SMTP where MailCore wasn’t running StartTLS before attempting to authenticate. Servers that require AUTH should now work great, and let me know of any problems.

Here’s a new website devoted to Kiwi. Expect more when we release the app.

E-mail clients are interesting in that it’s a somewhat religious decision, comparable to choosing a text editor or operating system; everyone has an opinion and their own feature that’s always a deal breaker. I’ve used quite a few clients, and I’ve never been happy with the IMAP implementations, atleast with regards to Mac e-mail clients, there might be some shining stars on other platforms that I don’t know of, (if you know of one let me know). I’ve been told that Mulberry had the best IMAP support, but screenshots of Mulberry were enough to keep me away. Sadly, the company went out of business and nothing more came of the client. There isn’t much in the way of great IMAP clients. Mail.app is slowly getting better, Thunderbird is probably the best out there right now, but it’s not completely OS X native, and most other clients don’t even support IMAP. This brings up the interesting question:

Where are the kick ass clients?

Part of the reason few excellent IMAP clients exist is due to the complexity of the IMAP spec. POP3 has been a lower fruit for developers to pick, and consequently POP3 support has been much more widespread. Not only is it difficult to get IMAP completely right, but it is a totally different paradigm then POP3, so in order to support IMAP and POP3 increased complexity is required. In fact, many e-mail clients start out POP3 only and add IMAP as patch later, which is causes some of the wonky IMAP behavior. However, it’s understandable that developers don’t conquer IMAP once you compare the specifications of IMAP vs. POP3:

IMAP Spec (RFC 3501) 6051 Lines

POP3 Spec (RFC 1939) 1291 Lines

The IMAP spec is far larger than the corresponding POP3 spec, so it’s not surprising that it’s a bitch to get right. Even then, that is just one RFC; there are other RFC’s for IMAP which add extensions like IDLE support, UIDPLUS, the UNSELECT command, and more (check out IMAP Connection if you’re interested).

Those 6000 lines describe just the basic protocol, people also expect stability, speed, and, disconnected support in a modern client. IMAP has proven itself to be stable, and fast, but disconnected support is a curious issue. IMAP wasn’t designed to support disconnected access, but rather an always on connection, but wait…

What’s online, offline and disconnected?

Let me regress for a moment and explain. An offline client uses POP3, you connect to the server, and download your mail to the local machine. Whenever you read your e-mail it’s being fetched from your local computer. An online client is what IMAP was first designed as, when you read your e-mail you always have an active internet connection, your e-mail stays on the server and messages are downloaded on demand. A disconnected client is a fusion of the two mail models. In a disconnected client your local copy of e-mail is synchronized with the server. You can still manipulate your mail and the client is smart enough to copy your changes to the server when you have an internet connection again. Because of the flexibility it’s not surprising that many users have come to expect disconnected mode.

Where does this leave us?

Disconnected mode is very convenient but it’s difficult to get right. The IMAP protocol is very difficult to use in disconnected mode because it wasn’t originally designed for that purpose. In IMAP, every message has a UID (Unique Identifier) and a every folder has a UIDValidity value, the combination of these two is unique across the account. This does make it easier to program a disconnected e-mail client, but the problem is that the UID’s aren’t always static, and some servers are better with this than others. What can happen is that the server might decide that it’s time to change the message UID’s for whatever reason, and then you need to purge your local cache and download everything in that mailbox again. If the developer fails to verify the UID before executing a command on the server, some very nasty things can happen. For example, you might have a UID stored for a message that the user would like to delete, so you send a delete command to the server, but the UID validity is never checked. If the UID has changed, in the best case the delete operation will fail, but in the worst case it could remove a message that is now using the other message’s old UID. This is of course, not cool, and it would be nice if UID’s didn’t change. Some servers are crappy in that they change the UID’s far more than is necessary while others attempt to keep the UID constant for as long as possible. And just as it is hard to build an IMAP client, it’s even more of a bitch to build the server.

Another problem with IMAP is that it’s very time consuming to update all off the message flags. What do I mean by message flags? I’m taking about the flags that mark if a message has been read, replied to, forwarded, marked for deletion etc. There are some tricks out there using UID’s to only download new messages, but to make sure you have the latest flags you have to iterate over every single message, and this can be very time consuming for a large mailbox with thousands of messages.

What next?

What I propose instead of time consuming iterations is a time stamp. You send the server a time stamp, like for example the last time you connected to the server, and it plays back every change it has made to the mailbox since that time. So syncing becomes much easier, connect to the server and send the time stamp, the server sends you everything that has changed since that time. No more time consuming flags updates, or UID tricks. It would be even nicer if we could do away with IMAP and replace it with something simpler, and more generic. Why not write a generic synchronization server like .Mac that you can use to synchronize objects with? Then add support for time stamps and we can start syncing everything on to a central server. Then we can sync all kinds of data across computers just by using the generic protocol. Oh, and let me know if I missed an IMAP detail, or there is some magic trick I don’t know about it.

Want to know how many lines of code your project is? Run this command:

find . -name '*.[hm]' -exec cat {} \; | wc -l

Note: Header files in your project build directories can skew the result, you might want to empty out your build directories before running this. Change the file extensions in [hm] if you want to run this one something other than a Cocoa project.

As you probably already know, I’m working on an e-mail client known as Kiwi. Kiwi is not going to be the end all e-mail client, and for many people Mail.app might be a better option. What Kiwi is will be, a sharper tool in an advanced user’s bag of tricks. Part of the problem with designing for the more experienced computer user is that they expect more. They expect more functionality, more options, more customizability. In particular, making an app super customizable is not easy to do, and Kiwi will probably not be perfect as a 1.0. However, I hope to ease the pain of customization by providing a Lua glue layer in a few key places.

What’s Lua? It’s a small scripting language built to provide a language for configuration files, and places where code changes frequently. It’s becoming increasinly popular, especially in the gaming industry and with companies like Adobe (Lightroom makes heavy use of Lua). So I’m going to inject a little Lua glue into Kiwi, and just a little. So if, for example, you don’t like how Kiwi parses and renders the e-mail output, you can edit some CSS and Lua and have a different view of your e-mail. Or maybe you want to create a complex action that you can run on selected e-mails, jump in and fiddle with things via Lua.

So yes, it might be very cool, but don’t expect too much. What I’m doing here is throwing around ideas, and I can’t guarantee when it will be released, and what will be in it. Either way, let me know what you think.

Today I had a weird problem where copying and pasting stopped working. Since I really dislike restarting, I messed around and found a solution:

Run this in the Terminal: ps -x | grep "/System/Library/CoreServices/pb[s]"

I got a result like this, if you don’t get any results skip the next step: 20829 p0- SN 0:00.28 /System/Library/CoreServices/pbs

So it looks like it’s working fine, but for some reason it’s hung up. So kill the server: kill 20829

Now restart it: nohup /System/Library/CoreServices/pbs &

Now copy and paste should work again, if it doesn’t work in an app, try restarting the app, that should do the trick.

MailCore 0.2.3 has just been uploaded. Here’s some release notes:

• The CTCoreMessage object now has a flag attribute, it holds flags like unread, forwarded, new..etc
• More documentation has been added. I added a small section on sending e-mail.
• MailCore has now been released under the BSD license
• @executable_path/../Frameworks/ is no longer the default for the installation location. If you are going to be placing MailCore inside your application bundle you will need to run the install_name_tool on the library, please look at the man page for more information.

Due to the way Spotlight is designed, it is very difficult to index monolithic databases, as in all the data stored in one file. Spotlight works by monitoring the file system for modifications, like a file being created or a file being saved. When a file system event occurs, Spotlight identifies the type of file and uses it’s corresponding Spotlight plugin, if available, to extract data from the file. This works great for things like Word documents, text files, and any other kind of document based application.

For applications which use databases, Spotlight indexing is a lot harder to pull off. For example, iCal uses a database to store your calendar events, but iCal is searchable by Spotlight, how can this be? What iCal does is very hackish, for every event in the database it creates a corresponding file in your Library folder which can be indexed by Spotlight.

I think a cleaner option would be for Apple to implement a system service similar to printer spooling. If an application that uses a database updates data, it can post via a notification or some other mechanism to the Spotlight spooler the modified data. When Spotlight is free, it can index the data and add it to it’s repository. Along side the data fed into the Spotlight spooler, there could be some kind of index that Spotlight could pass into the host application when that search result is opened.

What’s also interesting, is that despite the fact that Apple has introduced CoreData to developers, they haven’t provided an easy system for developers to index their databases. Hopefully, a system like the above would help alleviate the problem and eliminate the trickery with files.

MailCore 0.2 adds a few bug fixes and it’s built as a universal binary. Most importantly I fixed a few SMTP/MIME related bugs. Now that those bugs are fixed, SMTP works great. I have two working examples, InboxLister, which is a minimal IMAP client in like 15 lines of code and MessageSender, which sends e-mail in very few lines of code as well. The documentation is nearly complete, you can check it out here. I will be adding some more detail to the introduction page and the exceptions page in the future, but all of the pages on the classes are complete. The newest release is available precompiled here. Note: I haven’t implemented message flags yet, but that is next on my list.

Most people probably aren’t going to care about this, but I want to share this so no one else has to go through the same pain. A few things are fairly hackish, but hey, it works. Much thanks goes to Nicholas Riley in helping me figure this out.

1. Download the cyrus-sasl tarball from http://asg.web.cmu.edu/sasl/sasl-library.html


2. The newest version of GCC is much stricter when compiling so you’ll have to comment out line 112 in file lib/saslint.h, like so:
   
   //extern sasl_global_callbacks_t global_callbacks;


3. DO NOT run configure by itself, instead execute this:
   
   env CFLAGS="-isysroot /Developer/SDKs/MacOSX10.4u.sdk -arch ppc -arch i386" ./configure --disable-dependency-tracking


4. Open libtool in a text editor and change this (should be about line 177):
   
   # Commands used to build and install a shared archive. archive_cmds="\$CC \$(test x\$module = xyes && echo -bundle || echo -dynamiclib) \$allow_undefined_flag -o \$lib \$libobjs \$deplibs \$linkopts \$(test x\$module != xyes && echo -install_name \$rpath/\$soname \$tmp_verstring)"
   
   To this:
   
   # Commands used to build and install a shared archive. -archive_cmds="\$CC -Wl,-syslibroot,/Developer/SDKs/MacOSX10.4u.sdk -arch ppc -arch i386 \$(test x\$module = xyes && echo -bundle || echo -dynamiclib) \$allow_undefined_flag -o \$lib \$libobjs \$deplibs \$linkopts \$(test x\$module != xyes && echo -install_name \$rpath/\$soname \$tmp_verstring)"
   
   
5. Now run make, and it should compile

A few links from around the web

Hex Fiend http://weblog.scifihifi.com/2006/03/29/hex-fiend/ A new hex editor that is really fast, even with large files.

appscript http://freespace.virgin.net/hamish.sanderson/appscript.html A MacPython to Apple event bridge, great for people like me who despise Applescript.

Building a Sample Core Data Application http://developer.apple.com/cocoa/coredatatutorial/index.html An introductory screencast tutorial to Core Data by Rentzsch.

PyInjector http://end.com/speth/wp-content/pyinjector.html Injects a Python command interpreter and object browser into a running app. This is really cool if you’re into Python.

Pandora http://pandora.com/ Online streaming radio site that is awesome for finding new music.

Tour de Babel http://www.cabochon.com/~stevey/blog-rants/tour-de-babel.html Some interesting commentary on the pluses and minuses of different languages like C, C++, Lisp, Python, Perl, Ruby..etc.

After I read a bit about how great Lua is as an embedded and configuration language I decided to give it a try. I read through a few online tutorials, and I browsed the Lua docs, but I really can’t read books on the computer. So I bought the “Programming in Lua” book by Roberto Ierusalimschy. I’m just a little ways into the book, but I’m impressed by the Lua language. It’s very simple, compact, easy to use, and best of all it’s great for extending existing apps. In fact, while working on Kiwi I wrote a few pieces of code in Objective-C that are bound to change frequently, so much so that I was considering how I could make it pluggable. For example, before a message is displayed it goes through a little bit of preprocessing with regex’s and such. One of the preprocessors pulls out the >’s from the e-mail and replaces the chunk with a <blockquote>…</blockquote>. By doing this it makes it easy for the style of the output to be configured using a WebView and some CSS. However, some people might not want to strip out the >’s, and maybe they want to do some extra processing before the message is displayed. I think the perfect solution for this is going to be Lua and an Objective-C bridge. Things can be easily configured and changed while maintaining it’s simplicity and speed, for now though, I’m going to start with inserting Lua in a few simple places and I’ll see where it takes me :)

I’m working on packaging up my Objective-C interface to the e-mail library Libetpan so that users of my framework don’t have to install Libetpan. I setup targets in Xcode to build Libetpan and then I added the built static library to my framework target. However, when building my test cases, which rely on the framework, I kept getting linker errors, something about a dylib. Hmm, that’s strange, because I put the libetpan.a static library in the Xcode project, not the dynamically linked library. After about an hour of pulling my hair out I finally figured it out. When you add a library to Xcode it adds the path of the folder the library resides in to Xcode/s list of library paths to search. If you have two of the same library, one static and the other dynamic in that folder, Xcode will always choose the .dylib over a static library. I solved the problem by deleting all of the dylibs from the folder, now my framework links properly.

Today I uploaded an alpha version of a framework I am building known as MailCore. It’s very alpha, so be careful. I am putting it up because, well…release early and release often No really, my hope is that other developers can take a look and give me feedback. The framework in it’s current form is distributed just as source, but you can find out more here and you can read the unfinished documentation.

By writing this framework, I’ve written a Cocoa layer on top of the Libetpan framework, and I’ve made it easy for other developers to integrate this framework into their projects. Up until this point it’s been hard for Cocoa developers to work with IMAP and SMTP, this framework should make it much easier. This framework is somewhat low level though, it’s designed to be fast and it is not multithreaded so any calls could potentially block waiting for network activity. I’m working on another framework known as KiwiKit which is built on top of this framework and will be providing a more high level abstraction and internal threading. So stay tuned as I finish MailCore and I lay the foundation for KiwiKit.

The toolbar alone makes this app unusable. Everytime you click on something the toolbar totally rearranges itself, talk about confusing! Not only that, it makes it really difficult to “hardwire” the toolbar locations into your brain.

#macsb iron coder contest. looks very cool. Too bad school is taking up so much of my time.

Somehow I managed to install Adobe’s PDF Viewer as a web plugin. Now Safari no longer uses it’s built-in PDF viewer which is significantly faster that Adobe’s plugin. If you run into a similar situation here is how to remove the plugin:

Delete the file AdobePDFViewer.plugin from the folder /Library/Internet Plug-ins

Very cool stuff. I hope we can use it in Kiwi. http://www.tumultco.com/blog/?p=43

In Kiwi, the plan is to have very extensive support for searching e-mail. I think archiving e-mail and search are the two areas that are really going to make Kiwi shine. First off, we are going to support two different kinds of searching: server side and local. IMAP allows server side searches where the server returns the results to you. This is particulary nice for searching the bodies of messages that you have yet to download, like for a mailing list. On top of that we are going to have a more powerful and flexible local search that searches your database of e-mail. This is one of the many reasons I decided to go with CoreData as the data store for Kiwi. Since CoreData sits right on top of SQLlite, Kiwi will have power of a relational database sitting underneath which will provide for complex queries.

I’ve been working on stress testing Kiwi. I setup an e-mail account and signed up to all of Apple’s mailing lists and a bunch of Linux mailing lists. I’ve collected about 10,000 messages so far. We are aiming at handling 20,000 messages in an IMAP folder (not per account, per folder) as efficiently and quickly as possible. Right now we can handle 10,000 without too much trouble. Displaying the messages is instantaneous and downloading them is blazing fast. We can download the headers for 10,000 messages at the rate of about 400 messages per second. The only part that is somewhat slow is downloading the list of messages from the server, once we have that list, Kiwi flies. We’ll be working on improving this in the near feature as well.

So my question to everyone is, what is the maximum number of messages that you have in a folder?