Thursday, March 31, 2005

Partial Replacing Lambda?

Wonder if partial functions are going to replace lambda in python3000.

partial(operator.add, 1) isn't such an attractive substitute for lambda x: x+1, but that's more about operator than it is about partial().

Maybe partial()+1 should be supported using operator overloading. It'd be especially nice to be able to do partial()+x+y instead of partial(operator.add, x+y)

This form should probably be a differently named built-in, but built-ins aren't cheap, and the no argument constructor isn't otherwise meangingful.

Tuesday, March 29, 2005

Better Spam Protection Implementation, part 2

This post is the second part of my implementation of Better Spam Protection; you should start with the first part.

Run the following in your account's shell. It will create a directory .tmda and
its contents, and it will append to (or create) your .fetchmailrc.

You'll have to customize at least places that the word "your" occurs in .tmda/config and .fetchmailrc.

You'll also have to configure your mail client to use tmda-sendmail instead of the actual sendmail, or else configure it to use tofmipd, that is port 8025 on localhost for the smtp server.

#START OF SCRIPT 2

mkdir $HOME/.tmda
cd $HOME/.tmda
mkdir filters lists logs pendings responses
touch filters/incoming filters/outgoing logs/in logs/out logs/debug

tmda-keygen | head -3 | tail -1 > crypt_key

echo yourlocallogin:yourlocalpass > tofmipd

cat >>$HOME/.fetchmailrc <<EOF
poll yourpopserver.com protocol pop3 username youruser password yourpass
mda "xtmdaheader youruser@userhost.com | SENDER=%F tmda-filter"
EOF

chmod 600 . crypt_key tofmipd $HOME/.fetchmailrc

cat >config <<EOF
USERNAME = 'youruser'
HOSTNAME = 'yourhost.com'
RECIPIENT_DELIMITER = '+' #sendmail&postfix default to +, qmail defaults to -

#mta (for fetchmail)
MAIL_TRANSFER_AGENT='sendmail'
RECIPIENT_HEADER = 'X-RecFor-Recipient'

#default tagging method
ACTION_OUTGOING='keyword=sender'

#features
ACTION_HEADER_INCOMING = 1
X_TMDA_IN_SUBJECT = 1

#paths
d = os.path.expanduser('~/')+'.tmda/'
CONFIRM_APPEND = d+ 'lists/whitelist_confirmed'
PENDING_WHITELIST_APPEND = d+ 'lists/whitelist'
LOGFILE_INCOMING = d+ 'logs/in'
LOGFILE_OUTGOING = d+ 'logs/out'
LOGFILE_DEBUG = d+ 'logs/debug'
DELIVERY = os.environ['MAIL']
#DELIVERY = '|procmail'

#use the following line to prevent sending out challenges
#ACTION_EXPIRED_DATED=ACTION_FAIL_DATED=ACTION_FAIL_KEYWORD=ACTION_INCOMING='hold'
EOF

#END OF SCRIPT 2

Better Spam Protection Implementation, part 1

As promised, I'm posting instructions for implementing Better Spam Protection.

These instructions are intended for people who know what they're doing. These instructions describe my setup, but I didn't try using them to reproduce my setup, so I may have missed something. Because blogspot doesn't support attachments, you'll have to copy from the article text. You can either paste directly into a shell, or into a new file which you can then run.

This post describes what you should do under root, or at least in an administrator capacity. The next post describes what you should do under your user account.

First install tmda, then run the following script as root. Only its first step is necessary for all configurations.

You're going to need sendmail installed and functioning. To configure sendmail on a new fedora installation you usually just have to set your isp's smtp server on the line of /etc/mail/sendmail.cf that starts with "DS".

If you don't use pop (or imap), you won't need the xtmdaheader stuff. If you've been using your mail gui to download pop mail, you'll need to make sure that you have fetchmail installed, so that you can start using that instead.

If your mail clients support using any path to the sendmail binary, you won't need the tofmipd stuff.

The following script will patch /usr/bin/tmda-inject to enable it to use "sender keywords" and it will install:
/usr/local/bin/tmda-address-random - program to easily create throwaway addresses
/usr/local/xtmdaheader - for fetchmail support
/usr/local/src/xtmdaheader-1.1.c - source code for previous
/etc/init.d/tofmipd - runs tmda-ofmipd on startup, to support mail clients that can't use tmda-sendmail

#START OF SCRIPT 1

cat <<EOF | patch /usr/bin/tmda-inject
*** tmda-inject.orig 2005-03-23 15:44:49.417614904 -0500
--- tmda-inject 2005-03-23 15:44:26.931033384 -0500
***************
*** 202,207 ****
--- 202,209 ----
cookie_option + '@' + hostname
elif cookie_type in ('kw','keyword') and cookie_option:
# Send a message with a tagged (keyword) address.
+ if cookie_option == 'sender':
+ cookie_option = md5.new(to_address).hexdigest()[:6]
field = Cookie.make_keyword_address(from_address, cookie_option)
elif not cookie_type:
# cookie_type == None means field is a text string
EOF

cat >/usr/local/bin/tmda-address-random <<EOF
#!/usr/bin/python
import os, string
print "Keyword for the new address, or just hit Enter to randomly generate:"
keyword = raw_input()
os.system('clear')
if not keyword:
keyword = file('/dev/urandom').read(8).translate(((string.letters+string.digits)*5)[:256])
os.system("tmda-address -k "+keyword)
raw_input()
EOF

cat >/etc/init.d/tofmipd <<EOF
#!/bin/bash
# description: tofmipd
# chkconfig: 2345 90 10
# source function library
. /etc/rc.d/init.d/functions

RETVAL=0

start() {
action $"Running tofmipd: " su - yourlocalaccount -c tmda-ofmipd
RETVAL=$?
}

stop() {
action $"Killing tofmipd: "
killproc tofmipd
RETVAL=$?
}

restart() {
stop
start
}

case "$1" in
start)
start
;;
stop)
stop
;;
restart|force-reload)
restart
;;
reload)
;;
condrestart)
[ -f "$lockfile" ] && restart
;;
*)
echo $"Usage: $0 {start|stop|restart|reload|force-reload|condrestart}"
exit 1
esac

exit $RETVAL
EOF
chkconfig --add tofmipd
chkconfig tofmipd on

cat >/usr/local/src/xtmdaheader-1.1.c <<EOF
// Free to use, copy, modify or whatever.
// Copyright (c) by Hannu Kotipalo
// Version 1.1
// 1.0-1.1:
// - To and CC - headers checked

#include <stdio.h>
#define false 0
#define true (!false)
#define bool unsigned short
#define bufsize 1024

char InputLine[bufsize],ReceivedFor[bufsize];
char ToAddress[bufsize],CcAddress[bufsize],TempAddress[bufsize];
char UserName[bufsize];
char DomainName[bufsize];

bool IsEmptyLine(char*line)
{
while (*line)
{
if (*line>' ') return false;
line++;
}
return true;
}

unsigned int FindAt(char* src)
{ // Attention: does not find, if @ is at first char, src[0]
unsigned int i;
for (i=0; src[i]; i++)
{
if (i >= bufsize) return 0;
if (src[i]=='@') return i;
}
return 0;
}

bool IsEmailChar(char c)
{
return (
((c>'?') && (c<='~')) ||
((c>'"') && (c<',')) ||
((c>',') && (c<';')) ||
(c=='=') ||
(c=='!'));
}

char * CopyEmailAddr(char* dest, char* src)
{
unsigned int i,j;
i = 0;
j = FindAt(src);
if (j)
{
while (j && src[j] && IsEmailChar(src[j]) ) j--;
if (!IsEmailChar(src[j])) j++;
while (IsEmailChar(src[j]))
{
if (i++ >= bufsize) break;
*dest++ = src[j++];
}
*dest='\0';
return &src[j];
}
return NULL;
}

void strccpy(char* dest, char* src, char delim)
{
unsigned int i;
i=0;
do
{
if (i++ >= bufsize) break;
*dest++ = *src++;
} while (*src && (*src != delim));
*dest++ = '\0';
}

char LowerCase(char c)
{
if ((c >= 'A') &&
(c <= 'Z'))
c |= 0x20;
return c;
}


bool strsubcmp(char *mainstr, char* substr)
{
while (*substr)
{
if (LowerCase(*mainstr++) != LowerCase(*substr++)) return false;
}
return true;
}

#define H_Unknown 0
#define H_Received 1
#define H_Received_for 2
#define H_To 3
#define H_Cc 4

int main(int argc, char **argv)
{
unsigned int state;
char * r_for;
char * linebrowse;
unsigned int i,ULen,DLen;
char * arg;
ReceivedFor[0] = '\0';
ToAddress[0] = '\0';
CcAddress[0] = '\0';
ULen=0;
DLen=0;
if (argc>1)
{
arg = argv[1];
i = FindAt(arg);
if (arg[i]=='@')
{
strncpy(UserName,arg,i);
strccpy(DomainName,&arg[i+1],' ');
ULen=strlen(UserName);
DLen=strlen(DomainName);
}
//printf("<Debug: Name: %s Domain: %s>",UserName,DomainName);

}
state = H_Unknown;
while (!feof(stdin))
{
fgets(InputLine,bufsize,stdin);
if (IsEmptyLine(InputLine))
{
printf("X-RecFor-Recipient: ");
if (ToAddress[0])
printf("%s",ToAddress);
else
if (CcAddress[0])
printf("%s",CcAddress);
else
printf("%s",ReceivedFor);
printf("\n\n");
break;
}
printf("%s",InputLine);
if (InputLine[0]>' ')
{ // Enter a header
if (strncmp("Received:",InputLine,9)==0)
{
//printf("<Debug:Found Received-header>\n");
state = H_Received;
}
else
if (strncmp("To:",InputLine,3)==0)
{
//printf("<Debug:Found To-header>\n");
state = H_To;
}
else
if (strncmp("Cc:",InputLine,3)==0)
{
//printf("<<ebug:Found Cc-header>\n"); state = H_Cc; } else state = H_Unknown; } switch (state) { case H_Received: if (r_for=strstr(InputLine,"for")) { //printf("<Debug:Found email!>\n"); r_for+=3; if (CopyEmailAddr(ReceivedFor,r_for)==NULL) state = H_Received_for; } break; case H_Received_for: // check one line for email address CopyEmailAddr(ReceivedFor,InputLine); state = H_Received; break; case H_To: if (ULen) { linebrowse = &InputLine[0]; while (linebrowse = CopyEmailAddr(TempAddress,linebrowse)) { i = FindAt(TempAddress); if (strsubcmp(TempAddress,UserName)) { if (strsubcmp(&TempAddress[i+1],DomainName)) { //printf("\n");
CopyEmailAddr(ToAddress,TempAddress);
}
}
}
}
break;
case H_Cc:
if (ULen)
{
linebrowse = &InputLine[0];
while (linebrowse = CopyEmailAddr(TempAddress,linebrowse))
{
i = FindAt(TempAddress);
if (strsubcmp(TempAddress,UserName))
{
if (strsubcmp(&TempAddress[i+1],DomainName))
{
//printf("<Debug:Found email!>\n");
CopyEmailAddr(CcAddress,TempAddress);
}
}
}
}
break;

}
}
while (!feof(stdin))
{
InputLine[0]='\0';
fgets(InputLine,bufsize,stdin);
printf("%s",InputLine);
}
return (0);
}

EOF
cc /usr/local/src/xtmdaheader-1.1.c -o /usr/local/bin/xtmdaheader

#END OF SCRIPT 1

Wednesday, March 23, 2005

Humble and Smart

The more you learn about something, the more you realize how little you know about it.

This isn't just philosophy; when people know nothing about a thing, they assume that there's "nothing to it". Then they confidently make decisions based on incomplete information. (It could be because interesting elements exploit the interplay of the simple and the complex. Or it could just be because small details aren't visible at a distance.)

Paradoxically, bright people are more susceptible to this thinking. Bright people are accustomed to quickly understanding ideas outside of their field of expertise, and this understanding is often superior to experts'. A bright person can start to believe (subconsciously) that much of the work in a field is characterized by lack of understanding.

This has ramifications for religious leadership. That religious leaders have insight into issues not technically within the sphere of religious practice is a doctrine of Judaism, and probably other religions as well. It's only logical; following Judaism is about living wisely, and someone with a deeper understanding of Judaism should have attained greater wisdom. In order to be truly wise, however, a person must be aware of his limitations. Maybe this awareness can be achieved by encountering sufficiently dramatic examples (for example the abuse cases involving religious leaders in recent years), or maybe it requires learning a little bit about alot of different things.

(It is troubling to think that G-d confers special wisdom in a way that is completely independent of the natural working of the world. For one thing, thinking that you've got supernatural aid from G-d certainly exacerbates the problem described above. For another, there is no good way to tell when you got it, and when you don't. "Trust in God, but steer away from the rocks", applies here as well.

At any rate, I don't think Judaism encourages completely ignoring the rules of the physical world. There's a gemara in niddah that gives advice about achieving various goals; in each case the gemara advises both acting according to natural way of the world, and also praying for success. In this era characterized by hester panim ("G-d hiding His face"), we certainly shouldn't abdicate the responsibility to do our hishtadlus (effort), especially where other people are involved.)

Friday, March 18, 2005

Idiomatic Python

Programming languages, like human languages, each have their own idiomatic usage. Though there are many different ways to express a single thing, some patterns of expression just feel much more natural.

The idiomatic style of python is beautiful.

Here's an example. You'd like to allow categorizing emails using their subjects, by prefixing the normal subject with the category and a colon. So for a message with subject "game on tuesday", you could use subject "sports: game on tuesday". If no category is specified, you'd like to use "misc". Here's the Pythonic way to do it:
try:
category, subject = headers['Subject'].split(':')
except:
category, subject = 'misc', headers['Subject']

There are a few different features at work here:
  • "It's easier to ask forgiveness than permission"
  • Exceptions are used heavily
  • Language support for collections is simple and easy

Friday, March 11, 2005

"Startups" and Small Businesses

At least since the dot-com bubble, the idea of the startup has been enshrouded in glamor. Most recently Paul Graham posted How to Start a Startup, but his advice is for dot-com-bubble type startups. He suggests that starting a company means compressing a career of 40 years into 4, and assumes that every new business will either flop or become an industry giant.

It could be that when Graham says "startup", he means a company that would rather flop than settle for staying small. Only talking about these startups is strange for a few reasons. Many companies consider themselves successful without making a public stock offering, or being bought out for piles of cash. How about creating a successful small company first, on the way to becoming an industry giant later? And which ideas and strategies are appropriate for which goal? These questions are a lot more relevant today than how to handle a Venture Capitalist imposed CEO.

After all, though more people are employed by big companies, there are many more small companies than big ones.

Wednesday, March 09, 2005

Note to Future Self

Whenever I put something down in an unusual place, there's a strong chance I will "lose" it.

Hopefully when I'm older I'll look back at this note and realize that I'm not going senile.

Tuesday, March 08, 2005

Ipod Shuffle for Lectures

The IPod Shuffle is a wonderful device. One more feature would make it perfect for listening to lecture series (or books on tape or language courses), without bloating its wonderfully simple interface.

The IPod Shuffle should remember the current "song" after it's turned off.

So apparently this is supported for purchases from ITunes and Audible. In order to get it to work with anything else, you have to use faac to create .m4b files.

UPDATE: It works great. If you're using linux, get gnupod, mpg321, and faac. Then you can do something like:
mkdir /mnt/ipod 2>/dev/null
mount /dev/sda1 /mnt/ipod
for mp3 in *.mp3
do
m4b=${mp3%.*}.m4b
mpg321 -w $mp3 | faac -b 10 -c 3500 - -o $m4b
gnupod_addsong.pl -m /mnt/ipod $m4b
done
mktunes.pl -m /mnt/ipod
umount /mnt/ipod
Incidentally, it would be great if gnupod would expose a filesystem-type api to the ipod using fuse.

UPDATE#2: Don't use gnupod; use the iPod shuffle Database Builder.

Thursday, March 03, 2005

Warn of Unsaved Changes Javascript

OK, this will get all the javascript out of my system. This one is intensely useful for web "applications". It warns the user if she tries to leave the form with unsubmitted changes, whether leaving by moving to a different page or by closing the window. It requires a recent firefox or ie browser.
<body onLoad="lookForChanges()" onBeforeUnload="return warnOfUnsavedChanges()">
<form>
<select name=a multiple>
<option value=1>1
<option value=2>2
<option value=3>3
</select>
<input name=b value=123>
<input type=submit>
</form>

<script>
var changed = 0;
function recordChange() {
changed = 1;
}
function recordChangeIfChangeKey(myevent) {
if (myevent.which && !myevent.ctrlKey && !myevent.ctrlKey)
recordChange(myevent);
}
function ignoreChange() {
changed = 0;
}
function lookForChanges() {
var origfunc;
for (i = 0; i < document.forms.length; i++) {
for (j = 0; j < document.forms[i].elements.length; j++) {
var formField=document.forms[i].elements[j];
var formFieldType=formField.type.toLowerCase();
if (formFieldType == 'checkbox' || formFieldType == 'radio') {
addHandler(formField, 'click', recordChange);
} else if (formFieldType == 'text' || formFieldType == 'textarea') {
if (formField.attachEvent) {
addHandler(formField, 'keypress', recordChange);
} else {
addHandler(formField, 'keypress', recordChangeIfChangeKey);
}
} else if (formFieldType == 'select-multiple' || formFieldType == 'select-one') {
addHandler(formField, 'change', recordChange);
}
}
addHandler(document.forms[i], 'submit', ignoreChange);
}
}
function warnOfUnsavedChanges() {
if (changed) {
if ("event" in window) //ie
event.returnValue = 'You have unsaved changes on this page, which will be discarded if you leave now. Click "Cancel" in order to save them first.';
else //netscape
return false;
}
}
function addHandler(target, eventName, handler) {
if (target.attachEvent) {
target.attachEvent('on'+eventName, handler);
} else {
target.addEventListener(eventName, handler, false);
}
}
</script>

Wednesday, March 02, 2005

Telephone Shell Javascript Widget

Here's some javascript I wrote to coerce uniformly styled phone numbers. If you don't use any punctuation, it defaults to american style. You can enter unrestricted text after the number. You should be able to copy the following into an html file to try it out.
<form>
phone #<input onkeypress="return phone_only(event)">
</form>

<script>
function phone_only(myevent) {
mykey = myevent.keyCode || myevent.which; //ie||netscape
myfield = myevent.srcElement || myevent.target; //ie||netscape
if (mykey == 8) //backspace (netscape only)
return true;
f = myfield.value;
g = myfield.value;
ndigits = f.replace(/-/,'').length;
ngroupdigits = g.replace(/.*-/,'').length;
if (ndigits == 0) {
if (50 <= mykey && mykey <= 57) { //2-9, can't start with 0 or 1
return true;
} else {
return false;
}
} else if (ndigits <= 7) { //only need 2 hyphens: 123-456-7
if (32 <= mykey && mykey <= 47 && ngroupdigits != 0) { //punctuation
myfield.value += "-";
return false;
} else if (48 <= mykey && mykey <= 57) { //0-9
if ((ngroupdigits % 4) == 3) {
myfield.value += "-";
}
return true;
} else {
return false;
}
} else {
return true;
}
}
</script>

Tuesday, March 01, 2005

Forms Support in Javascript, not Template

Turns out it's pretty simple to populate forms generically in javascript. This means that your template engine doesn't need special html forms support; you can just use a pair of templating "for" loops to create the javascript data structure. You could even grab the data using xmlhttprequest. Html made the the mess, so html can clean it up.

Here's an example form, followed by the code. You should be able to copy into a file and try it.
<form>
<input name=a>
<input name=b type=checkbox value=x1>
<input name=b type=checkbox value=y1>
<input name=b type=checkbox value=z1>
<select name=c multiple>
<option value=x2>X2
<option value=y2>Y2
<option value=z2>Z2
</select>
<button onClick="populateForm(this.form, {'a':'x0',b:['x1','z1'],'c':['x2','z2']}); return false">Populate</button>
<input type=reset>
</form>

<script>
function populateForm(myForm,myHash) {
for (var k in myForm) {
if (!(k in myForm) || !(k in myHash)) continue;
if (typeof myHash[k] == 'string')
myHash[k] = [myHash[k]];
if (myForm[k].type == 'text') {
myForm[k].value = myHash[k][0];
} else {
var field = 'type' in myForm[k] && myForm[k].type.match('^select') ? 'selected' : 'checked';
var selected = Array();
for (var i=0; i<myHash[k].length; i++)
selected[myHash[k][i]] = 1;
for (var i=0; i<myForm[k].length; i++)
myForm[k][i][field] = myForm[k][i].value in selected;
}
}
}
</script>