It’s been 48 hours since we screwed up DNS and, bingo, our metering is taking a big pop, up 20% in the last hour. I’ll try to grab an image of it a little later.

Here’s what we’ve been able to piece together. The correct DNS is slowly disseminating through the many tiers/branches of domain registries stacked around the globe. In some cases, the servers update every 2 hours, in some cases every 48 hours, and in some cases longer (a week?) We’ve tried a bunch of tricks and tactics to speed things up yesterday and are continuing today. As far as we can tell, 85% of blog readers can now see the adstrips/ads and Blogads.com. That’s up from 60% at the low. (One datapoint is that ad purchases have not slacked off, though I still can’t see Blogads.com from my home computer.) In theory, we should see another improvement today (assuming there are more servers out there with 48-hour caches than we’d originally anticipated). But, theory doesn’t buy lunch, so we’re continuing to look for ways to manually speed the process along. If you have suggestions, we’d love to hear them. I’m very frustrated and apologize. We’ll be issuing make-goods to all advertisers.

In part, this error and a couple of other recent stumbles may be a sign that our Budapest team is overstretched. As a company we tend to try to have more balls in the air than hands. That’s great and keeps you motivated and driving, but when you slip up, it’s ugly. Driving to normalize slightly, we added an additional programmer a month ago and he’s pulled into the swing very quickly. Now we’ve moving towards hiring another strong coder.

For numbers geeks, I don’t know if you can make sense of this graph of one of our server’s bandwidth consumption, but if you compare like time periods, we’re doing roughly 15% better today than yesterday.

I’ve just published the logos that came in during the initial “private submission” phase of our logo collaboration. There are some interesting logos that should not be overlooked. Head over and leave your comments if they catch your eye. Here’s one by Greg Stobb. And a bunch by Stephanie Kloss, particularly this one. Finally, there’s this bunch by Mikel Browning, including this and this.

Since we ended up with more logos than we anticipated and a wider variety of comments, we’re considering creating a shortlist and inviting public comment on that list. OK idea?

We screwed up our domain name registry entry yesterday afternoon, which means we made our servers’ addresses invisible to much of the Internet. A stupid human error which should not have occured. I went home last night thinking everything would be ok in 30 minutes, and didn’t blog about the problem because I was unable to access the blog server myself. The error propagated very quickly, but the correction has taken longer to spread. (As my colleague noted, bad news travels faster than good news. ) We are seeing the correct DNS slowly propagate (judging from our bandwidth metering and spot checks) and know that a growing number of people can view blogads. We’re guessing that by noon 90% of the Internet will be able to see blogads. I apologize for the problem and will pass along more information if I learn anything new.

We spent the weekend hiking in Western NC. The weather was perfect — 75 and low humidity. And the wildflowers were… wild. We waded through what seemed like hundreds of different species. Beneath rocks in the torrent way up near Pot Cove Gap we found a few giant salamanders and cray fish. How do they manage to get that high?

Jeff Jarvis asks his readers whether he should accept a blogad for a t-shirt bashing FEMA. So far, his readers say no, 9 to 0.

Chug Roberts, who has Blogads running in many corners of the blogosphere for his publications, suggests that advertisers donate space to relief efforts. He’s been replacing his own ads with appeals for the Salvation Army.

Rogers Cadenhead is catching some flak for mooting the idea that liberals and conservatives stage a friendly competition to raise money for Katrina relief. FWIW, I’ve heard this suggestion from people on both sides of the aisle and thought it was a good idea, though the various efforts have diverged at this point and a horse race would be difficult. Having said that, I’m waiting to hear from the folks who work with MercyCorps to talk about separate landing pages for various networks, including John Hawkin’s conservative network, who are already running MercyCorps ads.

At this point, the liberal network has raised $157,000 for Katrina relief.

We’re having trouble right now with the servers at Voxel.net that host our images. We’re trying to get things straightened out. I apologize for the problems. If you are an advertiser, we’ll be crediting you a day. If you are a blogger, the easiest approach is defering your currently running ads until this freak storm passes. Again, I’m very sorry for the problem.

Update 5.15 I called Voxel 30 minutes ago and heard that I’d be getting an answer shortly. I guess shortly might be longly. The good news, at least until Tuesday morning, is that as folks go home for the long weekend, bandwidth consumption is dropping off so our load balancing is doing a more adequate job. At this point, though Voxel’s problems continue, everyone’s adstrips should be working Ok. If not, please drop us a line.

Update 6.30 Tracked down someone Voxel and he was super helpful. We’ll have the problem solved tomorrow morning.

At least that is what this graph shows.

We’ve spent the morning playing with the UI of the logo contest. Cleaned up the navigation a little and added a logo with most comments page and recent comments page. We’ve seen a couple of instances in which a logo has gotten a comment and then the designer has created a new version building on the feedback — maybe the new pages will accelerate discussion and collaboration. (My wife agrees with Jeff Jarvis, BTW.)

Tessa Wegert spots a key trend that I’ve been jawing about at conferences… DIY advertising, which is quietly transforming advertising. Wegert focuses on the word of mouth angle, but her point also applies to advertising purchase and distribution models. Just as blogging upends publishing, tools like Google Adwords, Blogads and Adbrite make it simple for advertisers to place their own ads. We’re doing to the ad industry what Dell did to computer sales and distribution.

As Wegert says, all is not lost for media buyers. But the game is changing:

In many ways, online media has become a self-service communications channel. Advertisers, even those with little online experience, are discovering what consumers have known for years: at its core, the Internet is a medium “for the people.” It doesn’t necessitate assistance from the pros. One can develop ad creative using a home software program, launch a site using inexpensive Web design and hosting tools, and promote one’s initiative with blogs, discussion boards, and good old word of mouth — all without traditional advertising agencies, media brokers, or placements experts.

Where does that leave media buyers, whose livelihoods depend on the demand for ad placement expertise? There’s no denying the need for evolution. Today’s buyers and planners must prove their worth by showing versatility and demonstrating expertise in new formats to makes themselves indispensable. Though it’s certainly possible to work without media buyers, they possess a knowledge of the Internet space and its users that professional and amateur advertisers alike would be remiss not to exploit. Having access to DIY online advertising is one thing. Making it work is quite another.