According to the official blog there's a rolling restart underway. Details are sketchy, but when they're this poorly announced they usually mean it's an exploit fix and we won't get a lot of additional details, although it is also setting up for the new search, fixing sale auction IDs and (most interesting) allowing inventory databases to switched in and out without booting everyone.

The restart is running North to South, so jump North if you get caught by it. It started at 2:17pm and will run until about 6pm.

I've been teleporting around all evening on a window-shopping spree. Suddenly one of my group IMs buzzed to life with "Is it only me or can no one teleport?" Over the next few minutes we've had no ability to build, search is borked (nothing shows up even if you search for sex with mature checked, it must be bad!) and people that are relying on the dataserver for reading notecards and customer information are finding that's not working either.

No word from the official blog as yet, but your second life may be rather adversely affected. I may not see the resolution of this one, it's getting kind of late for me.

[UPDATE] Official blog catches up, adds shopping, loading money and rezzing no copy items to list of things with problems.

[UPDATE] According to the main blog this has been fixed. I've not yet been back in world to check, but we had a rough two hours looking at the time it was declared fixed.

At 8:00AM SLT (US Pacific) this-morning, with 37,419 users logged into the Second Life grid, something began to go wrong. Reports began coming in of teleportation problems, difficulties accessing inventory, search failures, and difficulties logging in.

At 10:10PM things started to worsen and users began logging off in large numbers. Linden Lab shut down all search facilities at that time, which suggests that there are problems with load on the database systems. Recently this sort of load has been caused by everything from misconfigured systems to hardware failures, so at this moment there's not a lot of information to go on.

[Update: 11:36AM SLT - Linden Lab reports that the problem has been found and fixed, but isn't divulging the details.]

Frontier Linden advises that in order to address issues with the grid (we presume that these are among the issues to which they are referring) Linden Lab has shut down Search functions for Places and Groups as of 1:40AM SLT (US Pacific).

Additionally, many European users (all the affected ones seem to be routing through Level 3) are once again unable to access Second Life web servers or the Second Life grid. The first major occurrence of that was on Friday during the time that the Support Portal was offline, then a protracted period on Saturday, and now we are informed, that they have been locked out since some time before midnight Saturday, and that those connectivity problems are ongoing.

[Update: 5PM SLT - Search is back on, having been off for the busiest day of the week - but no word from Linden Lab as to whether the problems that they were trying to solve are themselves resolved]

See? Someone's listening! Everett Linden today posted a detailed and thorough description of last week's search problems; What happened, what seemed to cause it, what was done, and when. It also acknowledges that Linden Lab could have been more forthcoming with communications during the period.

It's terrific work, and we'd love to see more of this sort of thing. It provides valuable insights into the architecture, failure provisioning and the genuine challenges involved in keeping a platform like Second Life up and running.

Teleports, asset uploads, the Second Life map, and most of all the search functions all suddenly began functioning relatively smoothly just before 3am SLT (US Pacific time). Granted, login concurrency was just past ebb, but the sudden improvement in function was sudden and dramatic.

Linden Lab have said nothing about it, so it is possible that this isn't because the problem has been fixed, and that it may recur again shortly, as the grid starts to get busy again as we get into Friday. Everyone will be watching to see, that's for sure.

The infrastructure of Second Life is still struggling to cope. Despite relatively magnificent peak performance in the last couple of weeks, basic systems are suddenly struggling to cope with loads that are half that. Search systems are still down, payments and teleportation remains erratic. Map texture loading is hit and miss.

The severity of these problems changes. At some times, only searches are problematic, while at others, hardly anything seems to be working. Linden Lab say that they are aware of the problem and don't require any more support requests on these issues to add to the workload. They're still working on it, but aren't divulging any clues as to the nature of the problem, or even that they know what the cause is at the present time.

Just before 7am SLT (US Pacific) this-morning, residents began reporting a number of issues on the Second Life grid. Primarily the issues seem to center around the transfer of object and inventory to and from people and other objects. Searching for people and places, classified ads, and - well, all search functions are down. Group member interfaces are not functioning.

When the problems commenced, login concurrency was at 31,884. It has already slid to 30,627 and continues to fall. Linden Lab has not posted any acknowledgement or notice about the difficulties at this time.

[Update: 7:30am - Seems like the entire grid suddenly stalled for 30 seconds, then search functions were restored, and login concurrency is climbing sharply.]

One of the highlights of Meta Linden's office hours today was the confirmation from Zee and Meta that -- as we suspected -- parcel traffic figures would likely remain. Traffic would no longer be considered as a factor in ranking search results, but the figure will remain in some form to parcel owners.

Meta went on to poll attendees about whether the traffic figure per parcel should be visible only to the owner/group or to anyone who checked. The answer for those present was overwhelmingly in favor of retaining it as visible to all who wanted to check it.

The redesign and rewrite of the search system apparently is ongoing and expected to take several more weeks before it is rolled out.

At 5:10pm SLT (US Pacific) Search > Places and teleportation went off the air, though sim border crossings and logins seem to be working as usual. There are some suggestions that inworld Instant Messaging may have partially failed as well, but that is harder to check. Additionally there are reports of the map acting up.

No word from Linden Lab on this one, yet. But there's no easy way to report this to them so they may not be aware of the problem yet. This failure does not come at a peak period. We're currently on the slide downwards, about 3 hours after peak load.

[Update (5:40pm): After 30 minutes of issues, teleportation has started working again, and Search > Places started returning results a couple minutes later.]

When I login to Second Life, I have goals in mind. These might be social goals: to hang out with friends, to participate in a game with or against other people, to meet new friends. Potentially they are business goals: conducting interviews, gathering quotes, business meetings. All these goals have something in common: in order to be realized, they require that people be able to hook up in-world and interact with each other.

When the presence mesh has been non-functional in the past, and for a large chunk of the last few hours (though it's back now), we have had a preview of what it's going to be like on Wednesday when we can no longer see who is online, except the people already on our friends list. In a word? Disappointing. Suddenly it becomes that much harder to determine when other people are reachable. Will we be left casting hopeful little Instant Messages into the waves like a disconsolate fisherman?

Linden Lab are continuing with their efforts to reduce or eliminate factors that contribute to unnecessary database load. The database load is the most commonly cited reason for partial or total grid failures. The next phase is set for the next release (which does not presently have an announced release date, though earlier comments on the Linden Blog suggested the possibility of January 31).

These changes involve the way searches for people take place in Second Life. Apparently searching based on the Online status of avatars is causing significant amounts of load on the database system. Accordingly, Linden Lab are planning changes that will reduce the load by partially or wholly eliminating presence information in profiles and searches.

James Linden apologises for the breakage in the search code last week where he set the system to search with the keywords ORed together so it would match parcels listing any one of the provided words, causing your search to expand, rather than narrow, as you added more words.

1.13.2 includes assorted changes for that, but only finds words that are next to each-other (sometimes called adjacent word searching, sometimes called phrase searching), so it will find your exact phrase - if it exists, in the word order you provide. Nevertheless, this is probably an improvement.

James promises more sophisticated search operators as a part of an ongoing overhaul of the search system.

Yesterday's grid problems continued throughout the night and on into today. At 11:09 AM SLT, Search > Places was disabled, and residents were advised to remove attachments from their avatars if they were having difficulty teleporting.

After five hours of investigation and work 164 regions were taken offline for rerouting through one of the new bandwidth circuits between the colocation facilities, a process that took slightly less than 45 minutes.

Jean Linden reports that Linden Dollar balances are not updating properly for some while after logging in and that there are reports of failures in search and teleportation mechanics. These issues "are being investigated", and may well be load-related, though with 17,000 residents on the grid right now, the load should be relatively stable.

Only one of these systems has any connection to the new communications protocols, so the odds of these issues being connected to issues with the new protocols are low.