Please Note: Blog posts are not selected, edited or screened by Seeking Alpha editors.

Loss of Data on SA: Mechanical or Procedural Error Seems Likely.

I recently confirmed to myself that comments were be "lost" from articles in SA. The comments still exist (the ones I could see) under the SA user comments section and point to the articles or instablogs correctly. When you click the "view article" link and then search for the comment in the article or instablog it may not be found.

I did my usual "brain dump" communique to SA to make them aware.

I'll update this and the link in the chat to it as things warrant.


Sent To SA on 12/10

Hi Koby and Boaz,

While reviewing some of the older posts of folks I've begun following, I encountered comments attached to the SA user that did not appear in the article when I clicked through to view the article.

This is probably *very* widespread, as there are many possible causes and I saw it on quite a few pages on this one user.

For confirmation of the problem, go to user Doubleguns and go to the last page of his comments. Select a comment or few and click through to view the article. Then use your browser's search facility to find all occurrences of Doubleguns. There will be several instances where the comment you linked *from* does not appear in the article.

Boaz will be glad to know that this could cause some folks to believe that comments were intentionally deleted because the user did not go back in their comment stream and note the comment was still attached to their user ID and correctly deduce that a mechanical problem was the cause (apparently).

Anyway, potential causes are manifold: hardware crash without adequate RAID facilities to assure "lossless crash and recovery", coding errors, incorrect restore and reconstruction of data from backups, (possible) incorrect use of SA editing facilities with SA, ...

NOTE: the following is the natural output of an "old hand" that sounds like I'm trying to tell you how to do your job. Being as everyone in the world is younger, and maybe less experienced, this is unavoidable. I always operate in "brain dump" mode.

I offer my apologies in advance for any angst you may experience at this presumptuous action.

Assuming you run some RDBM system (or even a hierarchical one, but few do these days), you need to run an SQL query that makes a sorted list of all comments with the user as root and a matching one with the article as root. Your programmer will know the needed sort hierarchy, Then a mechanical match of comments found in the two should produce a difference list which can be formatted for input to a mechanical update to the RDBM. Of course, I'm thinking in *IX terms and have no idea if these facilities are easily available on whatever platform you use.

An inspection of code may be warranted along with "boundary condition" tests to locate/eliminate code as a potential culprit.

In my opinion, your hardware platform should include a hardware raid controller (real hardware, not "fake raid" such as provided by many embedded controllers and accessible via BIOS setup) with on-board cache and battery backup to allow complete flushing of cache prior to actual spin-down.

A journaling file system would also provide a lot of additional robustness.

I will not address other potential processes, such as periodic "live snapshot" and backup since I've no idea if your configurations can support this facility.

Regardless, I would suggest that any reload/reconstruct processes that are used should be followed by thorough consistency checks as errors detected immediately are usually much easier to correct than those which have "aged gracefully in obscurity".

If you have any comments or questions, I'll be happy to converse further with you.


Sent 12/11 a little later

I almost forgot: the mechanical rebuild may be just a simple rebuild of index(es), depending on your specific RDBM system and they way you have the relations structured and maintained.


12/11 Boaz responded that since it was technical, he'd let Koby handle it.


Sent 12/14


In my prior post I forgot to mention possible race conditions and transaction commit failures as possible causes for the data losses.

Also, I've done an instablog, because a couple of folks asked to be kept informed, here

A comment there seems to confirm that a mechanical problem exists.

If more convenient for you than e-mail, I'll keep an eye on it and you can comment there as you desire and I'll respond as appropriate.


12/16 sent this to

I posted some potential issues directly to Koby and Boaz 12/10-12/14. Usually I hear something back fairly quickly.

Any word? Confirmed problem? Investigating?