Fatal errors may cause infinite Feeds batch runs and exasperate you!

Every developer knows that kind of situations, where you spend hours chasing a problem, that suddenly appeared, and the more you debug and dive into fixing it, the more you get confused and unnerved. And when you finally have found the solution, it turns out, that it was a very simple and stupid bug, you've created unwittingly - the sort of cause, you haven't ever thought about, and you feel like bumping your head hard onto the wall...

Well, exactly such a situation happened to me yesterday, when I was working on a Feed importer for a customer - and I think, that one can get into exactly the same situation very easily. And I'll bet, many others would also get easily misleaded. So I want to share my experience with the world, so that others may prevent a similar situation :-)

The ingredients of this mean problem

Under certain circumstances the most easiest-to-fix problem can disguise itself and bring you a lot of troubles. In my situation, these factors together resulted in a "toxic cocktail":

• a tiny change, that I've tested successfully
• a even more tiny change to the first one AFTER testing (change a function name)
• a failed FTP transfer (which I did not recognize)
• another, little bit bigger change (which was innocent)
• running the importer via AJAX (Batch API) - thus not seeing the full page reload
• seeing strange results that seems to be related to the bigger change

My Feeds importer was already in use and tested. I only had to do a very small configuration change, and a very small improvement in my hook_feeds_presave implementation, and - in order to improve the performance signficantly - implement hook_feeds_after_parse and do some filtering of duplicate entries (that would not have been possible in that form with a Feeds Tamper plugin).

First, I have changed the configuration, then wrote the small change in the presave hook, afterwards tested it successfully, and then decided to move forward to the next problem. But before I actually did, I've just changed a function name in a helper class because I felt that name would be semantically more correct and better. I've did the change to both affected files. And there was really no need to test it again... So I started to implement my filter in the hooks_feeds_after_parse hook.

But there was one thing I have missed after changing my function name earlier: the FTP transfer of the helper class failed, so that an ordinary "Fatal error: Call to undefined method" will be caused when running the importer the next time.

You may ask, how I could not recognize that...

Well, the first reason is, that a fatal error was raised, and not an execption. Exceptions would have been caught and logged by Feeds Processor. The error just ends the request abruptly. And as the Batch API didn't get the response, that the job has finished, it just calls it another time. If you do not abort the import run forcefully, it would run infinitely!

But in my case, the problem was even more mean. As I mentioned already, I've implemented a duplicate filter, whose cache is emptied after finishing an import run successfully. So it happened, that every chunk of the CSV file was processed exactly two times: the first time, the fatal error happened, but the second time my filter already marked the unique key column as processed, so that it excluded the row from the result set. So, there was no infinite run, but on the other side, none of my items got updated.

For sure, I've blamed the filter and/or Feeds for causing the problem. First, I was thinking of a concurrent execution problem. I was already debugging deeply in various Feeds functions, when I saw, that after skipping saving the entity and also my custom processing in the presave hook, the filter worked correctly, and no chunk was processed twice. But I was already on the wrong trail and still needed some time to find out, what was really causing the problem.

Lessons learned

1. If your Feeds importer is running endlessly, or causing some other strange behaviour like processing the same items multiple times, you should first check your browser's console and the responses of your requests!!
2. I sometimes really hate PHP for not being a strongly typed language such as Java (as well as being interpreted and not compiled...)
3. even the tiniest and simpliest changes can have a huge impact and cause big problems...