Large gallery - blank page returned/script terminates

Hi guys,
Got a query here, and I think I might also know why!

I have hosting with servage.net and installed phpalbum 0.4.1.14, which ran fine, but as the gallery has increased in size (this is for a photographer, so raw images are 1-2Mb in size, and about 30-40 categories of 100-200 pictures in each) the site has stopped working.

Basically when you hit the gallery it waits for a minute or so and then returns a blank page, however checking, the script isn't actually returning a blank page, its obviously php/apache timing out/killing the operation (see the lynx output below).
I have removed the URL to the site, as this is adult in nature and do not want to publically post/promote this on your forums.

What I think might be the problem though is the cache folder. I have cache enabled, and have noticed that in the cache folder there aren't any sub-folders or anything, all the files are in a flat entire directory, which I suspect will be the problem as my cache folder will be huge based on the number of images, etc that are in all the folders, etc. Perhaps the cache should use folder structures to mirror the raw image store?

Output test:

lynx --source http://www.thesite.net/auto_gallery/

Looking up www.thesite.net
Making HTTP connection to www.thesite.net
Sending HTTP request.
HTTP request sent; waiting for response.
HTTP/1.1 302 Found
Data transfer complete
HTTP/1.1 302 Found
Location URL is not absolute.
Using http://www.thesite.net/auto_gallery/main.php?cmd=album
Looking up www.thesite.net
Making HTTP connection to www.thesite.net
Sending HTTP request.
HTTP request sent; waiting for response.
Alert!: Unexpected network read error; connection aborted.
Can't Access `http://www.thesite.net/auto_gallery/main.php?cmd=album'
Alert!: Unable to access document.

lynx: Can't access startfile

Hi, I have almost 4000

Hi,

I have almost 4000 pictures in my personal gallery and I have no problems with it. When developing I am testing it with 10000 picture gallery and only 8MB memory limit and it works. You can get problems if too many pictures are in one single directory. But if you have enough memory limit it should be ok too.

So, I am not sure what is causing your problem and so no idea how to help you. In your output protocol I can see some network read error which is normally not there if only some PHP problem occurs. Could you may be post your phpinfo() page of your PHP installation?

Patrik

Hiya, thanks for the

Hiya, thanks for the info,
Yes I'm a little confused too as it did work initially.

I don't believe its a network error as have tried this from several network connections now (I'm a network engineer for an ISP), and get the same result which makes me think the php process/thread is being terminated which results in the error I posted.
So I'm a little puzzled.

PHP info outputs (I've chopped the bits I'm assuming you need):

Configure Command './configure' '--with-config-file-path=/imports/lib/' '--with-openssl' '--with-dom' '--with-iconv' '--with-curl' '--with-mysql=/usr/local' '--with-gettext' '--enable-mbstring' '--with-xml' '--enable-track-vars' '--with-apache=../apache' '--enable-exif' '--with-jpeg-dir' '--with-zlib-dir' '--with-png-dir' '--with-gd' '--enable-gd-native-ttf' '--with-gettext' '--with-freetype-dir' '--enable-ftp' '--with-pear' '--with-pspell' '--with-mysqli=../mysql-5.0.15' '--with-ldap' '--with-ldap-sasl' '--with-imap=../imap-2004g' '--with-kerberos' '--with-imap-ssl' '--with-libxml' '--enable-xslt' '--with-xslt-sablot' '--with-dom-xslt' '--with-expat-dir=/usr' '--enable-bcmath' '--with-mime-magic' '--enable-calendar' '--with-mcrypt' '--with-mhash'

display_errors On On
display_startup_errors Off Off
file_uploads On On
gpc_order GPC GPC
magic_quotes_gpc On On
magic_quotes_runtime Off Off
magic_quotes_sybase Off Off
max_execution_time 30 30
max_input_time -1 -1
open_basedir no value no value
output_buffering no value no value
output_handler no value no value
post_max_size 8M 8M
precision 12 12
register_argc_argv On On
register_globals On On
report_memleaks On On
safe_mode Off On
upload_max_filesize 100M 100M
upload_tmp_dir /upload_tmp /upload_tmp
user_dir no value no value
variables_order EGPCS EGPCS
xmlrpc_error_number 0 0
xmlrpc_errors Off Off
y2k_compliance Off Off

not sure what other info I can provide, I'm fairly sure this problem is specific to this host however I cannot see/figure out why!

Right, to try and narrow

Right,
to try and narrow down where the problem was I tried this, I edited the config.php and changed the image folder.
Tried to access the gallery, and got the error saying the images folder didn't exist.
So I created the folder, and set it to 777.
Then tried to access the gallery, same problem blank page as the script timed out.
So I'm now in via FTP trying to delete the contents of the cache folder, and its struggling via FTP due to the size of that folder, so I'm now more and more convinced that php will also be struggling as a php opendir or similar folder scan would probably consume quite a bit of resource parsing that cache folder.

I'm now deleting the contents of the cache and will report back if this solves the problem.

Deleting the cache via FTP

Deleting the cache via FTP is not the best idea as FTP is really slow when deleting too many files. You should use some basic ftp client where you can switch off prompting with "prompt" command and delete all files with one command i.e. "delete *". Just be sure to be in the right folder.

PHP is quite fast in deleting the cache too if you have acces to PA Setup section.

For your information, in config.php is not the Image folder but the DATA folder which should be chmodded to 777.

The image folder should be set up in PA Setup->Main Setup , as well as cache directory, but you probably know this already.

But I still have no idea what could be your problem. Did you tried to setup it first with only few pictures?

Right, Results are quite

Right,
Results are quite interesting here.
After deleting the entire cache folder contents, no difference, gallery doesnt appear (when I say gallery, I even try going straight into the setup area via a saved link to /auto_gallery/main.php?cmd=setup and it does the same blank page/fail to load)

So I then decided to clear the data_folder out entirely too, this brought the thing back to life, so I could get into the setup area.
As soon as I swapped the paths back to where my images were stored it died again and didn't come back, so it does appear to be struggling with parsing/reading the image folders.

I'm going to add some debugging to the script so that I can see what its processing before it hangs.

sorry forgot to mention, yes

sorry forgot to mention, yes initially it worked fine with about 10-20 images, and it seems like as more and more were added via ftp it started to degrade then completely stop as it has done now. So it is definately a volume issue.

Ok, now I probably know what

Ok, now I probably know what is your problem. In your PHP.ini you have maximum execution time only 30s. This is normally enough but if you have very much pictures or some slower machine it could be not enough to scan all folders for all pictures.
After 30s it is then probably interrupted by system and if the database is in incorrect status it could cause what you are describing here.

With enough execution time not even 10k pictures is a problem for PA. But they should not be in one directory as this could cost quite much performance.

Ah right, That makes sense,

Ah right,
That makes sense, as I've added a lot of debugging and testing code to the main.php now and yes, it keeps getting stuck on the auto generation at each call.

I've also removed auto generation when I call the setup pages, so I can always get into that section, and have now set it to not auto generate on the fly, so I need to manually run that to at least keep the front-end running without the timeout.

I'll modify the php timeout in an htaccess file and try running the update, is there anything I should do to try and recover or clear out the database (and not loose my other settings?)???

Thanks for your continued help :-)

Normally it should be able

Normally it should be able to continue with scanning of directories on the place where it stopped due to time out. But I never tested this, either it will work or not, you will see.

Right, I've brought the

Right,
I've brought the gallery back online, so I cleared all caches, and then set all gallery folders to permission 000 so the scripts couldn't access them.
I then changed one-by-one and ran a 'scan directories now' so that it had a chance to do one folder at a time.
Even after adding
php_value max_execution_time 190
The script kept stopping and leaving at a blank page.
Checking the folders of images, one folder could contain up to around 150 images, each image being between 1Mb and 3Mb, so not really huge, but this seemed to cause the script problems.
Whenever the script failed with a blank screen, going back no other operations worked, so checking in the data_dir I was finding .lock files, which if I deleted these manually, it released the DB and allowed me to run a scan again, which generally succeded the second time. So it loooks like when the DB locks like this, it causes the script to completely hang without a solution, unless you manually delete the locks.

So my main changes (I'm happy to submit a diff if its of any use):
* Stopped automatic directory scanning (set via a variable) so that it doesn't do the 24hr rescan (scan_photos_directories) [you can still do a manual forced scan via the setup area]
* Added a global debug flag and added a lot of debug outputs to the main.php script (and suitable flush statements for browser/stream flush) to help diagnose where problems may be happening (again debug set via a variable flag)
* When debug mode is enabled, disables the cache to again assist with diagnostics

Thanks for the help tracking this one down Patrik, I'm going to continue and work on the script to try and get my friends site working smoothly (mainly so that on error it will return whatever it can, rather than blank pages/timouts).

Kind regards.

Andy

Next thing you could try is

Next thing you could try is switch off Use of IPTC. I already have had some problems with bad characters in IPTC descriptions which on default is used by PA.

This switch you can find in Setup->Main Setup. If it alone does not help you can try to delete data dir, switch off IPTC and setup photo dir for scanning.

Patrik

Yep, tried that and still

Yep, tried that and still having problems.
The problems all seem down to the database locking (which I know is a tricky concept in programming terms!).
As what I'm finding is that when the website comes to a halt and just sits timing out, its generally because there is a .lock file in the data dir that doesn't seem to get removed no matter how long I leave it, so I suspect the dblock function that is supposed to check for a stale lock file isn't running correctly (this is a symptom of the problem, as obviously ideally the main scan should complete, but I cannot seem to get this to complete no matter what memory or time limits I allow the script to consume).

I'll continue to tweak at this and see if I can figure out where its halting.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.


style="display:inline-block;width:468px;height:60px"
data-ad-client="ca-pub-8698264690166658"
data-ad-slot="4417389723">